Large model agent construction method and system, refrigeration equipment and storage medium

Abstract

The invention provides a large model intelligent agent construction method and system, refrigeration equipment and a storage medium, multi-modal data is effectively divided into action data and state data by efficiently integrating and processing the multi-modal data, and in order to enhance the decision and prediction ability of an intelligent refrigerator, the multi-modal data is effectively divided into action data and state data. Historical action and state data and real-time data are migrated and fused, the comprehensive application of the data is optimized, and the prediction accuracy is improved. On the basis, data weight and network model parameter migration are optimized through decoder processing of a multi-head attention mechanism, so that excellent adaptability and response capability can be shown in the face of complex user requirements and environment changes.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a large model agent construction method and system, a refrigeration equipment and a storage medium. BACKGROUND

[0002] In the operation of intelligent devices, inefficient strategies often become the main problem that restricts performance and user experience. Inefficient strategies can cause intelligent devices to be unable to respond quickly to user needs or adapt to environmental changes. For example, the device may delay adjusting its operating state to adapt to new temperature or humidity requirements, thereby affecting overall performance and efficiency. Secondly, such inefficient strategies can also lead to waste of energy and computing resources, as the device may run excessively or start frequently when it is not needed, without achieving the purpose of optimizing resource use. SUMMARY

[0003] The purpose of the present application is to provide a large model agent construction method and system, a refrigeration equipment and a storage medium.

[0004] The present application provides a large model agent construction method, characterized in that it comprises the following steps:

[0005] Obtaining multi-modal data, the multi-modal data comprising at least two of voice data, text data, image data, audio data and video data;

[0006] Processing and converting the multi-modal data;

[0007] Classifying the converted multi-modal data into action data and state data;

[0008] Obtaining historical multi-modal data and classifying it into historical action data and historical state data;

[0009] Migrating and fusing the action data and the state data, the historical action data and the historical state data;

[0010] Migrating and merging data weights and network model parameters, determining dynamic types through a classifier, and obtaining the agent action output result;

[0011] Constructing a reward model for feedback;

[0012] Outputting the agent prediction target content.

[0013] As a further improvement of the present application, the classification of the converted multi-modal data into action data and state data specifically comprises:

[0014] The converted multi-modal data is classified into action data and state data by using a text classification model, the action data including text indicating or commanding, and the state data including text describing system or environment state.

[0015] As a further improvement of the present application, the historical multi-modal data is obtained and classified into historical action data and historical state data, specifically including:

[0016] The historical multi-modal data is migrated to a unified feature space;

[0017] Data features of the historical multi-modal data are extracted;

[0018] The extracted data features are classified by a classifier into state process features and action process features, obtaining the historical action data and the historical state data.

[0019] As a further improvement of the present application, the action data and the state data, the historical action data and the historical state data are migrated and fused, specifically including:

[0020] Data representations of the action data and the state data, the historical action data and the historical state data are obtained;

[0021] The action data and the state data, the historical action data and the historical state data are migrated and fused.

[0022] As a further improvement of the present application, the data weights and the network model parameters are migrated and fused, and the dynamic type is determined by a classifier, obtaining the agent action output result, specifically including:

[0023] The data weights and the network model parameters are migrated by a decoder with a multi-head attention mechanism to obtain a migration strategy feature vector;

[0024] The migration strategy feature vector is input into a classifier to determine the dynamic type or the action information output, obtaining the agent action output result.

[0025] As a further improvement of the present application, the reward model is constructed for feedback, specifically including:

[0026] Reward text is obtained and encoded;

[0027] The encoded reward data is input into a classifier to identify and determine different types of reward information.

[0028] As a further improvement of the present application, the agent prediction target content is output, specifically including:

[0029] The type of information that the agent needs to convey is determined;

[0030] Based on the information type, a corresponding display mode is selected to convey the prediction result of the agent.

[0031] As a further improvement of the present application, real-time multi-modal data is acquired, specifically including:

[0032] Multi-modal data acquisition channel initialization;

[0033] Acquiring the multi-modal data collected by the data collection device, and / or

[0034] Acquiring the multi-modal data transmitted from the customer terminal.

[0035] As a further improvement of the present application, the multi-modal data is preprocessed and converted, specifically including:

[0036] Cleaning and format conversion of the original multi-modal data;

[0037] Voice and video separation is performed on the video data to obtain separated video voice data and video image data;

[0038] Preprocessing of text data;

[0039] Standardization adjustment processing of image data and video image data.

[0040] As a further improvement of the present application, the multi-modal data is preprocessed and converted, specifically including:

[0041] Extracting features of the voice data and video voice data to obtain voice features;

[0042] Inputting the voice features into a speech recognition convolutional neural network to obtain output text data;

[0043] Aligning the output text data sequence with the voice feature sequence through a connection time series classification method;

[0044] After the output text data is combined through a fully connected layer, the voice text data is output to a classification function for score calculation.

[0045] The present application also provides a large model agent construction system, which comprises:

[0046] A data acquisition module configured to acquire multi-modal data, the multi-modal data including at least two of voice data, text data, image data, audio data, and video data;

[0047] A data processing module configured to process and convert the multi-modal data; and classify the converted multi-modal data into action data and state data.

[0048] The historical data processing module is configured to acquire historical multimodal data and classify it into historical action data and historical state data.

[0049] The data migration and fusion module is configured to migrate and fuse the action data and the state data, the historical action data and the historical state data; migrate and merge the data weights and network model parameters; and determine the dynamic type through a classifier to obtain the action output result of the agent.

[0050] The reward feedback module is configured to build a reward model for feedback.

[0051] An output module is configured to output the predicted target content of the agent.

[0052] As a further improvement of the present invention, the data processing module uses a text classification model to divide the converted multimodal data into action data and state data. The action data includes text indicating or commanding, and the state data includes text describing the state of the system or environment.

[0053] As a further improvement of the present invention, the historical data processing module is configured to migrate historical multimodal data to a unified feature space; extract data features from the historical multimodal data; and classify the extracted data features into state process features and action process features by a classifier to obtain the historical action data and the historical state data.

[0054] The present invention also provides a refrigeration device, comprising:

[0055] Memory, used to store executable instructions;

[0056] The processor, when running the executable instructions stored in the memory, implements the above-described method for constructing large-scale intelligent agents.

[0057] The present invention also provides a computer-readable storage medium storing executable instructions that, when executed by a processor, implement the above-described method for constructing large-scale intelligent agents.

[0058] The beneficial effects of this invention are as follows: By efficiently integrating and processing multimodal data, this invention effectively divides multimodal data into action data and state data. Furthermore, to enhance the decision-making and predictive capabilities of the smart refrigerator, historical action and state data are migrated and fused with real-time data, optimizing the comprehensive application of data and improving prediction accuracy. Based on this, through decoder processing using a multi-head attention mechanism, the migration of data weights and network model parameters is optimized, enabling it to exhibit excellent adaptability and responsiveness in the face of complex user needs and environmental changes. Attached Figure Description

[0059] Figure 1 This is a schematic diagram of the steps in a method for constructing a large model intelligent agent according to an embodiment of the present invention.

[0060] Figure 2 This is a schematic diagram of the preprocessing steps for multimodal data according to one embodiment of the present invention.

[0061] Figure 3 This is a schematic diagram illustrating the steps of converting voice data into text data in one embodiment of the present invention.

[0062] Figure 4 This is a schematic diagram illustrating the steps of converting image data into text data according to one embodiment of the present invention.

[0063] Figure 5 This is a schematic diagram of step S4 of the large model intelligent agent construction method in one embodiment of the present invention.

[0064] Figure 6 This is a block diagram of a large model intelligent agent construction system according to an embodiment of the present invention. Detailed Implementation

[0065] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0066] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0067] This embodiment provides a method for constructing large-scale intelligent agents. By utilizing real-time, offline, and historical data, combined with multi-agent and deep reinforcement learning, it optimizes the accuracy and efficiency of data migration and information interaction, and achieves efficient data weight and network model parameter migration. This addresses the problem of low policy efficiency in large-scale intelligent agents, improves the agent's model parameters, data representation, and migration capabilities, thereby enhancing the efficiency and quality of information interaction.

[0068] In this embodiment, a smart refrigerator is used as an application example to illustrate the implementation of this technology. The multimodal data acquisition and processing flow has been specifically optimized for the specific usage environment of the smart refrigerator, in order to meet the specific needs of refrigeration equipment in terms of information interaction and service operation.

[0069] like Figure 1 As shown, this embodiment provides a method for constructing a large-scale intelligent agent, which includes the following steps:

[0070] S1: Acquire multimodal data, which includes at least two of the following: speech data, text data, image data, audio data, and video data.

[0071] S2: Process and transform the multimodal data.

[0072] S3: Classify the transformed multimodal data into action data and state data.

[0073] S4: Obtain historical multimodal data and classify it into historical action data and historical state data.

[0074] S5: Migrate and merge the action data and the state data, the historical action data and the historical state data.

[0075] S6: Transfer and merge the data weights and network model parameters, determine the dynamic type through the classifier, and obtain the action output result of the agent.

[0076] S7: Build a reward model for feedback.

[0077] S8: Output the predicted target content of the intelligent agent.

[0078] Step S1 specifically includes:

[0079] Multimodal data acquisition channel initialization; acquiring multimodal data collected by the data acquisition device, and / or acquiring multimodal data transmitted from the client terminal.

[0080] In this embodiment, multimodal data includes speech data, text data, image data, audio data, and video data.

[0081] Various multimodal data input methods have been set up to ensure the effective collection and transmission of different types of data. For smart refrigerators, multimodal information is collected by combining multiple data acquisition methods, including real-time and offline audio, text, and video information. Diverse data access methods can adapt to different usage scenarios and user needs, thereby improving the convenience and efficiency of user interaction.

[0082] In this embodiment, the smart refrigerator collects the user's voice information in real time through a built-in microphone and pickup, while video and image information are captured by a camera inside the refrigerator. Furthermore, the user's voice, video, and text information can also be transmitted through various client devices connected to the refrigerator, such as smartphones, tablets, smart speakers, smart bracelets, or Bluetooth headsets. Users can input multimodal information into these devices, which then transmit the information to the smart refrigerator via wireless technologies such as WiFi or Bluetooth.

[0083] By integrating multiple information formats such as text, images, and sound, multimodal data provides richer and more comprehensive content than single-format information. This comprehensive utilization of information greatly enhances the ability of large models to recognize and understand complex phenomena. Multimodal data integrates information from different sources, comprehensively showcasing multiple aspects of a thing or event, helping models to delve deeper into key features and correlations, thus improving accuracy in prediction or classification tasks. When single-modal information is affected by noise, occlusion, or other interference, the complementary information in multimodal data helps reduce these effects, enhancing the robustness of the model.

[0084] Step S1 establishes a comprehensive, efficient, and reliable data input channel, providing a solid foundation for in-depth information exchange between refrigeration equipment and users, and providing high-quality input data for subsequent data processing, analysis, and training of large-scale models.

[0085] like Figure 2 As shown, in step S2, the multimodal data is first preprocessed, which specifically includes:

[0086] S201: Clean and convert the format of the raw multimodal data.

[0087] S202: Perform audio-video separation on the video data to obtain the separated audio-video data and video-image data.

[0088] S203: Preprocess the text data.

[0089] S204: Standardize and adjust image data and video image data.

[0090] For video data, audio and image content are segmented to extract audio streams and visual elements separately. This allows different data types to undergo specific subsequent processing and analysis. This can be accomplished using existing software tools and programming libraries, or by writing specific scripts to customize audio and video processing steps to meet automation and special needs. The segmented pure audio portion is suitable for speech recognition and sentiment analysis, while the image content separated from the video can be used for tasks such as image recognition and object tracking. This segmentation method allows us to selectively choose the most appropriate techniques and algorithms to process this data, thereby improving the efficiency and accuracy of data processing.

[0091] For text data, preprocessing is performed to clean and organize the text, making it more suitable for efficient analysis and model training. Preprocessing includes steps such as removing useless words, deleting duplicate content, and word segmentation. Eliminating these elements helps to streamline the dataset, speed up processing, and improve the accuracy of model training. Duplicate information in the text increases processing complexity and may introduce bias during training. By identifying and removing this redundant information, the quality and diversity of the dataset can be ensured. Furthermore, breaking down the text into smaller units, such as words or phrases, and performing operations such as stemming, lemmatization, and punctuation removal significantly improve the efficiency and quality of data processing, laying a solid foundation for subsequent text analysis, feature extraction, and model training.

[0092] For image data and image data extracted from videos, standardization processing is performed to improve the efficiency of model training. Processing measures include adjusting image size, standardization, and noise removal. By unifying the size and resolution of image and video frames, and adjusting the pixel values ​​of image data to a certain standard range, the training process can converge faster and the model's stability can be improved.

[0093] like Figure 3 As shown, after preprocessing the multimodal data, the data undergoes a transformation process to convert speech data into text data. Specifically, this includes:

[0094] S205: Extract features from speech data and video speech data to obtain speech features.

[0095] S206: Input the speech features into the speech recognition convolutional neural network to obtain the output text data.

[0096] S207: Align the output text data sequence with the speech feature sequence using a concatenated temporal classification method.

[0097] S208: The output text data is combined through a fully connected layer and then output to a classification function to calculate the score and obtain the speech text data.

[0098] When processing voice information from refrigeration equipment such as refrigerators, connection-time classification is employed to effectively convert the voice information into text. This method is particularly suitable for scenarios with inconsistent input and output sequence lengths and exhibits excellent performance in continuous speech recognition. By assigning labels to each time frame in the sequence, connection-time classification enables large models to automatically identify the correspondence between the input speech and the generated text.

[0099] Furthermore, the constructed deep learning model integrates recurrent neural networks (RNNs) and convolutional neural networks (CNNs), combining a loss function for temporal classification with an attention mechanism. The CNNs capture local features in the speech signal, while the RNNs process sequential data and resolve temporal dependencies. The attention mechanism enhances the model's focus on key information within the speech signal.

[0100] By implementing connection-based temporal classification technology and an end-to-end deep learning model that integrates recurrent and convolutional networks, the accuracy of converting speech data into text in refrigeration equipment has been significantly improved, providing strong technical support for achieving efficient and accurate multimodal interaction.

[0101] like Figure 4 As shown, the data conversion process, specifically converting image data into text data, includes:

[0102] S209: Obtain the image feature sequence of image data and video image data.

[0103] S210: Image semantic representation is obtained through a large model of distillation-diffusion deep fusion based on spatiotemporal and long-distance dependent features.

[0104] S211: Output image semantics by connecting temporal classification and fully connected layers.

[0105] Specifically, step S210 includes:

[0106] S2101: After the large-scale model of distillation-diffusion fusion is trained, the teacher model is introduced into a time- and space-continuous student model to obtain a student model with the same parameters as the teacher model.

[0107] S2102: Convert the large distillation model into a student model with discrete time steps and a short number of steps, and repeat the above steps until the teacher model is distilled to half the number of steps of the student model, i.e., N / 2.

[0108] When extracting and recognizing text content from image and video data, considering the complexity of recognizing text in speaker images, such as varying sentence lengths, vocabulary diversity, and contextual relationships between sentences, a large model combining spatiotemporal relationships and long-term dependency attributes is adopted. This model utilizes deep transfer and knowledge distillation techniques to track features that change over time in images and effectively handle the contextual information of long texts.

[0109] In the knowledge distillation stage, the complex "teacher" model simplifies its knowledge and passes it to a lighter "student" model. Through deep fusion technology, the outputs of the teacher and student models are further integrated, improving the model's ability to handle time series and contextual dependencies, and enhancing the efficiency of mining semantic features in sentence sequences.

[0110] Multi-source text data obtained from multimodal data processing are aggregated into a unified dataset. Data integration constructs a high-quality and content-rich training set, supporting the construction of large models and further improving the efficiency of model training and generation tasks.

[0111] Step S3 specifically includes:

[0112] The transformed multimodal data is divided into action data and state data using a text classification model. The action data includes text indicating or commanding actions, and the state data includes text describing the state of the system or environment.

[0113] The action and state data classifier is responsible for effectively classifying the input text data, ensuring that the data is correctly understood and processed according to its action and state attributes. Action data contains instructions or commands; in the field of refrigeration equipment, this could include phrases such as "turn on the refrigerator" or "set the temperature," which are specific operations that the system needs to perform. State data describes the current state of the system or its environment; in the field of refrigeration equipment, this could include phrases such as "the refrigerator is currently at 5 degrees Celsius" or "the door is currently open," information that helps the system assess the current situation.

[0114] A classifier based on a fast text classification model, such as the Fast-Text classifier, is used for classification. The Fast-Text classifier uses n-gram features and hierarchical classification techniques, can handle large-scale text data, is suitable for multiple languages, and is particularly suitable for fast and efficient text classification. It can be applied to scenarios that require quickly extracting key information from short texts.

[0115] Data classification can employ supervised or semi-supervised learning methods. When using supervised learning, the classifier learns from a training dataset already labeled with the correct action and state categories, resulting in high classification accuracy. Semi-supervised learning combines labeled and unlabeled data to improve model performance in environments with limited data. By utilizing a large amount of unlabeled data, the model's generalization ability can be enhanced when data labeling is costly or when available labeled data is limited.

[0116] like Figure 5 As shown, step S4 specifically includes:

[0117] S41: Migrate historical multimodal data to a unified feature space.

[0118] S42: Extract data features from historical multimodal data.

[0119] S43: The extracted data features are classified by a classifier into state process features and action process features to obtain the historical action data and the historical state data.

[0120] Training agents using historical data provides a wealth of information and experience, offering valuable resources for training large models. Historical data typically encompasses diverse scenarios and cases, increasing the diversity of training samples and thus contributing to improved generalization and prediction accuracy of large models.

[0121] The migration of historical action and status data integrates data collected at different points in time into a continuous timeline. This includes historical data collected from various sensors and user inputs, such as temperature settings, door open / close status, and usage patterns. Historical data is integrated using data migration techniques, ensuring that the data maintains its time-series integrity and relevance in the new analytical environment.

[0122] By projecting all historical state and action data into a unified feature space, this process ensures that data from different time points and sources can be compared and processed within the same analytical framework through standardized data formats and scales. This facilitates subsequent data analysis tasks such as pattern recognition and trend prediction.

[0123] Deep convolutional networks can be used to extract data features from historical multimodal data. These features can better reflect the essential attributes of historical multimodal data, thereby enabling classifiers to classify the historical multimodal data.

[0124] By applying classifiers to process extracted features, we can efficiently obtain quantified values ​​of state and dynamic process features, such as probability distributions. These quantified values ​​reflect the probability and importance of various states and actions, providing data support for system decision-making. Probability distributions can describe statistical predictions of future states and action outcomes. In an agent, it can assess the probability of various results caused by different actions and adjust its response to optimize operational efficiency and effectiveness, such as adjusting resource allocation in advance when predicting peak demand. As the environment and input data change, the system can use real-time data to update its probability distribution to reflect the latest information, maintaining the adaptability and timeliness of its decisions.

[0125] Step S5 specifically includes:

[0126] Acquire data representations of action data, state data, historical action data, and historical state data; migrate and merge action data, state data, historical action data, and historical state data.

[0127] In intelligent agents, the transfer and merging of action and state data ensures data continuity and consistency. Using neural network models with encoders, such as autoencoders or variational autoencoders, high-dimensional state and action data can be effectively transformed into feature vectors. Data compression helps extract the most informative features from the data, reducing processing complexity while retaining key information.

[0128] In this implementation, data migration and merging can be achieved using Markov process computation. Markov process computation, which handles the migration or transfer of real-time, offline, and historical state and action data, is a method based on statistical models. It assumes that the future state of the system depends only on the current state (Markov property), making it suitable for describing random changes in state over continuous or discrete time processes. By constructing a state transition matrix, the probability distribution of the next state is predicted, thereby achieving mathematical modeling and prediction of system behavior.

[0129] By migrating and fusing state and action data, comprehensive data synchronization is achieved. Processed state and action data are integrated into a unified data structure, allowing the system to simultaneously consider state changes and action feedback. After data fusion, the interactions and dependencies between states and actions can be analyzed more accurately, improving the accuracy and efficiency of decision-making algorithms.

[0130] In smart home systems such as cooling equipment, by integrating users' daily activity data (actions) and environmental data (status), environmental settings, such as temperature, lighting, and security systems, can be adjusted more intelligently to adapt to users' behavioral patterns and preferences. The migration and merging of action and status data not only enhances the depth and breadth of data processing but also provides a unified analytical foundation for the system, making decision-making more data-driven and responses more timely and accurate.

[0131] Step S6 specifically includes:

[0132] S61: The transfer strategy feature vector is obtained by transferring the data weights and network model parameters through a decoder with a multi-head attention mechanism.

[0133] S62: The action output of the agent is obtained by passing the transfer strategy feature vector through the classifier to determine the dynamic type or action information.

[0134] In step S61, a Transformer decoder with a multi-head attention mechanism can be used to transfer data weights and network model parameters. In the Transformer architecture, multi-head attention allows the model to simultaneously focus on multiple aspects of the input data in different subspaces, enabling it to capture relevant information from a broader context and improve its understanding of complex data patterns. In applications involving action and state data fusion, the most critical information influencing the current decision can be identified and emphasized. After processing the fused action and state data through the Transformer decoder, a transfer policy vector is generated, containing optimized data weights, network layer number, and parameter weights. These vectors provide specific guidance for the agent's operations, making its behavior more aligned with the current environment and task requirements.

[0135] After obtaining the transfer policy vectors, the intelligent system inputs these vectors into the classifier. The classifier predicts the dynamic type or specific action to be taken based on these vectors, and determines the behavior of the agent in a specific situation.

[0136] In smart refrigerators, a multi-head attention mechanism is used to analyze and understand user voice commands and various sensor data on the refrigerator's internal state (such as changes in temperature, humidity, and stored contents). This mechanism can simultaneously attend to multiple input sources, ensuring the refrigerator fully understands and adapts to the user's specific needs. The Transformer decoder processes the fused motion data (such as user commands to adjust the set temperature) and state data (such as current temperature, humidity, and food storage conditions) to generate migration strategy vectors. These vectors help the smart refrigerator precisely adjust its operating strategies, such as automatically adjusting cooling intensity or changing cooling modes to maintain food freshness.

[0137] By analyzing the transfer policy vector using a classifier, a smart refrigerator can determine when a specific operation needs to be performed, such as activating defrost mode, issuing a maintenance reminder, or adjusting the internal temperature setting. This allows the refrigerator to not only respond to direct user commands but also predict and meet unexpressed needs.

[0138] Step S7 specifically includes:

[0139] The reward text is obtained and encoded; the encoded reward data is then input into a classifier to identify and distinguish different types of reward information.

[0140] Building reward models helps guide the behavior of intelligent agents, shaping desired action patterns through rewards or punishments. For refrigeration equipment such as smart refrigerators, reward models can be used to optimize equipment operation, ensure maximum energy efficiency, and improve user satisfaction.

[0141] Acquiring and collecting reward text data in smart refrigerator applications includes user feedback, device performance logs, and energy consumption records. For example, positive user feedback on the refrigerator's cooling effect can be considered positive reward text data. Data collection can be accomplished through online surveys, direct feedback from the user interface, or automatic recording of device operation data.

[0142] Once the reward text data is collected, it is encoded using a variable encoder. A variable encoder is a neural network that generates effective data representations by reconstructing the input by learning a compressed representation of the input data.

[0143] Finally, the encoded reward data is fed into a classifier, which analyzes data features to determine which behaviors should be rewarded, penalized, or ignored. This information can then be used to adjust the agent's behavioral decision-making model to achieve optimal operational efficiency and user satisfaction. The reward model helps the device learn how to adjust its operating modes based on user preferences, such as adjusting temperature settings or automatically reordering refrigerator contents, to improve usability and satisfaction.

[0144] Step S8 specifically includes:

[0145] Determine the type of information the agent needs to convey; based on the information type, select the corresponding display method to convey the agent's prediction results.

[0146] Smart devices predict user intentions and needs by analyzing user input (such as voice commands and text input). This input is then translated into specific operational instructions, such as adjusting the temperature or activating a specific function, or into useful feedback information, such as status reports and prompts.

[0147] The prediction results can be presented in various ways to suit different usage scenarios and user preferences. This includes displaying them on the cooling equipment's built-in screen or through external devices such as smartphones and tablets.

[0148] The display methods built into the device may include text and image information on the built-in screen, while the display of external devices utilizes external connections or networks to transmit information.

[0149] In summary, this implementation method efficiently integrates and processes multimodal data, effectively dividing it into action data and state data. To enhance the decision-making and predictive capabilities of the smart refrigerator, historical action and state data are migrated and fused with real-time data, optimizing the comprehensive application of data and improving prediction accuracy. Furthermore, through a multi-head attention mechanism decoder, the migration of data weights and network model parameters is optimized, enabling the refrigerator to exhibit superior adaptability and responsiveness in the face of complex user needs and environmental changes.

[0150] like Figure 6 As shown, this embodiment also provides a large-scale model intelligent agent construction system, which includes:

[0151] The data acquisition module 1 is configured to acquire multimodal data, which includes at least two of the following: voice data, text data, image data, audio data, and video data.

[0152] Data processing module 2 is configured to process and transform the multimodal data; and classify the transformed multimodal data into action data and state data.

[0153] Historical data processing module 3 is configured to acquire historical multimodal data and classify it into historical action data and historical state data;

[0154] The data migration and fusion module 4 is configured to migrate and fuse the action data and the state data, the historical action data and the historical state data; migrate and merge the data weights and network model parameters; and determine the dynamic type through a classifier to obtain the action output result of the agent.

[0155] Reward feedback module 5 is configured to build a reward model for feedback;

[0156] Output module 6 is configured to output the predicted target content of the agent.

[0157] Furthermore, the data processing module 2 uses a text classification model to divide the converted multimodal data into action data and state data. The action data includes text indicating or commanding, and the state data includes text describing the state of the system or environment.

[0158] Furthermore, the historical data processing module 3 is configured to migrate the historical multimodal data to a unified feature space; extract the data features of the historical multimodal data; and classify the extracted data features into state process features and action process features using a classifier to obtain the historical action data and the historical state data.

[0159] This embodiment also provides a refrigeration device, including:

[0160] Memory, used to store executable instructions;

[0161] The processor, when running the executable instructions stored in the memory, implements the above-described method for constructing large-scale intelligent agents.

[0162] This embodiment also provides a computer-readable storage medium storing executable instructions that, when executed by a processor, implement the above-described large model intelligent agent construction method.

[0163] It should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This way of describing the specification is only for clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

[0164] The detailed descriptions listed above are merely specific descriptions of feasible implementations of the present invention and are not intended to limit the scope of protection of the present invention. All equivalent implementations or modifications made without departing from the spirit of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for constructing a large-scale intelligent agent, characterized in that, Including the following steps: Acquire multimodal data, wherein the multimodal data includes at least two of the following: speech data, text data, image data, audio data, and video data; The multimodal data is processed and transformed; The transformed multimodal data is classified into action data and state data; Acquire historical multimodal data and classify it into historical action data and historical state data; The action data and the state data, as well as the historical action data and the historical state data, are migrated and fused. The data weights and network model parameters are transferred and merged, and the dynamic type is determined by a classifier to obtain the action output result of the agent; Build a reward model for feedback; Output the predicted target content of the agent.

2. The method for constructing a large-scale intelligent agent according to claim 1, characterized in that, The process of classifying the converted multimodal data into action data and state data specifically includes: The transformed multimodal data is divided into action data and state data using a text classification model. The action data includes text indicating or commanding actions, and the state data includes text describing the state of the system or environment.

3. The method for constructing a large-scale intelligent agent according to claim 1, characterized in that, The acquisition of historical multimodal data, and its classification into historical action data and historical state data, specifically includes: Migrate historical multimodal data to a unified feature space; Extract data features from historical multimodal data; The extracted data features are classified by a classifier into state process features and action process features to obtain the historical action data and the historical state data.

4. The method for constructing a large-scale intelligent agent according to claim 1, characterized in that, The migration and fusion of the action data and the state data, the historical action data and the historical state data specifically includes: Acquire data representations of action data, state data, historical action data, and historical state data; The action data and status data, as well as historical action data and historical status data, are migrated, merged, and integrated.

5. The method for constructing a large-scale intelligent agent according to claim 1, characterized in that, The process of transferring and merging data weights and network model parameters, and determining the dynamic type through a classifier to obtain the agent's action output results specifically includes: The transfer strategy feature vector is obtained by transferring the data weights and network model parameters through a decoder with a multi-head attention mechanism. The action output of the agent is obtained by using the feature vector of the transfer strategy to determine the dynamic type or action information through the output of the classifier.

6. The method for constructing a large-scale intelligent agent according to claim 1, characterized in that, The construction of the reward model for feedback specifically includes: Obtain the reward text and encode it; The encoded reward data is input into a classifier to identify and distinguish different types of reward information.

7. The method for constructing a large-scale intelligent agent according to claim 1, characterized in that, The output of the agent's predicted target content specifically includes: Determine the type of information the agent needs to convey; Based on the type of information, select the corresponding display method to convey the agent's prediction results.

8. The method for constructing a large-scale intelligent agent according to claim 1, characterized in that, Real-time acquisition of multimodal data, specifically including: Multimodal data acquisition channel initialization; The multimodal data acquired by the data acquisition device, and / or The multimodal data is acquired from the client terminal.

9. The method for constructing a large-scale intelligent agent according to claim 8, characterized in that, The preprocessing and transformation of the multimodal data specifically includes: Clean and convert the format of the raw multimodal data; The video data is subjected to audio-video separation to obtain separated video-audio data and video-image data; Preprocess the text data; Standardize and adjust image and video image data.

10. The method for constructing a large-scale intelligent agent according to claim 9, characterized in that, The preprocessing and transformation of the multimodal data further includes: Extract features from the speech data and video speech data to obtain speech features; The speech features are input into a speech recognition convolutional neural network to obtain the output text data; The output text data sequence is aligned with the speech feature sequence using a connection-time classification method; The output text data is combined through a fully connected layer and then output to a classification function to calculate the score, thus obtaining the speech text data.

11. A large-scale intelligent agent construction system, characterized in that, include: The data acquisition module is configured to acquire multimodal data, which includes at least two of the following: voice data, text data, image data, audio data, and video data. A data processing module is configured to process and transform the multimodal data; and classify the transformed multimodal data into action data and state data. The historical data processing module is configured to acquire historical multimodal data and classify it into historical action data and historical state data. A data migration and fusion module is configured to migrate and fuse the action data and the state data, as well as the historical action data and the historical state data. The data weights and network model parameters are transferred and merged, and the dynamic type is determined by a classifier to obtain the action output result of the agent; The reward feedback module is configured to build a reward model for feedback. An output module is configured to output the predicted target content of the agent.

12. The large-scale intelligent agent construction system according to claim 11, characterized in that, The data processing module uses a text classification model to divide the converted multimodal data into action data and state data. The action data includes text indicating or commanding, and the state data includes text describing the state of the system or environment.

13. The large-scale intelligent agent construction system according to claim 11, characterized in that, The historical data processing module is configured to migrate historical multimodal data to a unified feature space; extract data features from the historical multimodal data; and classify the extracted data features into state process features and action process features using a classifier to obtain the historical action data and the historical state data.

14. A refrigeration device, comprising: Memory, used to store executable instructions; A processor, when executing executable instructions stored in the memory, implements the large model agent construction method according to any one of claims 1 to 10.

15. A computer-readable storage medium storing executable instructions that, when executed by a processor, implement the large model intelligent agent construction method according to any one of claims 1 to 10.

Images