Multimodal large model construction method, system, refrigeration equipment, and storage medium

Abstract

The present invention provides a multimodal large model construction method, system, refrigeration equipment, and storage medium. First, by utilizing multimodal data and performing targeted processing on different types of data, a rich and high-quality corpus is prepared for large model training. Then, corresponding training corpora are generated according to different learning needs, and features are extracted and analyzed for the generated training corpora. These features are then compared with the existing feature library to determine and update the content of the feature library. In addition, a word-level semantic compression large model is used to optimize the prompt generation process, and the processed data encoding is input into the large model for pre-training. Through this collaborative workflow that integrates feature extraction, prompt generation, and large model training, the training efficiency and prediction accuracy of the model can be significantly improved. Not only is the computing resource utilization of the model optimized, but also the model's adaptability and generalization capabilities to new situations are enhanced.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a multi-modal large model construction method, system, refrigeration equipment, and storage medium. Background Art

[0002] Rapid development and application of deep learning and artificial intelligence technologies in the field of smart home appliances and refrigeration equipment. As smart devices become more and more popular, their requirements for data processing and interaction efficiency are also increasing. In the context of big data, how to improve the computing efficiency of large AI models and reduce model training and inference time has become an urgent problem to be solved. Especially in the field of smart home appliances, the equipment has high requirements for real-time performance, and the improvement of model efficiency directly affects the user experience. With the increasing complexity of smart home appliance functions, large AI models are required to not only handle single tasks, but also show good synergy in multi-task and multi-modal data processing. Summary of the invention

[0003] The object of the present invention is to provide a multi-modal large model construction method, system, refrigeration equipment, and storage medium.

[0004] The present invention provides a method for constructing a multimodal large model, which comprises the steps of:

[0005] Acquiring multimodal data, the multimodal data comprising at least two of voice data, text data, image data, audio data, and video data;

[0006] Preprocessing and converting the multimodal data, and fusing the converted multimodal data;

[0007] Generate a large model training corpus based on fused multimodal data;

[0008] Extract and analyze the features of the training corpus, compare the extracted features with the current feature library, and iteratively update and optimize the content of the current feature library;

[0009] Constructing prompt content based on the extracted features, and encoding and embedding the prompt content into a large model;

[0010] Training and adjusting the large model based on the prompt content;

[0011] Package and deploy the large model.

[0012] As a further improvement of the present invention, feature extraction and analysis are performed on the training corpus, the extracted features are compared with the current feature library, and the content of the current feature library is iteratively updated and optimized, specifically including:

[0013] Extract features from structured and unstructured data;

[0014] When the extracted features do not exist in the current feature library, the extracted features are encoded and added to the current feature library.

[0015] As a further improvement of the present invention, the step of constructing prompt content based on the extracted features, and encoding and embedding the prompt content into a large model specifically includes:

[0016] The prompt content data is compressed through a word-level semantic compression model, and a low-rank weight matrix decomposition model is used to represent the target weight matrix of the prompt content data as a sparse coefficient matrix and a dense weight matrix;

[0017] The compressed prompt content data and the encoded text data obtained by preprocessing the multimodal data are input into the embedding layer of the large model.

[0018] As a further improvement of the present invention, the acquiring of multimodal data specifically includes:

[0019] Initialize multimodal data acquisition channels;

[0020] Acquiring multimodal data collected by a data collection device, and / or,

[0021] Acquire multimodal data transmitted from a client terminal.

[0022] As a further improvement of the present invention, the multimodal data includes voice data, text data, image data, audio data and video data.

[0023] As a further improvement of the present invention, the preprocessing and converting of the multimodal data specifically includes:

[0024] Cleaning and formatting the multimodal data;

[0025] Separating the video data into audio and video to obtain separated video audio data and video image data;

[0026] Preprocess the text data;

[0027] Perform standardization and adjustment processing on image data and video image data.

[0028] As a further improvement of the present invention, the preprocessing and converting of the multimodal data further includes:

[0029] Extracting features of the voice data and video voice data to obtain voice features;

[0030] Inputting the speech features into a speech recognition convolutional neural network to obtain output text data;

[0031] Aligning the output text data sequence with the speech feature sequence by a connection temporal classification method;

[0032] The output text data is combined through a fully connected layer and then output to a classification function to calculate a score to obtain speech text data.

[0033] As a further improvement of the present invention, the preprocessing and converting of the multimodal data further includes:

[0034] Acquire image feature sequences of image data and video image data;

[0035] The image semantic representation is obtained by deep fusion of large models based on distillation diffusion and spatiotemporal and long-distance dependent features;

[0036] The image semantics is output by connecting the temporal classification and the fully connected layer.

[0037] As a further improvement of the present invention, the image semantic representation is obtained by using a distillation diffusion deep fusion large model based on spatiotemporal and long-distance dependent features, specifically including:

[0038] After the distillation diffusion deep fusion large model is trained, the teacher model is introduced into a student model that is continuous in time and space to obtain a student model with the same parameters as the teacher model;

[0039] Convert the distilled large model into a student model with discrete time steps and short steps, and repeat the above steps until the teacher model is distilled to half the steps of the student model.

[0040] As a further improvement of the present invention, it also includes:

[0041] Obtain historical multimodal data, vectorize the historical multimodal data, and map it to the same vector space for transfer learning.

[0042] As a further improvement of the present invention, the generating of a large model training corpus based on the fused multimodal data specifically includes:

[0043] According to the data training type, large model training corpus of supervised learning, semi-supervised learning and unsupervised learning types is generated.

[0044] As a further improvement of the present invention, the deploying and applying the large model includes:

[0045] Get a large model service application;

[0046] After review and authorization, the interface is connected to the big model and the big model is called;

[0047] Monitor the performance and status of large models in real time during use, and record call data and performance data.

[0048] The present invention also provides a multi-modal large model construction system, which includes:

[0049] a data acquisition module configured to acquire multimodal data, wherein the multimodal data includes at least two of voice data, text data, image data, audio data, and video data;

[0050] A data processing module, configured to preprocess and convert the multimodal data and fuse the converted multimodal data;

[0051] A corpus generation module, configured to generate a large model training corpus based on the fused multimodal data;

[0052] A feature library construction module is configured to extract and analyze features of the training corpus, compare the extracted features with the current feature library, iteratively update and optimize the content of the feature library; construct prompt content based on the extracted features, and encode and embed the prompt content into a large model; and train and adjust the large model based on the prompt content;

[0053] The big model application module is configured to package and deploy the big model.

[0054] The present invention also provides a refrigeration device, comprising:

[0055] A memory for storing executable instructions;

[0056] The processor is used to implement the above-mentioned multimodal large model construction method when running the executable instructions stored in the memory.

[0057] The present invention also provides a storage medium storing executable instructions, which implement the above-mentioned multimodal large model construction method when executed by a processor.

[0058] The beneficial effects of the present invention are as follows: the method of the present invention firstly utilizes multimodal data and performs targeted processing on different types of data to prepare a rich and high-quality corpus for large model training. Then, the corresponding training corpus is generated according to different learning needs, and the generated training corpus is subjected to feature extraction and analysis, and then these features are compared with the existing feature library to judge and update the content of the feature library. In addition, a word-level semantic compression large model is used to optimize the prompt generation process, and the processed data encoding is input into the large model for pre-training. Through this collaborative workflow integrating feature extraction, prompt generation and large model training, the training efficiency and prediction accuracy of the model can be significantly improved, which not only optimizes the computing resource utilization of the model, but also enhances the model's adaptability and generalization capabilities to new situations. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 It is a schematic diagram of the steps of a method for constructing a multimodal large model in one embodiment of the present invention;

[0060] Figure 2 is a schematic diagram of preprocessing steps of multimodal data in one embodiment of the present invention;

[0061] Figure 3 is a schematic diagram of steps for converting voice data into text data in one embodiment of the present invention;

[0062] Figure 4 is a schematic diagram of steps for converting image data into text data in one embodiment of the present invention;

[0063] Figure 5 is a schematic diagram of step S4 of a multimodal large model construction method in one embodiment of the present invention;

[0064] Figure 6 is a schematic diagram of step S5 of a multimodal large model construction method in one embodiment of the present invention;

[0065] Figure 7 It is a block diagram of a multimodal large model building system in one embodiment of the present invention. DETAILED DESCRIPTION

[0066] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in combination with the specific implementation methods of the present invention and the corresponding drawings. Obviously, the described implementation methods are only part of the implementation methods of the present invention, not all implementation methods. Based on the implementation methods of the present invention, all other implementation methods obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0067] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0068] This embodiment provides a method for constructing a multimodal large model. The large model collects multimodal data, generates training data after fusing the multimodal data, extracts and analyzes features of the generated training corpus, and then compares these features with the existing feature library to determine and update the content of the feature library. In addition, a word-level semantic compression large model is used to optimize the prompt generation process, and the processed data encoding is input into the large model for pre-training. Finally, the large model is ensured to achieve optimal performance through evaluation and fine-tuning. The overall method significantly improves the performance of deep learning models in complex data scenarios.

[0069] The method provided in this embodiment can be applied to a large model of interaction between smart electronic appliances and users. Exemplarily, in this embodiment, the method is described by taking a smart refrigerator as an example. Based on the usage scenario of the smart refrigerator, this embodiment adaptively optimizes the acquisition and processing method of multimodal data, which is suitable for the specific information interaction and service operation requirements of refrigeration equipment.

[0070] like Figure 1 FIG. 1 is a schematic diagram of the steps of a multimodal large model construction method provided by the present invention, which includes:

[0071] S1: Acquire multimodal data, where the multimodal data includes at least two of voice data, text data, image data, audio data, and video data.

[0072] S2: Preprocess and transform the multimodal data, and fuse the transformed multimodal data.

[0073] S3: Generate a large model training corpus based on the fused multimodal data.

[0074] S4: Extract and analyze features of the training corpus, compare the extracted features with the current feature library, and iteratively update and optimize the content of the feature library.

[0075] S5: Construct prompt content based on the extracted features, and encode and embed the prompt content into a large model.

[0076] S6: Package and deploy the application model.

[0077] In step S1, multimodal data is obtained, which specifically includes:

[0078] Initialize multimodal data acquisition channels;

[0079] Acquiring multimodal data collected by a data collection device, and / or,

[0080] Acquire multimodal data transmitted from a client terminal.

[0081] In this embodiment, the multimodal data includes voice data, text data, image data, audio data and video data.

[0082] First, initialize the channel for multimodal information data input to ensure smooth data collection and transmission.

[0083] For smart refrigerators, multimodal information collection relies on integrating multiple input methods, including voice, text, and video data collected in real time and offline. Diversified data access methods can adapt to different usage scenarios and user preferences, thereby improving the convenience and efficiency of user interaction.

[0084] In this embodiment, the smart refrigerator uses a built-in pickup and microphone array to collect the user's real-time voice information. At the same time, video and image data are collected through the camera inside the refrigerator. In addition, the user's voice, video and text information can also be transmitted through a client device connected to the refrigerator through a wireless communication protocol. These devices include smart devices with information sending capabilities such as smart phones, tablets, smart speakers, smart bracelets or Bluetooth headsets. Users can directly input multimodal data into these devices, and then the devices transmit the data to the smart refrigerator via wireless methods such as Wi-Fi or Bluetooth. Through this multi-channel data acquisition method, users can input information without facing the refrigerator, which greatly enhances the flexibility of data acquisition. This information acquisition method significantly improves the diversity and quality of data, which is conducive to improving the training effect of large models.

[0085] Multimodal data provides more comprehensive and detailed information than a single mode by integrating multiple data types such as text, images, and sound. The comprehensive use of information significantly enhances the insight of large models into complex phenomena. By combining different types of data, multimodal data can comprehensively describe the same object or event from multiple perspectives, which can help the model explore key features and internal connections more deeply, thereby achieving higher accuracy when performing prediction or classification tasks. In scenarios where a single data mode may be affected by noise, occlusion, or other disturbances, multimodal data helps mitigate the impact of these problems through its supplementary information and improves the stability of the model.

[0086] By establishing a comprehensive, efficient and robust data input channel, a solid foundation is laid for in-depth information interaction between refrigeration equipment and users, and high-quality input data is provided for subsequent data processing, analysis and large model training.

[0087] like Figure 2 As shown, in step S2, the multimodal data is preprocessed and converted, which specifically includes:

[0088] S201: Clean and format-convert multimodal data.

[0089] S202: Separate the video data into audio and video to obtain separated video audio data and video image data.

[0090] S203: Preprocess the text data.

[0091] S204: Performing standardization adjustment processing on the image data and the video image data.

[0092] It should be noted that, in step S2, there is no clear order between step S202 and step S203, and they can be performed in alternating order or simultaneously.

[0093] For video data, the audio and visual elements must be separated, and the pure audio part and visual content in the audio and video streams must be extracted respectively, so that their respective processing and analysis can be more targeted. Using existing software tools and development libraries, audio and video data can be effectively processed and edited. In addition, scripts can be written to customize audio and video processing methods based on specific needs and automated processes. The separated pure audio data is suitable for applications such as speech recognition and sentiment analysis, while the visual content can be used for tasks such as image recognition and object recognition. This separation allows the processing to apply the most suitable technologies and algorithms in a more targeted manner, thereby improving the efficiency and accuracy of data processing.

[0094] For text data, the preprocessing step aims to clean and optimize the text for more effective analysis and model training. This includes removing stop words, eliminating content duplication, and performing word segmentation. These steps help reduce the burden on the dataset, improve processing speed, and improve the accuracy of model training. Repetitive information in the text increases the difficulty of processing and may cause training bias. By removing these repetitive contents, the quality and diversity of the dataset are ensured. Word segmentation breaks the text into smaller units, such as words or phrases. In addition, text processing may also include steps such as stem extraction or word form restoration, and punctuation removal. Through these text data preprocessing operations, the quality and processing efficiency of the data are significantly improved, providing a solid foundation for subsequent text analysis, feature extraction, and model training.

[0095] For image data and separated video image data, normalization is a key step to improve the efficiency of large model training, including resizing, normalization, and noise elimination. By unifying all images and video frames to the same size and resolution and standardizing the pixel values ​​of these data to a certain range, the convergence process in model training can be accelerated and the stability of the model can be enhanced.

[0096] Through these normalization and adjustment measures, we can ensure that image and video data have a unified format and high quality before being sent to the large model, reduce inconsistencies during the training process, and thus improve the performance of the model in practical applications.

[0097] After preprocessing the multimodal data through the above steps, the multimodal data needs to be converted to uniformly convert the multimodal data into text data.

[0098] like Figure 3 As shown, converting voice data into text data specifically includes:

[0099] S205: Extract features of voice data and video voice data to obtain voice features.

[0100] S206: Input the speech features into the speech recognition convolutional neural network to obtain output text data.

[0101] S207: Align the output text data sequence with the speech feature sequence through a connection temporal classification method.

[0102] S208: After the output text data is combined through the fully connected layer, it is output to the classification function to calculate the score to obtain the speech text data.

[0103] When it comes to speech data from refrigeration equipment such as refrigerators, the influence of its spatiotemporal properties and contextual information is considered. In order to accurately convert these speech data into text, the end-to-end training technology of connected temporal classification is selected. This method is particularly suitable for continuous speech recognition tasks, and it can effectively handle situations where the length of the input and output sequences are inconsistent. By assigning labels to each frame in the sequence, this technology enables the model to learn the matching relationship between sound input and text output by itself.

[0104] In the constructed deep learning architecture, a combination of recurrent neural networks and convolutional neural networks is used, and a connection time series classification loss function and an attention mechanism are incorporated. The convolutional neural network is mainly used to grasp the detailed features in the speech signal, while the recurrent neural network is responsible for analyzing the temporal dependencies in the sequence, and the attention mechanism helps the model focus on the key information in the speech. Such an end-to-end deep learning model can deeply mine the original speech signal and capture important speech features about intonation, rhythm, and volume, which is crucial to improving the accuracy of speech recognition and understanding the user's intentions.

[0105] By introducing connection temporal classification technology and a deep learning model that combines recurrent and convolutional networks, the accuracy of converting refrigeration equipment voice data into text has been significantly improved, providing technical support for effective and accurate multimodal interaction.

[0106] like Figure 4As shown, converting image data into text data specifically includes:

[0107] S209: Acquire image feature sequences of image data and video image data.

[0108] S210: Obtain image semantic representation through a deep fusion model based on distillation diffusion of spatiotemporal and long-distance dependent features.

[0109] S211: Output the image semantics by connecting the temporal classification and the fully connected layer.

[0110] Specifically, step S210 obtains the image semantic representation through a distillation diffusion deep fusion large model based on spatiotemporal and long-distance dependent features, specifically including:

[0111] S2101: After the distillation diffusion deep fusion large model is trained, the teacher model is introduced into a student model that is continuous in time and space to obtain a student model with the same parameters as the teacher model.

[0112] S2102: Convert the distilled large model into a student model with discrete time steps and short steps, and repeat the above steps until the teacher model is distilled to half the steps of the student model.

[0113] Image features are extracted from image data and video image data to identify text content. Considering the complexity of identifying sentences from the image features of speakers, such as the length variation of sentences, the diversity of words, and the contextual relevance between sentences, a large model of deep fusion based on migration and distillation diffusion of spatiotemporal relationships and long-distance dependency features is adopted, so that the features that change over time in images and the contextual information maintained in long texts can be captured and analyzed. The large model of deep fusion of distillation diffusion transfers and simplifies the knowledge of a complex teacher model to a lighter student model through distillation, maintaining the performance of the teacher model while reducing the consumption of computing resources. The outputs of the teacher model and the student model are further integrated using deep fusion technology to optimize the large model's processing capabilities for time series and contextual dependencies, thereby improving the efficiency of mining semantic feature information in sentence sequences.

[0114] Through the above method, the large model can effectively extract and recognize text content from image data, especially process sentences containing complex semantics and long-distance dependencies, improving the accuracy and efficiency of identifying from image features to text content.

[0115] In step S2, after obtaining multi-source text data through multi-modal data processing, the text data from different sources are integrated together, and the effective text data are fused to form a unified data set, which provides a high-quality and information-rich training data set for large model construction, and can improve the efficiency of subsequent large model training and generation work.

[0116] Furthermore, in this embodiment, the method further comprises the steps of:

[0117] Collect and process historical multimodal data, convert it into vector form, and then perform transfer learning in the same vector space. Vectorization ensures that data from different times and devices can be compared and measured in a unified space, which establishes a common framework for further data analysis and application. Implementing transfer learning in this vector space can leverage the previous learning results of existing large models on similar tasks to speed up the processing of new tasks.

[0118] Historical data contains a wealth of information and experience, providing an extremely important resource for the training of large models. These data include a wide range of scenarios and examples, enriching the training samples, which in turn helps to enhance the versatility of large models and improve their prediction accuracy.

[0119] In step S3, the fused text data, image data, audio data, etc. are used to generate large model training corpora of supervised learning, semi-supervised learning and unsupervised learning types.

[0120] By training a comprehensive data set including supervised learning, semi-supervised learning and unsupervised learning materials, the expressiveness, adaptability and breadth of application of the large model can be effectively improved. Through the labeled data in supervised learning, the large model can accurately grasp the relationship between input and output, while integrating the data of semi-supervised and unsupervised learning enables the large model to explore the deep structure and pattern of the data. Adopting this diversified data strategy helps the large model identify more comprehensive data characteristics and enhance its generalization ability.

[0121] In real-world application scenarios, the amount of unlabeled data is usually huge, exceeding the amount of labeled data. Building a dataset containing semi-supervised and unsupervised learning materials can maximize the use of these unlabeled data. This not only greatly reduces the financial and time costs of data labeling, but also improves the learning efficiency and performance of large models by using a large amount of unlabeled data. Using these unlabeled data, especially in the framework of semi-supervised and unsupervised learning, large models can better adapt to new fields or tasks, and quickly adapt and improve performance through transfer learning to optimize their effects in new environments.

[0122] Furthermore, different types of corpora prompt the large model to learn at different levels, from mastering specific task knowledge through labeled data to using unlabeled data to gain insight into the general characteristics and distribution of data. This multi-task learning method allows progress on a certain task to promote the performance of other tasks, improving the adaptability and flexibility of the large model. By combining different types of training data, the large model not only learns clear annotation information, but also actively explores the underlying structure and laws of the data, thereby improving its interpretability and credibility.

[0123] like Figure 5 As shown, in step S4, the training corpus is subjected to feature extraction and analysis, the extracted features are compared with the current feature library, and the content of the feature library is iteratively updated and optimized, which specifically includes:

[0124] S41: Extract features from structured data and unstructured data.

[0125] S42: When the extracted feature does not exist in the current feature library, the extracted feature is encoded and added to the feature library.

[0126] For structured data, such as tables and records in databases, key features are extracted through statistical analysis, association rules and other methods, such as data distribution, central tendency, degree of dispersion, etc. For unstructured data, such as text, images, audio, etc., natural language processing technology is used to extract text features, computer vision technology is used to extract image features, and signal processing technology is used to extract audio features.

[0127] After feature extraction, determine whether the feature already exists. This ensures the uniqueness and effectiveness of the feature and avoids waste of resources caused by feature duplication. For data features identified as new, encode them and convert them into a format that can be understood and processed by the large model.

[0128] Through step S4, key features can be effectively extracted and analyzed from complex data sets, and the novelty and uniqueness of these features can be ensured, providing high-quality input for large model training.

[0129] like Figure 6 As shown, in step S5, prompt content is constructed based on the extracted features, and the prompt content is encoded and embedded in a large model, which specifically includes:

[0130] S51: compress the prompt content data through a word-level semantic compression model, and use a low-rank weight matrix decomposition model to represent the target weight matrix of the prompt content data as a sparse coefficient matrix and a dense weight matrix.

[0131] S52: Input the compressed prompt content data and the encoded text data obtained by preprocessing the multimodal data into the embedding layer of the large model.

[0132] A semantic compression large model suitable for word-level processing is selected so that when processing text data, the large model can identify and compress the semantic information between words, reduce the dimension and complexity of data representation, and thus improve processing speed and accuracy.

[0133] A low-rank weight matrix decomposition model is adopted. The target weight matrix of the prompt content data is divided into blocks, and each block is represented as a sparse coefficient matrix and a dense weight matrix, thereby reducing the computational complexity of the weight matrix.

[0134] Through step S51, in the process of optimizing data prompt generation, efficient use of computing resources and significant improvement in processing speed are achieved, while maintaining the semantic integrity and accuracy of data features.

[0135] The compressed prompt content data and the encoded text data obtained by preprocessing the multimodal data are converted into numerical vector representations, and the encoded data is used to pre-train the deep learning model. After the pre-training is completed, the performance of the large model is tested through a series of evaluation indicators, such as accuracy, recall rate, F1 score, etc. The evaluation process aims to fully understand the strengths and weaknesses of the large model and ensure that the large model can achieve the expected performance level on specific tasks.

[0136] Based on the evaluation results, the large model can be adjusted, such as adjusting the large model parameters, optimizing the training strategy, introducing task-specific data, etc., to further optimize its performance on specific tasks.

[0137] In step S4 and step S5, by building a collaborative mechanism that combines features, prompt content, and large model generation, the performance and efficiency of the overall system are effectively improved. By extracting useful information from multi-source heterogeneous data, the system can identify and extract the features that best represent the essence of the data. The feature extraction technology ensures that the data has been optimized and simplified before entering the model to adapt to subsequent processing steps. After feature extraction, the system uses these features to generate prompts. Prompts are used to guide the learning direction of the large model. Especially in the scenario of pre-trained models, the generation of prompts depends on the features extracted from the data. The generated prompts can help the model better understand the task requirements, thereby achieving more accurate learning and prediction in specific pre-training tasks. By integrating features and prompts into the training process of the large model, the model can not only benefit from basic pre-training, but also be fine-tuned and optimized according to specific application requirements. In this process, the generation of large models depends not only on the existing knowledge base and data, but also on the input prompts, which enables the model to be more focused and effective in processing specific types of tasks or data.

[0138] This collaborative workflow that integrates feature extraction, cue generation, and large model training can significantly improve the model's training efficiency and prediction accuracy. It not only optimizes the model's computing resource utilization, but also enhances the model's adaptability and generalization capabilities to new situations.

[0139] In step S6, encapsulating and deploying the application model, which specifically includes:

[0140] Select a large model that meets the expected effect, encapsulate it according to the multimodal data type of the large model, and deploy and publish the encapsulated large model.

[0141] Obtain a big model service application, review and authorize it, connect the interface to the big model, call the big model, monitor the performance and status of the big model in real time during use, and record the call data and performance data.

[0142] Encapsulating the trained large model involves integrating the model's weights, settings, and preprocessing processes to create an easy-to-call software development kit or application programming interface. For example, in specific data processing tasks such as speech recognition, the large model can be encapsulated into a service that complies with specific standards (such as the Media Resource Control Protocol, MRCP) so that it can be smoothly integrated with the voice platform and other applications. The encapsulation process also involves selecting the appropriate technology stack and protocol (such as HTTP, RESTful) to adapt to different usage environments and ensure that the large model can be effectively accessed over the network.

[0143] After deployment is complete, a detailed description, performance parameters, and usage conditions of each large model are provided to create a list of large models for users to choose from.

[0144] like Figure 7 As shown, this embodiment also provides a multi-modal large model construction system, which includes:

[0145] a data acquisition module configured to acquire multimodal data, the multimodal data including at least two of voice data, text data, image data, audio data, and video data;

[0146] A data processing module, which is configured to pre-process and convert the multimodal data and fuse the converted multimodal data;

[0147] A corpus generation module, which is configured to generate a large model training corpus based on the fused multimodal data;

[0148] The feature library construction module is configured to extract and analyze features of the training corpus, compare the extracted features with the current feature library, iteratively update and optimize the content of the current feature library; construct prompt content based on the extracted features, encode the prompt content and embed it into the large model; and train and adjust the large model based on the prompt content;

[0149] The large model application module is configured to package and deploy the application large model.

[0150] This embodiment also provides a refrigeration device, including:

[0151] A memory for storing executable instructions;

[0152] The processor is used to implement the above-mentioned multimodal large model construction method when running the executable instructions stored in the memory.

[0153] This embodiment also provides a storage medium storing executable instructions, which, when executed by a processor, implement the above-mentioned multimodal large model construction method.

[0154] In summary, this implementation first utilizes multimodal data and performs targeted processing on different types of data to prepare a rich and high-quality corpus for large model training. Then, the corresponding training corpus is generated according to different learning needs, and the features of the generated training corpus are extracted and analyzed. These features are then compared with the existing feature library to determine and update the content of the feature library. In addition, a word-level semantic compression large model is used to optimize the prompt generation process, and the processed data encoding is input into the large model for pre-training. Finally, the large model is ensured to achieve optimal performance through evaluation and fine-tuning. The overall method significantly improves the performance of deep learning models in complex data scenarios.

[0155] It should be understood that although this specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each implementation mode may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

[0156] The series of detailed descriptions listed above are only specific descriptions of feasible implementation methods of the present invention and are not intended to limit the scope of protection of the present invention. All equivalent implementation methods or changes that do not deviate from the technical spirit of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for constructing a multimodal large model, characterized in that: include: Acquiring multimodal data, the multimodal data comprising at least two of voice data, text data, image data, audio data, and video data; Preprocessing and converting the multimodal data, and fusing the converted multimodal data; Generate a large model training corpus based on fused multimodal data; Extract and analyze the features of the training corpus, compare the extracted features with the current feature library, and iteratively update and optimize the content of the current feature library; Constructing prompt content based on the extracted features, and encoding and embedding the prompt content into a large model; Encapsulate and deploy large application models; The encoding and large model embedding of the prompt content specifically includes: The prompt content data is compressed through a word-level semantic compression model, and a low-rank weight matrix decomposition model is used to represent the target weight matrix of the prompt content data as a sparse coefficient matrix and a dense weight matrix; Inputting the compressed prompt content data and the encoded text data obtained by preprocessing the multimodal data into the embedding layer of the large model; The extracting and analyzing the features of the training corpus, comparing the extracted features with the current feature library, and iteratively updating and optimizing the content of the current feature library specifically include: Extract features from structured and unstructured data; When the extracted features do not exist in the current feature library, the extracted features are encoded and added to the current feature library.

2. The multimodal large model construction method according to claim 1, characterized in that: The acquiring of multimodal data specifically includes: Initialize multimodal data acquisition channels; Acquiring multimodal data collected by a data collection device, and / or, Acquire multimodal data transmitted from a client terminal.

3. The multimodal large model construction method according to claim 1, characterized in that: The multimodal data includes voice data, text data, image data, audio data and video data.

4. The multimodal large model construction method according to claim 3, characterized in that: The preprocessing and converting of the multimodal data specifically includes: Cleaning and formatting the multimodal data; Separating the video data into audio and video to obtain separated video audio data and video image data; Preprocess the text data; Perform standardization and adjustment processing on image data and video image data.

5. The multimodal large model construction method according to claim 4, characterized in that: The preprocessing and converting of the multimodal data further includes: Extracting features of the voice data and video voice data to obtain voice features; Inputting the speech features into a speech recognition convolutional neural network to obtain output text data; Aligning the output text data sequence with the speech feature sequence by a connection temporal classification method; The output text data is combined through a fully connected layer and then output to a classification function to calculate a score to obtain speech text data.

6. The multimodal large model construction method according to claim 4, characterized in that: The preprocessing and converting of the multimodal data further includes: Acquire image feature sequences of image data and video image data; The image semantic representation is obtained through a large model of distillation diffusion deep fusion based on spatiotemporal and long-distance dependent features; The image semantics is output by connecting the temporal classification and the fully connected layer.

7. The multimodal large model construction method according to claim 6, characterized in that: The image semantic representation is obtained by using a distillation diffusion deep fusion large model based on spatiotemporal and long-distance dependent features, specifically including: After the distillation diffusion deep fusion large model is trained, the teacher model is introduced into a student model that is continuous in time and space to obtain a student model with the same parameters as the teacher model; Convert the distilled large model into a student model with discrete time steps and short steps, and repeat the above steps until the teacher model is distilled to half the steps of the student model.

8. The multimodal large model construction method according to claim 1, characterized in that: Also includes: Obtain historical multimodal data, vectorize the historical multimodal data, and map it to the same vector space for transfer learning.

9. The multimodal large model construction method according to claim 1, characterized in that: The generating of a large model training corpus based on the fused multimodal data specifically includes: Generate large model training corpora of supervised learning, semi-supervised learning and unsupervised learning types according to the data training type.

10. The multimodal large model construction method according to claim 1, characterized in that: The deploying and applying the large model includes: Get a large model service application; After review and authorization, the interface is connected to the big model and the big model is called; Monitor the performance and status of large models in real time during use, and record call data and performance data.

11. A multimodal large model construction system, characterized in that: include: a data acquisition module configured to acquire multimodal data, wherein the multimodal data includes at least two of voice data, text data, image data, audio data, and video data; A data processing module, configured to preprocess and convert the multimodal data and fuse the converted multimodal data; A corpus generation module, configured to generate a large model training corpus based on the fused multimodal data; A feature library construction module is configured to extract and analyze features of the training corpus, compare the extracted features with the current feature library, iteratively update and optimize the content of the feature library; construct prompt content based on the extracted features, and encode and embed the prompt content into a large model; Training and adjusting the large model based on the prompt content; The step of constructing prompt content based on the extracted features, encoding the prompt content and embedding the prompt content into a large model specifically includes: compressing the prompt content data through a word-level semantic compression large model, using a low-rank weight matrix decomposition large model, and expressing the target weight matrix of the prompt content data as a sparse coefficient matrix and a dense weight matrix; inputting the compressed prompt content data and the encoded text data obtained by preprocessing the multimodal data into the embedding layer of the large model; extracting and analyzing features of the training corpus, comparing the extracted features with the current feature library, and iteratively updating and optimizing the content of the current feature library specifically includes: extracting features from structured data and unstructured data; when the extracted features do not exist in the current feature library, encoding the extracted features and adding the extracted features to the current feature library; The big model application module is configured to package and deploy the big model.

12. A refrigeration device, characterized in that: include: A memory for storing executable instructions; A processor, configured to implement the multimodal large model construction method according to any one of claims 1 to 10 when running the executable instructions stored in the memory.

13. A storage medium storing executable instructions, characterized in that: When the executable instructions are executed by the processor, the multimodal large model construction method described in any one of claims 1 to 10 is implemented.

Images