Hybrid expert large model construction method and system, refrigeration equipment and storage medium

Abstract

The invention provides a hybrid expert large model construction method and system, refrigeration equipment and a storage medium, a forgetting gate control operation is introduced, the model can dynamically adjust information flow, centrally process information most critical to a prediction task and ignore irrelevant or redundant data, and the model is similar to gate control in a long-short-term memory network, so that the prediction efficiency is improved. However, the calculation efficiency and the resource utilization rate are improved. Besides, the model keeps the continuity of information in a deep network through residual connection, the problem of gradient disappearance in training is avoided, meanwhile, L1 norm normalization increases the sparsity of data and simplifies the learning process of the model, so that the model is easier to focus on important features, and the training efficiency is improved. The resource utilization rate of the large model is improved, and the problems of low calculation efficiency and insufficiency of the large model are solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a hybrid expert large model construction method and system, a refrigeration equipment and a storage medium. BACKGROUND

[0002] In the rapidly developing field of artificial intelligence, especially when dealing with multi-modal data, how to effectively improve the computing efficiency and resource utilization of large models has become an urgent problem to be solved. One of the main challenges is low computing efficiency and insufficient resource utilization. In addition, the complexity and large amount of data of large models often make parallel processing difficult, and data transmission and synchronization may become an efficiency bottleneck. In multi-task and multi-data source application scenarios, unreasonable resource allocation may also lead to resource surplus for some tasks and resource deficiency for other tasks, further affecting overall performance and response time. SUMMARY

[0003] The present application relates to the technical field of computer, in particular to a hybrid expert large model construction method and system, a refrigeration equipment and a storage medium.

[0004] The present application provides a hybrid expert large model construction method, which comprises the following steps:

[0005] Obtaining multi-modal data, the multi-modal data comprising at least two of voice data, text data, image data, audio data and video data;

[0006] Preprocessing and converting the multi-modal data, and fusing the multi-modal data;

[0007] Generating a large model training corpus based on the multi-modal data;

[0008] At the large model encoder end, inputting the training corpus for encoding processing, pruning the input data and output data of the embedding layer at the neuron, and pruning the activation layer;

[0009] At the large model decoder end, decoding and compressing the redundant data after encoding, and then outputting;

[0010] Deploying the large model.

[0011] As a further improvement of the present application, the pruning of the input data and output data of the embedding layer at the neuron specifically comprises:

[0012] Normalizing the input data and output data of the embedding layer, calculating the normalization value of each neuron, and when the preset embedding layer pruning threshold is not met, pruning and removing the corresponding row or column of the transpose matrix.

[0013] As a further improvement of the application, the pruning processing of the activation layer specifically includes:

[0014] The importance of the output data of the activation layer containing the multi-head attention mechanism is evaluated, and when the preset activation layer pruning threshold is not met, pruning processing is performed.

[0015] As a further improvement of the application, the compressed redundant data is output, specifically including:

[0016] The encoded text feature vectorization is subjected to semantic compression at the word segmentation level;

[0017] The compressed text feature vector is decoded using an autoregressive variable decoder to generate the final output data in text format.

[0018] As a further improvement of the application, when outputting voice and / or video data, it further includes:

[0019] The data with noise is denoised and restored and separated to obtain the denoised data decoding feature or vector;

[0020] The autoregressive decoding restores the voice and / or video data.

[0021] As a further improvement of the application, the multi-modal data is acquired in real time, specifically including:

[0022] The multi-modal data acquisition channel is initialized;

[0023] The multi-modal data collected by the data acquisition device is acquired, and / or

[0024] The multi-modal data transmitted from the customer terminal is acquired.

[0025] As a further improvement of the application, the multi-modal data includes voice data, text data, image data, audio data, and video data.

[0026] As a further improvement of the application, the multi-modal data is preprocessed and converted, specifically including:

[0027] The original multi-modal data is cleaned and format-converted;

[0028] The video data is subjected to voice and video separation to obtain separated video voice data and video image data;

[0029] The text data is preprocessed;

[0030] The image data and video image data are subjected to standardization adjustment processing.

[0031] As a further improvement of the application, the preprocessing and conversion of the multi-modal data further comprises:

[0032] Extracting the speech data and video speech data features to obtain speech features;

[0033] Inputting the speech features into a speech recognition convolutional neural network to obtain output text data;

[0034] Aligning the output text data sequence with the speech feature sequence through a connection time sequence classification method;

[0035] After combining the output text data through a full connection layer, outputting to a classification function to calculate scores to obtain speech text data.

[0036] As a further improvement of the application, the preprocessing and conversion of the multi-modal data further comprises:

[0037] Obtaining image feature sequences of image data and video image data;

[0038] Obtaining image semantic representations through a distillation diffusion deep fusion large model based on space-time and long-distance dependence features;

[0039] Outputting image semantics through a connection time sequence classification and a full connection layer.

[0040] As a further improvement of the application, the distillation diffusion deep fusion large model based on space-time and long-distance dependence features to obtain image semantic representations specifically comprises:

[0041] After the distillation diffusion deep fusion large model is trained, a teacher model is introduced into a time and space continuous student model to obtain a student model with the same parameters as the teacher model;

[0042] The distillation large model is converted into a student model with discrete time steps and short steps, and the above steps are repeated until the teacher model is distilled into a student model with half the number of steps, i.e. N / 2.

[0043] As a further improvement of the application, it further comprises:

[0044] Obtaining historical multi-modal data, and mapping the vectorized historical multi-modal data in the same vector space for transfer learning.

[0045] As a further improvement of the application, the generation of a large model training corpus based on the fused multi-modal data specifically comprises:

[0046] According to the data training type, generating large model training corpora of supervised learning, semi-supervised learning and unsupervised learning types.

[0047] As a further improvement of the present application, the deployment application of the large model further comprises:

[0048] Obtaining a large model service application;

[0049] After authorization, the interface is connected with the large model, and the large model is called.

[0050] The performance and state of the large model in the use process are monitored in real time, and the calling data and performance data are recorded.

[0051] The present application provides a kind of hybrid expert large model construction system, it includes:

[0052] Data acquisition module is configured to obtain multi-modal data, and the multi-modal data includes at least two of voice data, text data, image data, audio data and video data;

[0053] Data processing module is configured to preprocess and convert the multi-modal data, and fuse multi-modal data;

[0054] Corpus generation module is configured to generate large model training corpus based on the multi-modal data;

[0055] Large model optimization module is configured to encode the input training corpus at the large model encoder end, and the input data and output data of embedding layer are pruned in neuron, and the activation layer is pruned;After decoding and compressing redundant data, the encoded data are output at the large model decoder end;

[0056] Large model application module is configured to deploy and apply the large model.

[0057] The present application provides a kind of refrigeration equipment, comprising:

[0058] Memory for storing executable instructions;

[0059] Processor for running the executable instructions stored in the memory, to realize the above-mentioned hybrid expert large model construction method.

[0060] The present application provides a kind of computer readable storage medium, which stores executable instructions, and the executable instructions are executed by processor to realize the above-mentioned hybrid expert large model construction method.

[0061] The beneficial effects of this invention are as follows: By introducing forget gate control operations, the model can dynamically adjust the information flow, focusing on processing the most critical information for the prediction task and ignoring irrelevant or redundant data, similar to gate control in long short-term memory networks, but with a greater focus on improving computational efficiency and resource utilization. Furthermore, the model maintains the coherence of information in deep networks through residual connections, avoiding the gradient vanishing problem during training. Simultaneously, L1 norm normalization increases data sparsity, simplifies the model's learning process, and makes it easier for the model to focus on important features, improving the resource utilization of large models and solving the problems of low and insufficient computational efficiency in large models. Attached Figure Description

[0062] Figure 1 This is a schematic diagram of the steps in the method for constructing a hybrid expert large model according to one embodiment of the present invention.

[0063] Figure 2 This is a schematic diagram of the preprocessing steps for multimodal data according to one embodiment of the present invention.

[0064] Figure 3 This is a schematic diagram illustrating the steps of converting voice data into text data in one embodiment of the present invention.

[0065] Figure 4 This is a schematic diagram illustrating the steps of converting image data into text data according to one embodiment of the present invention.

[0066] Figure 5 This is a schematic diagram of step S4 of the hybrid expert large model construction method in one embodiment of the present invention.

[0067] Figure 6 This is a block diagram of a hybrid expert large model construction system according to one embodiment of the present invention. Detailed Implementation

[0068] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0069] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0070] This embodiment provides a hybrid expert large model construction method. By collecting multimodal data from different sources as input data, and utilizing forget gates to control computation and construct expert models, it improves the computational efficiency and computing power utilization of large models. For example, in this embodiment, a smart refrigerator is used as an example to illustrate this method.

[0071] like Figure 1 As shown, this embodiment provides a method for constructing a hybrid expert large model, which includes the following steps:

[0072] S1: Acquire multimodal data, which includes at least two of the following: speech data, text data, image data, audio data, and video data.

[0073] S2: Process and transform the multimodal data to obtain fused text data with prompts for engineering information.

[0074] S3: Control operations are performed through the forget gate to construct a hybrid expert large model.

[0075] S4: Output encoded data after performing residual and L1 norm normalization operations.

[0076] S5: Based on the output encoded data type, decode to generate corresponding data type for output.

[0077] S6: Deploy the large model described above.

[0078] Step S1 specifically includes:

[0079] Multimodal data acquisition channel initialization; acquiring multimodal data collected by the data acquisition device, and / or acquiring multimodal data transmitted from the client terminal.

[0080] In this embodiment, multimodal data includes speech data, text data, image data, audio data, and video data.

[0081] First, initialize the multimodal information data input channel to ensure smooth data acquisition and transmission.

[0082] To acquire multimodal information for smart refrigerators, it is necessary to integrate various input methods, such as real-time and offline voice, text, and video data. Diverse data access methods can adapt to different usage scenarios and user habits, thereby enhancing the convenience and efficiency of user interaction.

[0083] In this embodiment, the smart refrigerator collects real-time voice data from the user through built-in microphones and microphone arrays. Video and image data are collected via a camera inside the refrigerator. Furthermore, the user's voice, video, and text data can also be transmitted through client devices connected to the refrigerator. These devices include mobile phones, tablets, smart speakers, smart bracelets, or Bluetooth headsets—all smart electronic devices with information sending capabilities. Users can directly input multimodal data into these client devices, which then collect the data and transmit it to the smart refrigerator via wireless communication technologies such as WiFi or Bluetooth. This multi-channel data acquisition method eliminates the need for direct interaction between the user and the refrigerator, enabling more flexible multimodal data collection. This not only enriches the variety and quality of data but also improves the quality of data used for large-scale model training.

[0084] By integrating multiple data formats such as text, images, and sound, multimodal data provides more detailed and comprehensive information than single-modal data. This comprehensive utilization of information significantly improves the ability of large models to identify and understand complex phenomena. Multimodal data integrates different data sources, enabling a panoramic view of the same object or event from various perspectives, helping models to deeply explore key features and correlations, thereby improving accuracy when performing prediction or classification tasks. When single-modal data is affected by noise, occlusion, or other perturbations, the complementary information from multimodal data helps mitigate these effects and enhances model performance.

[0085] In summary, by constructing a comprehensive, efficient, and reliable data input channel in step S1, a solid foundation was laid for in-depth information interaction between refrigeration equipment and users, while also providing high-quality input data for subsequent data processing, analysis, and large-scale model training.

[0086] like Figure 2 As shown, in step S2, the multimodal data is first preprocessed, which specifically includes:

[0087] S201: Clean and convert the format of the raw multimodal data.

[0088] S202: Perform audio-video separation on the video data to obtain the separated audio-video data and video-image data.

[0089] S203: Preprocess the text data.

[0090] S204: Standardize and adjust image data and video image data.

[0091] When processing video data, the first step is to distinguish between the audio and image content in the video stream, extracting pure audio and visual elements separately for further specialized processing and analysis. This process can be performed using existing software tools and libraries. Furthermore, to meet specific needs and automate workflows, the audio and video data processing can be customized using programming scripts.

[0092] The purpose of preprocessing text data is to clean and organize the text, making it more suitable for effective analysis and model training. This includes steps such as removing irrelevant words, eliminating duplicates, and performing word segmentation. Removing these duplicates helps reduce the size of the dataset, speeds up processing, and improves the accuracy of model training. Duplicates in the text can increase processing complexity and introduce biases during training. Identifying and eliminating these duplicates ensures the high quality and diversity of the dataset. Word segmentation breaks the text down into smaller units, such as words or phrases, for further processing. Furthermore, text preprocessing may include operations such as stemming, lemma reconstruction, and punctuation removal. These steps significantly improve the efficiency and quality of text processing, laying the foundation for subsequent analysis, feature extraction, and model training.

[0093] For image data and image data extracted from videos, standardization processing is performed. This includes resizing images, normalizing, and removing noise. By unifying all image and video frames to the same size and resolution, and adjusting pixel values ​​to a standard range, it helps the model training process converge quickly and improves overall stability. These standardization and adjustment measures ensure that image and video data are consistent and of high quality before being input into the model, reducing uncertainty during training and improving the model's performance in practical applications.

[0094] like Figure 3 As shown, after preprocessing the multimodal data, the data undergoes a transformation process to convert speech data into text data. Specifically, this includes:

[0095] S205: Extract features from speech data and video speech data to obtain speech features.

[0096] S206: Input the speech features into the speech recognition convolutional neural network to obtain the output text data.

[0097] S207: Align the output text data sequence with the speech feature sequence using a concatenated temporal classification method.

[0098] S208: The output text data is combined through a fully connected layer and then output to a classification function to calculate the score and obtain the speech text data.

[0099] When processing voice data from refrigeration equipment such as refrigerators, its spatiotemporal characteristics and contextual relevance are taken into account. To accurately convert voice data into text, connection-based temporal classification is employed as an end-to-end learning method. This technique is particularly well-suited for handling the problem of mismatched input and output sequence lengths and performs exceptionally well in continuous speech recognition tasks. By assigning labels to each frame in the sequence, this technique enables the model to automatically recognize and learn the mapping relationship between the input speech and the generated text.

[0100] Furthermore, the constructed deep learning model combines recurrent neural networks (RNNs) and convolutional neural networks (CNNs), utilizing a loss function for temporal classification and an attention mechanism. The CNNs capture local features in the speech signal, while the RNNs process sequential data to understand temporal dependencies. The attention mechanism enhances the model's ability to focus on key information within the speech signal. This end-to-end model extracts rich, high-level information from the raw speech signal, including key speech features such as intonation, rhythm, and intensity, which is crucial for improving the accuracy of speech recognition and understanding user intent.

[0101] By using connection-time classification technology and an end-to-end deep learning model combining recurrent and convolutional networks, the accuracy of converting voice data from refrigeration equipment into text has been significantly improved, providing solid technical support for achieving efficient and accurate multimodal interaction.

[0102] like Figure 4 As shown, the data conversion process, specifically converting image data into text data, includes:

[0103] S209: Obtain the image feature sequence of image data and video image data.

[0104] S210: Image semantic representation is obtained through a large model of distillation-diffusion deep fusion based on spatiotemporal and long-distance dependent features.

[0105] S211: Output image semantics by connecting temporal classification and fully connected layers.

[0106] Specifically, step S210 includes:

[0107] S2101: After the large-scale model of distillation-diffusion fusion is trained, the teacher model is introduced into a time- and space-continuous student model to obtain a student model with the same parameters as the teacher model.

[0108] S2102: Convert the large distillation model into a student model with discrete time steps and a short number of steps, and repeat the above steps until the teacher model is distilled to half the number of steps of the student model, i.e., N / 2.

[0109] When extracting image features from image and video data and identifying text content from them, given the complexity of recognizing text sentences in speaker images, such as variable sentence length, vocabulary diversity, and contextual relationships between sentences, a large model combining deep transfer and knowledge distillation based on spatiotemporal relationships and long-term dependencies is employed. This model can capture features in images that evolve over time and handle contextual information in long texts.

[0110] In the knowledge distillation process, the knowledge of the complex "teacher" model is transferred to a lighter "student" model. This not only maintains the efficiency of the teacher model but also reduces the demand for computational resources. Deep fusion technology further integrates the outputs of the teacher and student models, optimizing the model's ability to handle time series and contextual dependencies, and improving the efficiency of mining semantic features from sentence sequences. Using this method, large models can effectively extract and recognize text content from image data, especially excelling at handling sentences with complex semantics and long-distance dependencies, significantly improving the accuracy and efficiency of image feature-to-text content recognition.

[0111] In step S2, the multi-source text data obtained after processing the multimodal data is integrated into a unified dataset. Then, a cue engineering task is performed on the resulting text data, ultimately yielding text data content with cue engineering information. Based on the key information and context obtained from the analysis, training prompts are constructed. These prompts may include specific task descriptions, expected operational results, user intentions, etc., to guide the model's learning direction during training. The generated prompts are accurately labeled and classified, including marking their applicable scenarios, expected user interactions, and the model's response methods. This helps the model better understand various interaction scenarios and corresponding behavioral patterns.

[0112] Furthermore, this embodiment also includes the step of:

[0113] Historical multimodal data is collected and vectorized, mapping them to a unified vector space for transfer learning. This approach enables comparisons of data collected from different time points and devices within the same vector space. This unified mapping mechanism ensures that all data can be evaluated and compared according to the same standards, providing a common foundation for in-depth data analysis and subsequent applications. Within this unified vector space, transfer learning techniques can be effectively applied, leveraging the learning experience of existing large models on similar tasks to improve the processing efficiency of new tasks.

[0114] Historical data accumulates a wealth of information and experience, providing resources for training large models. This data typically includes a variety of different scenarios and cases, enhancing the diversity of training samples and helping to improve the generalization ability and prediction accuracy of large models.

[0115] In step S3, the large model embedding layer encoder is first pruned, and the activation layer is pruned, specifically including:

[0116] The input and output data of the embedding layer are subjected to L1 norm normalization. The L1 norm normalization value of each neuron is calculated. When the preset embedding layer pruning threshold is not met, pruning is performed to remove the rows or columns of the corresponding transpose matrix.

[0117] The importance of the output data of the activation layer containing the multi-head attention mechanism is evaluated, and pruning is performed when the preset activation layer pruning threshold is not met.

[0118] Pruning the encoder in the embedding layer of a large model can optimize computational efficiency and resource consumption during training. In the embedding layer of a neural network, the input and output data of each neuron are first processed by residual normalization, using L1 norm normalization to standardize the data. Based on the normalized data, a pruning threshold is set for each neuron. The threshold is used to determine which neurons are important and which are not. If the normalized value of a neuron is lower than the set threshold, pruning is performed by removing the corresponding row or column from the weight matrix.

[0119] By pruning unnecessary neurons, the complexity of the model can be reduced, thereby lowering computational resource consumption and improving computational efficiency. This makes the model more lightweight and better suited for operation in resource-constrained environments.

[0120] Using L1 norm normalization to standardize data helps increase its sparsity. When dealing with high-dimensional data, L1 norm normalization can reduce the weights of some unimportant features to zero, thereby reducing the number of features and simplifying the model's complexity. By reducing the weights of unimportant features to zero, L1 norm normalization can reduce the impact of noisy data on the model, thus improving its stability and accuracy. Because it generates a sparse matrix, related computations are often faster, especially when using optimization algorithms such as gradient descent, as sparsity reduces computational and storage requirements. The simplified model typically has better generalization ability, focusing on more important features and ignoring unnecessary details and noise.

[0121] When performing activation layer pruning, multi-head self-attention mechanisms are pruned. Pruning involves removing unnecessary or redundant attention heads from the multi-head self-attention mechanism. Each head focuses on different information when processing the input data, but not all heads are equally important. By analyzing the contribution and effect of each head, it can be determined which heads are necessary and which can be removed.

[0122] Pruning activation layers by removing unimportant or redundant attention heads reduces model complexity, requiring fewer computational resources and running faster. Reduced complexity decreases overfitting to training data, thus improving generalization ability. Furthermore, by pruning unnecessary heads, the remaining heads are more likely to directly influence the model's predictions, increasing the transparency and interpretability of the model's decision-making process.

[0123] By using forgetting gates for control operations, a hybrid expert model is constructed, specifically including:

[0124] A gate control computation model with a forget gate is constructed to filter redundant data; multiple feedforward neural networks are constructed, and resources are allocated among the feedforward neural networks through dynamic token routing.

[0125] When constructing hybrid expert models, employing a forgetting gate control operation to build multiple feedforward neural networks not only optimizes the allocation of computational resources but also improves overall computational efficiency and utilization. In hybrid expert models, the forgetting gate control operation determines which information to retain or ignore in each expert network. This is similar to the gate control mechanism in LSTM (Long Short-Term Memory) networks, helping the model learn and remember important information more effectively while forgetting less important information.

[0126] By controlling computation through forget gates, the model can dynamically adjust the activation state of each expert network, allocating computational resources based on the characteristics and needs of the input data. This allows the model to optimize resource usage according to actual conditions, avoiding waste. Furthermore, the forget gate mechanism allows the model to flexibly expand or shrink its network capacity as needed, dynamically adjusting the number and size of experts based on system performance requirements or the complexity of a specific task. Forget gate control can also effectively reduce unnecessary computation at a given moment, activating only the expert networks most critical to the current task, thus reducing unnecessary computation and accelerating processing speed.

[0127] Multiple expert networks are trained to handle different types of tasks or subsets of data, and the forget gate ensures that only the network most relevant to the current input is activated each time. The flexibility based on the forget gate makes the resulting hybrid expert model particularly suitable for handling multi-task or highly complex problems.

[0128] By leveraging dynamic token routing, the model enables efficient parallel computation, allowing individual expert networks to process different data or tasks simultaneously. Since resources are allocated only to currently active expert networks, unused networks do not consume resources. This parallel computation significantly reduces memory consumption and computational resource requirements, improving the computational efficiency and utilization of large models.

[0129] Step S4 specifically includes:

[0130] S41: In the hybrid expert large model, perform residual and L1 norm normalization operations.

[0131] S42: Prune the activation layer of the hybrid expert large model.

[0132] S43: Repeat the residual and L1 norm normalization operations to perform feedforward fully connected layer operations.

[0133] S44: After performing residual and L1 norm normalization operations again, output the encoded data.

[0134] By directly adding the input to the output, residual connections help solve the vanishing gradient problem, enabling the effective training of deeper networks. During computation, the output of each layer includes not only the result of the transformation performed by that layer but also the original information passed directly from the previous layer. Residual connections ensure that information can be effectively transferred between multiple layers of the model. Especially in deep networks, residual connections help avoid the vanishing gradient problem, making the training of deep networks more feasible and stable.

[0135] L1 norm normalization is performed, which normalizes the L1 norm of each input feature vector in the model to 1. This promotes the sparsity of the model weights, allowing the model to focus more on important features and also helps reduce overfitting. Through normalization, the inputs to each layer maintain a relatively stable distribution, which helps accelerate the training process and improve the model's convergence speed and stability.

[0136] The multi-head self-attention mechanism of the activation layer is pruned again. By removing unimportant or redundant attention heads, the model is further simplified, requiring fewer computational resources and running faster. Reducing model complexity generally helps improve the model's generalization ability because it reduces overfitting to the training data.

[0137] Residual and L1 norm normalization operations are performed again before the feedforward fully connected layer to ensure that the data processed in each layer maintains a consistent quality and standard, thereby maintaining the stability and consistency of the model during deep propagation.

[0138] Feedforward fully connected layers can be used for processing, capturing the complex relationships between different features and performing corresponding transformations to adapt to specific task requirements. Before outputting the encoded data, a final residual and L1 norm normalization operation is performed to ensure the stability and reliability of the output data, reducing potential fluctuations in model prediction. A total of three residual and normalization operations are performed to ensure good consistency and predictive stability of the output data. After completing all necessary processing and operations, the model outputs the encoded result.

[0139] Step S5 specifically includes:

[0140] When outputting voice and / or video data, the noisy data is denoised, restored, and separated to obtain the undecoded features or vectors of the denoised data; autoregressive decoding is then performed to restore the voice and / or video data.

[0141] When outputting text data, semantic compression at the word segmentation level is performed on the encoded text feature vectors; an autoregressive variable decoder is used to decode the compressed text feature vectors to generate the final text format output data.

[0142] When outputting speech and / or video data, the Unet network model structure with a Refiner Layer performs the denoising and restoration separation tasks. The Unet network model has a symmetrical structure, including a contraction path and a symmetrical expansion path. The contraction path is mainly used to capture contextual information in the data, gradually reducing the spatial dimension of the data by applying convolution and pooling operations layer by layer. The expansion path gradually restores the details and spatial dimension of the data through upsampling and convolution operations. Each step fuses features with the output of the corresponding layer in the contraction path, thereby helping to recover local information in the data.

[0143] First, the noisy input speech or video data is processed through the shrinking path of the Unet model to extract key contextual information and reduce noise levels. At the end of the shrinking path, a Refiner Layer is introduced, designed to refine and improve the denoised features. This layer enhances the model's ability to distinguish between noise and signal, further reducing residual noise.

[0144] The denoised data features then enter the UNet expansion path. During this process, the model uses contextual information captured by the previous contraction path to guide the data reconstruction, gradually restoring the original size and details of the data through upsampling and convolution. In each upsampling step, features from the corresponding contraction path layer are fused to help the model reconstruct the data more accurately.

[0145] After feature restoration, an autoregressive decoder is used for final decoding of the data. Utilizing the restored features or vectors, noise-free speech or video data is gradually reconstructed. During autoregressive decoding, the model gradually reduces the influence of any residual noise, optimizing output quality by learning the autocorrelation in the data.

[0146] When processing text data, especially in the entire process from vectorizing encoded text features and semantic compression to the final output formatted text, using a word segmentation-level network model and an autoregressive variable decoder is a very effective strategy. This method can not only compress and simplify data but also maintain the semantic integrity of the text data. The following is a detailed expansion and explanation of this process:

[0147] When outputting text data, the encoded text data is vectorized and then further semantically compressed using a word segmentation-level network model. This semantic compression process can be achieved through a series of convolutional, pooling, or attention mechanisms. These mechanisms help the model focus on key parts of the text, such as important vocabulary and grammatical structures, while removing redundant or unimportant information.

[0148] The semantically compressed text feature vectors are then fed into an autoregressive variable decoder, which takes into account the previously generated outputs when generating text, thus ensuring the coherence and contextual correctness of the output text. The autoregressive model constructs a complete text step by step based on the currently generated sequence by progressively predicting the next word or symbol.

[0149] Step S6 specifically includes:

[0150] Before deployment, the model service needs to be applied for and configured, including determining the service type, scale, and expected performance metrics. Model authorization management needs to be implemented, including authenticating and authorizing users or systems using the model. A model interface needs to be developed to allow users or other systems to invoke the model for prediction or other operations.

[0151] After deploying the model, continuously monitor its performance and status, including tracking the accuracy of model predictions, response time, and system health. Monitoring data will help the maintenance team to promptly identify and resolve potential issues. The system should automatically log all model-related operations and events.

[0152] In summary, this implementation introduces forget gate control, enabling the model to dynamically adjust the information flow, focusing on processing the most critical information for the prediction task while ignoring irrelevant or redundant data. This is similar to gate control in Long Short-Term Memory (LSTM) networks, but with a greater emphasis on improving computational efficiency and resource utilization. Furthermore, the model maintains the coherence of information in deep networks through residual connections, avoiding the vanishing gradient problem during training. Simultaneously, L1 norm normalization increases data sparsity, simplifying the model's learning process and making it easier to focus on important features. This improves the resource utilization of large models and addresses the issues of low and insufficient computational efficiency in large models.

[0153] This embodiment provides a hybrid expert large model construction system, which includes:

[0154] The data acquisition module 1 is configured to acquire multimodal data, which includes at least two of the following: voice data, text data, image data, audio data, and video data.

[0155] Data processing module 2 is configured to process and transform the multimodal data to obtain processed image data and fused text data with prompting engineering information;

[0156] The Hybrid Expert Large Model Generation Module 3 uses a forget gate to control the computation and construct a hybrid expert large model; after performing residual and L1 norm normalization operations, it outputs encoded data.

[0157] Output module 4 is configured to decode and generate corresponding type data based on the output encoded data type for output;

[0158] Large model application module 5, which is configured to deploy the large model.

[0159] Furthermore, the hybrid expert large model generation module is configured to construct a gate control operation model with a forget gate, filter redundant data, and construct multiple feedforward neural networks; and to allocate resources among the feedforward neural networks through dynamic token routing.

[0160] Furthermore, the hybrid expert large model generation module is configured to perform residual and L1 norm normalization operations in the hybrid expert large model; perform pruning on the activation layer of the hybrid expert large model; repeat the residual and L1 norm normalization operations and perform feedforward fully connected layer operations; and output encoded data after performing residual and L1 norm normalization operations again.

[0161] This embodiment provides a refrigeration device, including:

[0162] Memory, used to store executable instructions;

[0163] The processor, when running the executable instructions stored in the memory, implements the above-described hybrid expert large model construction method.

[0164] This embodiment provides a computer-readable storage medium storing executable instructions that, when executed by a processor, implement the above-described hybrid expert large model construction method.

[0165] It should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This way of describing the specification is only for clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

[0166] The detailed descriptions listed above are merely specific descriptions of feasible implementations of the present invention and are not intended to limit the scope of protection of the present invention. All equivalent implementations or modifications made without departing from the spirit of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for constructing a large hybrid expert model, characterized in that, Including the following steps: Acquire multimodal data, wherein the multimodal data includes at least two of the following: speech data, text data, image data, audio data, and video data; The multimodal data is processed and transformed to obtain fused text data with prompts for engineering information; A hybrid expert large model is constructed by performing control operations through forget gates. After performing residual and L1 norm normalization operations, the encoded data is output. Based on the output encoded data type, decode to generate corresponding type data for output; Deploy the large model described above.

2. The method for constructing a hybrid expert large model according to claim 1, characterized in that, The control computation through the forgetting gate, and the construction of the hybrid expert large model, specifically includes: Construct a gate control computation model with a forget gate, filter out redundant data, and build multiple feedforward neural networks; Resource allocation is performed among the feedforward neural networks using dynamic token routing.

3. The method for constructing a hybrid expert large model according to claim 1, characterized in that, The process of outputting encoded data after performing residual and L1 norm normalization operations specifically includes: In the hybrid expert large model, residual and L1 norm normalization operations are performed; The activation layer of the hybrid expert large model is pruned; Repeat the residual and L1 norm normalization operations to perform feedforward fully connected layer operations; After performing residual and L1 norm normalization operations again, the encoded data is output.

4. The method for constructing a hybrid expert large model according to claim 1, characterized in that, Also includes: Pruning is performed on the large model embedding layer encoder and the activation layer.

5. The method for constructing a hybrid expert large model according to claim 4, characterized in that, The pruning process for the large model embedding layer encoder and the pruning process for the activation layer specifically include: The input and output data of the embedding layer are normalized using the L1 norm. The L1 norm normalization value of each neuron is calculated. When the preset embedding layer pruning threshold is not met, pruning is performed to remove the corresponding row or column of the transpose matrix. The importance of the output data of the activation layer containing the multi-head attention mechanism is evaluated, and pruning is performed when the preset activation layer pruning threshold is not met.

6. The method for constructing a hybrid expert large model according to claim 1, characterized in that, The process of decoding and generating corresponding data types based on the output encoded data type for output specifically includes: When outputting voice and / or video data Denoising, restoring, and separating noisy data to obtain undecoded features or vectors of the denoised data; Perform autoregressive decoding to restore audio and / or video data; When outputting text data Semantic compression at the word segmentation level is performed on the vectorized features of the encoded text. The compressed text feature vector is decoded using an autoregressive variable decoder to generate the final text-formatted output data.

7. The method for constructing a hybrid expert large model according to claim 1, characterized in that, Real-time acquisition of multimodal data, specifically including: Multimodal data acquisition channel initialization; The multimodal data acquired by the data acquisition device, and / or The multimodal data is acquired from the client terminal.

8. The method for constructing a hybrid expert large model according to claim 7, characterized in that, The preprocessing and transformation of the multimodal data specifically includes: Clean and convert the format of the raw multimodal data; The video data is subjected to audio-video separation to obtain separated video-audio data and video-image data; Preprocess the text data; Standardize and adjust image and video image data.

9. The method for constructing a hybrid expert large model according to claim 8, characterized in that, The preprocessing and transformation of the multimodal data further includes: Extract features from the speech data and video speech data to obtain speech features; The speech features are input into a speech recognition convolutional neural network to obtain the output text data; The output text data sequence is aligned with the speech feature sequence using a connection-time classification method; The output text data is combined through a fully connected layer and then output to a classification function to calculate the score, thus obtaining the speech text data.

10. The method for constructing a hybrid expert large model according to claim 1, characterized in that, Also includes: Acquire historical multimodal data, vectorize the historical multimodal data, and then map it to the same vector space for transfer learning.

11. A hybrid expert large model construction system, characterized in that, include: A data acquisition module is configured to acquire multimodal data, wherein the multimodal data includes at least two of the following: voice data, text data, image data, audio data, and video data; The data processing module is configured to process and transform the multimodal data to obtain processed image data and fused text data with prompting engineering information; The hybrid expert large model generation module uses a forget gate to control the computation and construct a hybrid expert large model; after performing residual and L1 norm normalization operations, it outputs encoded data. The output module is configured to decode and generate corresponding data based on the output encoded data type for output. The large model application module is configured to deploy the large model.

12. The hybrid expert large model construction system according to claim 11, characterized in that, The hybrid expert large model generation module is configured to construct a gate control operation model with a forget gate, filter redundant data, and construct multiple feedforward neural networks; and to allocate resources among the feedforward neural networks through dynamic token routing.

13. The hybrid expert large model construction system according to claim 11, characterized in that, The hybrid expert large model generation module is configured to perform residual and L1 norm normalization operations in the hybrid expert large model; perform pruning on the activation layer of the hybrid expert large model; repeat the residual and L1 norm normalization operations and perform feedforward fully connected layer operations; and output encoded data after performing residual and L1 norm normalization operations again.

14. A refrigeration device, comprising: Memory, used to store executable instructions; A processor, when executing executable instructions stored in the memory, implements the hybrid expert large model construction method according to any one of claims 1 to 10.

15. A computer-readable storage medium storing executable instructions that, when executed by a processor, implement the hybrid expert large model construction method according to any one of claims 1 to 10.

Images