Multi-modal fusion intelligent interaction large language model system
By introducing feature extraction, polynomial regression analysis, computing resource evaluation and resource dynamic regulation modules in the multimodal fusion system, the problem of excessive computing resource consumption in multimodal data fusion is solved, and the optimal use of resources and the improvement of system performance is achieved.
Patent Information
- Application Number
- CN202510255451.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-05
- Publication Date
- 2025-06-06
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
When existing multimodal fusion systems process and fuse multimodal data, computing resources are consumed too much, resulting in slow inference speed and waste of resources, especially on edge computing devices.
A multimodal fusion intelligent interactive large language model system is designed, including feature extraction module, polynomial regression analysis module, computing resource consumption evaluation module and resource dynamic regulation module. Through the collaborative work of these modules, the system can extract the characteristics of multimodal data, analyze computing resource consumption, identify resource bottlenecks, and dynamically adjust computing resource allocation to achieve optimal use of resources.
It effectively reduces the consumption of computing resources during multimodal data fusion, improves the system's response speed and processing capabilities, ensures that the system maintains stable operation under high load or high fluctuations, and improves system performance and efficiency.
Smart Images
Figure CN120106224A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data fusion, and in particular to a multi-modal fusion intelligent interactive large language model system. Background Art
[0002] The multimodal fusion intelligent interactive large language model system is an artificial intelligence system that combines multiple information input methods (such as text, voice, image, video, etc.). It can understand and generate cross-modal content through deep learning technology. The core of the system is a large language model (such as GPT, etc.), which provides a smarter and richer user interaction experience by integrating data from different modalities. For example, users can interact with the system not only through text input, but also through voice or image input. The system can understand different types of information and give corresponding feedback. This system aims to achieve a more natural and humanized intelligent interaction.
[0003] Existing technologies are mostly focused on single-modal applications, such as plain text or speech recognition technology. Existing large language models mainly rely on text data, processing and generating text through natural language processing (NLP) technology. However, with the rapid development of multimodal data, existing technologies are gradually evolving towards the integration of multiple data types. For example, image recognition technology and speech recognition technology have been widely used, but they usually operate independently and lack effective cross-modal information fusion. Therefore, how to organically integrate information from different modalities such as vision, hearing and language, and realize intelligent interaction, is an important challenge for the current development of technology.
[0004] The prior art has the following deficiencies:
[0005] Multimodal systems usually require more powerful computing power to process and fuse information from multiple modalities. Especially when it comes to large-scale deep neural networks, the training and inference process of processing multimodal data will greatly consume computing resources. Large-scale image data processing and language model training require a lot of memory, GPU and other hardware support, and the more complex the model, the more serious the inference speed and resource consumption problems. In some cases, even efficient algorithms may face the problem of not being able to meet real-time requirements, especially when executed on edge computing devices, where resource limitations are particularly obvious. Summary of the invention
[0006] The purpose of the present invention is to provide a multi-modal fusion intelligent interactive large language model system to address the deficiencies in the background technology.
[0007] In order to achieve the above-mentioned object, the present invention provides the following technical solutions: a multimodal fusion intelligent interactive large language model system, comprising a feature extraction module, a polynomial regression analysis module, a computing resource consumption evaluation module and a resource dynamic control module;
[0008] Feature extraction module: extracts the processing complexity features of each modality from multiple input modalities, including the voice data sampling rate fluctuation index, and the processing delay features of each modality, including the text processing response time deviation index;
[0009] Polynomial regression analysis module: a polynomial regression model is used to comprehensively analyze the voice data sampling rate fluctuation index and the text processing response time deviation index. The regression model models and analyzes the relationship between the input features of multiple modalities and their computing resource consumption;
[0010] Computing resource consumption assessment module: Based on the regression model analysis results, the severity of computing resource consumption of each modality in the cross-modal fusion process is assessed to identify the precise location of resource bottlenecks during model operation;
[0011] Resource dynamic control module: Based on the evaluation results of the computing resource consumption, a dynamic control mechanism is adopted to adjust the computing resource allocation in the system in real time, and dynamically allocate memory and GPU according to the changes in the severity of computing resource consumption of each modality during the cross-modal fusion process to achieve optimal resource utilization.
[0012] Preferably, in the feature extraction module, the length of each frame signal is set to N frame , the signal x(t) is divided into several frames of length N frame , and use the window function to perform windowing processing on each frame of signal. The expression is: frame (t) = x(t)·w(t); where w(t) is the window function. For each frame of the signal, the time domain signal is converted into a frequency domain signal through the fast Fourier transform. The output of FFT is a complex array, which represents the amplitude and phase of the signal at different frequencies. The expression is: X(f) = FFT(x frame (t)); where X(f) is the frequency domain signal, f is the frequency, and FFT returns a complex signal containing amplitude and phase information. For each frame, the frequency characteristics of the signal are analyzed by calculating the amplitude spectrum of the FFT result. The amplitude spectrum is calculated as: A(f) = |X(f) |; where A(f) is the amplitude spectrum corresponding to frequency f. The voice data sampling rate fluctuation index SRV is calculated as: SRV = Where A(f,k) is the amplitude at frequency f in the kth frame, μ is the average amplitude value of all frame spectra, and σ is the standard deviation of the amplitude of all frame spectra.
[0013] Preferably, the response time data of the system processing different text tasks is collected, and the response time T of the system processing N text tasks is recorded within a set period of time. 1 ,T 2 ,…,TN , according to the maximum and minimum values of the response time, the entire time range is divided into M intervals, and the size of the interval ΔT is calculated as follows: Where min(T) and max(T) are the minimum and maximum values of the response time series, respectively. M is the number of intervals. For each interval i, the number of times the response time falls into the interval is counted. i , and then calculate the probability p of each interval i , that is, the frequency of response time within the interval: Among them, f i is the number of times the response time falls into interval i, N is the total number of response time data points, once the probability distribution p of each interval is obtained 1 ,p 2 ,…,p M , calculate the Shannon entropy of the response time, the formula is: Where: H(X) is the Shannon entropy of the response time, and the text processing response time deviation index is calculated, and the expression is: Where SX is the text processing response time deviation index.
[0014] Preferably, in the polynomial regression analysis module, the voice data sampling rate fluctuation index and the text processing response time deviation index are converted into comprehensive feature vectors, and the comprehensive feature vectors are used as inputs of the machine learning model. The machine learning model uses each group of comprehensive feature vectors to predict the severity value label of the computing resource consumption of each modality in the cross-modal fusion process as the prediction target, and takes minimizing the sum of the prediction errors of the severity value labels of the computing resource consumption of each modality in all cross-modal fusion processes as the training target. The machine learning model is trained until the sum of the prediction errors converges, then the model training is stopped, and the severity value of the computing resource consumption of each modality in the cross-modal fusion process is determined according to the model output results, wherein the machine learning model is a polynomial regression model.
[0015] Preferably, in the computing resource consumption evaluation module, the acquired severity value of the computing resource consumption of each modality in the cross-modal fusion process is compared with the standard resource consumption reference threshold set according to historical data; if the severity value of the computing resource consumption of each modality in the cross-modal fusion process is greater than or equal to the standard resource consumption reference threshold, it means that the severity of the computing resource consumption of each modality in the cross-modal fusion process is high, and an early warning signal is generated at this time, and the system faces a performance bottleneck; if the severity value of the computing resource consumption of each modality in the cross-modal fusion process is less than the standard resource consumption reference threshold, it means that the severity of the computing resource consumption of each modality in the cross-modal fusion process is low, and no early warning signal is generated at this time, and there is no need to take optimization measures immediately.
[0016] Preferably, the resource dynamic control module is used to collect the severity values of the computing resource consumption of each modality in the cross-modal fusion process within a subsequent fixed time period, and establish a data set for the severity values of the computing resource consumption of each modality in the cross-modal fusion process that are less than the standard resource consumption reference threshold, calculate the mean and standard deviation of the data set, and after analyzing them, determine the changes in the severity of the computing resource consumption of each modality in the cross-modal fusion process, and dynamically allocate memory and GPU to achieve optimal resource utilization.
[0017] Preferably, if the mean of the severity values in the data set is greater than or equal to the reference threshold of the mean of the severity values, and the standard deviation of the severity values is less than the reference threshold of the standard deviation of the severity values, it indicates that the computing resource consumption of the modality is at a high level, but the fluctuation is small, and a slight resource adjustment is considered to maintain the current resource configuration or slightly increase the resource allocation;
[0018] If the mean of the severity values is greater than or equal to the reference threshold of the mean of the severity values, and the standard deviation of the severity values is greater than or equal to the reference threshold of the standard deviation of the severity values, it means that the computing resource consumption of the modality is at a high level and fluctuates greatly. At this time, resource allocation should be strengthened to cope with the load changes caused by fluctuations, and more memory and GPU resources need to be temporarily added;
[0019] If the mean of the severity values is less than the reference threshold of the mean of the severity values, and the standard deviation of the severity values is greater than or equal to the reference threshold of the standard deviation of the severity values, it means that the computing resource consumption of the mode is low, but the fluctuation is large. At this time, it is necessary to reduce resource usage and avoid waste by dynamically adjusting resource allocation to maintain system stability;
[0020] If the mean severity value is less than the reference threshold of the mean severity value, and the standard deviation of the severity value is less than the reference threshold of the standard deviation of the severity value, it means that the computing resource consumption of the mode is low and the fluctuation is small. At this time, maintain the current resource configuration without making adjustments to keep the system running smoothly.
[0021] In the above technical solution, the technical effects and advantages provided by the present invention are:
[0022] 1. The present invention introduces a feature extraction module, a polynomial regression analysis module, a computing resource consumption assessment module and a resource dynamic control module. The present invention can accurately assess and dynamically adjust computing resource allocation in the process of multimodal data fusion. The feature extraction module analyzes the characteristics of voice data sampling rate fluctuations and text processing delays, and combines the polynomial regression analysis model to accurately predict the computing resource consumption of each modality to ensure the efficient execution of the cross-modal fusion process. The computing resource assessment and control module can automatically generate a warning signal when resource consumption is too high, and optimize memory and GPU resource allocation in real time through a dynamic control mechanism, thereby effectively avoiding system bottlenecks and improving the system's response speed and processing capabilities.
[0023] 2. The present invention greatly improves the resource utilization efficiency and real-time performance of multimodal systems, and is particularly suitable for large-scale deep learning models and edge computing device environments. Through intelligent resource consumption evaluation and regulation mechanisms, the system can automatically adjust resource configuration according to actual load conditions, which can not only optimize resource consumption, but also ensure that the system maintains stable operation under high load or high volatility conditions, thereby maximizing system performance. This invention can not only meet the needs of large-scale data processing, but also cope with computing resource limitations in complex environments, and has significant practical application value and market prospects. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.
[0025] Figure 1 It is a system module diagram of the present invention. DETAILED DESCRIPTION
[0026] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0027] For examples, see Figure 1 As shown, the multimodal fusion intelligent interactive large language model system described in this embodiment includes a feature extraction module, a polynomial regression analysis module, a computing resource consumption evaluation module, and a resource dynamic control module;
[0028] Feature extraction module: extracts the processing complexity features of each modality from multiple input modalities, including the voice data sampling rate fluctuation index, and the processing delay features of each modality, including the text processing response time deviation index;
[0029] Polynomial regression analysis module: a polynomial regression model is used to comprehensively analyze the voice data sampling rate fluctuation index and the text processing response time deviation index. The regression model models and analyzes the relationship between the input features of multiple modalities and their computing resource consumption;
[0030] Computing resource consumption assessment module: Based on the regression model analysis results, the severity of computing resource consumption of each modality in the cross-modal fusion process is assessed to identify the precise location of resource bottlenecks during model operation;
[0031] Resource dynamic control module: Based on the evaluation results of the computing resource consumption, a dynamic control mechanism is adopted to adjust the computing resource allocation in the system in real time, and dynamically allocate memory and GPU according to the changes in the severity of computing resource consumption of each modality during the cross-modal fusion process to achieve optimal resource utilization.
[0032] In the feature extraction module, the feature extraction module plays a vital role in the multimodal fusion intelligent interactive large language model system. Its main function is to extract features that can reflect the computing resource requirements of each modality from different input modalities (such as voice, text, image, etc.). These features include processing complexity features and processing delay features, which provide necessary data support for subsequent regression analysis, resource consumption evaluation and regulation.
[0033] The processing complexity feature refers to the complexity required for each modality in calculation, that is, the difficulty of data processing, the amount of calculation, and the consumption of system resources by the characteristics of the modality. Different modalities have different data formats and processing requirements, so their complexity features are also different. The following are the processing complexity feature extraction methods for several common modalities:
[0034] Sampling rate fluctuation index: The sampling rate of speech data often fluctuates in practical applications, especially when processing speech data of different quality or collected by different speech collection devices. The sampling rate fluctuation index measures the degree of variation of the sampling rate of speech signals in different time periods or different audio segments. Highly volatile speech data usually requires more computing resources to process, especially in speech recognition, feature extraction and other links. Calculation method: By calculating the difference in sampling rate of each time period in the speech data, the fluctuation index of the audio stream can be obtained. Speech data with large fluctuations often means higher computational complexity.
[0035] First, prepare the data of the speech signal. This can be the original audio signal obtained from a microphone or other speech acquisition device. The signal is usually a time series, which represents the amplitude of the audio signal in continuous time. In order to eliminate the impact caused by volume differences, the audio signal is normalized. That is, the maximum or minimum value of the signal is adjusted to a certain standard range. The normalized signal x(t) is adjusted to the range of [-1,1]. In order to adapt to the calculation of FFT, the signal is divided into small frames, each of which contains a fixed length of sampling points. Generally speaking, speech signals are non-stationary, so the signal needs to be framed, and the length of each frame is usually between 20ms and 40ms.
[0036] Set the length of each frame signal to N frame , the signal will be divided into several frames with a length of N frame , and use a window function (such as Hamming window or Hanning window) to perform windowing on each frame of the signal to reduce spectrum leakage. The expression is: frame (t) = x(t)·w(t); where w(t) is a window function. Commonly used window functions include the Hamming window
[0037] For each frame of signal, the time domain signal is converted into a frequency domain signal through fast Fourier transform. The output of FFT is a complex array, which represents the amplitude and phase of the signal at different frequencies. The expression is: X(f) = FFT(x frame (t)); where X(f) is the frequency domain signal, f is the frequency, and FFT returns a complex signal containing amplitude and phase information. For each frame, the frequency characteristics of the signal are analyzed by calculating the amplitude spectrum of the FFT result. The amplitude spectrum is calculated as: A(f) = |X(f)| where A(f) is the amplitude spectrum corresponding to frequency f. By calculating the energy distribution of each frequency in each frame of the spectrum, the fluctuation characteristics of the sampling rate are analyzed. The fluctuation of the sampling rate is usually manifested as the energy difference between the low-frequency region (such as the DC component) and the high-frequency region. Therefore, the energy fluctuation index of each frequency band can be calculated, and the voice data sampling rate fluctuation index SRV can be calculated. The expression is: Where A(f,k) is the amplitude at frequency f in the kth frame, μ is the average amplitude value of all frame spectra, and σ is the standard deviation of the amplitude of all frame spectra.
[0038] The processing delay feature refers to the delay feature of each modal data during the processing process. It reflects the time required from input data to output results, and is directly related to the real-time performance and responsiveness of the system. The delay feature extraction method for each modality is different. The following are several common modes of processing delay feature extraction methods:
[0039] Response time deviation index: The processing delay of text data is usually affected by model complexity, data volume, and hardware performance. The processing of long or complex text may result in a longer response time, which affects the user experience. Calculation method: By recording the response time between each text input and model output, and calculating the fluctuation and deviation of the response time, the processing delay characteristics of the text are obtained.
[0040] Collect the response time data of the system processing different text tasks. Set a period of time and record the response time T of the system processing N text tasks. 1 ,T 2 ,…,T N , these response times are usually expressed in milliseconds or other time units. According to the maximum and minimum values of the response time, the entire time range is divided into M intervals. The size of the interval is calculated as follows: Where min(T) and max(T) are the minimum and maximum values of the response time series, respectively. M is the number of intervals. For each interval i, the number of times the response time falls into the interval is counted. i , and then calculate the probability p of each interval i , that is, the frequency of response time in this interval: Among them, f i is the number of times the response time falls into interval i, N is the total number of response time data points, once the probability distribution p of each interval is obtained 1 ,p 2 ,…,p M , calculate the Shannon entropy of the response time, the formula is: Where: H(X) is the Shannon entropy of the response time, which indicates the uncertainty of the response time distribution. The text processing response time deviation index is calculated as: Where SX is the text processing response time deviation index.
[0041] In the polynomial regression analysis module, the voice data sampling rate fluctuation index and the text processing response time deviation index are converted into comprehensive feature vectors, and the comprehensive feature vectors are used as the input of the machine learning model. The machine learning model uses each group of comprehensive feature vectors to predict the severity value label of the computing resource consumption of each modality in the cross-modal fusion process as the prediction target, and takes minimizing the sum of the prediction errors of the severity value labels of the computing resource consumption of each modality in all cross-modal fusion processes as the training target. The machine learning model is trained until the sum of the prediction errors converges, then the model training is stopped, and the severity value of the computing resource consumption of each modality in the cross-modal fusion process is determined according to the model output results, wherein the machine learning model is a polynomial regression model.
[0042] The method for obtaining the severity value of the computing resource consumption of each modality in the cross-modal fusion process is: from the comprehensive feature vector training data of the trained machine learning model, obtain the corresponding function expression: LR=F(SRV, SX); where F is the output function of the model, SRV is the speech data sampling rate fluctuation index, SX is the text processing response time deviation index, and LR is the severity value of the computing resource consumption of each modality in the cross-modal fusion process.
[0043] In the computing resource consumption assessment module, the acquired severity value of the computing resource consumption of each modality in the cross-modal fusion process is compared with the standard resource consumption reference threshold set according to historical data. If the severity value of the computing resource consumption of each modality in the cross-modal fusion process is greater than or equal to the standard resource consumption reference threshold, it means that the severity of the computing resource consumption of each modality in the cross-modal fusion process is high. At this time, an early warning signal is generated and the system faces a performance bottleneck. If the severity value of the computing resource consumption of each modality in the cross-modal fusion process is less than the standard resource consumption reference threshold, it means that the severity of the computing resource consumption of each modality in the cross-modal fusion process is low. At this time, no early warning signal is generated and no optimization measures need to be taken immediately.
[0044] The resource dynamic control module is used to collect the severity values of the computing resource consumption of each modality in the cross-modal fusion process in a subsequent fixed time period, and establish a data set for the severity values of the computing resource consumption of each modality in the cross-modal fusion process that are less than the standard resource consumption reference threshold, calculate the mean and standard deviation of the data set, and analyze it to determine the changes in the severity of the computing resource consumption of each modality in the cross-modal fusion process, and dynamically allocate memory and GPU to achieve optimal resource usage.
[0045] If the mean of the severity values in the data set is greater than or equal to the reference threshold of the mean of the severity values, and the standard deviation of the severity values is less than the reference threshold of the standard deviation of the severity values, it means that the computing resource consumption of the modality is at a high level, but the fluctuation is small. Consider making a slight resource adjustment, maintaining the current resource configuration or slightly increasing the resource allocation;
[0046] If the mean of the severity values is greater than or equal to the reference threshold of the mean of the severity values, and the standard deviation of the severity values is greater than or equal to the reference threshold of the standard deviation of the severity values, it means that the computing resource consumption of the modality is at a high level and fluctuates greatly. At this time, resource allocation should be strengthened to cope with the load changes caused by fluctuations, and more memory and GPU resources need to be temporarily added;
[0047] If the mean of the severity values is less than the reference threshold of the mean of the severity values, and the standard deviation of the severity values is greater than or equal to the reference threshold of the standard deviation of the severity values, it means that the computing resource consumption of the mode is low, but the fluctuation is large. At this time, it is necessary to reduce resource usage and avoid waste by dynamically adjusting resource allocation to maintain system stability;
[0048] If the mean severity value is less than the reference threshold of the mean severity value, and the standard deviation of the severity value is less than the reference threshold of the standard deviation of the severity value, it means that the computing resource consumption of the mode is low and the fluctuation is small. At this time, maintain the current resource configuration without making adjustments to keep the system running smoothly.
[0049] In this embodiment, first, the feature extraction module extracts the processing complexity features of each modality from multiple input modalities, such as the sampling rate fluctuation index of voice data and the response time deviation index of text processing. Then, the polynomial regression analysis module is used to comprehensively analyze these features and establish a regression model to reveal the relationship between the input features of different modalities and their computing resource consumption. Based on the analysis results of the regression model, the computing resource consumption evaluation module can evaluate the severity of resource consumption of each modality in the cross-modal fusion process and identify the precise location of the resource bottleneck. Finally, the resource dynamic control module adjusts the computing resource allocation in the system in real time according to the evaluation results, and dynamically allocates hardware resources such as memory and GPU to ensure the optimal use of system resources. Through this series of steps, the system can effectively solve the problems of excessive computing resource consumption and insufficient real-time performance, and improve the performance and efficiency of the multimodal intelligent interaction system.
[0050] The above description is only a specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.
Claims
1. A multi-modal fusion intelligent interactive large language model system, characterized by: It includes feature extraction module, polynomial regression analysis module, computing resource consumption evaluation module and resource dynamic control module; Feature extraction module: extracts the processing complexity features of each modality from multiple input modalities, including the voice data sampling rate fluctuation index, and the processing delay features of each modality, including the text processing response time deviation index; Polynomial regression analysis module: a polynomial regression model is used to comprehensively analyze the voice data sampling rate fluctuation index and the text processing response time deviation index. The regression model models and analyzes the relationship between the input features of multiple modalities and their computing resource consumption; Computing resource consumption assessment module: Based on the regression model analysis results, the severity of computing resource consumption of each modality in the cross-modal fusion process is assessed to identify the precise location of resource bottlenecks during model operation; Resource dynamic control module: Based on the evaluation results of the computing resource consumption, a dynamic control mechanism is adopted to adjust the computing resource allocation in the system in real time, and dynamically allocate memory and GPU according to the changes in the severity of computing resource consumption of each modality during the cross-modal fusion process to achieve optimal resource utilization.
2. The multimodal fusion intelligent interactive large language model system according to claim 1, characterized in that: In the feature extraction module, the length of each frame signal is set to N frame , the signal x(t) is divided into several frames of length N frame , and use the window function to perform windowing processing on each frame of signal. The expression is: frame (t) = x(t)·w(t); where w(t) is the window function. For each frame of the signal, the time domain signal is converted into a frequency domain signal through the fast Fourier transform. The output of FFT is a complex array, which represents the amplitude and phase of the signal at different frequencies. The expression is: X(f) = FFT(x frame (t)); where X(f) is the frequency domain signal, f is the frequency, and FFT returns a complex signal containing amplitude and phase information. For each frame, the frequency characteristics of the signal are analyzed by calculating the amplitude spectrum of the FFT result. The amplitude spectrum is calculated as: A(f) = |X(f)|; where A(f) is the amplitude spectrum corresponding to frequency f. The voice data sampling rate fluctuation index SRV is calculated as follows: Where A(f,k) is the amplitude at frequency f in the kth frame, μ is the average amplitude value of all frame spectra, and σ is the standard deviation of the amplitude of all frame spectra.
3. The multimodal fusion intelligent interactive large language model system according to claim 2, characterized in that: Collect the response time data of the system processing different text tasks. Set a period of time and record the response time T1, T2, …, T of the system processing N text tasks. N , according to the maximum and minimum values of the response time, the entire time range is divided into M intervals, and the size of the interval ΔT is calculated as follows: Where min(T) and max(T) are the minimum and maximum values of the response time series, respectively. M is the number of intervals. For each interval i, the number of times the response time falls into the interval is counted. i , and then calculate the probability p of each interval i , that is, the frequency of response time within the interval: Among them, f i is the number of times the response time falls into interval i, N is the total number of response time data points, once the probability distribution p1, p2, …, p M , calculate the Shannon entropy of the response time, the formula is: Where: H(X) is the Shannon entropy of the response time, and the text processing response time deviation index is calculated, and the expression is: Where SX is the text processing response time deviation index.
4. The multimodal fusion intelligent interactive large language model system according to claim 3, characterized in that: In the polynomial regression analysis module, the voice data sampling rate fluctuation index and the text processing response time deviation index are converted into comprehensive feature vectors, and the comprehensive feature vectors are used as the input of the machine learning model. The machine learning model uses each group of comprehensive feature vectors to predict the severity value label of the computing resource consumption of each modality in the cross-modal fusion process as the prediction target, and takes minimizing the sum of the prediction errors of the severity value labels of the computing resource consumption of each modality in all cross-modal fusion processes as the training target. The machine learning model is trained until the sum of the prediction errors converges, then the model training is stopped, and the severity value of the computing resource consumption of each modality in the cross-modal fusion process is determined according to the model output results, wherein the machine learning model is a polynomial regression model.
5. The multimodal fusion intelligent interactive large language model system according to claim 4, characterized in that: In the computing resource consumption evaluation module, the acquired computing resource consumption severity value of each modality in the cross-modal fusion process is compared with the standard resource consumption reference threshold set according to historical data. If the computing resource consumption severity value of each modality in the cross-modal fusion process is greater than or equal to the standard resource consumption reference threshold, it means that the computing resource consumption severity of each modality in the cross-modal fusion process is high. At this time, an early warning signal is generated and the system faces a performance bottleneck. If the severity value of the computing resource consumption of each modality in the cross-modal fusion process is less than the standard resource consumption reference threshold, it means that the severity of the computing resource consumption of each modality in the cross-modal fusion process is low. At this time, no warning signal is generated and no optimization measures need to be taken immediately.
6. The multimodal fusion intelligent interactive large language model system according to claim 5, characterized in that: The resource dynamic control module is used to collect the severity values of the computing resource consumption of each modality in the cross-modal fusion process in a subsequent fixed time period, and establish a data set for the severity values of the computing resource consumption of each modality in the cross-modal fusion process that are less than the standard resource consumption reference threshold, calculate the mean and standard deviation of the data set, and analyze it to determine the changes in the severity of the computing resource consumption of each modality in the cross-modal fusion process, and dynamically allocate memory and GPU to achieve optimal resource usage.
7. The multimodal fusion intelligent interactive large language model system according to claim 6, characterized in that: If the mean of the severity values in the data set is greater than or equal to the reference threshold of the mean of the severity values, and the standard deviation of the severity values is less than the reference threshold of the standard deviation of the severity values, it means that the computing resource consumption of the modality is at a high level, but the fluctuation is small. Consider making a slight resource adjustment, maintaining the current resource configuration or slightly increasing the resource allocation; If the mean of the severity values is greater than or equal to the reference threshold of the mean of the severity values, and the standard deviation of the severity values is greater than or equal to the reference threshold of the standard deviation of the severity values, it means that the computing resource consumption of the modality is at a high level and fluctuates greatly. At this time, resource allocation should be strengthened to cope with the load changes caused by fluctuations, and more memory and GPU resources need to be temporarily added; If the mean of the severity values is less than the reference threshold of the mean of the severity values, and the standard deviation of the severity values is greater than or equal to the reference threshold of the standard deviation of the severity values, it means that the computing resource consumption of the mode is low, but the fluctuation is large. At this time, it is necessary to reduce resource usage and avoid waste by dynamically adjusting resource allocation to maintain system stability; If the mean severity value is less than the reference threshold of the mean severity value, and the standard deviation of the severity value is less than the reference threshold of the standard deviation of the severity value, it means that the computing resource consumption of the mode is low and the fluctuation is small. At this time, maintain the current resource configuration without making adjustments to keep the system running smoothly.