Multimodal large model driven ai base platform system and method
Patent Information
- Application Number
- CN202510256614.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-05
- Publication Date
- 2026-10-16
AI Technical Summary
传统单模态处理方法(如仅处理文本、图像或音频)在面对复杂任务时往往难以满足需求，而多模态处理可以综合多种数据类型的信息，为任务提供更全面和准确的支持
[0033] This invention discloses a multimodal, large-model-driven AI platform system and method, relating to the field of artificial intelligence technology, to address the problem that single-modal processing methods are insufficient. It includes a data input and preprocessing module, a modal feature extraction and semantic alignment module, a model loading and scheduling module, an inference and result generation module, a user interaction and feedback module, and a data storage module, with signal connections between the modules. The data input and preprocessing module implements dynamic adaptive data cleaning and enhancement. The modal feature extraction and semantic alignment module combines a cross-modal alignment mechanism to achieve semantic consistency mapping. The model loading and scheduling module performs efficient model loading and distributed scheduling. The inference and result generation module performs inference and implements result post-processing and multimodal fusion output. The user interaction and feedback module dynamically optimizes model performance based on online learning. This invention provides a multimodal AI platform suitable for multi-task inference and analysis scenarios.
Smart Images
Figure CN122927484A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology, and more specifically, to a multimodal large-model driven AI foundation platform system and method. Background Technology
[0002] With the rapid development of artificial intelligence (AI) technology, multimodal data processing has become an important direction for intelligent systems. Traditional single-modal processing methods (such as processing only text, images, or audio) often fall short of the requirements when facing complex tasks, while multimodal processing can integrate information from multiple data types to provide more comprehensive and accurate support for tasks. For example, in autonomous driving scenarios, multiple sensors such as cameras, radar, and lidar work together to build a complete environmental perception model; in the field of intelligent healthcare, text medical records, medical images, and patient voice descriptions together constitute important evidence for diagnosis.
[0003] Currently, large multimodal models are gradually becoming the mainstream approach for multimodal data processing. These models, based on deep learning, integrate data features from different modalities through a unified framework, achieving cross-modal semantic alignment and unified inference. This approach significantly improves the accuracy and robustness of multimodal tasks. However, multimodal data processing technology still faces the following challenges:
[0004] To address the above problems, this invention proposes a solution. Summary of the Invention
[0005] To overcome the aforementioned deficiencies of the prior art, embodiments of the present invention provide a multimodal large model-driven AI base platform system and method to solve the problems mentioned in the background art.
[0006] To achieve the above objectives, the present invention provides the following technical solution:
[0007] In a preferred embodiment, it includes: a data input and preprocessing module, a modal feature extraction and semantic alignment module, a model loading and scheduling module, an inference and result generation module, a user interaction and feedback module, and a data storage module, with signal connections between the modules;
[0008] The data input and preprocessing module receives and processes multimodal data from multiple sources;
[0009] The modal feature extraction and semantic alignment module extracts modal features from multimodal data and aligns them to a shared semantic space;
[0010] The model loading and scheduling module loads the appropriate pre-trained or fine-tuned model according to the task requirements, and completes task scheduling and resource allocation through distributed computing.
[0011] The reasoning and results generation module performs large-scale model reasoning tasks and generates the final results.
[0012] The user interaction and feedback module provides users with task configuration, result viewing, and feedback mechanisms, and supports model optimization and updates based on feedback.
[0013] The data storage module is used to store all data during the processing.
[0014] In a preferred embodiment, the data input and preprocessing module selects an appropriate access method based on the modal characteristics of the multimodal data: text data is input via file or API in structured JSON format; image and video data is uploaded via form upload or direct transmission of binary stream; audio data is input in real time in streaming media format; data upload for non-real-time tasks is implemented via REST API; and data upload for real-time tasks is implemented via WebSocket.
[0015] The data input and preprocessing module uses OpenCV, Pillow, or Librosa tools to perform denoising, normalization, rotation enhancement, and noise filtering on the multimodal data, and then stores the processed multimodal data in the data storage module.
[0016] In a preferred embodiment, the data input and preprocessing module obtains the brightness distribution, contrast range, and color saturation of image data by calculating the brightness histogram of each image and extracting the data features of the statistical modal data of pixel intensity; it obtains the sentence length distribution, word frequency distribution, and grammatical complexity of text data by statistically analyzing the number of words and sentence length of each text sample and constructing a word frequency matrix; it obtains the frequency distribution, signal-to-noise ratio, and dynamic range of audio data by extracting the spectrum through short-time Fourier transform and calculating the signal-to-noise ratio; and it obtains the frame rate consistency, scene change frequency, and motion vector distribution of video data by extracting the number of frames per second and detecting the changes between consecutive frames.
[0017] The data input and preprocessing module introduces the kernel density estimation (KDE) algorithm to model the distribution of multimodal data, analyze the distribution pattern of modal data, and select an appropriate preprocessing method for each modality based on the results of the distribution model.
[0018] The data input and preprocessing module evaluates the preprocessing effect based on the model inference results, and adjusts the parameters of the distributed modeling and preprocessing process by monitoring the inference performance of the model under different preprocessing conditions.
[0019] In a preferred embodiment, the modality feature extraction and semantic alignment module extracts modality features from each modality data using deep learning models BERT, ResNet, and OpenL3, and uses a multimodal Transformer cross-modal alignment model to map the modality feature data of these different modalities into a shared semantic space. It also employs a multimodal attention mechanism to ensure semantic consistency between different modalities.
[0020] The modal feature extraction and semantic alignment module performs correlation analysis on the modal features of multimodal data, and evaluates the interference effect of multimodal data by calculating the cosine similarity and mutual information between modalities; specifically based on the formula... Where A and B are two modal feature vectors, A·B is the dot product, and ||A|| and ||B|| are the magnitudes;
[0021] The modal feature extraction and semantic alignment module assigns different weights to each modal feature based on the magnitude of the interference effect using a weighted average or adaptive weighting algorithm. It also introduces a dynamic weighting mechanism based on the quantified interference effect, adjusting the feature weights according to the needs of the task, the interference effect of the modality, and the importance of the modality.
[0022] The modal feature extraction and semantic alignment module uses the deep learning model Multimodal Transformer to nonlinearly fuse the modal features of multimodal data;
[0023] The modality feature extraction and semantic alignment module employs a self-attention mechanism, specifically based on the formula... Where Q represents the modal feature query, K represents the modal feature key, and V represents the modal feature value; add residual joins and LayerNorm, specifically according to formula F. output =LayerNorm(F input +Attention(Q, K, V)) forces modal features to be semantically consistent in high-dimensional space through two layers of linear transformation and the non-linear activation function ReLU, as well as through an alignment loss function.
[0024] In a preferred implementation, the model loading and scheduling module saves the TensorFlow model as a SavedModel format, configures TensorFlow Serving, specifies the model path and version management, and loads and infers the model using a REST API or gRPC interface.
[0025] The model loading and scheduling module uses the Ray task scheduling framework for distributed computing scheduling, and dynamically allocates computing resources in conjunction with load balancing strategies.
[0026] In a preferred embodiment, the inference and result generation module accelerates inference through the ONNX Runtime inference engine, which supports multiple hardware and multiple backends, performs post-processing on the results in conjunction with the rule engine, and uses the Bokeh visualization tool to generate interactive charts and graphs to display the distribution of classification results and confidence curves.
[0027] In a preferred embodiment, the user interaction and feedback module uses React to develop the front-end interface, establishes a WebSocket connection to update the task status in real time, records user feedback, and completes model optimization and updates through the online learning framework PyTorchLightning.
[0028] In a preferred embodiment
[0029] Step 1: Receive multi-source multimodal data, model the data distribution based on kernel density estimation (KDE), dynamically select an appropriate preprocessing method, and optimize the preprocessing process through model inference feedback;
[0030] Step 2: Use a dedicated deep learning model to extract modal features, use a multimodal alignment model to map them to a shared semantic space, and dynamically adjust the modal feature weights to optimize semantic consistency;
[0031] Step 3: Load the adaptation model according to the task requirements, dynamically schedule resources through the distributed computing framework Ray, and optimize computing efficiency using load balancing strategies;
[0032] Step 4: Execute inference tasks through the accelerated inference engine ONNX Runtime, combine the results of rule engine and consistency check post-processing, integrate multimodal results, and iteratively optimize model performance through user feedback.
[0033] This invention discloses a multimodal, large-model-driven AI platform system and method, relating to the field of artificial intelligence technology, to address the problem that single-modal processing methods are insufficient. It includes a data input and preprocessing module, a modal feature extraction and semantic alignment module, a model loading and scheduling module, an inference and result generation module, a user interaction and feedback module, and a data storage module, with signal connections between the modules. The data input and preprocessing module implements dynamic adaptive data cleaning and enhancement. The modal feature extraction and semantic alignment module combines a cross-modal alignment mechanism to achieve semantic consistency mapping. The model loading and scheduling module performs efficient model loading and distributed scheduling. The inference and result generation module performs inference and implements result post-processing and multimodal fusion output. The user interaction and feedback module dynamically optimizes model performance based on online learning. This invention provides a multimodal AI platform suitable for multi-task inference and analysis scenarios. Attached Figure Description
[0034] Figure 1This is a schematic diagram of the system structure of the multimodal large model-driven AI base platform of the present invention.
[0035] Figure 2 This is a flowchart illustrating the operation of the multimodal large model-driven AI base platform method of the present invention. Detailed Implementation
[0036] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0037] Example
[0038] This invention discloses a multimodal large model-driven AI foundation platform system, including: a data input and preprocessing module, a modal feature extraction and semantic alignment module, a model loading and scheduling module, an inference and result generation module, a user interaction and feedback module, and a data storage module, with signal connections between the modules.
[0039] The data storage module is used to store all data during the processing.
[0040] The data input and preprocessing module receives and processes multimodal data (text, images, audio, and video) from multiple sources, providing high-quality, standardized input data for subsequent tasks.
[0041] First, the data input and preprocessing module designs an appropriate access method based on the modal characteristics of the data. In this embodiment, text data is input in a structured format (such as JSON or XML) via file or API; image and video files are uploaded via form upload or direct transmission of binary stream; audio data is input in real time in the form of streaming media; data upload for non-real-time tasks is realized through the data interface protocol REST API; and data upload for real-time tasks is realized through WebSocket.
[0042] Furthermore, the data input and preprocessing module uses OpenCV, Pillow, or Librosa tools to perform denoising, normalization, rotation enhancement, and noise filtering operations on the multimodal data, completing the cleaning and enhancement of the multimodal data, and storing the processed multimodal data in the data storage module.
[0043] It should be noted that traditional data preprocessing methods employ fixed procedures, failing to dynamically optimize based on the distribution of different modalities. This leads to insufficient or excessive data cleaning, impacting model performance. In this embodiment, a distribution evaluation algorithm, kernel density estimation (KDE), is introduced to model multimodal data based on the characteristics of different modalities, capturing data distribution features such as image brightness distribution and text sentence length distribution. Based on the distribution model, appropriate preprocessing tools are selected. For example, an adaptive noise suppression algorithm (such as Wiener Filter) is introduced for noisy audio data, and a segmentation strategy is applied to long text data to improve processing efficiency. By comparing the model inference performance of the preprocessed data, the parameters of the distribution modeling and preprocessing process are adjusted, forming a closed-loop data optimization mechanism.
[0044] First, the data input and preprocessing module extracts data features from the acquired multimodal data. Specifically, it calculates the brightness histogram of each image to extract statistical data features of pixel intensity, obtaining the brightness distribution, contrast range, and color saturation of the image data; it constructs a word frequency matrix by counting the number of words and sentence length of each text sample to obtain the sentence length distribution, word frequency distribution, and grammatical complexity of the text data; it extracts the spectrum by performing a short-time Fourier transform (STFT) and calculates the signal-to-noise ratio to obtain the frequency distribution, signal-to-noise ratio (SNR), and dynamic range of the audio data; and it extracts the frames per second and detects the changes between consecutive frames to obtain the frame rate consistency, scene change frequency, and motion vector distribution of the video data.
[0045] Furthermore, the data input and preprocessing module introduces distribution evaluation algorithms (such as kernel density estimation, KDE) to model the distribution of the data and analyze the distribution pattern of the data.
[0046] Specifically, KDE is a non-parametric density estimation method that can estimate the probability density function of data without assuming any distributional information, making it particularly suitable for multimodal data. For example, for image data, KDE can estimate the density function of brightness distribution to determine whether the image brightness is uniform and whether there are excessively dark or bright areas; for text data, KDE can estimate the distribution of sentence length to analyze whether there are excessively long or short texts; for audio data, KDE helps estimate frequency distribution and determine the range of noise or signal strength fluctuations.
[0047] Furthermore, the data input and preprocessing module selects appropriate preprocessing methods for each modality of data based on the results of the distribution model. For example, for image data, when the KDE model shows uneven brightness distribution, a contrast enhancement algorithm is selected, or automatic enhancement is performed on areas with low brightness; for audio data with high noise, based on the distribution model, the adaptive noise suppression algorithm Wiener Filter is introduced to denoise the audio signal by estimating the statistical characteristics of the noise, thereby improving the quality of the audio data; for text data, when the KDE model detects a long-tail phenomenon in the sentence length distribution, a segmentation strategy is selected to divide the excessively long text into several segments for processing, thereby improving processing efficiency.
[0048] Furthermore, after data preprocessing, the data input and preprocessing module evaluates the preprocessing effect based on the model's inference results. By monitoring the model's inference performance under different preprocessing conditions (such as accuracy, recall, and inference time), the data input and preprocessing module adjusts the parameters of the distributed modeling and preprocessing process. For example, when processing an audio recognition task, after KDE estimation, significant noise is found in the audio data. Based on this, the data input and preprocessing module selects the Wiener Filter for noise reduction. After the preprocessed audio signal is input into the model, the inference accuracy improves. If the accuracy does not improve significantly, it may be due to a mismatch in the noise suppression algorithm parameters. The data input and preprocessing module will automatically adjust the noise estimation parameters and re-evaluate the model performance until optimal results are achieved.
[0049] It's important to note that the entire process is dynamically adaptive. It not only selects the most suitable processing method based on the data distribution characteristics before preprocessing, but also optimizes the entire process through model inference performance feedback. After each round of optimization, the data preprocessing effect will better match the model's requirements, gradually improving the processing effect and inference performance of the data input and preprocessing modules.
[0050] The modality feature extraction and semantic alignment module extracts modality features from multimodal data and aligns them to a shared semantic space, providing a foundation for unified analysis and reasoning. Specifically, it uses specialized deep learning models (such as BERT, ResNet, and OpenL3) to extract modality feature vectors and employs multimodal alignment models (such as CLIP or Multimodal Transformer) to achieve cross-modal semantic alignment. The output modality features are stored in the data storage module.
[0051] First, the modality feature extraction and semantic alignment module extracts modality features from the data of each modality using deep learning models (such as BERT, ResNet, and OpenL3). The modality feature extraction methods for different modalities are as follows:
[0052] Text feature extraction: The BERT model is used, which employs the Transformer architecture and a multi-layer self-attention mechanism to learn the relationships between words in the text and long-range dependency information, thereby extracting high-dimensional feature vectors from the text and capturing the contextual information of the text.
[0053] Image feature extraction: The ResNet model is used to extract visual features from the image through multi-layer convolution and pooling operations, including shape, color and texture, and a fully connected layer is used to output a high-dimensional feature vector of the image.
[0054] Audio feature extraction: Using the OpenL3 model, modal features containing information such as rhythm, pitch, and timbre of the audio signal are extracted by convolving the Mel frequency cepstral coefficients (MFCC) of the audio time-frequency graph.
[0055] Furthermore, the modal feature extraction and semantic alignment module employs CLIP or a multimodal Transformer cross-modal alignment model to map modal feature data from different modalities into a shared semantic space. Within this space, modal features from different modalities, such as text, images, and audio, can be compared and fused. Contrastive learning or multimodal attention mechanisms are used to ensure semantic consistency between different modalities.
[0056] It should be noted that in multimodal data processing, features from different modalities may interfere with each other (e.g., noise in video data may weaken the relevance of text). Direct concatenation or simple alignment methods ignore the potential complex nonlinear relationships between modalities, leading to redundancy or loss of fused features. In this embodiment, a feature optimization and dynamic adjustment mechanism based on intermodal interference effects is designed. By quantifying the intermodal interference effects, the feature weighting strategy is dynamically optimized to ensure that the fused features are more semantically consistent. Through the dynamic adjustment mechanism of interference effects, the modal feature extraction and semantic alignment modules can adjust the importance distribution of modal features according to different tasks, improving adaptability to diverse tasks.
[0057] First, correlation analysis is performed on the modal features of the multimodal data. This analysis assesses the interference effect by calculating the similarity between modalities (e.g., cosine similarity, mutual information, etc.). High correlation indicates that the two modal features are complementary, while low correlation suggests potential interference. The modal feature extraction and semantic alignment module calculates the correlation between different modal features using statistical methods (e.g., Pearson correlation coefficient) or deep learning-based attention mechanisms.
[0058] Specifically, the cosine similarity between modal features is calculated to measure their semantic relevance, based on the formula... Here, A and B are two modal feature vectors, A·B is their dot product, and ||A|| and ||B|| are their moduli (i.e., the Euclidean norm of the vectors). When the correlation between the features of two modalities is low (for example, the correlation between images and audio is weak), it indicates that there is interference between them; when the correlation between the features of two modalities is high, it indicates that there is strong semantic consistency between them, and they complement each other.
[0059] Furthermore, based on the quantified interference effect, the modality feature extraction and semantic alignment module assigns different weights to the features of each modality. This process ensures that each modality occupies an appropriate proportion in the final fused features by introducing a dynamic weighting mechanism. The dynamic weighting mechanism adjusts the feature weights based on the needs of the task, the interference effect of the modality, and the importance of the modality.
[0060] Specifically, the modality feature extraction and semantic alignment module assigns different weights to each modality feature based on the magnitude of interference effects using weighted averaging or adaptive weighting algorithms. For example, in sentiment analysis tasks, audio features may be more critical than image features (voice emotion is more obvious), so the weight of audio features is dynamically increased. Conversely, when image features have high noise, their weight is reduced. Simultaneously, the modality feature extraction and semantic alignment module adjusts the weights of modality features according to the task objective based on task priority and modality contribution through a task-adaptive weighting mechanism. For example, in tasks combining text and images, the module may prioritize text features, while in tasks combining audio and video, the weights of audio and video features are increased.
[0061] Furthermore, after dynamically optimizing the feature weighting, the modal feature extraction and semantic alignment module uses a deep learning model (such as a multimodal Transformer) to nonlinearly fuse the modal features of the multimodal data to generate a high-dimensional joint feature vector.
[0062] Specifically, before entering the Transformer model, the modal feature extraction and semantic alignment module performs simple concatenation, joining modal feature vectors from different modalities along a certain dimension to form a larger vector. For example, image features are 2048-dimensional, text features are 768-dimensional, and audio features are 512-dimensional; the concatenated vector will be a 3328-dimensional vector. This vector contains weighted features from the three modalities. However, since the Transformer itself does not depend on the temporal or positional order of the data, positional encoding needs to be added to the concatenated vector to ensure that the model understands the arrangement order between the different modalities.
[0063] The core of the multimodal Transformer lies in using the attention mechanism (Self-Attention) to capture complex nonlinear relationships between different modalities;
[0064] Specifically, the modality feature extraction and semantic alignment module captures global dependencies between modalities through a self-attention mechanism, based on the formula... Where Q represents the query of the modal feature, K represents the key of the modal feature, and V represents the value of the modal feature;
[0065] Furthermore, residual connections and LayerNorms are added to enhance model stability and feature representation capabilities, specifically based on formula F. output =LayerNorm(F input The algorithm employs a +Attention(Q, K, V) approach, and further enhances feature representation through two layers of linear transformation and a non-linear activation function (ReLU). Finally, an alignment loss function is used to force semantic consistency of modal features in high-dimensional space.
[0066] Furthermore, after completing deep fusion and semantic alignment, the modality feature extraction and semantic alignment module reallocates modality weights according to the task objective (such as classification or generation), adjusts the final feature distribution, and balances the loss functions of each task through a multi-task framework with shared parameters, further optimizing the output features for different tasks.
[0067] The model loading and scheduling module loads the appropriate pre-trained or fine-tuned model according to the task requirements, and completes task scheduling and resource allocation through distributed computing.
[0068] First, the model loading and scheduling module implements model loading through the TensorFlow Serving framework, which is designed specifically for TensorFlow models and supports efficient online inference. Specifically, the model loading and scheduling module saves the TensorFlow model as a SavedModel format, configures TensorFlow Serving, specifies the model path and version management, and finally uses the REST API or gRPC interface for loading and inference.
[0069] Furthermore, the model loading and scheduling module employs the Ray task scheduling framework, which supports lightweight task distribution and efficient resource scheduling, for distributed computing scheduling, dynamically allocating computing resources in conjunction with load balancing strategies. Specifically, the model loading and scheduling module starts the distributed computing cluster through Ray's initialization interface. The Ray cluster includes a master node and several worker nodes. The master node is responsible for task scheduling, and the worker nodes are responsible for executing specific computing tasks. Simultaneously, each node registers its own resource information (such as the number of CPU cores, the number of GPUs, and memory size) with the master node upon startup, forming a global resource view. Before task submission, the task logic and input data are encapsulated into Ray's remote tasks. Ray automatically schedules the task to an idle worker node for execution. Finally, the model loading and scheduling module utilizes Ray's built-in distributed object store to achieve efficient data sharing between tasks, avoiding unnecessary data copying.
[0070] Ray periodically collects resource usage data from each node (such as CPU and GPU utilization, memory usage, etc.) and aggregates the data to the master node. It also monitors the task queue length of each node to determine its current load status. For computationally intensive tasks, GPU nodes are prioritized. For lightweight tasks, CPU nodes are prioritized to avoid wasting GPU resources.
[0071] The reasoning and results generation module performs large model reasoning tasks and generates final results (such as classification labels, generated images, and text reports).
[0072] First, the inference and result generation modules accelerate inference through the ONNX Runtime inference engine, which supports various hardware (CPU, GPU, TPU) and backends (such as CUDA, TensorRT). Combined with a rule engine, the modules perform post-processing of the results, specifically configuring rule sets for different tasks. For example, for classification tasks, probability thresholds are set to determine the final classification label; for generation tasks, rules are applied to check the coherence of generated text or repair defects in generated images. Simultaneously, alerts or requests for re-examination are triggered for abnormal results (such as classification probabilities below a set threshold). Consistency checks are performed on multimodal results, such as comparing whether the text description matches the content of the generated image.
[0073] Furthermore, the inference and result generation module supports the integration and visualization of multimodal results (e.g., via Bokeh or D3.js). Specifically, the inference and result generation module unifies results from different modalities (such as images and text) into a single multimodal output. For classification tasks, it uses a voting mechanism or confidence weighting strategy to fuse classification results from various modalities; for generation tasks, it combines text reports with visualized images to generate more interpretable multimodal outputs; and for video and audio tasks, it analyzes the consistency of results over time. Finally, the Bokeh visualization tool is used to generate interactive charts and graphs to display the distribution of classification results and confidence curves.
[0074] The user interaction and feedback module provides users with task configuration, result viewing, and feedback mechanisms, while also supporting model optimization and updates based on feedback.
[0075] First, the user interaction and feedback module uses React, a platform well-suited for building dynamic interactions and complex state management, to develop the front-end interface. By establishing a WebSocket connection, it updates the task status in real time, for example: displaying a real-time progress bar or status label (such as "Processing" or "Completed"); notifying the user when a task fails and providing a resubmission option; and sending a push notification upon task completion. State management tools (such as Redux or Vuex) ensure that task state is shared and synchronized across multiple UI components, enabling cross-component data caching and improving UI responsiveness.
[0076] Furthermore, user feedback is recorded, and model optimization and updates are performed using an online learning framework (such as PyTorch Lightning). Specifically, the user interaction and feedback module uses the PyTorch Lightning framework to load user-annotated data into the optimized model in real time via a data pipeline. The model is trained by dynamically adjusting parameters, such as performing incremental training using user feedback data and adjusting the learning rate and loss function based on the distribution of feedback data. The optimized model seamlessly replaces the old version, and users are notified of the updates (such as improved performance metrics).
[0077] This invention also relates to a method for a multimodal large model-driven AI foundation platform;
[0078] The specific process is as follows:
[0079] Step 1: Receive multi-source, multi-modal data, model the data distribution based on kernel density estimation (KDE), dynamically select an appropriate preprocessing method, and optimize the preprocessing process through model inference feedback.
[0080] Step 2: Extract modal features using a dedicated deep learning model, map them to a shared semantic space using a multimodal alignment model, and dynamically adjust the modal feature weights to optimize semantic consistency.
[0081] Step 3: Load the adaptation model according to the task requirements, dynamically schedule resources through the distributed computing framework Ray, and optimize computing efficiency using load balancing strategies.
[0082] Step 4: Execute inference tasks through the accelerated inference engine ONNX Runtime, combine the results of rule engine and consistency check post-processing, integrate multimodal results, and iteratively optimize model performance through user feedback.
[0083] The above formulas are all dimensionless calculations. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters in the formulas are set by those skilled in the art according to the actual situation.
[0084] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, in the form of a computer program product.
[0085] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and inventive constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0086] In addition, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.
[0087] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
[0088] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A multimodal large-model driven AI foundation platform system, characterized in that... ； It includes: a data input and preprocessing module, a modal feature extraction and semantic alignment module, a model loading and scheduling module, an inference and result generation module, a user interaction and feedback module, and a data storage module, as well as signal connections between the modules; The data input and preprocessing module receives and processes multimodal data from multiple sources; The modal feature extraction and semantic alignment module extracts modal features from multimodal data and aligns them to a shared semantic space; The model loading and scheduling module loads the appropriate pre-trained or fine-tuned model according to the task requirements, and completes task scheduling and resource allocation through distributed computing. The reasoning and results generation module performs large-scale model reasoning tasks and generates the final results. The user interaction and feedback module provides users with task configuration, result viewing, and feedback mechanisms, and supports model optimization and updates based on feedback. The data storage module is used to store all data during the processing.
2. The multimodal large model-driven AI foundation platform system according to claim 1, characterized in that: The data input and preprocessing module selects the appropriate access method based on the modal characteristics of the multimodal data: text data is input via file or API in structured JSON format; image and video data are uploaded via form upload or direct transmission of binary stream; Audio data is transmitted in real time in the form of streaming media; non-real-time task data upload is realized through the REST API data interface protocol; and real-time task data upload is realized through WebSocket. The data input and preprocessing module uses OpenCV, Pillow, or Librosa tools to perform denoising, normalization, rotation enhancement, and noise filtering on the multimodal data, and then stores the processed multimodal data in the data storage module.
3. The multimodal large model-driven AI foundation platform system according to claim 2, characterized in that: The data input and preprocessing module calculates the brightness histogram of each image and extracts the statistical modal data features of pixel intensity to obtain the brightness distribution, contrast range, and color saturation of the image data; it constructs a word frequency matrix by counting the number of words and sentence length of each text sample to obtain the sentence length distribution, word frequency distribution, and grammatical complexity of the text data; it extracts the spectrum by performing short-time Fourier transform and calculates the signal-to-noise ratio to obtain the frequency distribution, signal-to-noise ratio, and dynamic range of the audio data; and it extracts the frames per second and detects the changes between consecutive frames to obtain the frame rate consistency, scene change frequency, and motion vector distribution of the video data. The data input and preprocessing module introduces the kernel density estimation (KDE) algorithm to model the distribution of multimodal data, analyze the distribution pattern of modal data, and select an appropriate preprocessing method for each modality based on the results of the distribution model. The data input and preprocessing module evaluates the preprocessing effect based on the model inference results, and adjusts the parameters of the distributed modeling and preprocessing process by monitoring the inference performance of the model under different preprocessing conditions.
4. The multimodal large model-driven AI base platform system according to claim 3, characterized in that; The modality feature extraction and semantic alignment module extracts modality features from the data of each modality by using deep learning models BERT, ResNet, and OpenL3. It adopts a multimodal Transformer cross-modal alignment model to map the modality feature data of these different modalities into a shared semantic space. It also adopts a multimodal attention mechanism to ensure semantic consistency between different modalities. The modal feature extraction and semantic alignment module performs correlation analysis on the modal features of multimodal data, and evaluates the interference effect of multimodal data by calculating the cosine similarity and mutual information between modalities; specifically based on the formula... Where A and B are two modal feature vectors, A·B is the dot product, and ||A|| and ||B|| are the magnitudes; The modal feature extraction and semantic alignment module assigns different weights to each modal feature based on the magnitude of the interference effect using a weighted average or adaptive weighting algorithm. It also introduces a dynamic weighting mechanism based on the quantified interference effect, adjusting the feature weights according to the needs of the task, the interference effect of the modality, and the importance of the modality. The modal feature extraction and semantic alignment module uses the deep learning model Multimodal Transformer to nonlinearly fuse the modal features of multimodal data; The modality feature extraction and semantic alignment module employs a self-attention mechanism, specifically based on the formula... Where Q represents the modal feature query, K represents the modal feature key, and V represents the modal feature value; add residual joins and LayerNorm, specifically according to formula F. output =LayerNorm(F input +Attention(Q, K, V) also forces modal features to be semantically consistent in high-dimensional space by aligning the loss function.
5. The multimodal large model-driven AI base platform system according to claim 4, characterized in that: The model loading and scheduling module saves TensorFlow models in SavedModel format, configures TensorFlow Serving, specifies the model path and version management, and loads and infers using REST API or gRPC interface; The model loading and scheduling module uses the Ray task scheduling framework for distributed computing scheduling, and dynamically allocates computing resources in conjunction with load balancing strategies.
6. The multimodal large model-driven AI base platform system according to claim 5, characterized in that: The inference and result generation module accelerates inference by using the ONNX Runtime inference engine, which supports multiple hardware and backends. It combines the rule engine to post-process the results and uses the Bokeh visualization tool to generate interactive charts and graphs to display the distribution of classification results and confidence curves.
7. The multimodal large model-driven AI foundation platform system according to claim 6, characterized in that: The user interaction and feedback module uses React to develop the front-end interface. It establishes a WebSocket connection to update the task status in real time, records user feedback, and uses the online learning framework PyTorch Lightning to optimize and update the model.
8. A method for an AI foundation platform driven by a multimodal large model, characterized in that, Step 1: Receive multi-source multimodal data, model the data distribution based on kernel density estimation (KDE), dynamically select an appropriate preprocessing method, and optimize the preprocessing process through model inference feedback; Step 2: Use a dedicated deep learning model to extract modal features, use a multimodal alignment model to map them to a shared semantic space, and dynamically adjust the modal feature weights to optimize semantic consistency; Step 3: Load the adaptation model according to the task requirements, dynamically schedule resources through the distributed computing framework Ray, and optimize computing efficiency using load balancing strategies; Step 4: Execute inference tasks through the accelerated inference engine ONNX Runtime, combine the results of rule engine and consistency check post-processing, integrate multimodal results, and iteratively optimize model performance through user feedback.