Multi-mode cooperative channel processing system
The multimodal collaborative channel processing system solves the problems of low fusion efficiency and high latency in multimodal systems under data heterogeneity and high concurrency scenarios, and realizes efficient and low-latency multimodal data processing and decision-making, thereby improving the system's collaborative processing capabilities and data consistency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-21
- Publication Date
- 2026-03-10
AI Technical Summary
Existing multimodal systems have significant bottlenecks in handling heterogeneous data, real-time fusion algorithms, and high-concurrency scenarios. Semantic gaps between modalities lead to low feature fusion efficiency, dynamic weight learning relies on complex models, consumes large computational resources, distributed communication latency is difficult to meet low-latency requirements, and high data annotation costs and privacy leakage risks limit application expansion.
A multimodal collaborative channel processing system is adopted, including a multimodal data acquisition and preprocessing layer, a feature extraction and encoding layer, a cross-modal feature fusion and alignment layer, a collaborative processing layer, a distributed real-time communication and messaging layer, a business logic layer, and an application display and interaction layer. Features are extracted through deep learning models, dynamic weights and attention mechanisms optimize spatiotemporal alignment, distributed communication supports low-latency interaction, collaborative processing improves decision accuracy, and conflict resolution ensures data consistency.
It enables efficient collaborative processing of multimodal data, improves feature fusion efficiency, reduces latency, enhances decision-making accuracy, supports low-latency interaction for tens of thousands of concurrent users, and ensures data consistency and privacy security.
Smart Images
Figure BDA0005645167130000021 
Figure BDA0005645167130000051 
Figure FDA0005645167120000021
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of information processing, in particular to a multi-modal collaborative channel processing system. BACKGROUND
[0002] As a key branch of artificial intelligence, multi-modal data processing technology realizes efficient processing of complex tasks in intelligent security, medical image analysis, autonomous driving decision-making, and smart city management by fusing visual, auditory, and textual multi-source heterogeneous data. Traditional single-modal systems only process a single data type, making it difficult to capture cross-modal associated semantics. Early multi-modal solutions mostly use simple splicing or fixed weight fusion, resulting in low spatio-temporal alignment accuracy and high semantic ambiguity. In recent years, deep learning models such as CNN have made breakthroughs in image feature extraction, RNN / LSTM for processing time-series audio, and Transformer for processing textual sequences, but there are still technical gaps in cross-modal feature alignment in the time-space dimension, dynamic weight adaptive adjustment, real-time fusion efficiency, and other aspects.
[0003] Current multi-modal systems have significant bottlenecks in data heterogeneity processing, real-time fusion algorithms, and high-concurrency scenarios. The semantic gap between modalities leads to low feature fusion efficiency, dynamic weight learning relies on complex models, and the consumption of computing resources is large. Distributed communication latency is difficult to meet the low-latency demand, and the high cost of data labeling and the risk of privacy leakage limit the expansion of applications. SUMMARY
[0004] The present application aims to provide a multi-modal collaborative channel processing system to solve the problems raised in the background.
[0005] To achieve the above-mentioned purpose, the present application provides the following technical solution: a multi-modal collaborative channel processing system, which comprises a multi-modal data acquisition and preprocessing layer, a feature extraction and encoding layer, a cross-modal feature fusion and alignment layer, a collaborative processing layer, a distributed real-time communication and message layer, a business logic layer, and an application display and interaction layer. The multi-modal data acquisition and preprocessing layer is composed of a multi-modal sensor interface, a data preprocessing module, a data cache module, a data quality evaluation module, and a metadata management module.
[0006] Preferably, the feature extraction and encoding layer includes visual feature extraction, auditory feature extraction, textual feature extraction, feature encoding module, feature dimension reduction module, and modality-specific detection module.
[0007] Preferably, the feature extraction and encoding layer uses a deep learning model.
[0008] Preferably, the cross-modal feature fusion and alignment layer includes a multi-modal feature alignment module, a fusion algorithm module, a weight learning module, a fusion evaluation module, and a cross-modal attention mechanism.
[0009] Preferably, the cross-modal feature fusion alignment layer adopts a weighted feature fusion formula, and the weighted feature fusion formula is as follows:
[0010] F = a · V + b · A + g · T;
[0011] In the formula, F is a fused feature vector, representing a unified representation of multi-modal data, V is a visual feature vector, a feature extracted from visual data, A is an auditory feature vector (a multi-dimensional vector), a feature extracted from auditory data, T is a text feature vector, a feature extracted from text data, a is a dynamic weight coefficient of a visual mode, calculated in real time according to a signal-to-noise ratio, quality or importance of the visual data, and ranges from 0 to 1, b is a dynamic weight coefficient of an auditory mode, calculated in real time according to a signal-to-noise ratio, quality or importance of the auditory data, and ranges from 0 to 1, and g is a dynamic weight coefficient of a text mode, calculated in real time according to a signal-to-noise ratio, quality or importance of the text data, and ranges from 0 to 1.
[0012] Preferably, the collaborative processing layer is composed of an intention recognition, context awareness, decision making, collaborative filtering engine, decision optimization module and context memory network.
[0013] Preferably, the collaborative processing layer adopts a multi-modal decision algorithm, and the algorithm is as follows:
[0014]
[0015] In the formula, R uv is a collaborative rating of a user u to an item v, representing a predicted interest or preference degree of the user to the item, u is a user identifier, representing a specific user in the system, v is an item identifier, representing a specific item in the system, n is a total number of feature dimensions, representing a length of a user and item feature vector, i is a summation index, from 1 to n, traversing each feature dimension, P ui is a value of the user on the i-th feature, which is a component of the user feature vector, representing an attribute of the user on the feature, Q vi is a value of the item on the i-th feature, which is a component of the item feature vector, representing an attribute of the item on the feature.
[0016] Preferably, the distributed real-time communication and message layer includes a WebSocket service, a message queue, a connection management module, a message routing module and a load balancing module.
[0017] Preferably, the business logic layer includes a collaborative task management, a user state synchronization, a distributed lock manager, a state persistence module and a conflict resolution module.
[0018] Preferably, the application display and interaction layer includes a front-end interface, data visualization, an interaction log module, a real-time feedback module, and a multi-device adaptation module.
[0019] Compared with the prior art, the present application has the following advantages:
[0020] The present application realizes efficient collaborative processing of multi-modal data through a longitudinal layered architecture and a horizontal collaborative mechanism. The acquisition and preprocessing layer guarantees standardized input of multi-source heterogeneous data. The feature extraction and encoding layer extracts and unifies the feature format through a deep learning model. The cross-modal fusion and alignment layer optimizes the spatio-temporal alignment and semantic consistency using dynamic weights and attention mechanisms. The collaborative processing layer improves the intent recognition accuracy by combining context memory and decision optimization. The distributed communication layer supports low-latency interaction for tens of thousands of concurrent users. The business logic layer guarantees data consistency through conflict resolution and state persistence. Finally, the application display and interaction layer realizes real-time feedback and visualization through multi-device adaptation, fully solving the bottleneck problems of low fusion efficiency, high latency, and insufficient decision accuracy in the prior art. DETAILED DESCRIPTION
[0021] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the embodiments in the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.
[0022] A multi-modal collaborative channel processing system, which comprises a multi-modal data acquisition and preprocessing layer, a feature extraction and encoding layer, a cross-modal feature fusion and alignment layer, a collaborative processing layer, a distributed real-time communication and message layer, a business logic layer, and an application display and interaction layer. The multi-modal data acquisition and preprocessing layer is composed of a multi-modal sensor interface, a data preprocessing module, a data cache module, a data quality evaluation module, and a metadata management module. The multi-modal sensor interface sends raw data to the data preprocessing module for cleaning, and then the data cache module temporarily stores the data. After the data quality evaluation module checks the data quality, the metadata management module adds descriptive information. The entire unit acquires multi-source data such as vision, hearing, and text, performs denoising and normalization preprocessing, and provides standardized input through caching, quality evaluation, and metadata management.
[0023] The feature extraction coding layer includes visual feature extraction (CNN), auditory feature extraction (RNN), text feature extraction (Transformer), feature coding module, feature dimension reduction module and modality-specific detection module, which extracts high-level features of each modality using deep learning model, each feature extraction module (CNN, RNN, Transformer) outputs features to the feature coding module for standardization, the feature dimension reduction module reduces the dimension, and the modality-specific detection module identifies key features and performs coding, dimension reduction and specificity detection, and outputs a unified format of feature vectors.
[0024] The cross-modal feature fusion alignment layer includes a multi-modal feature alignment module, a fusion algorithm module, a weight learning module, a fusion evaluation module and a cross-modal attention mechanism. The multi-modal feature alignment module aligns the features in space and time, uses a weighted feature fusion formula, the fusion algorithm module performs fusion, the weight learning module dynamically adjusts the weight, the fusion evaluation module verifies the fusion effect, and the cross-modal attention mechanism enhances important features. Through feature alignment and fusion algorithm, cross-modal fusion is realized, combined with weight learning, evaluation and attention mechanism to dynamically optimize the fusion process. The weighted feature fusion formula is as follows:
[0025] F = a V + b A + g T;
[0026] In the formula: F is the fused feature vector, representing the unified representation of multi-modal data, V is the visual feature vector, the feature extracted from visual data, A is the auditory feature vector (multi-dimensional vector), the feature extracted from auditory data, T is the text feature vector, the feature extracted from text data, a is the dynamic weight coefficient of the visual modality, calculated in real time according to the signal-to-noise ratio, quality or importance of the visual data, ranging from [0, 1], b is the dynamic weight coefficient of the auditory modality, calculated in real time according to the signal-to-noise ratio, quality or importance of the auditory data, ranging from [0, 1], g is the dynamic weight coefficient of the text modality, calculated in real time according to the signal-to-noise ratio, quality or importance of the text data, ranging from [0, 1]. Note: The weight coefficients are adaptively adjusted by the weight learning module to ensure that a + b + g = 1 or are normalized according to the scene.
[0027] The collaborative processing layer is composed of intent recognition, context awareness, decision making, collaborative filtering engine, decision optimization module and context memory network. The intent recognition and context awareness modules output to the decision making module, the collaborative filtering engine provides recommendation calculation, which is a multi-modal decision algorithm, the decision optimization module adjusts the strategy, and the context memory network stores historical information. Based on the fused features, the user's intention is analyzed, the collaborative strategy is generated combined with the historical context and collaborative filtering, and the decision accuracy is improved through the optimization and memory network. The multi-modal decision algorithm is as follows:
[0028]
[0029] In the formula: R uv is the collaborative rating of user u to item v, representing the predicted interest or preference degree of the user to the item, u is the user identifier, representing a specific user in the system, v is the item identifier, representing a specific item in the system, n is the total number of feature dimensions, representing the length of the user and item feature vectors, i is the summation index, from 1 to n, traversing each feature dimension, P ui is the value of the user on the i-th feature, which is a component of the user feature vector, representing the attribute of the user on the feature, Q vi is the value of the item on the i-th feature, which is a component of the item feature vector, representing the attribute of the item on the feature, the formula is based on the idea of matrix decomposition, calculating the collaborative rating through user-item feature inner product, supporting million-level feature dimension calculation, used for recommendation system or collaborative decision-making.
[0030] The distributed real-time communication and message layer includes WebSocket service, message queue (NATS), connection management module, message routing module and load balancing module, the WebSocket service handles client connections, the message queue (NATS) manages message flow, the connection management module maintains session state, the message routing module directs messages, and the load balancing module distributes traffic, based on Go goroutines to achieve ten thousand concurrent connections, through connection management, message routing and load balancing to ensure low latency communication, making message delay controlled within 50ms.
[0031] The business logic layer includes collaborative task management, user state synchronization, distributed lock manager, state persistence module and conflict resolution module, the collaborative task management module assigns tasks to the user state synchronization module, the distributed lock manager handles concurrent access, the state persistence module saves data, and the conflict resolution module handles inconsistencies, using distributed locks and state persistence to handle task conflicts and state synchronization, through the conflict resolution mechanism to ensure data consistency.
[0032] The application display and interaction layer includes front-end interface, data visualization, interaction log module, real-time feedback module and multi-device adaptation module, the front-end interface receives user input and calls the data visualization module, the interaction log module records operations, the real-time feedback module provides immediate response, and the multi-device adaptation module adjusts the interface layout, used to provide Web / mobile interactive interface, supporting real-time data visualization, operation playback, log recording, feedback and multi-device adaptation.
[0033] Longitudinal flow: Data flow starts from the multi-modal data acquisition and pre-processing layer, goes through the feature extraction and encoding layer, the cross-modal feature fusion and alignment layer, the collaborative processing layer, the distributed real-time communication and message layer, the business logic layer, and the application display and interaction layer in sequence, and finally reaches the multi-modal application display and interaction layer. This sequence ensures the pipelining and efficiency of data processing.
[0034] Lateral collaboration: The distributed real-time communication and message layer communicates with all other layers bidirectionally, especially interacts closely with the intelligent collaborative decision and context processing layer and the collaborative business logic and state management layer, supports real-time message passing and state synchronization, in addition, cross-layer modules such as the weight learning module (fusion layer) and the context memory network (decision layer) have direct data exchange.
[0035] Multi-modal data acquisition and pre-processing layer → feature extraction and encoding layer → cross-modal feature fusion and alignment layer → collaborative processing layer → business logic layer → application display and interaction layer. The distributed real-time communication and message layer runs throughout, providing real-time communication support for each layer.
[0036] It should be noted that the relational terms herein such as first and second and the like are used solely to distinguish one entity or action from another, without necessarily requiring or implying any such actual relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element preceded by "comprises... a" does not, without more constraints, foreclose the existence of additional identical elements in the process, method, article, or apparatus that comprises the recited element.
[0037] Although embodiments of the present application have been shown and described, it is to be understood that various modifications, substitutions, replacements and variations can be made to these embodiments without departing from the principles and spirit of the present application, the scope of which is defined by the appended claims and their equivalents.
Claims
1. A multi-modal coordinated channel processing system, characterized by: The system comprises a multi-modal data acquisition and preprocessing layer, a feature extraction and coding layer, a cross-modal feature fusion and alignment layer, a collaborative processing layer, a distributed real-time communication and message layer, a business logic layer, and an application display and interaction layer.
2. The multi-modal coordinated channel processing system of claim 1, wherein: The feature extraction and coding layer comprises visual feature extraction, auditory feature extraction, text feature extraction, feature coding modules, feature dimension reduction modules, and modality-specific detection modules.
3. The multi-modal coordinated channel processing system of claim 1, wherein: The feature extraction and coding layer adopts a deep learning model.
4. The multi-modal coordinated channel processing system of claim 1, wherein: The cross-modal feature fusion and alignment layer comprises multi-modal feature alignment, fusion algorithm modules, weight learning modules, fusion evaluation modules, and cross-modal attention mechanisms.
5. The multi-modal coordinated channel processing system of claim 1, wherein: The cross-modal feature fusion and alignment layer adopts a weighted feature fusion formula, which is as follows: F=α·V+β·A+γ·T; In the formula, F is the fused feature vector, representing the unified representation of multi-modal data, V is the visual feature vector, extracted from visual data, A is the auditory feature vector (multi-dimensional vector), extracted from auditory data, T is the text feature vector, extracted from text data, α is the dynamic weight coefficient of the visual modality, calculated in real time according to the signal-to-noise ratio, quality or importance of the visual data, ranging from 0 to 1, β is the dynamic weight coefficient of the auditory modality, calculated in real time according to the signal-to-noise ratio, quality or importance of the auditory data, ranging from 0 to 1, and γ is the dynamic weight coefficient of the text modality, calculated in real time according to the signal-to-noise ratio, quality or importance of the text data, ranging from 0 to 1.
6. The multi-modal coordinated lane processing system of claim 1, wherein: The collaborative processing layer comprises intent recognition, context awareness, decision making, collaborative filtering engines, decision optimization modules, and context memory networks.
7. The multi-modal coordinated channel processing system of claim 1, wherein: The collaborative processing layer adopts a multi-modal decision algorithm, which is as follows: In the formula: R uv is the collaborative rating of the user u to the item v, representing the predicted interest or preference degree of the user to the item, u is the user identifier, representing a specific user in the system, v is the item identifier, representing a specific item in the system, n is the total number of feature dimensions, representing the length of the user and item feature vectors, i is the summation index, from 1 to n, traversing each feature dimension, P ui is the value of the user on the i-th feature, which is a component of the user feature vector, representing the attribute of the user on the feature, Q vi is the value of the item on the i-th feature, which is a component of the item feature vector, representing the attribute of the item on the feature.
8. The multi-modal coordinated channel processing system of claim 1, wherein: The distributed real-time communication and message layer comprises WebSocket services, message queues, connection management modules, message routing modules, and load balancing modules.
9. The multi-modal coordinated channel processing system of claim 1, wherein: The business logic layer comprises collaborative task management, user state synchronization, distributed lock managers, state persistence modules, and conflict resolution modules.
10. The multi-modal coordinated channel processing system of claim 1, wherein: The application display and interaction layer comprises front-end interfaces, data visualization, interaction log modules, real-time feedback modules, and multi-device adaptation modules.