Multimodal data processing methods, devices, electronic equipment and storage media

Abstract

This application relates to the field of data processing technology and discloses a multimodal data processing method, apparatus, electronic device, and storage medium. The method includes: acquiring multimodal input data, including first modality data and second modality data; extracting features from the first modality data and the second modality data respectively to obtain a first feature vector and a second feature vector; calculating a sensitivity score based on the first feature vector and the second feature vector to characterize the computational complexity requirements of the multimodal input data; determining a target sub-network from a network architecture library based on the sensitivity score; and performing inference and prediction on the multimodal input data through the target sub-network to obtain the multimodal data processing result. This method can be applied to business scenarios in smart healthcare and fintech. This application can dynamically allocate computing resources according to the characteristics of the input data, breaking the limitations of traditional static model architectures, effectively avoiding wasted computing power and reducing overall processing latency.

Description

Technical Field

[0001] This application relates to the field of data processing technology and can be applied to business scenarios in smart healthcare and fintech. In particular, it relates to a multimodal data processing method, device, electronic device, and storage medium. Background Technology

[0002] In edge deployment applications of multimodal large models, such as collaborative analysis of medical images and electronic medical records in the field of smart healthcare, or processing of on-site images and investigation texts for auto insurance claims in the field of finance and insurance, the most commonly used technology is to adopt a predefined static multimodal model architecture, or simply use static model compression technology to reduce the number of parameters to adapt to the computing power limitations of edge devices.

[0003] The inventors realized that the above-mentioned solution uses a fixed computation path and cannot dynamically adjust the allocation of computing resources according to the characteristics of the input data. This forces simple inputs and complex inputs to use the same amount of computation. This not only causes serious waste of computing power and power consumption on edge devices, but also makes the overall processing latency of the system too high, unable to meet the business scenarios with stringent real-time requirements. Summary of the Invention

[0004] This invention provides a multimodal data processing method, apparatus, electronic device, and storage medium to solve the technical problem that existing multimodal models, due to their static architecture, cannot dynamically allocate computing resources according to input features, resulting in serious waste of computing power and excessively high processing latency.

[0005] To achieve the above objectives, a first aspect of this application proposes a multimodal data processing method, the method comprising: Acquire multimodal input data, which includes first modal data and second modal data; Feature extraction is performed on the first modal data and the second modal data respectively to obtain a first feature vector and a second feature vector; Based on the first feature vector and the second feature vector, a sensitivity score is calculated to characterize the computational complexity requirements of the multimodal input data; Based on the sensitivity score, the corresponding target sub-network is determined from a preset network architecture library; The multimodal input data is inferred and predicted through the target sub-network to obtain the multimodal data processing result.

[0006] To achieve the above objectives, a second aspect of this application provides a multimodal data processing apparatus, the apparatus comprising: An acquisition module is used to acquire multimodal input data, wherein the multimodal input data includes at least first modal data and second modal data; The feature extraction module is used to extract features from the first modal data and the second modal data respectively to obtain a first feature vector and a second feature vector; The calculation module is used to calculate a sensitivity score based on the first feature vector and the second feature vector to characterize the computational complexity requirements of the multimodal input data. The determination module is used to determine the corresponding target sub-network from a preset network architecture library based on the sensitivity score; The inference and prediction module is used to perform inference and prediction on the multimodal input data through the target sub-network to obtain the multimodal data processing result.

[0007] To achieve the above objectives, a third aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect.

[0008] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect.

[0009] In the aforementioned multimodal data processing method, apparatus, electronic device, and storage medium, the following steps are taken: First, multimodal input data, including first modal data and second modal data, is acquired; features are extracted from the first and second modal data respectively to obtain a first feature vector and a second feature vector; based on the first and second feature vectors, a sensitivity score is calculated to characterize the computational complexity requirements of the multimodal input data; according to the sensitivity score, a corresponding target sub-network is determined from a preset network architecture library; finally, inference and prediction are performed on the multimodal input data through the target sub-network to obtain the multimodal data processing result. In this application, for multimodal data processing needs in business scenarios such as smart healthcare and fintech, a scheme based on dynamic computing power scheduling based on input features can be used. First, features are extracted and fused from different modal data to obtain a sensitivity score that accurately characterizes the computational complexity requirements of the current input data. Then, based on this score, the most suitable target sub-network is dynamically matched and determined from a preset network architecture library to perform inference. This solution breaks the traditional fixed computing path limitation, effectively distinguishes between simple and complex inputs, and then adaptively allocates computing resources on demand. It can solve the technical problem that existing multimodal models cannot dynamically allocate computing resources according to input characteristics due to their static architecture, which leads to serious waste of computing power on edge devices and excessively high overall processing latency. This solution can meet the needs of various business scenarios with stringent real-time requirements. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a schematic diagram of an application environment for a multimodal data processing method according to an embodiment of the present invention; Figure 2 This is a flowchart of the multimodal data processing method provided in the embodiments of this application; Figure 3 This is another flowchart of the multimodal data processing method provided in the embodiments of this application; Figure 4 This is another flowchart of the multimodal data processing method provided in the embodiments of this application; Figure 5 This is another flowchart of the multimodal data processing method provided in the embodiments of this application; Figure 6 This is another flowchart of the multimodal data processing method provided in the embodiments of this application; Figure 7 This is another flowchart of the multimodal data processing method provided in the embodiments of this application; Figure 8 This is another flowchart of the multimodal data processing method provided in the embodiments of this application; Figure 9 This is a schematic diagram of the structure of the multimodal data processing device provided in the embodiments of this application; Figure 10 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention; Figure 11 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0013] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0014] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0015] In edge deployment applications of multimodal large models, such as collaborative analysis of medical images and electronic medical records in the field of smart healthcare, or processing of on-site images and investigation texts for auto insurance claims in the field of finance and insurance, the most commonly used technology is to adopt a predefined static multimodal model architecture, or simply use static model compression technology to reduce the number of parameters to adapt to the computing power limitations of edge devices.

[0016] The inventors realized that the above-mentioned solution uses a fixed computation path and cannot dynamically adjust the allocation of computing resources according to the characteristics of the input data. This forces simple inputs and complex inputs to use the same amount of computation. This not only causes serious waste of computing power and power consumption on edge devices, but also makes the overall processing latency of the system too high, unable to meet the business scenarios with stringent real-time requirements.

[0017] Based on this, embodiments of this application provide a multimodal data processing method, apparatus, electronic device, and storage medium, aiming to solve the technical problem that existing multimodal models, due to their static architecture, cannot dynamically allocate computing resources according to input features, resulting in serious waste of computing power and excessively high processing latency.

[0018] The multimodal data processing method, apparatus, electronic device, and storage medium provided in this application are specifically described through the following embodiments. First, the multimodal data processing method in this application is described.

[0019] The multimodal data processing method provided in this application can be applied to, for example, Figure 1In this application environment, the client communicates with the server via a network. The server can obtain multimodal input data from the client, which includes at least first modality data and second modality data. Features are extracted from the first and second modality data to obtain a first feature vector and a second feature vector, respectively. Based on the first and second feature vectors, a sensitivity score is calculated to characterize the computational complexity requirements of the multimodal input data. According to the sensitivity score, a corresponding target sub-network is determined from a pre-defined network architecture library. The multimodal input data is then inferred and predicted using the target sub-network to obtain the multimodal data processing result, which is then fed back to the client. In this application, for multimodal data processing needs in business scenarios such as smart healthcare and fintech, a scheme based on dynamic computing power scheduling based on input features can be used. First, features are extracted and fused from different modal data to obtain a sensitivity score that accurately characterizes the computational complexity requirements of the current input data. Then, based on this score, the most suitable target sub-network is dynamically matched and determined from a pre-defined network architecture library to perform inference. This solution breaks the limitations of traditional fixed computing paths, enabling the system to effectively distinguish between simple and complex inputs and adaptively allocate computing resources on demand. It addresses the technical problem of existing multimodal models, which, due to their static architecture, cannot dynamically allocate computing resources based on input characteristics, leading to significant wasted computing power on edge devices and excessively high overall processing latency. This allows the solution to meet the demands of various business scenarios with stringent real-time requirements. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, portable wearable devices, and portable medical terminals. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The following detailed description of specific embodiments further illustrates this application.

[0020] Figure 2 This is an optional flowchart of the multimodal data processing method provided in the embodiments of this application. Figure 2 The method may include, but is not limited to, steps S201 to S205.

[0021] Step S201: Obtain multimodal input data, which includes first modal data and second modal data.

[0022] Step S202: Perform feature extraction on the first modality data and the second modality data respectively to obtain the first feature vector and the second feature vector.

[0023] Step S203: Based on the first feature vector and the second feature vector, calculate the sensitivity score used to characterize the computational complexity requirements of multimodal input data.

[0024] Step S204: Based on the sensitivity score, determine the corresponding target sub-network from the preset network architecture library.

[0025] Step S205: Perform inference and prediction on the multimodal input data through the target sub-network to obtain the multimodal data processing result.

[0026] To address the technical challenges of wasted computing power and excessive processing latency caused by deploying existing static multimodal model architectures at the edge, this application proposes a solution that dynamically adjusts the allocation of computing resources based on the specific characteristics of multimodal input data. This solution adaptively matches and calls the most suitable target sub-network from a pre-defined network architecture library for inference and prediction, thereby achieving optimal allocation of computing resources while ensuring processing accuracy. The details are described below.

[0027] In step S201 of some embodiments, multimodal input data refers to a data set collected by different types of data sources or sensors that can collectively describe the same physical object or business scenario from multiple dimensions. In practical edge computing scenarios (such as smart medical terminals or financial claims mobile devices), the first modality data can be visual data, and the second modality data can be textual data. These heterogeneous data are received through front-end data acquisition interfaces, sensors, or network communication modules and used as the basic input carrier for subsequent dynamic architecture search and inference analysis. By introducing data from at least two modalities, richer contextual information can be obtained than from a single data source, thereby providing multi-dimensional objective data for subsequent accurate evaluation.

[0028] In step S202 of some embodiments, feature extraction refers to the process of extracting key information that can characterize the intrinsic attributes of the original unstructured input data using a preset deep learning model, and transforming it into a vector representation that can be efficiently processed by a computer. Since the first modality data and the second modality data differ significantly in data structure and expression, independent processing channels are used for feature extraction on each. The core purpose of this step is not to directly output the final business result, but to transform the original heterogeneous multimodal data into a unified structured feature vector form, namely the first feature vector and the second feature vector, so that the system can quantify and extract the basic data based on dimensions such as the complexity, density, or sequence features of the data itself, to determine the computational scale required for subsequent inference.

[0029] Please see Figure 3 In some embodiments, the first modal data is visual image data and the second modal data is text data. Step S202 may include, but is not limited to, steps S301 to S302.

[0030] Step S301: The visual image data is processed by a pre-trained convolutional neural network to extract multidimensional statistical features used to characterize the computational complexity of the image as the first feature vector.

[0031] Step S302: Encode the text data with word vectors based on the pre-trained micro-language model, and calculate the word vector entropy value and sequence length coefficient used to characterize the semantic complexity of the text as the second feature vector.

[0032] To quantify the inherent complexity differences of heterogeneous multimodal input data and thus address the technical challenge of unbalanced computational power allocation caused by the lack of feature perception in traditional multimodal models, this application proposes a solution that constructs independent feature extraction channels for visual and textual modalities. This allows for the extraction of key feature vectors representing computational and semantic complexity from the underlying attributes of the data, providing a reliable data foundation for dynamic adaptive scheduling of computational power, as described below.

[0033] In step S301 of some embodiments, the pre-trained convolutional neural network is a lightweight network structure pre-trained on a large number of general image datasets and possessing basic image feature extraction capabilities, such as a three-layer lightweight convolutional neural network. When visual image data as the first modality data is received, the pre-trained convolutional neural network does not parse the deep, specific business semantics of the image, such as identifying specific lesions or vehicle parts. Instead, it quantifies the image computational complexity in terms of the underlying pixel distribution and physical structure. This complexity determines the computational overhead required for subsequent deep feature alignment and inference. To this end, a set of multidimensional statistical features is extracted. These features together constitute the first feature vector, which can be mathematically represented as a multidimensional feature vector. This feature vector contains information in three key dimensions: image resolution reflects the absolute pixel space size of the input image; a larger size means a greater amount of matrix multiplication and addition operations are required to process the receptive field. Edge density characterizes the richness of high-frequency gradient information in the image; images with denser edges usually contain more complex texture details, requiring activation of deeper network layers to extract effective features to avoid information loss. Color entropy measures the disorder and information entropy of the image's color distribution; images with richer colors and stronger contrast correspond to higher entropy values. By extracting statistical features across these multiple dimensions, the spatial and pixel processing difficulty of visual image data can be comprehensively and objectively transformed into a vector form that can be directly computed and quantified by a computer.

[0034] In step S302 of some embodiments, the pre-trained miniature language model can be a miniaturized BERT-like language model, a lightweight natural language processing architecture that has undergone parameter pruning and model distillation, possessing basic word meaning mapping and encoding capabilities with extremely low memory overhead. Upon receiving text data as second-modality data, the miniature language model is first used to segment the input character sequence, mapping each independent word to a word vector in a high-dimensional continuous space. Similar to visual images, which emphasize physical structural properties, the processing difficulty of text data mainly depends on its semantic complexity. To quantify this semantic complexity, two core quantification parameters need to be calculated: First, the word vector entropy value. This parameter is obtained by calculating the dispersion and information content of the word vector distribution corresponding to the entire input text sequence. A higher word vector entropy value indicates that the semantic concepts involved in the text content are more obscure, the vocabulary is denser, or the internal logical relationships are more complex, inevitably requiring a stronger text encoding branch for feature mining. Second, the sequence length coefficient. This coefficient represents the overall contextual span of the input text. Since the time and space complexity of long texts increases significantly with length when performing self-attention and other mechanisms, the sequence length coefficient directly reflects the memory consumption limit of sequence processing. The obtained word vector entropy value and sequence length coefficient are extracted and combined to construct a second feature vector specifically representing the difficulty of text modality processing, thus completing the quantification of the complexity of text data processing.

[0035] Through steps S301 and S302 described above, this embodiment of the application employs a dual-channel extraction mechanism that quantifies the underlying computational complexity of visual and textual modalities, addressing their physical and semantic heterogeneity. By acquiring multidimensional statistical features including image resolution, edge density, and color entropy, as well as textual features including word vector entropy values ​​and sequence length coefficients, the original multimodal input data is transformed into feature vectors that objectively represent computational complexity requirements. This design provides a structured data foundation for subsequent system computational sensitivity scoring, enabling edge computing devices to dynamically allocate corresponding computing resources based on the actual feature complexity of the input data. This effectively reduces ineffective computational power consumption and lowers overall inference latency during multimodal data processing.

[0036] In step S203 of some embodiments, the sensitivity score is used to objectively measure how much computational resources are required when performing inference and prediction on the current input multimodal data. After receiving the first feature vector and the second feature vector, a preset fusion mechanism is used to comprehensively analyze and map the feature dimensions of these two different modalities to assess whether the current input scenario belongs to a simple scenario that requires lightweight processing or a complex scenario that requires high-precision model processing. Through this cross-modal feature fusion and computation, a standardized sensitivity score is output, which directly reflects the computational power requirements of the current multimodal input data on the network depth and width.

[0037] Please see Figure 4 In some embodiments, step S203 may include, but is not limited to, steps S401 to S404.

[0038] Step S401: Normalize the multidimensional statistical features to obtain standard visual features.

[0039] Step S402 involves performing reverse mapping and linear normalization on the word vector entropy value and sequence length coefficient to obtain standard text features.

[0040] Step S403: Obtain the current computing resource utilization rate of the execution device, and determine the dynamic adjustment factor based on the computing resource utilization rate.

[0041] Step S404: Based on the dynamic adjustment factor and the preset modal weights, the standard visual features and standard text features are weighted and fused to generate a sensitivity score.

[0042] To quantify and integrate the complexity features of heterogeneous modalities such as visual and textual data, and to achieve unified measurement of cross-modal complexity and adaptive adaptation to device operating states, thereby solving the technical challenges of single-modal-dominated misjudgments and unquantifiable scoring in traditional multimodal systems, this application proposes a solution that introduces a dual-channel feature standardization mechanism and a device resource awareness mechanism. This solution uses a preset dynamic weight calculation formula to weight and fuse features from various modalities to generate a sensitivity score that can drive model architecture switching, as described below.

[0043] In step S401 of some embodiments, because the multidimensional statistical features extracted from the visual channel (which differ significantly in numerical range and physical dimensions) are directly used in subsequent weighted calculations, feature branches with larger absolute values ​​will dominate the calculation, thus masking the role of other effective features. Therefore, it is necessary to normalize the visual feature vector containing multidimensional features. Specifically, for any _th_ feature vector, 3D features The formula for processing is as follows:

[0044] in and These represent the minimum and maximum values ​​of the feature in the historical sample data, respectively. Through the above processing, the dimensional differences between the features are eliminated, and they are uniformly mapped to a preset dimensionless numerical range. This yields standard visual features that can be compared proportionally, providing benchmark data for the effective alignment and fusion of cross-modal features.

[0045] In step S402 of some embodiments, during the evaluation of text modalities, the increasing logic of the original word vector entropy value and the sequence length coefficient in a physically meaningful sense needs to be consistent with the direction of computational sensitivity. To assign higher sensitivity weights to high-entropy text and long sequence inputs, a reverse mapping operation is performed on the word vector entropy value, i.e., through the formula... The entropy features obtained after inverse computation are obtained; simultaneously, a linear normalization operation is performed on the sequence length coefficient to smoothly map it to the normalization interval. Through the above inverse mapping and linear stretching calculation, the original text-related attribute parameters are transformed into standard text features with the same numerical evaluation scale as standard visual features, ensuring that the computational complexity of text semantics can be quantitatively expressed in a structured and standardized manner.

[0046] In step S403 of some embodiments, the dynamic adjustment factor This is an adaptive weighting parameter whose core purpose is to dynamically intervene and adjust the system's sensitivity threshold to data complexity based on the real-time load pressure of edge computing devices. In practice, the computing resource utilization of the current execution device (such as the central processing unit of a portable medical terminal or mobile claims device) is collected in real time, with CPU utilization being a typical monitoring indicator. Based on a preset hardware resource allocation strategy, if the current CPU utilization is detected to be greater than a preset high-load threshold, such as 75%, a smaller dynamic adjustment factor (such as 0.8) is determined to proactively reduce the system's tendency to respond to high-sensitivity assessments, preventing the device from crashing due to forced invocation of high-precision, heavily loaded networks. Conversely, if the CPU utilization is less than a preset idle threshold (such as 50%), a larger dynamic adjustment factor (such as 90%) is determined, allowing for high sensitivity to data complexity when computing power is sufficient, enabling rapid initiation of heavy-duty networks for high-precision processing. This mechanism enhances the overall data processing system's hardware adaptive protection capabilities in complex environments.

[0047] In step S404 of some embodiments, standard visual features are acquired synchronously. Standard text features and and the dynamic adjustment factor determined at the current moment. Then, the final sensitivity score s is generated using a preset fusion formula. Specifically, the mathematical model for this fusion calculation can be expressed as:

[0048] In the above calculation process, constants 0.3 and 0.7 constitute the preset modal weights, used to define the relative influence ratio of visual feature information and text feature information on the overall task computational complexity; while 0.6 and 0.4 are the combined weights of text internal features, used to reflect the relative contribution of entropy value and length coefficient. Furthermore, to ensure the numerical stability of the system architecture switching decision, after completing the weighted fusion, a boundary mapping operation is performed on the calculation results through a clamping function, i.e. The final sensitivity score is strictly constrained to the range [0,1]. This sensitivity score, as the core quantitative indicator for allocating computing power, will be output to subsequent processing modules to drive the network architecture library to switch the target sub-network.

[0049] Through steps S401 to S404, this embodiment of the application normalizes and reverse-maps multidimensional statistical features, word vector entropy values, and sequence length coefficients, unifying the dimensions of multimodal heterogeneous data and achieving numerical fusion of visual and textual features. Simultaneously, this scheme incorporates a dynamic adjustment factor based on the current device's computing resource utilization in the scoring calculation, enabling the generated sensitivity score to comprehensively reflect the processing complexity of the input data and the real-time load status of the device. This mechanism reduces scoring bias caused by excessively large absolute values ​​of single-modal features, allowing for a balance between data processing needs and the carrying capacity of the underlying devices when selecting computational paths, thereby improving the rationality of system computing power allocation.

[0050] In step S204 of some embodiments, the network architecture library is a predefined and maintained collection of backbone network models with different computational complexities, parameter scales, and processing capabilities. The target sub-network refers to a specific computational path or model branch selected from the architecture library that matches the computational complexity requirement represented by the sensitivity score. Specifically, the obtained sensitivity score is compared and verified with preset matching rules or thresholds. When the sensitivity score indicates high complexity of the current input data, a complex network with high parameter count and strong feature extraction capability is determined from the network architecture library as the target sub-network; conversely, when the sensitivity score indicates low data complexity, a lightweight network with fewer parameters and low power consumption is determined as the target sub-network. This step endows the system with dynamic architecture search and adaptive routing capabilities, ensuring the matching of computing power supply with actual data requirements.

[0051] Please see Figure 5In some embodiments, the network architecture library contains multiple candidate subnetworks with different computational complexities, and step S204 may include, but is not limited to, steps S501 to S503.

[0052] Step S501: If the sensitivity score is greater than the first preset threshold, then select the first candidate subnetwork as the target subnetwork.

[0053] Step S502: If the sensitivity score is less than or equal to the first preset threshold and greater than the second preset threshold, then the second candidate subnetwork is selected as the target subnetwork.

[0054] In step S503, if the sensitivity score is less than or equal to the second preset threshold, then the third candidate subnetwork is selected as the target subnetwork.

[0055] To address the technical challenge of uneven computational power distribution in static models under dynamic loads, which can easily lead to system overload or resource idleness, this application's embodiments pre-define multiple candidate sub-networks with gradient computational complexity in a network architecture library. Multi-level threshold judgments can be used to perform interval mapping on sensitivity scores, adaptively determining the optimal computational path for the current scenario, as described below.

[0056] In step S501 of some embodiments, the first preset threshold is a set high-level limit value for identifying highly complex input scenes (e.g., it can be set to 0.7). It is determined whether the received sensitivity score s is greater than the first preset threshold. When s > 0.7, it indicates that the current visual image contains extremely high resolution or complex texture details, and / or the text data contains high-density semantic logic. To ensure that the accuracy of feature alignment and processing does not decrease, it is necessary to call a deep network with the strongest feature extraction and nonlinear mapping capabilities. Therefore, a first candidate sub-network (e.g., a high-precision network branch constructed from the deep residual module of ResNet) is retrieved from a preset network architecture library as the target sub-network. This first candidate sub-network has a large number of network layers and parameters, providing sufficient model capacity and computational power support for highly complex multimodal inference.

[0057] In step S502 of some embodiments, the second preset threshold is a low-order boundary value used to distinguish between normal scenes and minimalist scenes, for example, it can be set to 0.3. The current multimodal input data is of medium complexity, meaning it is neither overly complex nor extremely simple. Under this typical load, continuing to call the first candidate sub-network would result in unnecessary computational waste, while calling a network with too few parameters might lead to insufficient accuracy. Therefore, a second candidate sub-network (e.g., a backbone network built on the MobileNet architecture) is selected as the target sub-network. The second candidate sub-network has been moderately reduced in network depth and the number of feature channels, aiming to achieve a balance between performance accuracy and device inference power consumption to meet the data processing needs of most standard business scenarios.

[0058] In step S503 of some embodiments, when When the received image has simple background features and very short text semantics, it indicates a low-complexity data processing task. For such minimal inputs, the computation path is routed to a third candidate sub-network (e.g., a network architecture composed of basic lightweight modules like ShuffleNet). This third candidate sub-network has a minimal number of parameters in the architecture library, resulting in minimal computational and energy consumption. Through this interval-based degradation call strategy, the computational complexity of the first, second, and third candidate sub-networks is ensured to decrease sequentially. This tiered architecture library design allows the system to accurately locate the network branch with the best parameter range based on the specific sensitivity score.

[0059] In step S205 of some embodiments, inference prediction refers to the process of using a determined neural network structure to perform business logic calculations, classifications, recognitions, or generation operations on the current input data. After determining a specific target sub-network, the multimodal input data is propagated forward along a specific computational path formed by the target sub-network. Since the target sub-network is the optimal computational architecture determined based on the characteristics of the input data, it can ensure that complex data obtains sufficient computational depth to achieve the expected high accuracy during inference, while allowing simple data to pass quickly through shallow networks to avoid unnecessary computational overhead. The final output of the multimodal data processing results can be presented in the corresponding business data form according to the specific application scenario, such as diagnostic classification results or loss assessment reports.

[0060] Please see Figure 6 In some embodiments, step S205 may include, but is not limited to, steps S601 to S604.

[0061] Step S601: If a switch in the target sub-network is detected relative to the network architecture of the previous moment, then obtain the number of output channels of the previous network architecture and the number of input channels of the current target sub-network.

[0062] In step S602, if the number of output channels is greater than the number of input channels, a downsampling operation is performed through the dynamic adaptation layer to compress the channels.

[0063] In step S603, if the number of output channels is less than the number of input channels, an upsampling operation is performed through the dynamic adaptation layer to expand the channels.

[0064] Step S604: Input the channel-aligned feature data into the target sub-network for inference and prediction.

[0065] In some embodiments, to address the issue of inference interruption and data loss caused by the mismatch in input and output feature dimensions between different network architectures when switching based on dynamic sensitivity scores, embodiments of this application can adaptively perform feature channel expansion or compression operations upon sensing an architecture switching event, thereby ensuring the continuity and integrity of multimodal data streams during cross-architecture inference, as described below.

[0066] In step S601 of some embodiments, during the continuous input and processing of multimodal data, the computation path is dynamically allocated based on the sensitivity score of the current input data. A cross-architecture switching event is triggered when the target sub-network selected at the current moment is inconsistent with the network architecture running at the previous moment. Since different network architectures have different feature dimension requirements in their design, directly passing the feature map of the previous network to the current network will lead to a dimension mismatch error. Therefore, two key architecture parameters are extracted first: one is the channel dimension size of the previous network architecture when outputting the feature map, and the other is the standard channel dimension size required by the currently activated target sub-network when receiving the feature map.

[0067] In step S602 of some embodiments, the Dynamic Adapter Layer (DAL) is a data dimension transformation module built internally by the system, which has trainable parameters and supports gradient backpropagation. When a comparison reveals that the number of output channels of the previous network is greater than the number of input channels of the current target sub-network, directly truncating the extra channels would lead to severe loss of feature semantic information. At this time, the system calls the downsampling operation within the Dynamic Adapter Layer to compress the channel dimension. Specifically, the downsampling operation uses convolutional kernels with preset weights to reduce the dimensionality of the input feature map in the channel dimension. While retaining core feature information, it reduces the number of channels to the number of input channels specified by the current target sub-network, thereby avoiding the accuracy loss caused by dimension truncation.

[0068] In step S603 of some embodiments, in actual dynamic network scheduling, there is often an upgrade and switch from a low-computing-power network to a high-computing-power network. For example, the previous time the MobileNetV3 structure with 128 output channels was used, while the current time it needs to be switched to a ResNet18 structure with 512 input channels. In such scenarios, the feature dimension output by the previous network cannot fill the input tensor of the current target sub-network. To this end, a dynamic adaptation layer is called to perform an upsampling operation to expand the channel dimension. Specifically, the system uses a convolutional layer structure containing trainable parameters to map the original features to a high-dimensional space through multiple steps of convolutional mapping. This channel expansion process maintains the spatial resolution of the feature map unchanged, and only performs reconstruction and dimensionality increase filling in the depth channel dimension.

[0069] In step S604 of some embodiments, after the downsampling channel compression or upsampling channel expansion processing of the dynamic adaptation layer in the above steps, the original feature data with dimensionality discontinuities caused by the architecture switch has been converted into a target format that meets the current requirements. This batch of feature data, which has undergone alignment verification in the depth dimension, is transmitted to the input port of the currently activated target sub-network. Since the data format of the input features accurately matches the expected format of the target sub-network, the target sub-network can seamlessly take over the data stream, continue to perform forward propagation matrix operations, and thus complete the inference output of the multimodal data processing results.

[0070] Please see Figure 7 In some embodiments, steps S205 may include, but are not limited to, steps S701 to S704.

[0071] Step S701: In response to the number of architecture changes reaching a preset architecture change threshold, retrieve the historical best feature snapshot that matches the current multimodal input data from the preset meta-knowledge buffer pool.

[0072] Step S702: Calculate the dynamic feature alignment error between the first modal feature and the second modal feature output by the current target sub-network.

[0073] Step S703: The current dynamic feature alignment error is weighted and fused with the extracted historical best feature snapshot to generate a cross-modal representation vector.

[0074] Step S704: Perform knowledge distillation training based on cross-modal representation vectors, and after training is completed, write the current model parameters and the corresponding feature alignment error as the updated knowledge snapshot into the meta-knowledge buffer pool.

[0075] To address the issues of cross-modal feature space shift and historical knowledge forgetting that easily occur when multimodal large models frequently and dynamically adjust their network architecture, and to achieve feature alignment and continuous evolution of the model over long-term operation, this application proposes a solution that introduces an independent offline knowledge distillation framework and a meta-knowledge buffer mechanism. This allows for dynamic alignment training of the current model using historically optimal features when specific triggering conditions are met, enabling knowledge updates and retention without affecting the real-time performance of the main inference process, as described below.

[0076] In step S701 of some embodiments, because the system frequently switches target sub-networks when processing data of different complexities, this repeated change in network depth can easily lead to spatial drift of the extracted feature representations in the cross-modal vector space. To suppress this drift, the system maintains a meta-knowledge buffer pool to persistently retain the best-performing feature knowledge records from past (e.g., the last 20) architecture changes. When the cumulative number of dynamic architecture compilation changes reaches a preset architecture change threshold (e.g., a cumulative 10 network architecture switches), the system triggers a non-real-time background maintenance mechanism. At this time, a fast nearest neighbor matching algorithm such as Locality Sensitive Hash (LSH) is used to perform similarity retrieval in the meta-knowledge buffer pool based on the feature distribution of the current multimodal input data. In this way, the historical best feature snapshot most similar to the current input scenario can be extracted. This snapshot contains the reference feature representation of the multimodal data in the historical best state, providing a historical knowledge benchmark for subsequent feature alignment and calibration.

[0077] In step S702 of some embodiments, in cross-modal processing tasks, features from different modalities need to be mapped to a common semantic feature space before joint inference can be performed. The first modal features output by the currently running target sub-network are obtained. With second modal features A dynamic feature alignment loss function is constructed to quantify the dispersion of the two features in the current common space. In one specific implementation, the formula for calculating the dynamic feature alignment error is expressed as:

[0078] In this formula, The dimension of the feature vector. This is a dynamically adjusted parameter that is automatically updated based on the variance of the feature distribution during the most recent architectural change. This is achieved by introducing an adaptive variance parameter. This error calculation mechanism can overcome the shortcomings of traditional fixed loss functions that are prone to failure under dynamic architectures, and objectively reflect the degree of alignment offset between the current first modal feature and the second modal feature.

[0079] In step S703 of some embodiments, after obtaining the dynamic feature alignment error of the current target sub-network and retrieving the matching historical best feature snapshot from the meta-knowledge buffer, preset fusion weights are assigned to the current alignment state and historical experience, respectively. The feature error vector reflecting the current network offset is then weighted and summed with the stable feature vector from the historical best feature snapshot. This fusion process uses alignment knowledge accumulated in similar input scenarios as prior guidance to correct the fluctuating feature representation caused by the architecture switch, thereby generating a cross-modal representation vector that takes into account both current input characteristics and historical stable experience. This vector contains calibration information for the current feature offset state of the model and is the core foundational data for subsequent offline feature optimization.

[0080] In step S704 of some embodiments, the generated cross-modal representation vector is used as the label signal, and the current network parameters are fine-tuned offline using a knowledge distillation algorithm. This allows the current target sub-network to absorb the feature distribution patterns of the historical optimal state, thereby reducing the distribution difference between the first and second modes in the feature space. Once the knowledge distillation training is complete and the model feature alignment performance reaches the convergence criterion, the optimized model parameters and the corresponding reduced feature alignment error are combined to generate a new knowledge record. This record is then written as an updated knowledge snapshot into the meta-knowledge buffer pool. If the buffer pool reaches its fixed capacity limit, old records are discarded according to the circular storage overwrite rule, thus achieving rolling updates of the system's underlying knowledge.

[0081] Please see Figure 8 In some embodiments, the method provided in this application also includes, but is not limited to, steps S801 to S803.

[0082] Step S801: In response to the detection of a network architecture change event and the execution device being in an idle state, obtain the parameter difference between the local network model and the cloud network model.

[0083] Step S802: Differential encoding is performed on the parameter difference to obtain the differential update parameter.

[0084] Step S803: Upload the differential update parameters that have completed differential encoding to the cloud server for global architecture synchronization update.

[0085] In step S801 of some embodiments, a network architecture change event refers to the dynamic switching of the target sub-network triggered by the system's perception of changes in input data characteristics during the initial processing of multimodal data. This could involve adding a deep network module or pruning some lightweight network layers. To ensure that the extraction and uploading of these model evolution parameters does not affect the response latency of the front-end real-time inference main process, the operating status of edge devices is monitored in real time. Only when the execution device meets preset idle state conditions, such as a CPU utilization rate continuously falling below a set threshold, and there is currently no continuous data input or user request, the parameters of the currently stably running local network model are read and compared with the parameters of the previous version to calculate the parameter difference. This parameter difference only includes incremental parameters where substantial changes have occurred in the network structure or weights have been updated, avoiding full extraction of the entire model file and providing a foundation for subsequent data compression.

[0086] In step S802 of some embodiments, firstly, the high-frequency, minute variation data in the parameter difference is losslessly statistically compressed using the Huffman coding algorithm to eliminate coding redundancy in the data representation, completing the first level of weight difference compression. Secondly, different importance weights are assigned to different functional layers in the network structure. For those key network layers that have the most significant impact on the overall feature extraction and decision-making of the model, specific priority markers are added to their parameters to ensure that these core parameters can be prioritized and sent when subsequent network transmission is limited; this is the second level of priority marking. Finally, the channel quality of the current network environment in which the edge device is located is evaluated in real time, and the quantization precision of the parameter data is adaptively changed according to the channel quality status. If the channel quality is poor, a lower quantization bit width is used to reduce the data volume; if the channel quality is good, a higher quantization bit width is used to ensure parameter precision. After the above progressive three-level coding process, the original parameter difference is transformed into a highly compressed differential update parameter.

[0087] In step S803 of some embodiments, after obtaining the highly compressed differential update parameters, they are encapsulated into a standardized differential update package and sent to the cloud through the communication channel between the edge device and the cloud server. Upon receiving the differential update package, the cloud server decodes and inverse-quantizes it to restore the parameter increments, and based on the global architecture evolution tree maintained in the cloud, merges these changed parameters into the corresponding global base model. Through this step, the cloud server can promptly grasp the latest network structure and weight distribution that each edge computing node adaptively evolves in actual business scenarios, thereby achieving cloud-edge synchronization of model state.

[0088] As can be seen, in the above solution, for the multimodal data processing needs in business scenarios such as smart healthcare and fintech, a dynamic computing power scheduling approach based on input features can be utilized. First, features are extracted and fused from different modal data to obtain a sensitivity score that accurately represents the computational complexity requirements of the current input data. Then, based on this score, the most suitable target sub-network is dynamically matched and determined from a pre-set network architecture library to perform inference. This solution breaks the limitations of traditional fixed computation paths, effectively distinguishing between simple and complex inputs, and then adaptively allocating computing resources on demand. It solves the technical problem that existing multimodal models, due to their static architecture, cannot dynamically allocate computing resources based on input features, leading to significant waste of computing power on edge devices and excessively high overall processing latency. This satisfies various business scenarios with stringent real-time requirements.

[0089] Please see Figure 9 This application also provides a multimodal data processing apparatus that can implement the above-described multimodal data processing method. The apparatus includes: The acquisition module is used to acquire multimodal input data, which includes at least first modal data and second modal data. The feature extraction module is used to extract features from the first modality data and the second modality data respectively, to obtain a first feature vector and a second feature vector; The calculation module is used to calculate a sensitivity score based on the first feature vector and the second feature vector to characterize the computational complexity requirements of multimodal input data. The determination module is used to determine the corresponding target sub-network from a preset network architecture library based on sensitivity scores; The inference and prediction module is used to perform inference and prediction on multimodal input data through the target subnetwork to obtain multimodal data processing results.

[0090] In some embodiments, the feature extraction module is specifically used for: Visual image data is processed by a pre-trained convolutional neural network, and multidimensional statistical features used to characterize the computational complexity of the image are extracted as the first feature vector; among them, the multidimensional statistical features include image resolution, edge density and color entropy; The text data is encoded using a pre-trained miniature language model, and the word vector entropy value and sequence length coefficient, which are used to characterize the semantic complexity of the text, are calculated as the second feature vector.

[0091] In some embodiments, the computing module is specifically used for: The multidimensional statistical features are normalized to obtain standard visual features; The word vector entropy value and sequence length coefficient are reverse mapped and linearly normalized to obtain standard text features; Obtain the current computing resource utilization rate of the execution device, and determine the dynamic adjustment factor based on the computing resource utilization rate; Based on dynamic adjustment factors and preset modal weights, standard visual features and standard text features are weighted and fused to generate a sensitivity score.

[0092] In some embodiments, the determining module is specifically used for: If the sensitivity score is greater than the first preset threshold, then the first candidate subnetwork is selected as the target subnetwork. If the sensitivity score is less than or equal to the first preset threshold and greater than the second preset threshold, then the second candidate subnetwork is selected as the target subnetwork. If the sensitivity score is less than or equal to the second preset threshold, the third candidate subnetwork is selected as the target subnetwork; the computational complexity of the first, second and third candidate subnetworks decreases in that order.

[0093] In some embodiments, the inference prediction module is specifically used for: If a switch in the target sub-network is detected relative to the network architecture of the previous time step, the number of output channels of the previous network architecture and the number of input channels of the current target sub-network are obtained. If the number of output channels is greater than the number of input channels, then channel compression is performed by performing downsampling through the dynamic adaptation layer; If the number of output channels is less than the number of input channels, then an upsampling operation is performed through the dynamic adaptation layer to expand the channels; The feature data with aligned channel numbers is input into the target subnetwork for inference and prediction.

[0094] In some embodiments, the inference prediction module is further configured to: In response to the number of architecture changes reaching a preset threshold, the system retrieves the historical best feature snapshot that matches the current multimodal input data from a preset meta-knowledge buffer pool. Calculate the dynamic feature alignment error between the first modality feature and the second modality feature output by the current target subnetwork; The current dynamic feature alignment error is weighted and fused with the extracted historical best feature snapshot to generate a cross-modal representation vector. Knowledge distillation training is performed based on cross-modal representation vectors, and after training is completed, the current model parameters and the corresponding feature alignment error are used as an updated knowledge snapshot and written into the meta-knowledge buffer pool.

[0095] In some embodiments, the inference prediction module is further configured to: When a network architecture change event is detected and the execution device is idle, the parameter difference between the local network model and the cloud network model is obtained. Differential encoding is performed on the parameter differences to obtain differential update parameters; wherein, differential encoding includes, in turn: weight difference compression based on Huffman coding, priority marking of key layer parameters, and dynamic quantization bit width adjustment based on the current network channel quality; The differential update parameters that have been completed with differential encoding are uploaded to the cloud server for global architecture synchronization update.

[0096] This invention provides a multimodal data processing device. First, it extracts and fuses features from different modal data to obtain a sensitivity score that accurately represents the computational complexity requirements of the current input data. Then, based on this score, it dynamically matches and determines the most suitable target sub-network from a pre-defined network architecture library to perform inference. This solution breaks the limitations of traditional fixed computational paths, effectively distinguishing between simple and complex inputs, and adaptively allocating computational resources as needed. It solves the technical problem of existing multimodal models, which use static architectures and cannot dynamically allocate computational resources based on input features, leading to significant waste of computing power on edge devices and excessively high overall processing latency. This allows it to meet the demands of various business scenarios with stringent real-time requirements.

[0097] Specific limitations regarding multimodal data processing can be found in the limitations of multimodal data processing methods described above, and will not be repeated here. Each module in the aforementioned multimodal data processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0098] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 10 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a multimodal data processing method on the server side.

[0099] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 11 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements a client-side function or step based on a multimodal data processing method.

[0100] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: Acquire multimodal input data, which includes first modal data and second modal data; Feature extraction is performed on the first modality data and the second modality data respectively to obtain the first feature vector and the second feature vector; Based on the first feature vector and the second feature vector, a sensitivity score is calculated to characterize the computational complexity requirements of multimodal input data. Based on the sensitivity score, the corresponding target sub-network is determined from the preset network architecture library; The multimodal data processing results are obtained by performing inference and prediction on the multimodal input data through the target subnetwork.

[0101] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: Acquire multimodal input data, which includes first modal data and second modal data; Feature extraction is performed on the first modality data and the second modality data respectively to obtain the first feature vector and the second feature vector; Based on the first feature vector and the second feature vector, a sensitivity score is calculated to characterize the computational complexity requirements of multimodal input data. Based on the sensitivity score, the corresponding target sub-network is determined from the preset network architecture library; The multimodal data processing results are obtained by performing inference and prediction on the multimodal input data through the target subnetwork.

[0102] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0103] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0104] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0105] It should be noted that any AI models, software tools, or components not belonging to this company appearing in the embodiments of this application are merely illustrative examples and do not represent actual use. All user personal information involved in the embodiments of this application has been authorized (with the knowledge and consent) by the relevant parties or has been fully authorized by all parties, and the executing entity may obtain it through various legal and compliant means. The collection, storage, use, processing, transmission, provision, and disclosure of the information, data, and signals involved all comply with relevant laws and regulations and do not violate public order and good morals.

[0106] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A multi-modal data processing method, characterized by, include: Acquire multimodal input data, which includes first modal data and second modal data; Feature extraction is performed on the first modal data and the second modal data respectively to obtain a first feature vector and a second feature vector; Based on the first feature vector and the second feature vector, a sensitivity score is calculated to characterize the computational complexity requirements of the multimodal input data; Based on the sensitivity score, the corresponding target sub-network is determined from a preset network architecture library; The multimodal input data is inferred and predicted through the target sub-network to obtain the multimodal data processing result.

2. The method according to claim 1, characterized in that, The first modal data is visual image data, and the second modal data is text data; the step of extracting features from the first modal data and the second modal data respectively to obtain a first feature vector and a second feature vector includes: The visual image data is processed by a pre-trained convolutional neural network to extract multidimensional statistical features that characterize the computational complexity of the image as the first feature vector; wherein, the multidimensional statistical features include image resolution, edge density, and color entropy. The text data is encoded using a pre-trained micro-language model, and the word vector entropy value and sequence length coefficient, which characterize the semantic complexity of the text, are calculated as the second feature vector.

3. The method according to claim 2, characterized in that, The step of calculating a sensitivity score based on the first feature vector and the second feature vector to characterize the computational complexity requirements of the multimodal input data includes: The multidimensional statistical features are normalized to obtain standard visual features; The word vector entropy value and the sequence length coefficient are reverse mapped and linearly normalized to obtain standard text features; Obtain the current computing resource utilization rate of the execution device, and determine a dynamic adjustment factor based on the computing resource utilization rate; Based on the dynamic adjustment factor and the preset modal weights, the standard visual features and the standard text features are weighted and fused to generate the sensitivity score.

4. The method according to claim 1, characterized in that, The network architecture library contains multiple candidate sub-networks with different computational complexities; determining the corresponding target sub-network from the preset network architecture library based on the sensitivity score includes: If the sensitivity score is greater than the first preset threshold, then the first candidate subnetwork is selected as the target subnetwork; If the sensitivity score is less than or equal to the first preset threshold and greater than the second preset threshold, then the second candidate subnetwork is selected as the target subnetwork. If the sensitivity score is less than or equal to the second preset threshold, then the third candidate subnetwork is selected as the target subnetwork; wherein the computational complexity of the first candidate subnetwork, the second candidate subnetwork and the third candidate subnetwork decreases in that order.

5. The method according to claim 1, characterized in that, The inference and prediction of the multimodal input data through the target sub-network includes: If a switch in the target sub-network is detected relative to the network architecture of the previous moment, the number of output channels of the previous network architecture and the number of input channels of the current target sub-network are obtained. If the number of output channels is greater than the number of input channels, then channel compression is performed by performing a downsampling operation through a dynamic adaptation layer; If the number of output channels is less than the number of input channels, then an upsampling operation is performed through the dynamic adaptation layer to expand the channels; The feature data with aligned channel numbers is input into the target subnetwork for inference and prediction.

6. The method according to claim 1, characterized in that, After performing inference prediction on the multimodal input data through the target sub-network, the method further includes: In response to the number of architecture changes reaching a preset threshold, the system retrieves the historical best feature snapshot that matches the current multimodal input data from a preset meta-knowledge buffer pool. Calculate the dynamic feature alignment error between the first modality feature and the second modality feature output by the current target sub-network; The current dynamic feature alignment error is weighted and fused with the extracted historical best feature snapshot to generate a cross-modal representation vector; Knowledge distillation training is performed based on the cross-modal representation vector, and after training is completed, the current model parameters and the corresponding feature alignment error are used as an updated knowledge snapshot and written into the meta-knowledge buffer pool.

7. The method according to claim 1, characterized in that, The method further includes: When a network architecture change event is detected and the execution device is idle, the parameter difference between the local network model and the cloud network model is obtained. The parameter differences are differentially encoded to obtain differential update parameters; wherein, the differential encoding includes, in sequence: weight difference compression based on Huffman coding, priority marking of key layer parameters, and dynamic quantization bit width adjustment based on the current network channel quality; The differential update parameters for completing the differential encoding are uploaded to the cloud server for global architecture synchronization update.

8. A multimodal data processing device, characterized in that, The device includes: An acquisition module is used to acquire multimodal input data, wherein the multimodal input data includes at least first modal data and second modal data; The feature extraction module is used to extract features from the first modal data and the second modal data respectively to obtain a first feature vector and a second feature vector; The calculation module is used to calculate a sensitivity score based on the first feature vector and the second feature vector to characterize the computational complexity requirements of the multimodal input data. The determination module is used to determine the corresponding target sub-network from a preset network architecture library based on the sensitivity score; The inference and prediction module is used to perform inference and prediction on the multimodal input data through the target sub-network to obtain the multimodal data processing result.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the multimodal data processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, characterized in that, when the computer program is executed by a processor, it implements the multimodal data processing method according to any one of claims 1 to 7.

Images