Data processing method, data processing model acquisition method, data processing model acquisition device, data processing equipment and medium

Abstract

The invention relates to the technical field of data processing, and discloses a data processing method, a data processing model obtaining method, a data processing model obtaining device, data processing equipment and a medium. According to the data processing model obtaining method, preprocessing is carried out on a multi-source heterogeneous data set, a standard data set is obtained, multi-modal feature extraction processing is carried out on the basis of the standard data set, and a multi-modal feature set is obtained; performing cross-modal alignment processing on the multi-modal feature set to generate a cross-modal alignment feature set, performing fusion processing based on the modal alignment feature set to generate a fusion feature set, and inputting the fusion feature set into the to-be-trained model to perform training to obtain a data processing model. According to the method provided by the invention, the adaptation capability of the data processing model to multi-source heterogeneous data and the expression capability of the data processing model to a high-dimensional structure are enhanced while the multi-modal semantic integrity is ensured, so that the semantic understanding and reasoning judgment performance of the model in a complex scene is remarkably improved.

Description

Technical Field

[0001] This application relates to the technical field of data processing, and in particular, to a data processing method, a method for obtaining a model thereof, a device, a device and a medium. Background Art

[0002] With the advent of the big data era and the rapid development of artificial intelligence technology, especially the widespread application of deep learning models, modern information systems have put forward higher requirements for multi-modal data processing. Multi-modal data refers to a data set that simultaneously includes different forms of expression such as text, images, audio, and video, and is widely present in industry scenarios such as medical, financial, industrial, and transportation. The sources of these data are complex, including both structured data (e.g., database tables) and unstructured data (e.g., graphic and text content) and semi-structured data (e.g., JSON, XML formats), and there are significant differences in data structure, encoding method, and semantic expression.

[0003] Currently, multi-modal data processing still mainly relies on traditional data access and integration methods, such as API (Application Programming Interface) calls and ETL (Extract-Transform-Load) processes. Although these methods are relatively mature in structured data processing, there are obvious bottlenecks in processing data with diverse modalities and heterogeneous structures: (1) strong data heterogeneity and lack of a unified access and alignment mechanism; (2) inability to meet the requirements of deep learning models for joint modeling and feature fusion; (3) lack of a cloud-edge collaborative processing mechanism, which is prone to latency and bandwidth bottlenecks; (4) lack of effective modality-level permissions and security protection in data processing involving sensitive information; (5) lack of a unified standard between different industry systems, making it difficult to achieve cross-system interoperability.

[0004] To solve the above problems, current research has proposed a variety of multi-modal modeling strategies. As Figure 1 shown, existing solutions mainly include two typical structures: a post-fusion architecture (as shown in a in Figure 1 ) and a mid-fusion architecture (as shown in b in Figure 1 ). In a as shown in Figure 1 , each modality data (e.g., text and image) is respectively input into the corresponding single-modal model for processing, and the obtained feature results are uniformly modeled by a fusion model at the backend. The structure is simple but the interaction between modalities is limited; in b as shown in Figure 1 , the original high-level features and the extracted features are simultaneously input into the fusion model to enhance the semantic complementarity between modalities, but the requirements for feature alignment and fusion control ability are relatively high. Although the structures as shown in Figure 1 achieve multi-modal joint modeling to a certain extent, they still face problems such as a single fusion mechanism, inconsistent modality spaces, and unstable training.

[0005] Therefore, on the basis of coping with data modality diversity and structural heterogeneity, how to achieve unified processing, dynamic alignment, and efficient fusion of multi-modal data has become an important issue that needs to be solved urgently. Summary of the Invention

[0006] In view of this, embodiments of the present application provide a data processing method, a method for obtaining its model, a device, a device, and a medium, which can effectively solve problems such as scattered feature expressions, weak cross-modal data associations, low information utilization rates, and insufficient fusion accuracy caused by the lack of a deep semantic fusion mechanism between multi-source heterogeneous data.

[0007] In a first aspect, embodiments of the present application provide a method for obtaining a data processing model, including: Performing preprocessing on the original multi-source heterogeneous data set to obtain a standard data set; Based on the standard data set, performing multi-modal feature extraction processing to obtain a multi-modal feature set; Performing cross-modal alignment processing on the multi-modal feature set to generate a cross-modal alignment feature set; Based on the cross-modal alignment feature set, performing fusion processing to generate a fusion feature set; Inputting the fusion feature set into a model to be trained to perform training processing on the model to be trained to obtain a data processing model.

[0008] In some embodiments, the performing preprocessing on the original multi-source heterogeneous data set to obtain a standard data set includes: Performing normalization and enhancement processing on the image data in the original multi-source heterogeneous data set to obtain enhanced images; Performing word segmentation processing on the text data in the original multi-source heterogeneous data set, and mapping the word segmentation structure through a pre-trained word vector model to obtain text vectors; Performing standardization processing on the time series data in the original multi-source heterogeneous data set, and extracting Mel-frequency cepstral coefficients features based on the standardized time series data to obtain time-frequency features; Fusing the enhanced images, the text vectors, and the time-frequency features in a predetermined format to generate the standard data set.

[0009] In some embodiments, the based on the standard data set, performing multi-modal feature extraction processing to obtain a multi-modal feature set includes: Inputting the enhanced images in the standard data set into a convolutional neural network model to perform multi-layer feature extraction processing to obtain image semantic features; Inputting the text vectors in the standard data set into a pre-trained language model to obtain text semantic features; Input the time-frequency features in the standard dataset into a time series modeling network to extract time series features; Perform unified encoding processing on the image semantic features, the text semantic features, and the time series features to construct a multi-modal feature set.

[0010] In some embodiments, the performing cross-modal alignment processing on the multi-modal feature set to generate a cross-modal alignment feature set includes: Perform linear mapping operations on the image semantic features, the text semantic features, and the time series features representing the same semantics respectively to map the features of each modality to a feature space of the same dimension and obtain initial projection features; Perform normalization processing on the initial projection features to generate projection features with consistent scales; Calculate a contrast loss based on the projection features and optimize the trainable parameters in the linear mapping operation by using the contrast loss to generate the cross-modal alignment feature set.

[0011] In some embodiments, the performing fusion processing on the cross-modal alignment feature set to generate a fusion feature set includes: Perform attention weighting processing on the cross-modal alignment feature set to generate a weighted feature set; Perform gated adjustment processing on the weighted feature set to generate an adjustment feature set; Perform normalization processing on the adjustment feature set to generate the fusion feature set.

[0012] In some embodiments, the inputting the fusion feature set into a model to be trained, performing training processing on the model to be trained, and obtaining a data processing model includes: Input the fusion feature set into a model to be trained with a multi-modal input structure; In the model to be trained, process the input fusion feature set based on a fusion strategy, where the fusion strategy includes feature splicing fusion, cross-attention fusion, or decision integration fusion; Perform training processing on the fusion feature set processed by the fusion strategy to obtain the data processing model.

[0013] In a second aspect, an embodiment of the present application provides a data processing method, including: Receive multi-modal data; Input the multi-modal data into a data processing model to perform inference processing and generate inference information; wherein, the data processing model is obtained by using the method described in the first aspect above.

[0014] In a third aspect, an embodiment of the present application provides a data processing model acquisition device, including: A data processing module for performing preprocessing on an original multi-source heterogeneous data set to obtain a standard data set; A feature extraction module for performing multi-modal feature extraction processing based on the standard data set to obtain a multi-modal feature set; An alignment processing module for performing cross-modal alignment processing on the multi-modal feature set to generate a cross-modal alignment feature set; A fusion processing module for performing fusion processing based on the cross-modal alignment feature set to generate a fusion feature set; A model training module for inputting the fusion feature set into a model to be trained and performing training processing on the model to be trained to obtain a data processing model.

[0015] In a fourth aspect, an embodiment of the present application provides a computer device, which includes a processor and a memory. The memory stores a computer program, and the processor is configured to execute the computer program to implement the data processing model acquisition method in the first aspect above.

[0016] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, and when the computer program is executed on a processor, the data processing model acquisition method in the first aspect above is implemented.

[0017] The embodiments of the present application have the following beneficial effects: The data processing model acquisition method of the present application obtains a standard data set with a unified structure and a standardized format by performing preprocessing operations on the original data, ensuring that the multi-modal data has a unified input basis; performs intra-modal feature extraction based on the standard data set, and maps the features of each modality to a unified semantic space through a cross-modal alignment mechanism, improving the collaboration between modalities; further adopts a fusion strategy to integrate the aligned features, enhancing the expression integrity and complementarity of the multi-modal features; finally, inputs the fusion features into the model to be trained to complete the unified training and modeling of the multi-modal information. The data processing model acquisition method of the present application not only realizes the unified processing and deep collaborative fusion of multi-modal data, improves the consistency and expression integrity of feature modeling, but also significantly enhances the training efficiency and cross-modal reasoning ability of the model, providing key support for building a high-precision and strong-adaptability multi-modal intelligent system. Description of the Drawings

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings required in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as limiting the scope. For those of ordinary skill in the art, other related drawings can be obtained based on these drawings without creative efforts.

[0019] Figure 1 Shows a schematic diagram of the post - fusion architecture and the mid - fusion architecture in the multi - modal modeling strategy in the background art of the present application; Figure 2 Shows a flowchart in the method for obtaining a data processing model according to an embodiment of the present application; Figure 3 Shows another flowchart in the method for obtaining a data processing model according to an embodiment of the present application; Figure 4 Shows a schematic structural diagram in the method for obtaining a data processing model according to an embodiment of the present application. Detailed implementation manners

[0020] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments.

[0021] Generally, the components of the embodiments of the present application described and shown in the accompanying drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application claimed, but only represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of protection of the present application.

[0022] In the following text, the terms "including", "having" and their cognates that can be used in various embodiments of the present application are only intended to represent specific features, numbers, steps, operations, elements, components or combinations of the foregoing items, and should not be construed as first excluding the existence of one or more other features, numbers, steps, operations, elements, components or combinations of the foregoing items or increasing the possibility of one or more features, numbers, steps, operations, elements, components or combinations of the foregoing items. In addition, the terms "first", "second", "third", etc. are only used for distinguishing descriptions and cannot be understood as indicating or implying relative importance.

[0023] Unless otherwise limited, all terms (including technical terms and scientific terms) used here have the same meaning as commonly understood by those of ordinary skill in the art to which the various embodiments of the present application belong. The terms (such as those defined in a commonly used dictionary) will be interpreted as having the same meaning as the contextual meaning in the relevant technical field and will not be interpreted as having an idealized meaning or being too formal, unless clearly defined in the various embodiments of the present application.

[0024] Next, some embodiments of the present application will be described in detail in conjunction with the accompanying drawings. Without conflict, the following embodiments and the features in the embodiments can be combined with each other.

[0025] Considering the problems in the prior art, such as the scattered feature expressions caused by the lack of a deep semantic fusion mechanism between multi-source heterogeneous data, weak cross-modal data association, low information utilization rate, and insufficient fusion accuracy. Therefore, a method for obtaining a data processing model is proposed. By constructing a standardized preprocessing process for multi-source heterogeneous data, the consistency of different modal data in format and dimension is ensured; by performing deep feature extraction on each modal data and mapping it to a unified semantic space, effective alignment between multi-modal features is achieved; further, a fusion strategy is introduced to integrate the effectively aligned features, and a deep model is trained based on the fusion features, thereby obtaining a data processing model with cross-modal understanding and reasoning capabilities.

[0026] The following will illustrate the method for obtaining the data processing model with some specific embodiments.

[0027] Figure 2 FIG. shows a flowchart of the method for obtaining the data processing model according to an embodiment of the present application. Exemplarily, the method for obtaining the data processing model includes the following steps: Step S100, perform preprocessing on the original multi-source heterogeneous data set to obtain a standard data set.

[0028] In this embodiment, the original multi-source heterogeneous data set refers to a data set that contains multiple data modalities, different sources, and different formats uploaded through various data import methods (such as API, file upload, database query, etc.), and usually includes types such as images, texts, and time series signals. Since there are significant differences in data dimensions, representation structures, and coding methods among various modalities, they cannot be directly used as a unified input for model training or reasoning. Therefore, preprocessing operations are performed on this data set to convert data of different modalities into a unified representation form with compatible structures and consistent semantics. Among them, preprocessing refers to a series of data standardization, feature extraction, and format conversion operations performed on various modal data, which are used to eliminate modal differences and improve structural unity. Through the above preprocessing process, a standard data set can be generated, and the standard data set is a fusion input set with a unified format and numerical range.

[0029] In an alternative embodiment, step S100 includes: S101, perform normalization and enhancement processing on the image data in the original multi-source heterogeneous data set to obtain enhanced images.

[0030] Among them, the original multi-source heterogeneous dataset can be obtained through multiple paths. For example, (1) database data, including user input information and sensor reported data. (2) Hardware data, generated by various sensors or Internet of Things devices. (3) Webpage and public data, including webpage content obtained by crawlers, government open data, and enterprise shared data. (4) API interface data, referring to shared information from external systems, commonly seen in cross-industry platform integration, with semi-fixed structure. (5) File data, including unstructured content such as documents, emails, pictures, audio, and video.

[0031] Exemplarily, for image modality data, first perform normalization processing to scale the pixel values to the range [0, 1], or perform standardization processing according to predefined mean and standard deviation (e.g., the mean and variance of the ImageNet dataset) to eliminate interference factors such as brightness and contrast differences between different images. Subsequently, adopt an image enhancement strategy to perform operations such as rotation, flipping, and scaling on the image data to expand the diversity of training samples and improve the generalization ability of the model. After processing, the enhanced image is output.

[0032] S102, perform word segmentation on the text data in the original multi-source heterogeneous dataset, and map the word segmentation structure through a pre-trained word vector model to obtain text vectors.

[0033] Exemplarily, for text modality data, first perform word segmentation on the text sentence to extract word-level or word-character-level language units; then map the word segmentation result to a continuous embedded vector representation through a pre-trained word embedding model (e.g., Word2Vec, GloVe, or BERT), thereby obtaining a structured text semantic vector with semantic layer features that can be perceived by the model. In an optional implementation manner, to further optimize the representation effect, layer normalization (LayerNorm) or batch normalization (BatchNorm) can be added after word embedding to adaptively adjust the distribution characteristics of the embedded vector.

[0034] S103, perform standardization processing on the time series data in the original multi-source heterogeneous dataset, and extract Mel-frequency cepstral features based on the standardized time series data to obtain time-frequency features.

[0035] Exemplarily, for time series modality data (e.g., speech or sensor data), first perform standardization processing in ways such as sliding window or whole segment, e.g., perform Z-score normalization or dynamic range compression to solve the problem of inconsistent data amplitudes under different devices or time periods. Subsequently, extract Mel-frequency cepstral coefficients (MFCC) based on the normalized time series signal to capture the energy distribution characteristics of the audio in the time-frequency domain, and then obtain the time-frequency feature representation.

[0036] S104. Combine the enhanced image, text vector, and time-frequency features according to a predetermined format to generate a standard dataset.

[0037] Finally, preprocess the results of the above three types of modal data - enhanced images, text vectors, and time-frequency features - according to a preset format. For example, construct a unified data object or embedding structure to make them have the same dimension or common input format, which serves as the basic input for subsequent multi-modal feature alignment, fusion, and modeling, forming a standard dataset with a standardized structure.

[0038] Step S200. Based on the standard dataset, perform multi-modal feature extraction processing to obtain a multi-modal feature set.

[0039] In this embodiment, perform multi-modal feature extraction processing based on the standard dataset to obtain a multi-modal feature set. Among them, multi-modal feature extraction refers to using a feature encoding model that matches the modal attributes of different types of modal data to extract their semantic representations; the multi-modal feature set refers to a feature expression set obtained by unifying the feature vectors extracted from image, text, and time-series modal data in terms of structure and dimension, which is used for subsequent alignment and fusion processing. Demonstratively, based on the input data of three types of modalities, namely images, text, and time series, perform corresponding feature extraction operations respectively to generate feature representations with consistent structures.

[0040] In an alternative embodiment, as Figure 3 shown, step S200 includes: S201. Input the enhanced image in the standard dataset into a convolutional neural network model and perform multi-layer feature extraction processing to obtain image semantic features.

[0041] Among them, the image semantic feature refers to a representation tensor formed by extracting features from the image modality through a deep neural network (such as CNN, ResNet), which is used to describe the high-order information expression of the image in the spatial dimension and semantic dimension. The convolutional neural network model refers to a neural network structure that encodes the spatial structure information of the image layer by layer through operations such as stacking convolutional layers, activation layers, and pooling layers. Specifically, input the enhanced image in the standard dataset into the convolutional neural network model, extract multi-level spatial features such as edges, textures, and regions through layer-by-layer convolution, and map them to a fixed-length image semantic feature vector through a fully connected layer.

[0042] In an alternative embodiment, the image semantic features can be divided into multiple levels according to the depth of the network structure: the shallow convolutional kernels can extract low-level visual information such as edges, textures, and color distributions; the intermediate layer can identify local structural features, for example, the components or geometric contours of an object; the deep layer can extract high-level features with global semantic context, such as semantic concepts of concrete objects like "car tires" and "dog ears". In terms of the feature representation form, a Convolutional Neural Network (CNN) usually outputs a spatial pyramid feature map, whose dimensional structure is a three-dimensional tensor of H×W×C (where H is the height, W is the width, and C is the number of channels), retaining the location information of the image in the spatial structure. If a Vision Transformer (ViT) model is adopted, the image will be divided into several image patches, each patch is encoded through the self-attention mechanism, and the output feature sequence includes a special [CLS] token (Classification Token) for overall semantic expression. For example, in the typical object detection model Faster R-CNN, the RoI Align (Region of Interest Align) operation is used to extract a region feature tensor of a fixed size from the image. This feature not only contains the category information of the object but also retains the corresponding location information, facilitating subsequent detection or localization processing.

[0043] S202, input the text vectors in the standard dataset into the pre-trained language model to obtain text semantic features.

[0044] Among them, the text semantic features refer to the continuous vectors obtained by embedding word vectors and context modeling of natural language fragments, which are used to express the semantic relationship and context dependence between words. The pre-trained language model refers to the model structure used to map discrete text inputs into continuous vector representations, which can capture the semantics and context relationships of word sequences. Specifically, input the text vectors in the standard dataset into the pre-trained language model (for example, the BERT model), first perform context modeling on the word sequence, and then convert it into a text semantic feature vector through the embedding mapping function.

[0045] Furthermore, text semantic features can be divided into two major categories: word-level embeddings and sentence-level embeddings. Among them, there are two main methods for word-level representation: one is to use static word vector models, such as Word2Vec (word-to-vector) or GloVe (global vector representation). This type of model assigns a fixed vector representation to each word, which is simple and efficient but cannot capture the semantic changes of words in different contexts. The other method is to use dynamic context pre-trained language models, such as BERT (Bidirectional Encoder Representations from Transformers). This type of model can dynamically generate word representations according to the context, and is more suitable for dealing with semantic ambiguity problems and performing context reasoning. Sentence-level representation methods rely on different sequence models to extract the semantic features of the entire sentence. Traditional methods include recurrent neural networks (RNNs) and their variants, such as long short-term memory networks (LSTMs). These models can form the semantic representation of the entire sentence by extracting the final hidden state or performing average pooling operations on the hidden states. In modern Transformer architectures, a special [CLS] token is usually used as the summary semantic representation of the entire sentence, or average pooling is performed on the entire sequence.

[0046] For example, in a text classification task, the [CLS] vector output by the BERT model can be directly used for sentence-level classification judgment. This method has strong semantic compression and classification capabilities and can effectively capture the overall semantic information of the sentence.

[0047] S203: Input the time-frequency features in the standard dataset into the time series modeling network to extract time series features.

[0048] Among them, time series features refer to the semantic feature representations extracted from audio signals or sensor data that can reflect the time domain and frequency domain structures. The time series modeling network refers to a neural network structure dedicated to modeling time-dependent relationships, including gated recurrent networks (such as GRUs) or convolutional time series networks, etc. Specifically, the time-frequency features in the standard dataset can be input into the time series modeling network to model the time structure of the signal, thereby extracting time series feature vectors containing time domain variations and frequency domain features.

[0049] For example, basic temporal features can be obtained by extracting Mel-Frequency Cepstral Coefficients (MFCC) or Short-Time Fourier Transform (STFT). These features reflect the variation patterns of the signal in the time and frequency domains. At the advanced modeling level, network models specifically designed for processing temporal data can also be incorporated. For example, WaveNet is a speech modeling network based on dilated convolutions, suitable for constructing deep phoneme-level embedding features; while ECAPA-TDNN (Emphasized Channel Attention, Propagation and Aggregation in Time Delay Neural Network) is a speaker recognition network that emphasizes channel attention and is used to generate deep speaker-level embedding features.

[0050] For sensor-based tasks such as action recognition and vital sign detection, recurrent neural networks (such as LSTM) can be used to extract periodic structures, abnormal fluctuations, or other key temporal domain patterns in the data. For example, in the SlowFast model structure, the model is divided into two paths: the slow path is used to extract low-frequency appearance features, and the fast path is used to capture high-frequency dynamic information. Finally, dual modeling of the temporal modality is completed through feature fusion.

[0051] S204, perform unified encoding processing on the image semantic features, text semantic features, and temporal features to construct a multi-modal feature set.

[0052] Specifically, after obtaining the image semantic features, text semantic features, and temporal features, perform unified encoding processing on each of them, map them to the same feature space, and construct a multi-modal feature set.

[0053] Step S300, perform cross-modal alignment processing on the multi-modal feature set to generate a cross-modal alignment feature set.

[0054] Among them, cross-modal alignment processing refers to performing mapping, normalization, and optimization operations on feature representations from different modalities but expressing the same semantics through a neural network structure, so that they achieve consistent structures and close distributions in the unified feature space for subsequent feature fusion and inference; the cross-modal alignment feature set refers to the set of feature representations obtained after the above alignment processing, which has unified dimensions and scales and can be cross-modally compared and combined. Demonstratively, in this embodiment, based on the three types of features of image semantic features, text semantic features, and temporal features, unified mapping, distribution normalization, representation optimization, and fusion operations are respectively performed to generate a cross-modal alignment feature set.

[0055] In an alternative embodiment, step S300 includes: S301, respectively perform linear mapping operations on the image semantic features, text semantic features, and temporal features representing the same semantics, so as to map each modality feature to a feature space of the same dimension and obtain initial projection features.

[0056] Among them, the linear mapping operation refers to mapping different modality features to the same dimension space through a learnable fully connected layer or lightweight convolution structure, forming a unified feature representation that can be compared in structure. Demonstratively, use a fully connected layer or lightweight convolution layer to uniformly map the image semantic features, text semantic features, and temporal features representing the same semantics into initial projection features of the same dimension (for example, 256 dimensions) to ensure that subsequent calculations have a vector-level alignment basis. For example, after the image features are extracted by CNN, the dimension is adjusted through a linear layer, and the text features are linearly mapped after BERT.

[0057] S302, perform normalization processing on the initial projection features to generate projection features with consistent scales.

[0058] Among them, the core goal of the normalization processing is to adjust the scale of the projection features through a modality normalization mechanism. Specifically, this process aims to eliminate the amplitude differences of the feature values between different modalities, so that they have a consistent numerical distribution in the unified space. To achieve the above goal, trainable parameters (for example, scaling factor γ and bias β) can be introduced. The role of these parameters is to dynamically adjust the scale of the initial projection features, thereby completing the dynamic recalibration in the dimension space. Finally, projection features with consistent scales are generated.

[0059] S303, calculate a contrastive loss based on the projection features, and use the contrastive loss to optimize the trainable parameters in the linear mapping operation to generate a cross-modal alignment feature set.

[0060] Among them, the contrast loss is a loss function used to measure the similarity and difference between features of different modalities. In the cross-modal alignment task, the goal of the contrast loss is to reduce the distance between features with the same semantics but different modalities, while increasing the distance between features with different semantics but possibly the same modality. By calculating the contrast loss based on the projected features, the effect of the current feature alignment can be quantitatively evaluated, and the subsequent optimization process can be guided. Then, in order to further improve the quality of feature alignment, the trainable parameters in the linear mapping operation need to be optimized. The trainable parameters here usually include the weight matrix and bias vector in the linear mapping operation, which determine how to map the original features of different modalities to a unified feature space. By using the contrast loss as the optimization objective, these parameters can be dynamically adjusted to make the generated projected features more in line with the requirements of cross-modal alignment. Finally, after the above optimization process, a set of cross-modal aligned feature sets is finally generated. The characteristics of this set of feature sets are that even if they come from different modalities (such as images, texts, audios, etc.), as long as they represent the same semantics, their feature representations will be as close as possible.

[0061] Step S400: Perform fusion processing based on the cross-modal aligned feature set to generate a fused feature set.

[0062] In this embodiment, fusion processing is performed based on the cross-modal aligned feature set to generate a fused feature set. Among them, fusion processing refers to further integrating the information of each modality to form a unified expression vector after the features of each modality are aligned; the fused feature set refers to the set of unified feature vectors representing each sample generated after fusion processing, which is used as the direct input for subsequent model training. Specifically, this embodiment adopts a fusion structure that combines the attention mechanism and the gating mechanism to enhance the dynamic control ability of the weights between modalities during fusion, and ensures the consistency of the feature structure through normalization.

[0063] In an alternative embodiment, step S400 includes: S401: Perform attention weighting processing on the cross-modal aligned feature set to generate a weighted feature set.

[0064] Among them, attention weighting processing refers to dynamically allocating weights according to the importance of features of different modalities to the current task to improve the effectiveness of information aggregation. Specifically, the cross-modal aligned feature set is input into the multi-head attention mechanism module, and attention scores are constructed through queries (Query), keys (Key), and values (Value) to generate the attention weights of the features of each modality, and a weighted summation operation is performed on each feature based on the weights to output a weighted feature set.

[0065] S402: Perform gating adjustment processing on the weighted feature set to generate an adjusted feature set.

[0066] Among them, the gated adjustment process refers to using a learnable control factor to adjust the weighted features to suppress the interference of irrelevant or noisy modalities. Specifically, the weighted feature set is input into the gating module, and the gating coefficients are generated through the Sigmoid or Softmax function to perform weighted suppression or enhancement on the features of each channel or modality dimension, and the adjusted feature set is output.

[0067] S403. Perform a normalization process on the adjusted feature set to generate a fused feature set.

[0068] Among them, the normalization process refers to maintaining the consistency of the fused features in terms of numerical scale and distribution through standardization operations, avoiding the influence of distribution drift between different samples on model training. Demonstratively, perform layer normalization or batch normalization operations on the adjusted feature set to keep the fused feature samples unified in terms of mean and variance, and output the fused feature set.

[0069] Step S500. Input the fused feature set into the model to be trained, perform a training process on the model to be trained, and obtain a data processing model.

[0070] In this embodiment, the fused feature set is input into the model to be trained, and a training process is performed on the model to be trained to obtain a data processing model. Among them, the data processing model refers to a deep learning model that can simultaneously accept multi-modal fused feature inputs and perform inference and judgment based on the learned representations; this model constructs a parameter mapping relationship through the training stage and has generalization and decision-making capabilities in tasks. Demonstratively, by inputting the fused feature set into a model with a specific structure and combining a fusion strategy for structural integration, the parameters are continuously updated during the training process, and finally a data processing model is obtained as the output.

[0071] In an alternative embodiment, step S500 includes: S501. Input the fused feature set into the model to be trained with a multi-modal input structure.

[0072] Among them, the model to be trained refers to a neural network model with multiple input interfaces and a fusion structure that can accept multi-modal fused feature inputs with compatible structures and perform semantic integration and decision-making learning processes internally. Specifically, input the fused feature set into the model to be trained, and each modal fused feature is respectively input into the corresponding input branch according to the structural requirements. The model can adopt structures that support multi-modal inputs such as multi-input Transformer networks and bilinear pooling networks to ensure the independent expression and joint modeling capabilities of each modal feature.

[0073] S502. In the model to be trained, process the input fused feature set based on a fusion strategy, where the fusion strategy includes feature concatenation fusion, cross-attention fusion, or decision integration fusion. Here, the fusion strategy refers to a predefined structural logic for performing feature-level or decision-level fusion on multimodal inputs within the model, aiming to achieve collaboration and enhancement among multimodal information. Specifically, feature integration can be performed within the model based on a preset fusion strategy, which includes but is not limited to: (1) Feature concatenation fusion: directly concatenate the input features of each modality in the dimension to form a joint representation; (2) Cross-attention fusion: construct an interactive attention mechanism between modalities to achieve weighted mutual guidance of information between modalities; (3) Decision integration fusion: input the features of each modality into sub-models for independent inference, and finally integrate the outputs of each sub-model according to rules to generate a final decision.

[0074] S503. Perform training processing on the fused feature set processed by the fusion strategy to obtain a data processing model.

[0075] Here, the training processing refers to the process of iteratively updating the gradients of the parameters of the model to be trained based on input samples by specifying a loss function and an optimization algorithm. Specifically, input the fused feature set processed by the fusion strategy into the model to be trained, construct a supervised loss function based on a predefined task (such as classification or regression), for example, cross-entropy loss or mean squared error, and perform parameter training iteration in combination with a gradient descent-based optimization algorithm (such as Adam or SGD). After multiple rounds of training, output the data processing model as the final model result.

[0076] This application also provides a data processing method. Exemplarily, the data processing method includes: receiving multimodal data, inputting the multimodal data into the data processing model to perform inference processing and generate inference information; where the data processing model is obtained by the method of the above embodiment and has the ability to process multimodal inputs such as images, texts, and time series signals.

[0077] It can be understood that the received multimodal data can be input by the user in the display interface of the terminal device or obtained from other devices, which is not limited here. Further, in an optional implementation, the user can also view the generated inference information in the display interface of the terminal device.

[0078] Figure 4 FIG. shows a schematic structural diagram of a data processing model acquisition device according to an embodiment of the present application. Exemplarily, the data processing model acquisition device 100 includes: A data processing module 110 is configured to perform preprocessing on an original multi-source heterogeneous data set to obtain a standard data set; a feature extraction module 120 is configured to perform multi-modal feature extraction processing based on the standard data set to obtain a multi-modal feature set; an alignment processing module 130 is configured to perform cross-modal alignment processing on the multi-modal feature set to generate a cross-modal aligned feature set; a fusion processing module 140 is configured to perform fusion processing based on the cross-modal aligned feature set to generate a fused feature set; a model training module 150 is configured to input the fused feature set into a model to be trained, and perform training processing on the model to be trained to obtain a data processing model.

[0079] It can be understood that the device in this embodiment corresponds to the method in the above embodiment, and the optional items in the above embodiment are also applicable to this embodiment, so they will not be described repeatedly here.

[0080] The present application also provides a computer device. Exemplarily, the computer device includes a processor and a memory. The memory stores a computer program, and the processor runs the computer program to enable the computer device to execute the functions of the above method or each module in the above device.

[0081] Among them, the processor can be an integrated circuit chip with signal processing capabilities. The processor can be a general-purpose processor, including at least one of a central processing unit (CPU), a graphics processing unit (GPU), a network processor (NP), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc., and can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application.

[0082] The memory can be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electric Erasable Programmable Read-Only Memory (EEPROM), etc. Among them, the memory is used to store a computer program, and after receiving an execution instruction, the processor can execute the computer program accordingly.

[0083] This application also provides a computer-readable storage medium for storing the computer program used in the above computer device. For example, the computer-readable storage medium may include, but is not limited to: various media such as USB flash drives, external hard drives, Read-Only Memory (ROM), Random Access Memory (RAM), magnetic disks, or optical discs that can store program codes.

[0084] In several embodiments provided by this application, it should be understood that the disclosed device and method can also be implemented in other ways. The device embodiments described above are merely illustrative. For example, the flowcharts and structure diagrams in the accompanying drawings show the possible architectures, functions, and operations of the device, method, and computer program product according to multiple embodiments of this application. In this regard, each block in the flowchart or block diagram can represent a module, program segment, or part of the code, and the module, program segment, or part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in an alternative implementation, the functions marked in the block can occur in a different order than that marked in the accompanying drawings. For example, two consecutive blocks can actually be executed substantially in parallel, and they can sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the structure diagram and / or flowchart, as well as the combination of blocks in the structure diagram and / or flowchart, can be implemented by a dedicated hardware-based system for performing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0085] In addition, in each embodiment of this application, each functional module or unit can be integrated together to form an independent part, or each module can exist separately, or two or more modules can be integrated to form an independent part.

[0086] When the above functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to enable a computer device (which can be a smart phone, a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of this application.

[0087] The above is only the specific implementation manner of this application, but the protection scope of this application is not limited thereto. Any person skilled in the art within the technical scope disclosed by this application can easily think of changes or substitutions, which should all be covered within the protection scope of this application.

Claims

1. A method for obtaining a data processing model, characterized in that, The method includes: Performing preprocessing on the original multi-source heterogeneous dataset to obtain a standard dataset; Based on the standard dataset, performing multi-modal feature extraction processing to obtain a multi-modal feature set; Performing cross-modal alignment processing on the multi-modal feature set to generate a cross-modal alignment feature set; Based on the cross-modal alignment feature set, performing fusion processing to generate a fusion feature set; Inputting the fusion feature set into a model to be trained to perform training processing on the model to be trained, and obtaining a data processing model.

2. The method for obtaining a data processing model according to claim 1, wherein The performing preprocessing on the original multi-source heterogeneous dataset to obtain a standard dataset includes: Normalizing and enhancing the image data in the original multi-source heterogeneous dataset to obtain enhanced images; Performing word segmentation on the text data in the original multi-source heterogeneous dataset, and mapping the word segmentation structure through a pre-trained word vector model to obtain text vectors; Normalizing the time series data in the original multi-source heterogeneous dataset, and extracting Mel-frequency cepstral coefficients based on the normalized time series data to obtain time-frequency features; Fusing the enhanced images, the text vectors, and the time-frequency features in a predetermined format to generate the standard dataset.

3. The method for obtaining a data processing model according to claim 2, wherein The based on the standard dataset, performing multi-modal feature extraction processing to obtain a multi-modal feature set includes: Inputting the enhanced images in the standard dataset into a convolutional neural network model to perform multi-layer feature extraction processing to obtain image semantic features; Inputting the text vectors in the standard dataset into a pre-trained language model to obtain text semantic features; Inputting the time-frequency features in the standard dataset into a time series modeling network to extract time series features; Performing unified encoding processing on the image semantic features, the text semantic features, and the time series features to construct a multi-modal feature set.

4. The method for obtaining a data processing model according to claim 3, wherein The performing cross-modal alignment processing on the multi-modal feature set to generate a cross-modal alignment feature set includes: Performing linear mapping operations on the image semantic features, the text semantic features, and the time series features representing the same semantics respectively to map each modal feature to a feature space of the same dimension to obtain initial projection features; Performing normalization processing on the initial projection features to generate projection features with consistent scales; Calculating a contrastive loss based on the projection features, and optimizing the trainable parameters in the linear mapping operation by using the contrastive loss to generate the cross-modal alignment feature set.

5. The method for obtaining a data processing model according to claim 1, wherein The based on the cross-modal alignment feature set, performing fusion processing to generate a fusion feature set includes: Performing attention weighting processing on the cross-modal alignment feature set to generate a weighted feature set; Performing gated adjustment processing on the weighted feature set to generate an adjusted feature set; Performing normalization processing on the adjusted feature set to generate the fusion feature set.

6. The method for obtaining a data processing model according to claim 1, wherein The inputting the fusion feature set into a model to be trained to perform training processing on the model to be trained to obtain a data processing model includes: Inputting the fusion feature set into a model to be trained with a multi-modal input structure; In the to-be-trained model, the input fused feature set is processed based on a fusion strategy, and the fusion strategy includes feature concatenation fusion, cross-attention fusion, or decision integration fusion; Performing training processing on the fused feature set processed by the fusion strategy to obtain the data processing model.

7. A data processing method, characterized in that, The method includes: Receiving multimodal data; Inputting the multimodal data into a data processing model to perform inference processing and generate inference information; wherein, the data processing model is obtained by using the method according to any one of claims 1 to 6.

8. A data processing model acquisition device, characterized in that Including: A data processing module, configured to perform preprocessing on an original multi-source heterogeneous data set to obtain a standard data set; A feature extraction module, configured to perform multimodal feature extraction processing based on the standard data set to obtain a multimodal feature set; An alignment processing module, configured to perform cross-modal alignment processing on the multimodal feature set to generate a cross-modal alignment feature set; A fusion processing module, configured to perform fusion processing based on the cross-modal alignment feature set to generate a fused feature set; A model training module, configured to input the fused feature set into a to-be-trained model, perform training processing on the to-be-trained model, and obtain a data processing model.

9. A computer device, characterized in that, The computer device includes a processor and a memory, the memory stores a computer program, and the processor is configured to execute the computer program to implement the data processing model acquisition method according to any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that It stores a computer program, and when the computer program is executed on a processor, it implements the data processing model acquisition method according to any one of claims 1 to 6.

Images