Heterogeneous data fusion apparatus, method and medium for ai large model pre-training

Abstract

The application discloses a heterogeneous data fusion device, method and medium for AI large model pre-training, and relates to the technical field of artificial intelligence. The heterogeneous data fusion device comprises a collection module, a processing module and an output module, the processing module comprises an identification and analysis unit and an alignment and evaluation unit, the output module comprises a filling unit and an index unit; the identification and analysis unit is used for identifying and analyzing heterogeneous data of different modalities, obtaining each standard data object, the alignment and evaluation unit is used for unifying semantic vectors of each standard data object and calculating quality scores of pairs of multi-modal data; the filling unit is used for filling the pairs of multi-modal data to corresponding slots of a semantic driving template, and attaching labels and quality scores to target samples, and the index unit is used for encapsulating the target samples into a standard data structure and constructing a multi-dimensional index of the target samples. The device can unify standards, accurately fuse multi-modal data, and improve processing efficiency and the quality of pre-training data.

Description

Heterogeneous data fusion apparatus, method and medium for pre-training large AI models Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a heterogeneous data fusion device, method and medium for pre-training large AI models. Background Technology

[0002] With the rapid development of artificial intelligence (AI) technology, basic models, represented by large language models and multimodal large models, are gradually becoming important carriers of general intelligence and can be widely applied in fields such as question answering systems, image and text understanding, and intelligent agent systems. In order to support the continuous improvement of these large models in semantic understanding, cross-modal alignment, and generalization ability, model training has placed higher demands on both the scale and quality of data.

[0003] In reality, available data covers a variety of forms, including web page text, social media content, image and text data, and audio and video recordings. It is characterized by diverse sources, complex modalities (such as text and speech), and inconsistent structures (such as HTML, JSON, and CSV). However, related technologies can only process large model pre-training data of a single modality, while supporting other modalities is weak and lacking a unified fusion mechanism, which seriously restricts the efficiency and quality of data processing. Summary of the Invention

[0004] In view of the above-mentioned defects or deficiencies in related technologies, it is desirable to provide a heterogeneous data fusion device, method and medium for pre-training large AI models, which can unify standards and accurately fuse multimodal data, thereby improving processing efficiency and the quality of pre-training data.

[0005] In a first aspect, this application provides a heterogeneous data fusion device for pre-training large AI models, the heterogeneous data fusion device comprising a data acquisition module, a processing module and an output module connected in sequence;

[0006] The acquisition module is used to acquire heterogeneous data from different sources and attach data metadata to the heterogeneous data. The heterogeneous data includes at least one of text data, image data, and audio data. The data metadata includes file extension, encoding format, and header binary signature.

[0007] The processing module includes an identification and parsing unit and an alignment and evaluation unit. The identification and parsing unit is used to identify the modality of the heterogeneous data based on the data element information, and to parse the heterogeneous data of different modalities to obtain each standard data object. The alignment and evaluation unit is used to unify the semantic vector of each standard data object to obtain multimodal data pairs, and to calculate the quality score of the multimodal data pairs.

[0008] The output module includes a filling unit and an indexing unit; the filling unit is used to fill the multimodal data pairs into the corresponding slots of the semantically driven template to obtain target samples, and to attach labels and the quality score to the target samples, wherein the labels include modal labels and semantic labels; the indexing unit is used to encapsulate the target samples into a standard data structure, and to construct a multi-dimensional index of the target samples according to the labels and the quality score.

[0009] Optionally, in some embodiments of this application, when identifying the modality of the heterogeneous data based on the data metadata, the identification and parsing unit is specifically used to extract the single modality representation vector of the heterogeneous data by utilizing each sub-channel in the feature extraction layer of the multi-channel modality discriminant network. Each sub-channel includes a text sub-channel, an image sub-channel, an audio sub-channel, and a structured sub-channel. The attention enhancement layer of the multi-channel modality discriminant network is used to adjust the feature weights of each sub-channel to obtain a mixed modality representation vector.

[0010] Furthermore, the modality probability distribution corresponding to the hybrid modality representation vector is calculated using the discriminant coding layer of the multi-channel modality discrimination network, and the modality with a probability greater than or equal to a preset threshold value is taken as the modality of the heterogeneous data.

[0011] Optionally, in some embodiments of this application, when adjusting the feature weights of each sub-channel using the attention enhancement layer of the multi-channel modality discriminant network, the recognition and parsing unit is specifically used to generate the context vector of the heterogeneous data;

[0012] Calculate the similarity between the context vector and each of the single modality representation vectors, and convert the similarity into the feature weights of the sub-channels. If the feature weights of the sub-channels are less than a preset threshold, then reduce the feature weights of the sub-channels.

[0013] Optionally, in some embodiments of this application, an incremental learning adaptation layer is provided after the feature extraction layer of the multi-channel modality discrimination network. The incremental learning adaptation layer uses the modality center vector prototype as a semantic benchmark and combines the extracted features of each sub-channel to adaptively update the new modality distribution.

[0014] Optionally, in some embodiments of this application, when unifying the semantic vectors of the standard data objects to obtain multimodal data pairs, the alignment evaluation unit is specifically used to encode each of the standard data objects into a single semantic vector;

[0015] The single semantic vector is mapped to a shared semantic space through a projection layer, which includes multiple linear projection sub-layers with consistent structure. The alignment loss function is then used to optimize the shared semantic space to obtain the multimodal data pair.

[0016] Optionally, the alignment loss function described in some embodiments of this application for:

[0017] ;

[0018] In the above formula, Indicates the first A single semantic vector for each modality. Indicates the first The first modality match A single semantic vector of a positive sample. This indicates the calculation of cosine similarity. Indicates the temperature coefficient. This represents the total number of individual semantic vectors. Represents the th element in the total number of single semantic vectors. A single semantic vector.

[0019] Optionally, in some embodiments of this application, before filling the multimodal data pairs into the corresponding slots of the semantic-driven template, the filling unit is further configured to obtain the target task input by the operation interface and generate the semantic-driven template based on the target task.

[0020] Optionally, in some embodiments of this application, the indexing unit is also used to adjust the sample distribution based on a preset sampling strategy during the pre-training of the AI ​​large model.

[0021] Secondly, this application provides a heterogeneous data fusion method for pre-training large AI models, wherein the heterogeneous data fusion method is used in any one of the heterogeneous data fusion apparatuses described in the first aspect, and the heterogeneous data fusion method includes:

[0022] Collect heterogeneous data from different sources and attach data metadata to the heterogeneous data. The heterogeneous data includes at least one of text data, image data, and audio data. The data metadata includes file extension, encoding format, and header binary signature.

[0023] After identifying the modalities of the heterogeneous data based on the data metadata, the heterogeneous data of different modalities are parsed to obtain each standard data object, and the semantic vectors of each standard data object are unified to obtain multimodal data pairs, and the quality scores of the multimodal data pairs are calculated.

[0024] The multimodal data pairs are filled into the corresponding slots of the semantically driven template to obtain the target sample. After obtaining the target sample, a label and the quality score are attached to the target sample. The label includes a modal label and a semantic label. The target sample is then encapsulated into a standard data structure, and a multidimensional index of the target sample is constructed according to the label and the quality score.

[0025] Thirdly, this application provides a computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement the steps of the heterogeneous data fusion method described in the second aspect.

[0026] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0027] This application provides a heterogeneous data fusion device, method, and medium for pre-training large AI models. A unified and complete heterogeneous data fusion framework is established through a sequentially connected acquisition module, processing module, and output module. The processing module includes an identification and parsing unit and an alignment evaluation unit, while the output module includes a filling unit and an indexing unit. In practical use, it can quickly acquire heterogeneous data from different sources and automatically add metadata to this heterogeneous data. This heterogeneous data includes at least one of text, image, and audio data, significantly broadening the data sources. Secondly, based on the metadata, the modalities of the heterogeneous data are accurately identified, and different modalities of heterogeneous data are parsed to obtain standard data objects, effectively solving the problem of large differences in heterogeneous data structures and laying the foundation for cross-modal semantic processing. Then, the semantic vectors of each standard data object are unified to achieve deep alignment, improving semantic consistency and model generalization ability. Finally, the quality scores of multimodal data pairs are quantified to avoid the negative impact of noise and redundant data on the pre-training effect of large models. Finally, the multimodal data pairs are automatically populated into the corresponding slots of the semantically driven template to obtain the target samples. Labels and quality scores are then attached to the target samples. These labels include modal labels and semantic labels, eliminating the need for manual annotation and significantly improving sample generation efficiency. At the same time, the target samples are encapsulated into a standard data structure, and a multi-dimensional index of the target samples is constructed according to the labels and quality scores, improving retrieval efficiency and facilitating flexible access. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 is a schematic diagram of the architecture of a heterogeneous data fusion device for AI large model pre-training provided in an embodiment of this application;

[0030] Figure 2 is a flowchart illustrating a heterogeneous data fusion method for pre-training large AI models provided in an embodiment of this application;

[0031] Figure 3 is a structural block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0032] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0033] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0034] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of this application can be combined with each other. The heterogeneous data fusion device, method and medium for AI large model pre-training provided by the embodiments of this application are described in detail below with reference to Figures 1 to 3.

[0035] Please refer to Figure 1, which is a schematic diagram of the architecture of a heterogeneous data fusion device for pre-training large AI models provided in an embodiment of this application. The heterogeneous data fusion device 10 includes an acquisition module 101, a processing module 102, and an output module 103 connected in sequence. The processing module 102 includes an identification and parsing unit 1021 and an alignment evaluation unit 1022, and the output module 103 includes a filling unit 1031 and an indexing unit 1032.

[0036] In practical use, the acquisition module 101 can acquire heterogeneous data from different sources and attach data metadata to this heterogeneous data. This heterogeneous data includes at least one of text data, image data, and audio data. The data metadata includes file extensions, encoding formats, header binary signatures, data sources, acquisition time, and acquisition batch numbers. The header binary signature refers to the characteristic pattern formed by the first few bytes of a file at the binary level; it is usually fixed and used to identify the file type. It should be noted that the acquisition module 101 supports data protocols such as HTTP, FTP, API, and message queues, as well as data formats such as HTML, JSON, XML, CSV, MP3, and JPEG. Data sources include, but are not limited to, web page text, social media content, structured data from industry knowledge bases, and image and audio file libraries.

[0037] The identification and parsing unit 1021 can identify the modalities of heterogeneous data based on data metadata, and parse heterogeneous data of different modalities to obtain standard data objects. The alignment and evaluation unit 1022 can unify the semantic vectors of each standard data object to obtain multimodal data pairs and calculate the quality score of the multimodal data pairs. Furthermore, the filling unit 1031 can fill the multimodal data pairs into the corresponding slots of the semantically driven template to obtain target samples, and attach labels and quality scores to the target samples. The labels include modal labels and semantic labels. The indexing unit 1032 can encapsulate the target samples into standard data structures and construct a multi-dimensional index of the target samples according to the labels and quality scores. For example, standard data structures include, but are not limited to, JSON, TFRecord, and PyTorch Dataset, and are compatible with the reading interfaces of mainstream training frameworks.

[0038] In some embodiments of this application, when identifying the modality of heterogeneous data based on data element information, the identification and parsing unit 1021 can specifically use data element information as initial assistance and utilize each sub-channel in the feature extraction layer of the multi-channel modality discriminant network to extract the single modality representation vector of the heterogeneous data. The multi-channel modality discriminant network is a lightweight deep learning model that needs to be pre-trained and supports learning on the job. Each sub-channel includes a text sub-channel, an image sub-channel, an audio sub-channel, and a structured sub-channel. For example, the text sub-channel can extract text features such as word frequency distribution, n-gram patterns, and nested syntactic structures; the image sub-channel can extract image features such as color histograms, texture features, resolution, and edge distribution; the audio sub-channel can obtain acoustic features such as Mel spectrograms and Mel frequency cepstral coefficients of audio signals; and the structured sub-channel can obtain structural features such as field arrangement patterns, nesting levels, key name distribution, and JSON / XML format density.

[0039] Then, the recognition and parsing unit 1021 can specifically utilize the attention enhancement layer of the multi-channel modality discriminant network to adjust the feature weights of each sub-channel, obtaining a mixed modality representation vector. This involves introducing a modality self-attention mechanism, which can handle situations where some data contains mixed modalities, such as image-text combinations or video-audio combinations. For example, when adjusting the feature weights of each sub-channel using the attention enhancement layer of the multi-channel modality discriminant network, the recognition and parsing unit 1021 can specifically generate a context vector for heterogeneous data. This means considering the input data as a whole and calculating the similarity between the context vector and each individual modality representation vector, such as cosine similarity, while converting the similarity into feature weights for each sub-channel. ,For example:

[0040] (1)

[0041] In equation (1), Indicates the first Cosine similarity of a single modality representation vector This represents the temperature coefficient, used to adjust the smoothness of the weight distribution. Represents the total cosine similarity. Represents the th element in the total number of cosine similarities. Cosine similarity. If the feature weights of the sub-channels... If the value is less than a preset threshold, the feature weight of the sub-channel will be reduced. For example, feature weights Multiply by an attenuation coefficient less than 1, thereby reducing interference from irrelevant modes.

[0042] Finally, the identification and parsing unit 1021 can specifically calculate the modality probability distribution corresponding to the mixed modality representation vector using the discriminant coding layer of the multi-channel modality discrimination network, for example, by outputting the modality probability distribution through Softmax or Sigmoid. , Represents the modal probability of the text. Represents the modal probability of an image. Represents the audio modal probability. Represents structured modal probabilities. It represents the probability of mixed modes, and modes with probabilities greater than or equal to a preset threshold value are considered as modes of heterogeneous data.

[0043] Furthermore, an incremental learning adaptation layer can be added after the feature extraction layer of the multi-channel modality discriminator network. This layer uses the modality prototype as a semantic benchmark and combines the extracted features of each sub-channel to adaptively update the distribution of new modalities. This supports catastrophic incremental training of new modalities and improves the robustness of modality recognition across time and data sources. For example, after a new modality sample is added, a contrastive learning method is used to adjust the original prototype spacing to avoid forgetting old modalities. At the same time, a knowledge distillation mechanism is introduced so that training new modalities does not destroy the existing model structure and supports training and inference simultaneously, making it suitable for real-time updates in actual data acquisition tasks.

[0044] In some embodiments of this application, when parsing heterogeneous data of different modalities to obtain standard data objects, for example, the identification and parsing unit 1021 can specifically perform HTML tag stripping, text segmentation and sentence segmentation, multilingual detection and encoding standardization on text data. For example, HTML tag stripping uses regular expression matching and HTML parsers such as Beautiful Soup to remove structural tags and retain pure text. Text segmentation and sentence segmentation uses language-adaptive sentence segmentation models such as NLTK and spaCy to split the text content according to semantic boundaries. Multilingual detection and encoding standardization use language recognition models such as FastText and CLD3 to label language types, uniformly encode to UTF-8 format, and unify uppercase and lowercase, symbol normalization, emoji escaping, and special character cleaning.

[0045] For example, the recognition and parsing unit 1021 can extract visual metadata such as EXIF ​​information, resolution, and color space from image data, including camera parameters, shooting time, and GPS coordinates; record and adjust the image to a fixed aspect ratio; recognize and convert it to standard RGB or YCbCr; and calculate basic visual features such as gray-level co-occurrence matrix and SIFT. Similarly, the recognition and parsing unit 1021 can extract acoustic features from audio data and call an automatic speech recognition model to generate corresponding text transcriptions. For example, it can extract features such as MFCC, Chroma, and SpectralContrast using Librosa or Open SMILE; call lightweight models such as Whisper-tiny and Wav2Vec2 to transcribe speech into text; identify language type and speaker labels to construct speech dialogue samples; and detect noise level and signal-to-noise ratio, providing optional quality indicators.

[0046] For example, the identification and parsing unit 1021 can specifically perform field semantic parsing and mapping on structured data, unifying field naming and data formats from different sources. For instance, it unifies field names through named entity recognition and field alignment rules, and automatically parses the field semantics of structured data from different sources based on a schema alignment method using multimodal field semantic embedding and distribution matching. This achieves accurate mapping and format unification of cross-source fields, effectively improving the automation and accuracy of multi-source data fusion. Another example is unifying time to ISO format, converting currency to a unified currency, unifying date strings to YYYY-MM-DD, merging synonymous fields such as phone and contact_number, deleting null values ​​or abnormal fields, and retaining field types, source tags, and optional original field mappings.

[0047] In some embodiments of this application, when unifying the semantic vectors of various standard data objects to obtain multimodal data pairs, the alignment evaluation unit 1022 can specifically encode each standard data object into a single semantic vector. For example, text is encoded into a semantic vector using a language model such as Transformer, images have visual semantic features extracted using a visual encoder such as CLIP, and audio is converted into a semantic vector using an acoustic encoder such as Wav2Vec. Then, the single semantic vector is mapped to a shared semantic space through a projection layer. This projection layer includes multiple linear projection sublayers with consistent structures. Each modal input corresponds to one projection sublayer. After weight transformation, a fixed-dimensional semantic vector is output, and optimization is performed in the shared semantic space using an alignment loss function to obtain multimodal data pairs.

[0048] For example, by introducing a contrastive learning mechanism to maximize the similarity of semantically consistent modal pairs while distancing semantically inconsistent modal pairs, data from different modalities can achieve high-quality alignment at the same semantic scale. This provides a unified representation foundation for subsequent cross-modal training, sample generation, and fusion. This process balances structural simplicity with cross-modal expressive power and is scalable. Then, an alignment loss function is used to reduce the distance between different modalities with the same semantic meaning, making semantically consistent multimodal pairs similar in a unified space and distancing inconsistent ones. For example, for semantically related multimodal pairs, such as text-image or text-audio pairs, their cosine similarity in a unified space is calculated, and contrastive learning is used to reduce the distance between positive sample pairs while distancing the distance between semantically unrelated negative sample pairs. Alignment loss function. It can be:

[0049] (2)

[0050] In equation (2), Indicates the first A single semantic vector for each modality. Indicates the first The first modality match A single semantic vector of a positive sample. This indicates the calculation of cosine similarity. Indicates the temperature coefficient. This represents the total number of individual semantic vectors. Represents the th element in the total number of single semantic vectors. A single semantic vector.

[0051] In some embodiments of this application, when calculating the quality score of a multimodal data pair, the alignment evaluation unit 1022 can specifically perform multi-indicator evaluation from dimensions such as content integrity, semantic consistency, noise interference level, modal consistency, and modal confidence. Each dimension is scored according to preset rules, and the quality score is calculated by weighted summation. For example, quality features of each modality in the multimodal data pair, such as text, image, and audio, can be extracted separately. These quality features include, but are not limited to, the number of non-empty standard fields, actual text length, actual image resolution, and actual audio duration. For example, for structured fields... , Indicates the number of non-empty standard fields. This indicates the number of preset standard fields, which is relevant for text. , It can be a linear mapping or a sigmoid mapping. Indicates the actual text length. This indicates the preset maximum text length. This indicates the preset minimum text length, while an existence indicator is used for images and audio. , The value is 1 if the content is present and 0 otherwise. Additionally, when the multimodal data pair contains images or audio, normalization can be performed by dividing the actual image resolution by a preset image resolution or the actual audio duration by a preset audio duration. It's understood that the preset value must encompass the actual value, ensuring the normalization result is between 0 and 1. This results in a normalization score for content integrity. ,in , , and It can be configured according to the specific task or the importance of the modality, making it more flexible.

[0052] For example, the values ​​of multimodal data pairs can be calculated separately. The semantic vector of each mode c in each mode. , And obtain the normalized score of semantic consistency based on equation (3). ,Right now:

[0053] (3)

[0054] In equation (3), ,if The result is the cosine similarity of the pair, and the result falls within the range of . The interval, and also for normalization to The interval can be mapped using equation (4), that is:

[0055] (4)

[0056] For example, the values ​​of multimodal data pairs can be calculated separately. The noise level of each modality within a given modality. For example, for text noise, the noise level of the text modality is obtained by statistically analyzing the proportion of non-printed characters, garbled characters, and misspelled characters in the text. , For image noise, the compression ratio is used as the noise level of the image mode. , And for audio noise, the reciprocal of the signal-to-noise ratio is used as the noise level of the audio mode. , Then the synthesized noise level ,in , , and The normalized score for noise interference can be set according to the specific task or the importance of the modality. .

[0057] For example, it can be determined whether a multimodal data pair covers the required modalities of the template by identifying the type of each modality. If it fully covers the modalities, then... The value is 1 if the expression is true and 0 otherwise. Simultaneously, multiple semantically closest samples are searched within the existing samples of the training set, and the cosine similarity between the multimodal data pair and these multiple samples is calculated. The maximum cosine similarity is then selected. To calculate , , In other words, the more similar a multimodal data pair is to existing samples, the lower its score, and consequently, the lower the normalized score for modality compatibility. ,in , and This can be configured according to specific tasks. Alternatively, the normalized score of modal confidence can be calculated based on the confidence scores returned by each modal component. , , , and All located in The interval, in which , , and Settings can be configured based on the specific task or the importance of the modality. Finally, a quality score is calculated. , , , , and The values ​​can be 0.25, 0.3, 0.2, 0.1, and 0.15, respectively, and can be adjusted according to the specific task.

[0058] In some embodiments of this application, before filling the multimodal data pairs into the corresponding slots of the semantically driven template, the filling unit 1031 can also obtain the target task input by the operation interface and generate the semantically driven template based on the target task. This improves task adaptability and can flexibly adapt to various task forms, such as image-text matching, audio-visual question answering, and multimodal completion. It achieves efficient and structured training sample construction without manual annotation, significantly reducing data preparation costs. For example, a semantically driven template is automatically generated or preset according to the target task, such as "Please describe the content of this image: <image>", or "After listening to this audio, can you summarize its main points? <audio>". Then, the multimodal data pairs are filled into the corresponding slots of the semantically driven template, generating natural language prompts and corresponding supervision signals, thereby constructing the input-output pairs required for the multimodal pre-training task, thus obtaining the target sample, and automatically attaching modal labels, semantic labels, and quality scores to meet diverse training needs.

[0059] In some embodiments of this application, the indexing unit 1032 can specifically establish a multi-level label and index structure according to dimensions such as modal labels, semantic labels, and quality scores, supporting fast retrieval through keywords, modal combinations, or semantic similarity. Furthermore, the indexing unit 1032 can also adjust the sample distribution based on a preset sampling strategy during the pre-training of large AI models, such as prioritizing the sampling of high-quality samples or samples from specific semantic domains, greatly improving training efficiency and model convergence speed. In addition, all data storage supports a distributed file system and allows online updates and hot loading of new samples during training, realizing a closed-loop linkage between model training and data evolution.

[0060] The heterogeneous data fusion device for pre-training large AI models provided in this application can quickly collect heterogeneous data from different sources and automatically add data metadata to the heterogeneous data. The heterogeneous data includes at least one of text data, image data, and audio data, which significantly broadens the data sources. Secondly, based on the data metadata, the modality of the heterogeneous data is accurately identified, and the heterogeneous data of different modalities are parsed to obtain each standard data object. This effectively solves the problem of large differences in the structure of heterogeneous data and lays the foundation for cross-modal semantic processing. Then, the semantic vectors of each standard data object are unified to achieve deep alignment, which improves semantic consistency and model generalization ability. Finally, the negative impact of noise and redundant data on the pre-training effect of large models is avoided by quantifying the quality score of multimodal data pairs. Finally, the multimodal data pairs are automatically populated into the corresponding slots of the semantically driven template to obtain the target samples. Labels and quality scores are then attached to the target samples. These labels include modal labels and semantic labels, eliminating the need for manual annotation and significantly improving sample generation efficiency. At the same time, the target samples are encapsulated into a standard data structure, and a multi-dimensional index of the target samples is constructed according to the labels and quality scores, improving retrieval efficiency and facilitating flexible access.

[0061] Based on the foregoing embodiments, this application provides a heterogeneous data fusion method for pre-training large AI models. This heterogeneous data fusion method can be used in the heterogeneous data fusion device 10 of the embodiment corresponding to FIG1. ​​Please refer to FIG2, which is a flowchart illustrating a heterogeneous data fusion method for pre-training large AI models provided in this application. The heterogeneous data fusion method includes:

[0062] S101, collect heterogeneous data from different sources and attach data metadata to the heterogeneous data. The heterogeneous data includes at least one of text data, image data and audio data. The data metadata includes file extension, encoding format and header binary signature.

[0063] S102, after identifying the modalities of heterogeneous data based on data element information, the heterogeneous data of different modalities are parsed to obtain each standard data object, and the semantic vectors of each standard data object are unified to obtain multimodal data pairs, and the quality score of the multimodal data pairs is calculated.

[0064] S103: Fill the corresponding slots of the semantically driven template with multimodal data pairs. After obtaining the target sample, attach labels and quality scores to the target sample. The labels include modal labels and semantic labels. Then, encapsulate the target sample into a standard data structure and construct a multi-dimensional index of the target sample according to the labels and quality scores.

[0065] It should be noted that the descriptions of the same steps and contents as in other embodiments in this embodiment can be found in the descriptions in other embodiments, and will not be repeated here.

[0066] The heterogeneous data fusion method for pre-training large AI models provided in this application can quickly collect heterogeneous data from different sources and automatically add data metadata to the heterogeneous data. The heterogeneous data includes at least one of text data, image data, and audio data, which significantly broadens the data sources. Secondly, based on the data metadata, the modality of the heterogeneous data is accurately identified, and the heterogeneous data of different modalities are parsed separately to obtain each standard data object. This effectively solves the problem of large differences in the structure of heterogeneous data and lays the foundation for cross-modal semantic processing. Then, the semantic vectors of each standard data object are unified to achieve deep alignment, which improves semantic consistency and model generalization ability. Finally, the negative impact of noise and redundant data on the pre-training effect of large models is avoided by quantifying the quality score of multimodal data pairs. Finally, the multimodal data pairs are automatically populated into the corresponding slots of the semantically driven template to obtain the target samples. Labels and quality scores are then attached to the target samples. These labels include modal labels and semantic labels, eliminating the need for manual annotation and significantly improving sample generation efficiency. At the same time, the target samples are encapsulated into a standard data structure, and a multi-dimensional index of the target samples is constructed according to the labels and quality scores, improving retrieval efficiency and facilitating flexible access.

[0067] Based on the foregoing embodiments, this application provides an electronic device. Referring to FIG3, the electronic device 20 may include a processor 201 and a memory 202. The memory 202 stores at least one instruction, at least one program, code set, or instruction set. The instruction, program, code set, or instruction set is loaded and executed by the processor 201 to implement the steps of the heterogeneous data fusion method of the embodiment corresponding to FIG2.

[0068] In another aspect, embodiments of this application provide a computer-readable storage medium for storing program code that executes any one of the implementation methods of the heterogeneous data fusion method corresponding to the aforementioned embodiment of FIG2.

[0069] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and modules described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0070] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed between each other can be through some interfaces, indirect coupling or communication connection between devices or modules, and can be electrical, mechanical, or other forms. Modules described as separate components may or may not be physically separate, and components shown as modules may or may not be physical units, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0071] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each module can exist physically separately, or two or more units can be integrated into one module. The integrated unit can be implemented in hardware or as a software functional unit. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0072] Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the heterogeneous data fusion method of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0073] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0074] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A heterogeneous data fusion device for pre-training large AI models, characterized in that, The heterogeneous data fusion device includes a data acquisition module, a processing module, and an output module connected in sequence. The data acquisition module acquires heterogeneous data from different sources and adds metadata to the heterogeneous data. The heterogeneous data includes at least one of text data, image data, and audio data. The metadata includes file extensions, encoding formats, and header binary signatures. The processing module includes a recognition and parsing unit and an alignment evaluation unit. The recognition and parsing unit identifies the modalities of the heterogeneous data based on the metadata and parses the heterogeneous data of different modalities to obtain standard data objects. The alignment evaluation unit unifies the semantic vectors of the standard data objects to obtain multimodal data pairs and calculates the quality score of the multimodal data pairs. The output module includes a filling unit and an indexing unit. The filling unit fills the multimodal data pairs into the corresponding slots of the semantically driven template to obtain target samples and outputs them to the target data. The target sample is labeled with a quality score, the labels including modality labels and semantic labels; the indexing unit is used to encapsulate the target sample into a standard data structure and construct a multi-dimensional index of the target sample according to the labels and the quality score; wherein, when identifying the modality of the heterogeneous data based on the data metadata, the identification and parsing unit is specifically used to extract the single modality representation vector of the heterogeneous data by using each sub-channel in the feature extraction layer of the multi-channel modality discriminant network, the sub-channels including text sub-channel, image sub-channel, audio sub-channel and structured sub-channel, and to adjust the feature weights of each sub-channel by using the attention enhancement layer of the multi-channel modality discriminant network to obtain a mixed modality representation vector; and to calculate the modality probability distribution corresponding to the mixed modality representation vector by using the discriminant coding layer of the multi-channel modality discriminant network, and to take the modality with a probability greater than or equal to a preset threshold value as the modality of the heterogeneous data.

2. The heterogeneous data fusion device according to claim 1, characterized in that, When adjusting the feature weights of each sub-channel using the attention enhancement layer of the multi-channel modality discriminant network, the recognition parsing unit is specifically used to generate the context vector of the heterogeneous data; calculate the similarity between the context vector and each single modality representation vector, and convert the similarity into the feature weight of the sub-channel. If the feature weight of the sub-channel is less than a preset threshold, the feature weight of the sub-channel is reduced.

3. The heterogeneous data fusion device according to claim 1, characterized in that, The multi-channel modality discrimination network is followed by an incremental learning adaptation layer after the feature extraction layer. The incremental learning adaptation layer uses the modality center vector prototype as a semantic benchmark and combines the extracted features of each sub-channel to adaptively update the new modality distribution.

4. The heterogeneous data fusion device according to claim 1, characterized in that, When unifying the semantic vectors of the standard data objects to obtain multimodal data pairs, the alignment evaluation unit is specifically used to encode each of the standard data objects into a single semantic vector; map the single semantic vector to a shared semantic space through a projection layer, the projection layer including multiple linear projection sub-layers with consistent structure, and optimize it in the shared semantic space using an alignment loss function to obtain the multimodal data pairs.

5. The heterogeneous data fusion device according to claim 4, characterized in that, The alignment loss function for: In the above formula, Indicates the first A single semantic vector for each modality. Indicates the first The first modality match A single semantic vector of a positive sample. This indicates the calculation of cosine similarity. Indicates the temperature coefficient. This represents the total number of individual semantic vectors. Represents the th element in the total number of single semantic vectors. A single semantic vector.

6. The heterogeneous data fusion apparatus according to any one of claims 1 to 5, characterized in that, Before filling the multimodal data pairs into the corresponding slots of the semantic-driven template, the filling unit is also used to obtain the target task input by the operation interface and generate the semantic-driven template based on the target task.

7. The heterogeneous data fusion device according to claim 6, characterized in that, The indexing unit is also used to adjust the sample distribution based on a preset sampling strategy during the pre-training of the AI ​​large model.

8. A heterogeneous data fusion method for pre-training large AI models, characterized in that, The heterogeneous data fusion method is used in any one of the heterogeneous data fusion apparatuses of claims 1 to 7. The heterogeneous data fusion method includes: collecting heterogeneous data from different sources and attaching data metadata to the heterogeneous data, wherein the heterogeneous data includes at least one of text data, image data, and audio data, and the data metadata includes file extensions, encoding formats, and header binary signatures; identifying the modalities of the heterogeneous data based on the data metadata, parsing the heterogeneous data of different modalities to obtain standard data objects, unifying the semantic vectors of the standard data objects to obtain multimodal data pairs, and calculating the quality score of the multimodal data pairs; filling the multimodal data pairs into corresponding slots of a semantically driven template to obtain target samples, and attaching labels and the quality score to the target samples. The labels include modal labels and semantic labels, and the target samples are encapsulated into a standard data structure. A multi-dimensional index of the target samples is constructed according to the labels and the quality scores. When identifying the modality of the heterogeneous data based on the data metadata, the single modality representation vector of the heterogeneous data is extracted by each sub-channel in the feature extraction layer of the multi-channel modality discriminant network. Each sub-channel includes a text sub-channel, an image sub-channel, an audio sub-channel, and a structured sub-channel. The feature weights of each sub-channel are adjusted by the attention enhancement layer of the multi-channel modality discriminant network to obtain a mixed modality representation vector. The modality probability distribution corresponding to the mixed modality representation vector is calculated by the discriminative coding layer of the multi-channel modality discriminant network, and the modality with a probability greater than or equal to a preset threshold value is taken as the modality of the heterogeneous data.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs, which can be executed by one or more processors to implement the steps of the heterogeneous data fusion method of claim 8.

Images