Brain-like multi-modal model end-to-end pre-training device and method

Abstract

The application belongs to the technical field of brain-like and multi-modal artificial intelligence, and particularly relates to a brain-like multi-modal model end-to-end pre-training device and method. The device comprises a multi-modal information source module, a multi-modal perception module, a multi-modal data processing module, a multi-modal data distillation module and a multi-modal model training module. The method comprises the following steps: observing a target to generate non-human factor information and interacting with a human to generate human factor information; acquiring human factor data and non-human factor data by a perception unit; processing the data by a data processing unit and integrating the data in a same storage space by a data integration unit; repeating the process to obtain a multi-modal data set; removing multi-modal data with inconsistent human factor semantics and non-human factor semantics from the data set by a multi-modal data distillation module; and completing brain-like multi-modal model training by using the distilled data set. The application migrates the multi-modal information representation and alignment mechanism in the human brain to the multi-modal model, effectively improving multi-modal alignment and cross-modal conversion performance.

Description

Technical Field

[0001] This application belongs to the field of brain-like and multimodal artificial intelligence technology, and in particular relates to an end-to-end pre-training device and method for brain-like multimodal models. Background Technology

[0002] Large-scale language models (LLMs) represent a new stage in the development of Natural Language Processing (NLP), signifying the shift in artificial intelligence from machine learning and deep learning towards Artificial General Intelligence (AGI). However, humans rely on multiple senses to jointly perceive the external environment, integrating information from various modalities to form a holistic and comprehensive understanding. Therefore, the next stage of development for large language models is multimodal large-scale models, which possess semantic understanding and reasoning capabilities across multiple modalities.

[0003] Currently, research on large-scale multimodal models is largely limited to pairwise modal matching of text-image, text-speech, or speech-image, and often uses text as the alignment benchmark. However, text struggles to provide a rich and global representation of other modal information. Nevertheless, the modal types of information are far more diverse than this, including electromagnetic waves and emotions. Continuously expanding the types of information modalities that large-scale models can process and improving their ability to jointly process multiple modalities is an inevitable trend.

[0004] Semantic alignment and interconversion of signals from different modalities are key issues that need to be addressed to achieve the aforementioned trends. Large-scale models typically employ a training paradigm of "pre-training + fine-tuning." Pre-training is a crucial technical process for models to internalize general knowledge and generalize across multiple modalities, and it is a prerequisite for model performance to generalize to specific tasks. Therefore, there is an urgent need to propose a pre-training solution that can unify, synthesize, and align multi-modal data as comprehensively as possible. Summary of the Invention

[0005] To address the problems existing in the prior art, this application provides an end-to-end pre-training device and method for brain-like multimodal models, aiming to provide a brain-like pre-training method and model framework for the multimodal alignment problem in the field of multimodal cognitive computing. It proposes a multimodal data distillation module for removing inconsistencies between human and non-human semantics in multimodal data; introduces a feature alignment unit in the multimodal training model for aligning human and non-human features; and extends this to a cross-modal conversion method, thereby promoting the development of multimodal general artificial intelligence.

[0006] Technical solution of the present invention:

[0007] A brain-like multimodal model pre-training device, the device comprising: a multimodal information source module (1), a multimodal perception module (2), a multimodal data processing module (3), a multimodal data distillation module (4), and a multimodal model training module (5);

[0008] The multimodal information source module (1) includes human-cause information sources and several non-human-cause information sources, which are used to provide human-cause information and non-human-cause information under the same observation target;

[0009] The multimodal sensing module (2) includes a human factor sensing unit and a non-human factor sensing unit, which are used to sense the human factor information and non-human factor information of the multimodal information source module, respectively, and convert them into data, thereby obtaining human factor data and non-human factor data under the same observation target, and providing them to the multimodal data processing module.

[0010] The multimodal data processing module (3) includes a human factors data processing unit, a non-human factors data processing unit and a data integration unit, which are used to process human factors data and non-human factors data respectively, and integrate the two into multimodal data, which is then provided to the multimodal data distillation module.

[0011] The multimodal data distillation module (4) is used to remove multimodal data that is inconsistent between human-cause semantics and non-human-cause semantics; the distilled multimodal data is provided to the multimodal model training module.

[0012] The multimodal model training module (5) is used to complete the training process of the brain-like multimodal model, including a data encoding unit, a feature extraction unit, a feature alignment unit, a cross-modal feature conversion unit, and a data output unit.

[0013] It should be noted that the human information source refers to the observer, generally referring to humans, and the causal information generated by the interaction between the observer and the observed target is defined as human information; the observed target generally refers to an objective entity or event, which can itself serve as a non-human information source to spontaneously generate non-human information; the data obtained by the sensing unit from perceiving human information is called human data, and the data obtained from perceiving non-human information is called non-human data.

[0014] This application provides a method for pre-training a brain-like multimodal model, the method comprising the following steps:

[0015] Step S101: In the multimodal information source module (1), the observer interacts with the observation target to generate human-cause information, while the observation target spontaneously generates non-human-cause information; the human-cause perception unit of the multimodal perception module (2) acquires human-cause data; the corresponding modal perception unit is selected according to the modal type of the non-human-cause information generated by the observation target to acquire the corresponding modal non-human-cause data; the observation target is changed and the above interaction and perception process is repeated to obtain a large amount of human-cause data and non-human-cause data;

[0016] It should be noted that, for the same observation target, the semantics carried or represented by the human and non-human data acquired during the interaction and perception process are similar or consistent.

[0017] In step S102, human factors data and non-human factors data are transmitted to the multimodal data processing module, where they are processed by the human factors data processing unit and the non-human factors data processing unit, respectively. Then, the data integration unit integrates the human factors data and non-human factors data obtained under the same observation target into the same storage space, i.e., the storage space is different for different observation targets. The above processing and integration process is repeated to obtain a multimodal dataset.

[0018] Step S103: If the semantics carried or represented by the human factors data and non-human factors data in a set of multimodal data are inconsistent, the multimodal data distillation module filters out the set of multimodal data, thereby improving the quality of the multimodal dataset.

[0019] Step S104: The distilled multimodal dataset is transmitted as training data of the brain-like multimodal model to the multimodal model training module to complete the pre-training process of the brain-like multimodal model.

[0020] This application also provides an electronic device, the electronic device comprising: a processor and a memory connected to the processor; the memory storing computer program instructions executable by the processor, the computer program instructions being executed by the processor to enable the processor to execute the neuromorphic multimodal model pre-training method provided in this application.

[0021] This application also provides a readable storage medium storing a computer program for execution by a processor to implement the neuromorphic multimodal model pre-training method provided in this application.

[0022] Beneficial effects of this invention:

[0023] Addressing the major technical bottlenecks in the development of multimodal models and general artificial intelligence, this application provides a brain-inspired intelligence solution that organically combines humans, physical reality, and multimodal information. Specifically, considering that the human brain possesses a natural, mature, and unified semantic representation mechanism and feature alignment mechanism for observing different modalities and providing different modal information in physical reality, this application effectively and indirectly empowers the training process of brain-inspired multimodal models with these representation and alignment mechanisms, thereby achieving semantic unification and alignment in the computer feature space and cross-modal conversion. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the structure of a brain-like multimodal model pre-training device according to an embodiment of the present invention;

[0025] Figure 2 This is a schematic diagram of the structure of the multimodal information source module, the multimodal sensing module, and the multimodal data processing module according to an embodiment of the present invention;

[0026] Figure 3 This is a schematic diagram of the structure of the multimodal data distillation module according to an embodiment of the present invention.

[0027] Figure 4 This is a schematic diagram of the structure of the multimodal model training module according to an embodiment of the present invention;

[0028] Figure 5 This is a flowchart illustrating a brain-like multimodal model pre-training method according to an embodiment of the present invention;

[0029] Figure 6 This is a schematic diagram illustrating the construction process of the multimodal data distillation module according to an embodiment of the present invention;

[0030] Figure 7 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0031] The technical solutions provided in this application will be further described below with reference to specific embodiments and accompanying drawings. The advantages and features of this application will become clearer with the following description.

[0032] Figure 1 The specific structure and connection relationship between the modules of a brain-like multimodal model pre-training device provided in this application embodiment include a multimodal information source module (1), a multimodal perception module (2), a multimodal data processing module (3), a multimodal data distillation module (4), and a multimodal model training module (5);

[0033] Figure 2 A schematic diagram of the structure of the multimodal information source module, the multimodal perception module, and the multimodal data processing module provided in the embodiments of this application, as well as the connection relationships between modules and units;

[0034] In the multimodal information source module, human-caused information sources interact with non-human-caused information sources, i.e., the observer interacts with the observed target; the interaction methods include sensory perception, semantic understanding, and semantic reproduction; the non-human-caused information spontaneously generated by the observed target includes, but is not limited to, text, one-dimensional time series (sound, electromagnetic waves), images, videos, etc.; human-caused information generated during the interaction process is specifically defined in this embodiment as the semantic mapping of the non-human-caused information generated by the observed target in the human brain; therefore, human-caused information is specifically the information transmitted by the observer's neurons, including but not limited to electroencephalography (EEG), etc.

[0035] It should be noted that observers must be physically healthy, possess basic common sense and cognitive abilities, maintain a seated or lying posture as much as possible during the observation process, and provide objective and truthful human factors information reflecting their perception, understanding, and reproduction of the observed target. In addition, in this embodiment, for non-human factors information generated by the observed target, only the modality with the most information is considered as non-human factors data. It is generally believed that the order of information content is video > image > one-dimensional time series > text.

[0036] The multimodal sensing module includes human-cause sensing units and non-human-cause sensing units.

[0037] As an example, the human factor sensing unit uses an EEG sensing device, including: an EEG sensor, a signal amplifier, and an A / D converter. Here, the EEG sensor uses non-invasive electrodes that directly contact the observer's scalp through conductive adhesive to sense brain waves. The number and placement of the electrodes can refer to the international 10-20 or 10-10 standard lead system. The sensed brain waves are amplified and converted into digital signals, i.e., human factor data, with a sampling frequency of 500Hz and d channels.

[0038] On the other hand, the non-human factor sensing unit includes non-human factor information sensing units of various modal types. The sensing unit is selected according to the modal type of the non-human factor information, that is, the i-th modal information corresponds to the i-th modal sensing unit, i = 1, ... m, where m is the total number of non-human factor information modal types; for example, sound information corresponds to sound sensor (microphone), visual information corresponds to image sensor (camera), etc.

[0039] The multimodal data processing module is used to process human factors data and non-human factors data, and integrate the two into multimodal data.

[0040] For human factors data, the human factors data processing unit sequentially performs operations such as filtering, temporal alignment, data truncation, baseline correction, artifact removal, and outlier removal.

[0041] As an embodiment, the filtering process includes: 8Hz high-pass filtering to attenuate brainwaves not generated by the observed target; 100Hz low-pass filtering to attenuate high-frequency noise; 48 to 52Hz notch filtering to attenuate power frequency interference. Finally, the main effective frequency bands of the human factor data are distributed in the ranges of 8 to 48Hz and 52 to 100Hz, that is, attention is paid to the Alpha wave, Beta wave and Gamma wave generated by the observer due to interaction;

[0042] Time domain alignment and data truncation: For the same observed target, align the start time and end time of the human factor data and non-human factor data in the time domain during the observation process, and only focus on the human factor data generated during the period when the observer is subjected to the non-human factor information and during the reaction period, and truncate the data of the remaining periods;

[0043] Baseline correction, taking the mean value of the human factor data when the observer is in a calm state in the above remaining periods as the baseline, and subtracting the mean value from the human factor data in the above action period and reaction period;

[0044] Artifact rejection: For the human factor data of d channels, the Independent Component Analysis (ICA, prior art) algorithm is used to obtain the first s (s<d) independent components, and the corresponding artifact-related components are rejected according to the standard reference data and distribution of electrooculogram artifacts, tongue electromyogram artifacts, electromyogram artifacts, etc., and the remaining components are recombined, so as to attenuate the influence of the above artifacts;

[0045] Outlier rejection: reject data points whose amplitude exceeds the threshold.

[0046] For the non-human factor data processing unit, filtering, time domain alignment, data truncation and outlier rejection are performed in sequence; the specific settings of filtering are determined according to the corresponding modal data, and low-pass filtering is generally used to reduce the influence of high-frequency noise or median filtering to smooth the data; align the start time and end time of the non-human factor data and the human factor data in the time domain during the observation process, and only focus on the non-human factor data during the period when the observer is subjected to the non-human factor information and during the reaction period, and truncate the data of the remaining periods; reject data points whose amplitude exceeds the threshold;

[0047] Data integration unit, which combines the human factor data and non-human factor data within the time domain of action of the same observed target into a set of multi-modal data, and stores it in the same storage space; further, the multi-modal data acted by different observed targets form a multi-modal dataset;

[0048] as Figure 3As shown, the multimodal data distillation module is used to remove multimodal data with inconsistent human and non-human semantics. It is a value-based reinforcement learning model, which includes: data encoding unit, feature extraction unit, agent unit, and environment. (This module can also be implemented based on a classification model; only one reinforcement learning scheme is given here.)

[0049] The data encoding unit is used to encode multimodal data into computable vectors; the multimodal data includes human factors data and non-human factors data of a certain modality; wherein, human factors data is encoded by the human factors data encoding unit; and non-human factors data is encoded by the corresponding modality data encoding unit according to the data modality type, for example, the first modality data is encoded by the first modality data encoding unit; the unit performs equal-separation discretization, vector encoding, and temporal encoding or temporal and positional encoding on the corresponding data; for example, temporal encoding is performed on text modality data and one-dimensional temporal modality data in human factors data and non-human factors data, and temporal and positional encoding is performed on image modality data and video modality data in non-human factors data;

[0050] The feature extraction unit is used to extract semantic features of the data, including a human factor feature extraction unit and a non-human factor data feature extraction unit. The input of the feature extraction unit is the output of the corresponding data encoding unit. Specifically, human factor data is extracted by the human factor feature extraction unit; non-human factor data is extracted by selecting the corresponding feature extraction unit according to the data modality type, for example, the first modality data is extracted by the first modality data encoding unit; human factor features and non-human factor features are extracted respectively.

[0051] The intelligent agent unit, corresponding to the modal type of non-human information, contains m agents. Each agent takes human features extracted by the feature extraction unit and non-human features of the corresponding modality as input. For example, the first modality feature extraction unit outputs to the first agent. The function of the agent is to evaluate the semantic relevance of human and non-human data based on the above features, output the relevance level to the environment, and receive a reward from the environment. The relevance levels include E level (relevance <50%), D level (relevance 50-70%), C level (relevance 70-80%), B level (relevance 80-90%), and A level (relevance 90-100%), that is, the action set of the agent is {A, B, C, D, E}.

[0052] The environment includes a multimodal dataset with known true relevance levels. The reward is calculated based on the true relevance level and the relevance level of the agent's output and fed back to the agent.

[0053] The learning objective of the agent is to minimize the deviation between the agent's output and the true level to obtain more rewards, thereby optimizing the agent's output; furthermore, the trained multimodal data distillation module can evaluate multimodal data with unknown relevant levels and retain multimodal data with relevant level A, thus completing the data distillation process;

[0054] The multimodal model training module, i.e., the structure and internal unit connections of the brain-like multimodal model, is as follows: Figure 4 As shown, it consists of a data encoding unit, a feature extraction unit, a feature alignment unit, a cross-modal feature conversion unit, and a data output unit;

[0055] It should be noted that the data encoding unit and feature extraction unit in this module are not the same units as those in the multimodal data distillation module, but the unit inputs, unit outputs and function settings are the same, which will not be described in detail here;

[0056] The feature alignment unit functions to map non-human features of a certain modality from the original modality's feature space to the human feature space, based on human features, for the same observed target. This aligns the non-human features with the semantic representation mechanism of the human brain, resulting in aligned non-human features. Corresponding to the modality type of non-human information, m modality feature alignment units are involved. Each modality feature alignment unit takes the outputs of the human feature extraction unit and the corresponding non-human feature extraction unit as input. For example, the outputs of the human feature extraction unit and the first modality feature extraction unit are input to the first modality feature alignment unit; or the output of the cross-modality feature transformation unit is used as input. Furthermore, there are some shared weight parameters among different modality feature alignment units.

[0057] It should be noted that the weight parameters refer to the learnable parameters in the modality feature alignment unit (such as the weights and biases of convolutional and fully connected layers in a CNN network). In the feature alignment unit, features obtained from data with different modalities but the same semantics have commonalities (e.g., descriptions of physical laws and geometric relationships), and can be processed using the same weight parameters. Therefore, feature alignment units of different modalities can share some weight parameters.

[0058] The data output unit functions to decode the aligned non-human features into non-human data of the corresponding modality. The data output unit takes the output of the corresponding feature alignment unit as input; for example, the first feature alignment unit corresponds to the first data output unit.

[0059] The cross-modal feature conversion unit takes the output of the feature extraction unit as input; its function is to convert the non-human features of a certain modality into non-human features of the target modality.

[0060] Figure 5A flowchart illustrating a brain-like multimodal model pre-training method provided in an embodiment of this application is shown.

[0061] Step S101: In the multimodal information source module (1), the observer interacts with the observation target to generate human-cause information, while the observation target spontaneously generates non-human-cause information; the human-cause perception unit of the multimodal perception module (2) acquires human-cause data; the corresponding modal perception unit is selected according to the modal type of the non-human-cause information generated by the observation target to acquire the corresponding modal non-human-cause data; the observation target is changed and the above interaction and perception process is repeated to obtain a large amount of human-cause data and non-human-cause data;

[0062] Specifically, for a given observation target, such as a video of duration T or time-varying image information, the human-caused perception unit and the video modal perception unit respectively obtain human-caused data h(t) and non-human-caused data u(t), where h(t) has d channels and a sampling frequency of 500Hz, and u(t) has a sampling frequency of 50Hz. For the observation target set consisting of all observation targets, the same observer can interact with all observation targets, or different observers can interact with different observation targets. Finally, the human-caused dataset H and the non-human-caused dataset U are obtained.

[0063] In step S102, human factors data and non-human factors data are transmitted to the multimodal data processing module, where they are processed by the human factors data processing unit and the non-human factors data processing unit, respectively. Then, the data integration unit integrates the human factors data and non-human factors data obtained under the same observation target into the same storage space, i.e., the storage space is different for different observation targets. The above processing and integration process is repeated to obtain a multimodal dataset.

[0064] Specifically, taking the data obtained from a certain observation target as an example, for human-caused data h(t), after filtering, the frequency domain is mainly distributed in the range of 8-48Hz and 52-100Hz. Then, after time-domain alignment and data truncation, t∈[T0,T], where T0 is the start time of the observation process, baseline correction, artifact removal, and outlier removal are then performed. For non-human-caused data u(t), each frame of video data is first smoothed after median filtering, then time-domain alignment and data truncation are performed to the same time domain range, and then outlier removal is performed. After processing, human-caused data h(t) and non-human-caused data u(t) are stored in the same storage space, forming a set of multimodal data {h(t), u(t)}. Furthermore, a large number of multimodal data are used to form a multimodal dataset {H, U}.

[0065] It should be noted that in each set of multimodal data in the multimodal dataset {H, U}, h(t) and u(t) must be data obtained for the same observation target.

[0066] Step S103: The multimodal dataset is processed by a multimodal data distillation module to remove multimodal data inconsistent between human and non-human semantics, thereby improving the quality of the multimodal dataset.

[0067] The construction process of the multimodal data distillation module is as follows: Figure 6 As shown, the specific steps are as follows:

[0068] Step S103-1: Construct a multimodal dataset with known correlation levels for training the agent.

[0069] For human factors data, unlike step S101 where only one observation process involves one observation target, in this step, one observation process involves two or more observation targets, and the proportion of time the observer interacts with a certain observation target is set according to the relevant level.

[0070] For example, given an observation process [T1, T2], the observation time of the primary observation target is w1*(T2-T1), and the observation time of the non-primary observation targets (one or more) is w2*(T2-T1). If the correlation level is set to E, then w1 = 0.4 and w2 = 0.6 are selected. If the correlation level is set to A, then w1 = 0.9 and w2 = 0.1 are selected. The method or order of switching observation targets is not specifically limited here.

[0071] For non-human data, within the same time domain, the corresponding modal sensing unit is used to sense only the main observation target, and the sensing time is (T2-T1). That is, the semantics of non-human data only involves the information generated by the main observation target.

[0072] Step S103-2: Set different time percentage weights, repeat step S103-1 to obtain multiple multimodal datasets with known relevance levels, and label the relevance levels of the multimodal datasets according to the set time percentage weights. After merging, obtain a multimodal dataset {H′, U′, L} with known relevance levels, where L is the relevance level. As an example, in {H′, U′, L}, the proportion of A-level multimodal data is set to 40%, B-level to 25%, C-level to 20%, D-level to 10%, and E-level to 5%.

[0073] It should be noted that a portion of the observation target set used in step S102 to construct the multimodal dataset {H, U} is used to construct the training data {H′, U′, L}.

[0074] It should be noted that in a set of multimodal data {h′, u′, l}∈{H′, U′, L}, the main observation target of h′ is the same as the observation target of u′;

[0075] Step S103-3: {H′, U′, L} is processed by the data encoding unit and the feature extraction unit to obtain human factors features and non-human factors features; for example, a set of multimodal data {h′, u′, l}, where l is the relevance level; for human factors data h′, the human factors data encoding unit first discretizes it into N sub-data with the same sequence length to obtain the data sequence. N is determined based on the total duration of the observation process, and then encoded by vector E. h′ Convert a data sequence into a computable vector. Finally, a temporal code S is added to characterize the temporal information of the sub-data in the data sequence, resulting in x. h′ =V h′ +S;

[0076] For non-human data u′(t), taking video modality as an example, it is also temporally separated into N image sequences. Then each image is equally divided into M block sequences. Then it is encoded by vector E u′ get Then, a position code P is added to characterize the sub-block u′. ij In block sequence The positional information of the sub-image in the image sequence is used, along with the addition of a temporal code S to characterize the temporal information of the sub-image within the image sequence, to obtain x. u′ =V u′ +P+S;

[0077] The x h′ and x u′ The features are processed separately by a human factor feature extraction unit and a non-human factor feature extraction unit; the human factor feature extraction unit and the non-human factor feature extraction unit can be composed of multiple layers of Transformer blocks stacked together, with the output features of the first layer serving as the input of the second layer, and so on; the number of stacking layers T is set according to the input data, for example, 3 layers for human factor data and 6 layers for non-human factor data of video modalities; thus, the human factor feature f is obtained. h′ =T *3 (x h′ ) and non-human characteristics f u′ =T *6 (x h′ );

[0078] It should be noted that Transformer blocks are a commonly used technique in the field of artificial intelligence, and will not be discussed in detail here; similarly, they can be replaced by convolutional neural networks or spiking neural networks (SNNs).

[0079] Steps S103-4: Human and non-human features are input into the corresponding agent, and the agent is trained by optimizing the reward function;

[0080] For example, based on the modal type of non-human features, the f mentioned in the previous example... h′ and f u′ The data are input to the agent corresponding to the video modality. The agent evaluates the correlation between human-caused data and non-human-caused data, outputs a correlation level y, and maximizes its own reward r. The reward function is set as follows:

[0081]

[0082] Where p is the probability that the agent corresponding to the video modality makes the correct evaluation. If the level is truly relevant, and the other agents are similar, then the overall reward is the sum of the rewards of all agents, and the overall reward is maximized through continuous iterative training.

[0083] Steps S103-5: For the multimodal dataset {H, U} with unknown relevance levels, the agent trained in the above iterations evaluates the dataset, filters and retains multimodal data with a relevance level of A, and obtains the distilled multimodal dataset.

[0084] Step S104, the distilled multimodal dataset {H - U - The data is processed by the data encoding unit, feature extraction unit, feature alignment unit, cross-modal feature conversion unit, and data output unit of the multimodal model training module, and the weight parameters of the above units are continuously adjusted according to the set loss function to complete the training of the brain-like multimodal model.

[0085] The data encoding unit and feature extraction unit of the multimodal model training module are identical in structure and function to the data encoding unit and feature extraction unit of the multimodal data distillation module in step S103, but the weight parameters are different.

[0086] For example, taking multimodal data composed of human factors data and video modal non-human factors data involved in steps S101, S102, and S103 as an example; the processing procedures of the data encoding unit and the feature extraction unit will not be described in detail, and human factors features are obtained after processing. Non-human characteristics

[0087] Furthermore, the feature alignment unit is composed of stacked Transformer blocks; for example, the feature alignment unit corresponding to the video modality is set to 3 layers, based on human factors features. Non-human characteristics As input, the output of each layer becomes the input of the next layer, and so on, to obtain the aligned non-human factors features.

[0088] Furthermore, the data output unit is composed of multiple layers of Transformer blocks T stacked together, with the output of the corresponding feature alignment unit as input; for example, the data output unit corresponding to the video modality is set to 2 layers, with the aligned non-human factors as input, to obtain the video modality data d. video =T *2 (y);

[0089] The training process for the data encoding unit, feature extraction unit, feature alignment unit, and data output unit in the multimodal model training module provided in this application, i.e., the training process of the brain-like multimodal model, is as follows:

[0090] S104-1, Training data encoding unit, feature extraction unit;

[0091] Specifically, the data encoding unit and the feature extraction unit can be trained on the corresponding single modality data. For example, the first modality data encoding unit and the first modality feature extraction unit are trained in a self-supervised manner only on a large amount of unlabeled data of the first modality type, and the same applies to the other modalities.

[0092] It should be noted that only the parameters of the vector encoding in the data encoding unit need to be trained. The training objective is to map the data into a computable vector suitable for the feature extraction unit. The training objective of the feature extraction unit is to obtain the ability to fully extract shallow texture features and deep semantic features of the data.

[0093] It should be noted that self-supervised training is a common technique in the field of artificial intelligence. Data encoding and feature extraction are universal, and the loss function used in self-supervised training will not be designed in detail here.

[0094] S104-2, Training Feature Alignment Unit;

[0095] Based on the distilled multimodal dataset {H - U -}, semantically similar or consistent multimodal data are used as the training data for each modality feature alignment unit; for example, taking the i-th modality as an example, according to the functional description of the i-th modality feature alignment unit, the non-human features obtained by the aforementioned unit are... The i-th modal feature space is approximated and aligned to the human factor feature space, and the final output is the aligned non-human factor feature y. i To achieve the above objectives, a contrastive learning loss function is adopted. Here, taking the alignment of the i-th modality's non-human features with human features as an example, the loss function is designed as follows:

[0096]

[0097] Where the function This is used to assess the similarity of two features in their statistical distributions. Cosine similarity can be used for calculation; the higher the similarity, the closer the function value is to 1, and vice versa. Therefore, for a batch size of b containing b groups of multimodal data during training, the total contrastive loss is as follows:

[0098]

[0099] Where α1 represents the number of multimodal data points in a batch_size that are aligned from the first modality feature to the human factor feature, and so on. Therefore, ∑α i =b; Thus, through the feature alignment unit, the brain-like multimodal model aligns the features extracted from non-human data of different modalities into the human feature space;

[0100] It should be noted that the weight parameters of the above data encoding unit, feature extraction unit, and feature alignment unit remain unchanged after training is completed.

[0101] S104-3, Training data output unit;

[0102] Without loss of generality, for a certain non-human data u1∈U, without limiting its mode, similar to the method described in step S103, it is equally divided into N sub-data of the same size to obtain a data sequence. By predicting the loss function This enables the data output layer to generate data; specifically, it encodes, extracts features, and aligns the jq-th to j-th sub-data of the data sequence to obtain non-human features y. (i-q:i) This enables the brain-like multimodal model to be based on y (j-q:j) Able to predict the i-th sub-data u 1(j+1) The prediction loss function is designed as follows:

[0103]

[0104] Where q represents the length of the context window on which the brain-like multimodal model's prediction data depends. <j,P(u 1(j+1) |y (j-q:j) ) represents the conditional probability of predicting the i-th sub-data.

[0105] For a batch size of b non-human data points during training, the total prediction loss is... It is the sum of the prediction losses of all non-human factor data;

[0106] In summary, by combining self-supervised training commonly used in the field of artificial intelligence, and by continuously adjusting the weight parameters of the data encoding unit, feature extraction unit, feature alignment unit, and data output unit of the multimodal model training module (i.e., the brain-like multimodal model) using the contrastive learning loss function and prediction loss function provided in this application, the total loss function is:

[0107] It should be noted that the brain-like multimodal model provided in this application is mainly used for multimodal data generation. It can also be used to design other downstream task interfaces and corresponding loss functions to replace the data output unit, thereby realizing other types of downstream tasks, such as classification tasks.

[0108] Furthermore, if the brain-like multimodal model determines that cross-modal transformation is required based on the model input, then the function of the cross-modal feature conversion unit is executed; for example, generating an image scene based on text description, converting from text modality to image modality; or summarizing an image scene, converting from image modality to text modality. The functional implementation process is as follows:

[0109] The cross-modal feature transformation unit does not contain weight parameters that need to be trained, so it does not need to participate in the training process of the brain-like multimodal model;

[0110] For ease of explanation, let's assume, for example, that the distilled multimodal dataset {H} - U - In the context of}, there are n sets of multimodal data for the i-th modality, which are non-human factors data, i.e., there are n sets of non-human factors features and human factors features (f) for the i-th modality. i u，i f i h ), i = 1, ..., n, and the two features in each group are semantically similar or consistent;

[0111] It should be noted that when the weight parameters of the brain-like multimodal model are fixed, the latent pattern M mapped from human factor data to human factor characteristics... h It is fixed, representing the latent pattern mapped from the non-human data of the i-th modality to non-human features. It is also fixed; thus, it approximately reflects the inherent differences in information representation between human factors and non-human factors of the i-th modality. Furthermore, this inherent difference does not involve semantic differences; the approximation method is as follows: For inherent differences The n observations are used to eliminate the semantics of two features with the same semantics, thereby eliminating the influence of semantics and obtaining the inherent differences between modes;

[0112] Furthermore, if we use Kalman filtering to process n observations, the filtering formula is as follows:

[0113]

[0114] in, for The prior estimate, where A is the state transition matrix. for The posterior estimate, For the prior error covariance, P k For the error covariance, K k Let R be the Kalman gain. The covariance matrix of the observation noise, Z k for The observed value, i.e.

[0115] After Kalman filtering, the result is k posterior estimates Then it can be obtained by fitting a large number of posterior estimates. approximation

[0116] The same logic applies to the other modal types. Then the inherent differences in information representation of non-human factors of different modalities can be calculated. For example, the inherent difference between the i-th modality and the j-th modality is...

[0117] For a cross-modal transformation task, for example, given non-human factor data of the i-th modality... The goal is to convert it into non-human factor data of the j-th modality; u i After passing through the i-th modal data processing unit, data integration unit, i-th modal data encoding unit, and i-th modal feature extraction unit, the non-human factor feature f of the i-th modality is obtained. u，i Then the non-human characteristics of the j-th mode can be determined according to have to Then, the aligned features are obtained through the j-th modal feature alignment unit, and finally, the j-th modal data is output by the j-th modal data output unit.

[0118] The principle for implementing other types of modal transitions is the same as above, and will not be detailed here;

[0119] This application also provides an electronic device, such as... Figure 7 As shown; electronic devices are intended to represent various forms of digital computers, such as personal computers and cloud computing centers; specifically, they may include: processors, non-volatile memory, internal buses, I / O interfaces, and network interfaces; the processors, non-volatile memory, I / O interfaces, and network interfaces communicate with each other via the internal bus;

[0120] Processors include, but are not limited to, central processing units (CPUs), field-programmable gate arrays (FPGAs), graphics processing units (GPUs), and dedicated artificial intelligence chips; processors can call computer program instructions stored in non-volatile memory; when executed by the processor, the computer program instructions can specifically perform the aforementioned processing of human and non-human data, data integration, multimodal data distillation, and... Figure 5 The training steps for the neuromorphic multimodal model are shown below;

[0121] Non-volatile memory, such as disks, optical disks, etc.; may include a program storage area and a data storage area, wherein the program storage area stores computer program instructions that can be executed by the processor and the operating system, and the data storage area stores human and non-human data, as well as extracted human and non-human characteristics;

[0122] The I / O interface can connect to a multimodal sensing module and transfer the sensed human and non-human data to non-volatile memory. It can also connect to other components, such as displays and audio devices.

[0123] Network interfaces, such as network cards, switches, and routers, can download perceived human and non-human data through network resources; these resources can be provided by local area networks (LANs), wide area networks (WANs), and the Internet.

[0124] This application also provides a readable storage medium storing computer program instructions, which are executed by a processor to perform the aforementioned processing of human and non-human data, data integration, multimodal data distillation, and... Figure 5 The training steps for the brain-like multimodal model are shown.

[0125] The above description is merely a description of preferred embodiments of this application and is not intended to limit the scope of this application in any way. Any changes or modifications made by those skilled in the art based on the above-disclosed technical content should be considered as equivalent and valid embodiments and fall within the scope of protection of the technical solution of this application.

Claims

1. A brain-like multimodal model pre-training device, characterized in that, The device includes: a multimodal information source module (1), a multimodal perception module (2), a multimodal data processing module (3), a multimodal data distillation module (4), and a multimodal model training module (5); The multimodal information source module (1) includes human-cause information sources and several non-human-cause information sources, which are used to provide human-cause information and non-human-cause information under the same observation target; The multimodal sensing module (2) includes a human factor sensing unit and a non-human factor sensing unit, which are used to sense the human factor information and non-human factor information of the multimodal information source module, respectively, and convert them into data, thereby obtaining human factor data and non-human factor data under the same observation target, and providing them to the multimodal data processing module. The multimodal data processing module (3) includes a human factors data processing unit, a non-human factors data processing unit and a data integration unit, which are used to process human factors data and non-human factors data respectively, and integrate the two into multimodal data, which is then provided to the multimodal data distillation module. The multimodal data distillation module (4) is used to remove multimodal data that is inconsistent between human-cause semantics and non-human-cause semantics; the distilled multimodal data is provided to the multimodal model training module. The multimodal model training module (5) is used to complete the training process of the brain-like multimodal model, including a data encoding unit, a feature extraction unit, a feature alignment unit, a cross-modal feature conversion unit, and a data output unit. The feature alignment unit functions to map non-human features of a certain modality from the feature space of the original modality to the human feature space, based on human features, for the same observation target, thereby aligning with the semantic representation mechanism of the human brain to obtain aligned non-human features. Corresponding to the modality type of non-human information, m modality feature alignment units are involved. Each modality feature alignment unit takes the outputs of the human feature extraction unit and the corresponding non-human feature extraction unit as input; or takes the output of the cross-modality feature conversion unit as input. The data output unit is designed to decode the aligned non-human features into non-human data of the corresponding modality. The training method for the feature alignment unit is as follows: Based on distilled multimodal dataset Multimodal data with semantic similarity or consistency are used as the training data for each modality feature alignment unit; based on the functional description of the i-th modality feature alignment unit, non-human features are... The feature space of the i-th modality is approximately mapped to the human factor feature space, and the final output is the aligned non-human factor feature. ; To achieve the above objectives, a contrastive learning loss function is adopted; the alignment loss function for the i-th modality's non-human features and human features is designed as follows: Where the function This function is used to evaluate the similarity between two features in their statistical distribution. The higher the similarity, the closer the function value is to 1; conversely, the closer it is to 0. For a batch_size during training, it contains... For multimodal data sets, the total contrast loss is as follows: in Let represent the number of multimodal data points in a batch_size that are aligned from the first modality feature to the human factor feature; the rest are similarly represented. Then we have: Thus, through the feature alignment unit, the brain-like multimodal model aligns the features extracted from non-human data of different modalities into the human feature space.

2. The brain-like multimodal model pre-training device as described in claim 1, characterized in that, The multimodal data distillation module is used to remove multimodal data that is inconsistent between human and non-human semantics. It is a value-based reinforcement learning model, consisting of a data encoding unit, a feature extraction unit, an agent unit, and an environment. The data encoding unit is used to encode multimodal data into computable vectors; the multimodal data is human factor data and non-human factor data of a certain modality; the human factor data is encoded by the human factor data encoding unit; the non-human factor data is encoded by the corresponding modality data encoding unit according to the data modality type. The feature extraction unit is used to extract semantic features of the data, including human factor feature extraction unit and non-human factor data feature extraction unit; the input of the feature extraction unit is the output of the corresponding data encoding unit; specifically, human factor data is extracted by human factor feature extraction unit; non-human factor data is extracted by selecting the corresponding feature extraction unit according to the data modality type to obtain human factor features and non-human factor features respectively. The intelligent agent unit, corresponding to the modal type of non-human information, contains m intelligent agents. Each intelligent agent takes human features extracted by the feature extraction unit and non-human features of the corresponding modality as input. The function of the intelligent agent is to evaluate the semantic relevance of human data and non-human data based on the above features, output the relevance level to the environment and receive rewards from the environment. The relevance levels include level E (relevance < 50%), level D (relevance 50-70%), level C (relevance 70-80%), level B (relevance 80-90%), and level A (relevance 90-100%). That is, the action set of the intelligent agent is {A, B, C, D, E}. The environment includes a multimodal dataset with known true relevance levels. Rewards are calculated based on the true relevance levels and the relevance levels of the agent's output, and then fed back to the agent. The learning objective of the agent is to minimize the deviation between the agent's output and the true level to obtain more rewards, thereby optimizing the agent's output; furthermore, the multimodal data distillation module can evaluate multimodal data with unknown relevant levels and retain multimodal data with relevant levels of A.

3. The brain-like multimodal model pre-training device as described in claim 2, characterized in that, The construction process of the multimodal data distillation module is as follows: Step S103-1: Construct a multimodal dataset with known correlation levels for training the agent; A single observation process involves two or more observation targets, and the proportion of time the observer interacts with a particular observation target is set according to the relevant level. Step S103-2: Set different time proportion weights, repeat step S103-1 to obtain multiple multimodal datasets with known correlation levels, and label the correlation levels of the multimodal datasets according to the set time proportion weights. After merging, obtain a multimodal dataset with known correlation levels. ,in For the relevant level; It should be noted that a set of multimodal data middle Main observation target and The observation target is consistent; Step S103-3, After processing by the data encoding unit and the feature extraction unit, human-caused features and non-human-caused features are obtained; Steps S103-4: Human and non-human features are input into the corresponding agent, and the agent is trained by optimizing the reward function. Step S103-5, for multimodal datasets with unknown relevance levels The multimodal dataset is distilled by evaluating the data through iteratively trained agents, selecting and retaining the multimodal data with a relevance level of A. .

4. The brain-like multimodal model pre-training device as described in claim 1, characterized in that, The cross-modal feature conversion unit is specifically as follows: The cross-modal feature transformation unit does not contain weight parameters that need to be trained, so it does not need to participate in the training process of the brain-like multimodal model; Distilled multimodal dataset In this dataset, there are n sets of multimodal data where the non-human factor data is the i-th modality, meaning there are n sets of non-human factor features and human factor features for the i-th modality. , Furthermore, the two features in each group are semantically similar or consistent; Once the weight parameters of the brain-like multimodal model are fixed, the latent patterns mapped from human factor data to human factor characteristics are... It is fixed, representing the latent pattern mapped from the non-human data of the i-th modality to non-human features. It is also fixed; thus, it approximately reflects the inherent differences in information representation between human factors and non-human factors of the i-th modality. Furthermore, this inherent difference does not involve semantic differences; the approximation method is as follows: For inherent differences Given n observations, where the function This is used to eliminate the semantics of two features that are semantically identical, thereby eliminating the influence of semantics and revealing the inherent differences between modalities; Furthermore, Kalman filtering is used here to process the n observations, and the filtering formula is as follows: (5) in, for Prior estimates, Here is the state transition matrix. for The posterior estimate, For the prior error covariance, For error covariance, For Kalman gain, for The covariance matrix of the observation noise, for The observed value, i.e. ; After Kalman filtering, the result is of A posterior estimate Then it can be obtained by fitting a large number of posterior estimates. approximation ; The same applies to the other modal types. Then calculate the inherent differences in information representation of non-human features of different modalities; For a cross-modal transformation task, given non-human factor data of the i-th modality. , The non-human features of the i-th modality are obtained after passing through the i-th modality data processing unit, data integration unit, i-th modality data encoding unit, and i-th modality feature extraction unit. Then the non-human features of the (i+1)th mode can be determined according to ,have to Then, the aligned features are obtained through the (i+1)th modal feature alignment unit, and finally, the (i+1)th modal data is output by the (i+1)th modal data output unit.

5. A method for pre-training a brain-like multimodal model, based on the brain-like multimodal model pre-training device described in claim 1, characterized in that, The method includes the following steps: Step S101: In the multimodal information source module (1), the observer interacts with the observation target to generate human factor information, while the observation target spontaneously generates non-human factor information; the human factor sensing unit of the multimodal sensing module (2) acquires human factor data; according to the modal type of the non-human factor information generated by the observation target, the corresponding modal sensing unit is selected to acquire the non-human factor data of the corresponding modality; By changing the observation target and repeating the above interaction and perception process, a large amount of human and non-human factor data can be obtained. In step S102, human factors data and non-human factors data are transmitted to the multimodal data processing module, where they are processed by the human factors data processing unit and the non-human factors data processing unit, respectively. Then, the data integration unit integrates the human factors data and non-human factors data obtained under the same observation target into the same storage space, i.e., the storage space is different for different observation targets. The above processing and integration process is repeated to obtain a multimodal dataset. Step S103: If the semantics carried or represented by the human factors data and non-human factors data in a set of multimodal data are inconsistent, the multimodal data distillation module filters out the set of multimodal data, thereby improving the quality of the multimodal dataset. Step S104: The distilled multimodal dataset is transmitted as training data of the brain-like multimodal model to the multimodal model training module to complete the pre-training process of the brain-like multimodal model.

6. An electronic device, characterized in that, include: A processor and a memory connected to the processor; the memory stores computer program instructions that can be executed by the processor, the computer program instructions being executed by the processor to perform the method of claim 5.

7. A readable storage medium, characterized in that, The readable storage medium stores a computer program that is executed by a processor to perform the method of claim 5.

Images