AI data lake-based model optimization method, device, medium, and apparatus

Abstract

An AI data lake-based model optimization method, device, medium and equipment, relating to the technical field of models. The method comprises: obtaining first multi-modal data from an AI data lake, splitting the first multi-modal data into multiple segments to obtain a first segment sequence, and randomly adjusting the segment order of the first segment sequence to obtain a second segment sequence; generating a first masking strategy for the second segment sequence based on the first score and the second score through a reinforcement learning model, and performing masking processing on the second segment sequence based on the first masking strategy to obtain a third segment sequence; processing the third segment sequence through a multi-modal model to obtain a first processing result; performing loss calculation based on the first processing result and the first segment sequence, and updating the model parameters of the multi-modal model according to the loss calculation result. Thus, the masking strength can be dynamically adjusted according to the sample difficulty and the model capability, effectively improving the optimization efficiency of the multi-modal model.

Description

Technical Field

[0001] This content relates to the field of model technology, specifically to a model optimization method, apparatus, medium, and equipment based on an AI data lake. Background Technology

[0002] The field of multimodal learning has long suffered from the "shortcut effect" problem, which means that multimodal models tend to rely on information from a single modality for judgment during training, while ignoring information from other modalities, which seriously hinders the cross-modal reasoning ability of multimodal models.

[0003] In related technologies, jigsaw puzzle-style self-supervised learning enables multimodal models to perform cross-modal fusion inference, which alleviates the shortcut effect to some extent. However, it adopts a random masking strategy with fixed probabilities, which cannot adapt to the masking requirements of different training samples. For example, it may lead to excessive masking of simple samples, wasting computational resources, or insufficient masking of complex samples, which may prevent the full learning of cross-modal associations, thereby reducing the training efficiency of multimodal models. Summary of the Invention

[0004] This content section is provided to briefly introduce the concepts, which will be described in detail in the examples section later. This content section is not intended to identify key or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0005] Firstly, this content provides a model optimization method based on AI data lakes, including: First multimodal data is obtained from the AI ​​data lake, the first multimodal data is split into multiple segments to obtain a first segment sequence, and the segment order of the first segment sequence is randomly adjusted to obtain a second segment sequence; A first masking strategy is generated for the second segment sequence based on the first score and the second score by a reinforcement learning model, and the second segment sequence is masked based on the first masking strategy to obtain the third segment sequence. The first score is used to characterize the temporal reconstruction difficulty of the first multimodal data, and the second score is used to characterize the temporal reconstruction accuracy of the multimodal model before this training. The third segment sequence is processed by the multimodal model to obtain a first processing result. The first processing result includes first prediction information corresponding to each segment in the third segment sequence and second prediction information corresponding to the masked segment in the third segment sequence. The first prediction information is used to indicate the prediction temporal position of the corresponding segment in the first segment sequence. The second prediction information includes the prediction content obtained by the multimodal model after restoring the masked segment. Loss is calculated based on the first processing result and the first segment sequence, and the model parameters of the multimodal model are updated according to the loss calculation result.

[0006] Secondly, this content provides a model optimization device based on an AI data lake, including: The acquisition module is used to acquire first multimodal data from the AI ​​data lake, split the first multimodal data into multiple segments to obtain a first segment sequence, and randomly adjust the segment order of the first segment sequence to obtain a second segment sequence; The masking module is used to generate a first masking strategy for the second segment sequence based on a first score and a second score through a reinforcement learning model, and to mask the second segment sequence based on the first masking strategy to obtain a third segment sequence. The first score is used to characterize the temporal reconstruction difficulty of the first multimodal data, and the second score is used to characterize the temporal reconstruction accuracy of the multimodal model before this training. The processing module is used to process the third segment sequence through the multimodal model to obtain a first processing result. The first processing result includes first prediction information corresponding to each segment in the third segment sequence and second prediction information corresponding to the masked segment in the third segment sequence. The first prediction information is used to indicate the prediction temporal position of the corresponding segment in the first segment sequence. The second prediction information includes the prediction content obtained by the multimodal model after restoring the masked segment. The update module is used to perform loss calculation based on the first processing result and the first segment sequence, and update the model parameters of the multimodal model according to the loss calculation result.

[0007] Thirdly, this content provides a computer-readable medium having a computer program stored thereon that, when executed by a processing device, implements the steps of the method described in the first aspect.

[0008] Fourthly, this content provides an electronic device, including: A storage device on which computer programs are stored; A processing device for executing the computer program in the storage device to implement the steps of the method in the first aspect.

[0009] The above technical solution involves acquiring first multimodal data from an AI data lake, splitting it into multiple segments to obtain a first segment sequence, and randomly adjusting the order of these segments to obtain a second segment sequence. A reinforcement learning model then generates a first masking strategy for the second segment sequence based on first and second scores, and applies this strategy to mask the second segment sequence, resulting in a third segment sequence. The third segment sequence is then processed by the multimodal model to obtain a first processing result. Finally, loss calculation is performed based on the first processing result and the first segment sequence, and the model parameters are updated according to the loss calculation result. This method dynamically determines the appropriate masking strategy based on the temporal reconstruction difficulty of the multimodal data and the temporal reconstruction accuracy of the multimodal model. This allows for dynamic adjustment of the masking intensity according to the sample difficulty and model capability, achieving adaptive matching between sample difficulty and model capability. For example, it can guide the multimodal model to learn progressively from easy to difficult, effectively improving the optimization efficiency and cross-modal fusion depth of the multimodal model and further overcoming the shortcut effect. In addition, multimodal data in the AI ​​data lake can be fully utilized for model optimization training, effectively improving the data utilization efficiency of the AI ​​data lake.

[0010] Other features and advantages of the technical solution will be described in detail in the following examples section. Attached Figure Description

[0011] The above and other features, advantages, and aspects of the technical solution will become more apparent when considered in conjunction with the accompanying drawings and the following examples. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale. In the drawings: Figure 1 This is a schematic diagram illustrating an implementation environment according to an example.

[0012] Figure 2 This is a flowchart illustrating an AI data lake-based model optimization method, as shown in an example.

[0013] Figure 3 This is an interactive flowchart illustrating an adaptive multimodal self-supervised learning system based on reinforcement learning, as shown in an example.

[0014] Figure 4 This is a flowchart illustrating a model training process according to an example.

[0015] Figure 5 This is a schematic diagram of the structure of a model optimization device based on an AI data lake, as illustrated by an example.

[0016] Figure 6This is a schematic diagram of the structure of an electronic device as illustrated by an example. Detailed Implementation

[0017] The technical solution will now be described in more detail with reference to the accompanying drawings. Although certain scenarios are shown in the drawings, it should be understood that the technical solution can be implemented in various forms and should not be construed as limited to the scenarios described herein. Rather, these scenarios are provided to provide a more thorough and complete understanding of the technical solution. It should be understood that the accompanying drawings and the scenarios described are for illustrative purposes only and are not intended to limit the scope of protection of the technical solution.

[0018] It should be understood that the steps described in the method implementation may be performed in different orders and / or in parallel. Furthermore, the method implementation may include additional steps and / or omit the steps shown. The scope of the technical solution is not limited in this respect.

[0019] The term "comprising" and its variations as used herein can be open-ended, meaning "including but not limited to". The term "based on" can mean "at least partially based on". The term "one case" means "at least one case"; the term "another case" means "at least one additional case"; the term "some cases" means "at least some cases". Definitions of other terms will be given in the following description.

[0020] It should be noted that the concepts of "first" and "second" mentioned here are only used to distinguish different devices, modules or units, and are not used to limit the order of the functions performed by these devices, modules or units or their interdependencies.

[0021] It should be noted that the terms "one" and "more" used here are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0022] The names of messages or information exchanged between the multiple devices in the implementation are for illustrative purposes only and are not intended to limit the scope of these messages or information.

[0023] It is understandable that before using the technical solutions provided here, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in accordance with relevant laws and regulations, and their authorization should be obtained through appropriate means.

[0024] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose, based on the prompt message, whether to provide personal information to the software or hardware such as electronic devices, applications, servers, or storage media performing the operations described herein.

[0025] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0026] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of the technical solution. Other methods that comply with relevant laws and regulations may also be applied to the implementation of the technical solution.

[0027] At the same time, it is understood that the data involved in the technical solution (including but not limited to the data itself, the acquisition or use of the data) should comply with the requirements of relevant laws, regulations and related provisions.

[0028] Multimodal data lakes (or AI data lakes), as a unified unstructured data management architecture, are gradually becoming the foundational platform for carrying massive amounts of training data such as audio, video, images, and text. A typical multimodal data lake uses object storage as its foundation, providing capabilities such as a unified metadata catalog, multimodal data lineage tracking, and feature storage to support the training and iteration of AI models. However, current data lake capabilities mainly remain at the level of data storage and management. How to efficiently and self-supervisedly use the massive amounts of multimodal data in the data lake for model training, and how to write back intermediate features, quality labels, and temporal consistency annotations generated during training to the data lake to form a closed loop, are urgent technical problems that need to be solved.

[0029] With the development of large model technology, AI (Artificial Intelligence) agents driven by large models are widely used in various fields. Among them, large models can be large language models (LLM) or multimodal large models.

[0030] The "shortcut effect" has long been a problem in the field of multimodal learning. The shortcut effect refers to the tendency of multimodal models during training to rely on information from a single modality (usually visual) to make judgments, while actively ignoring information from other modalities (such as audio or text), forming a "lazy" reasoning approach. This phenomenon severely hinders the model's ability to truly understand complex scenes, especially in audio and video understanding tasks that require precise temporal alignment. Multimodal models often make judgments based on visual content, ignoring crucial temporal cues contained in the audio.

[0031] In related technologies, jigsaw puzzle-style self-supervised learning enables multimodal models to perform cross-modal fusion reasoning. The main steps are as follows: the input video is divided into multiple segments along the timeline, and the order of these segments is randomly shuffled; for the shuffled video sequence, several segments are randomly selected to mask one of their visual or audio information; the multimodal model's task is to restore the shuffled segment order and predict the content of the masked modality. Since visual and audio cues are scattered across different segments, the multimodal model must jointly utilize both audiovisual and visual information to accurately restore the temporal sequence; through this forced cross-modal reasoning training, the multimodal model overcomes its dependence on a single modality.

[0032] While the aforementioned approaches can mitigate the shortcut effect to some extent, their fixed-probability random masking strategy cannot adapt to the masking needs of different training samples. For example, it may lead to over-masking of simple samples, wasting computational resources, or insufficient masking of complex samples, failing to adequately learn cross-modal associations, thus reducing the training efficiency of the multimodal model. Furthermore, the training process often focuses on improving the inference quality of the multimodal model itself, failing to automatically detect and label segments with temporal logical contradictions between different modalities, thus making it unsuitable for direct application to downstream tasks such as fact-checking. Additionally, the training process lacks automatic difficulty adjustment capabilities based on the multimodal model's performance, failing to establish a learning path from simple to complex, further reducing the training efficiency of the multimodal model. Moreover, it cannot fully utilize data within the AI ​​data lake for optimized training of the multimodal model.

[0033] In view of this, this content provides a model optimization method, apparatus, medium and equipment based on AI data lake to solve the above problems.

[0034] The model optimization method based on AI data lake provided in this content can be executed by electronic devices, which can be provided as at least one of terminals and servers. Figure 1 This is an exemplary schematic diagram illustrating an implementation environment; see [link / reference]. Figure 1 The implementation environment includes: terminal 101 and server 102.

[0035] For example, a business system can be installed on terminal 101. The business system can connect to an agent associated with a multimodal model. The agent receives multimodal data processing tasks from the business system, sends the multimodal data processing tasks to server 102, and receives and displays the task processing results returned by server 102. Server 102 can be a backend server for the business system, providing backend services for the business system. Server 102 can also be a backend server for the agent, providing backend services for the agent, such as executing multimodal data processing tasks through the multimodal model, providing a training environment for the multimodal model, and so on.

[0036] In one scenario, terminal 101 provides users with the operation interface of the business system and the interaction interface with the intelligent analysis agent, while server 102 provides corresponding backend services and trains the multimodal model using multimodal data in the AI ​​data lake to optimize the multimodal model.

[0037] In other words, terminal 101 can be responsible for the front-end display, while server 102 can be responsible for the back-end data processing. Alternatively, terminal 101 can be responsible for some of the data processing. The specific configuration can be set according to requirements, and there are no restrictions on this.

[0038] Terminal 101 can be at least one of the following devices: smartphone, smartwatch, desktop computer, laptop, virtual reality terminal, augmented reality terminal, wireless terminal, and laptop computer. Terminal 101 has communication capabilities and can access wired or wireless networks. Terminal 101 can refer to one of multiple terminals, and those skilled in the art will understand that the number of such terminals can be more or less. Server 102 can be an independent physical server, a server cluster composed of multiple physical servers, or a distributed file system. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and artificial intelligence platforms.

[0039] For example, server 102 and terminal 101 are connected directly or indirectly via wired or wireless communication, without limitation.

[0040] Optionally, the number of servers 102 can be more or less, without limitation. Of course, servers 102 may also include other functional servers to provide more comprehensive and diversified services. Specifically, server 102 undertakes the primary computing work, and terminal 101 undertakes secondary computing work; or, server 102 undertakes secondary computing work, and terminal 101 undertakes primary computing work; or, server 102 or terminal 101 can each independently undertake computing work, without limitation. Furthermore, the backend servers for the aforementioned business system, intelligent analysis agent, and first model can be the same server or different servers, without limitation.

[0041] Figure 2 This is a flowchart illustrating an exemplary model optimization method based on an AI data lake. (Example:) Figure 2 As shown, the method may include the following steps: S201: Obtain the first multimodal data from the AI ​​data lake, split the first multimodal data into multiple segments to obtain the first segment sequence, and randomly adjust the segment order of the first segment sequence to obtain the second segment sequence.

[0042] For example, an AI data lake is a centralized distributed storage architecture for heterogeneous raw data across the entire domain. It embeds a vector storage engine and an AI automated data processing unit, and uniformly schedules heterogeneous computing power of CPU (Central Processing Unit) / GPU (Graphics Processing Unit). It is an integrated intelligent data processing system that realizes multimodal data lake governance, vector generation, large model training and fine-tuning, RAG (Retrieval-Augmented Generation) retrieval and data analysis in one stop.

[0043] like Figure 3 As shown, this content provides an adaptive multimodal self-supervised learning system based on reinforcement learning for self-supervised learning of multimodal models. The model optimization method provided in this content will be explained in detail below in conjunction with the system structure.

[0044] In one scenario, the first multimodal data includes data corresponding to the first modality and data corresponding to the second modality. The first multimodal data is split into multiple segments to obtain a first segment sequence, comprising: splitting the first multimodal data into multiple segments to obtain an initial segment sequence; extracting features from the initial segment sequence using a feature extractor corresponding to the first modality to obtain a first modality segment sequence; extracting features from the initial segment sequence using a feature extractor corresponding to the second modality to obtain a second modality segment sequence; the first modality segment sequence includes the feature vector of the first modality corresponding to each segment in the initial segment sequence; the second modality segment sequence includes the feature vector of the second modality corresponding to each segment in the initial segment sequence; the first segment includes both the first modality segment sequence and the second modality segment sequence.

[0045] For example, such as Figure 3 As shown, multimodal data can be obtained from the AI ​​data lake as training samples through the input layer. This multimodal data can be audio-text data, audio-visual data, audio-video data, image-text data, video-text data, or mixed data including audio, text, and video, etc., without limitation. It should be understood that the first and second modalities are for distinguishing different modalities of the multimodal data and do not limit the number of modalities. In practical applications, multimodal data can include even more modalities.

[0046] To facilitate understanding of the embodiments described herein, the following detailed embodiments will be provided using multimodal data as audio and video data.

[0047] For example, the audio and video data can be segmented into segments using the segment segmenter in the preprocessing module. This can be uniformly divided into T equal-length segments (the initial segment sequence), where T is a positive integer and can be set according to requirements. Visual frame features are then extracted for each segment using a visual frame feature extractor. Extracting audio spectral features based on audio feature extractors , i≤T. Sequence the visual features of T segments { , , ..., } (first modal segment sequence) and audio feature sequence { , , ..., The second modal segment sequence is used as the input to the subsequent modules. The first segment sequence can be understood as a feature sequence that maintains the original audio and video segment order.

[0048] It should be understood that the feature extractor in the preprocessing module is used to convert the original modal data into feature vectors of the corresponding modality, that is, to uniformly map the original data to the vector space, reduce the input dimension of subsequent modules, and thus effectively improve the processing efficiency of subsequent modules and the difference in low-level feature extraction.

[0049] Continue to refer to Figure 3 A random permutation index can be generated based on the segment order random shuffling device in the adaptive learning module. All segments of the first segment sequence are shuffled according to the random permutation index to obtain the second segment sequence. The second segment sequence can be understood as a feature sequence of shuffling the original audio and video segment order.

[0050] In one scenario, each segment in the second segment sequence is associated with a first position component and a second position component. The first position component represents the temporal position of the corresponding segment in the first segment sequence, and the second position component represents the temporal position of the corresponding segment in the second segment sequence.

[0051] For example, a learnable temporal position encoding can be attached to each segment in the second segment sequence, which can include two components: the original position encoding (the temporal position of the segment in the first segment sequence before shuffling) and the random position encoding (the temporal position of the segment in the second segment sequence). This dual position encoding allows the subsequent multimodal model to simultaneously perceive the current presentation order and the original temporal attribution of the segments, thereby enabling the multimodal model to explicitly learn positional mapping relationships.

[0052] S202: A first masking strategy for the second segment sequence is generated based on the first score and the second score using a reinforcement learning model, and the second segment sequence is masked based on the first masking strategy to obtain the third segment sequence. The first score is used to characterize the temporal reconstruction difficulty of the first multimodal data, and the second score is used to characterize the temporal reconstruction accuracy of the multimodal model before this training.

[0053] In one scenario, a first score is determined based on at least one of the following: modal content similarity, used to measure the degree of content similarity between different segments in the first segment sequence under the same modality; modal content change frequency, used to measure the degree of drastic content change within the same segment under the same modality in the first segment sequence; and cross-modal content correlation, used to measure the degree of content matching between different modalities within the same segment in the first segment sequence.

[0054] like Figure 3As shown, the difficulty estimator in the reinforcement learning scheduling layer can be a pre-trained model that receives feature sequences corresponding to unscrambled fragment sequences and outputs a scalar difficulty score to reflect the temporal reconstruction difficulty of the current multimodal data. The difficulty estimator is determined based on at least one of the following: modal content similarity, modal content change frequency, and cross-modal content correlation.

[0055] For example, taking audio and video data, modal content similarity can be visual similarity between segments, modal content change frequency can be audio content change frequency, and cross-modal content correlation can be audiovisual content correlation (which can be calculated from the shallow features of the multimodal encoder to obtain mutual information).

[0056] For example, the difficulty score can range from [0,1], where 0 represents extremely easy (significant differences between segments and clear temporal relationships) and 1 represents extremely difficult (segments are highly similar and require deep integration of audiovisual information). Generally, modal content similarity and cross-modal content correlation are positively correlated with the difficulty score, while the frequency of modal content changes is negatively correlated with the difficulty score.

[0057] This allows for the quantification of the temporal reconstruction difficulty of training samples, enabling the dynamic selection of masking strategies based on the temporal reconstruction difficulty of training samples, thereby improving the flexibility and efficiency of model optimization training.

[0058] In one scenario, the second score is determined as follows: A fifth segment sequence is obtained from the validation set. This fifth segment sequence is obtained by randomly adjusting the segment attributes of a sixth segment sequence, which is obtained by splitting the third multimodal data. A fourth prediction information is obtained by performing temporal prediction on the fifth segment sequence using a multimodal model. This fourth prediction information indicates the predicted temporal position of the corresponding segment within the sixth segment sequence. Based on the difference between the fourth prediction information and the sixth segment sequence, a fourth score is determined. This fourth score characterizes the temporal reconstruction accuracy of the multimodal model. The second score is then determined based on this fourth score.

[0059] For example, the second score can be understood as the recent performance of the multimodal model on the validation set. It can be expressed by the following formula:

[0060] in, This indicates the second score. The fourth score represents the accuracy of the time-series reconstruction task calculated on the validation set after the t-th training iteration of the multimodal model (i.e., the proportion of the original position of the segment predicted by the model to match the true position, or the negative value mapping of the cross-entropy loss). The accuracy of the time-series reconstruction task obtained from the previous validation can be directly used as the second score, or the average of the accuracy of the time-series reconstruction task obtained from multiple validations can be used as the second score. This represents the moving average of the accuracy calculated from the most recent W time series reconstruction tasks. W can be set according to requirements. Using a moving average can eliminate random fluctuations in a single evaluation and reflect the stable trend of the model's capabilities.

[0061] in addition, The range can be [0,1], as long as it matches the difficulty score. Directly using accuracy or average accuracy will naturally... Normalize to the [0,1] interval.

[0062] Alternatively, using the relative increase to reflect the change in the multimodal model from the start of training to the present, it can be expressed by the following formula:

[0063] in, This represents the accuracy rate when making random guesses. This represents the ideal upper limit, such as 1.

[0064] By analyzing the recent average accuracy of the multimodal model in time-series prediction experiments, the training difficulty of the multimodal model can be dynamically adjusted. That is, the policy network of the reinforcement learning model changes with... As the perceived multimodal model increases, its predictive power becomes stronger, leading to the selection of more challenging masking strategies, such as simultaneously masking sound and image, or masking more segments. If the size is not increased further, the policy network will choose a simple masking strategy, allowing the multimodal model to lay a good foundation.

[0065] It should be understood that both reinforcement learning models and difficulty assessment models can be continuously optimized using training data from multimodal models, and there are no restrictions on this.

[0066] Furthermore, the policy network Difficulty score for receiving and the recent performance of current multimodal models on the validation set. As joint state input Output masking strategy probability distribution And sample from it to obtain the masking decision for this training iteration.

[0067] Taking multimodal data as audio and video data as an example, based on the masking decision output by the policy network, the selected segments are subjected to corresponding modal masking operations to obtain a partially masked visual feature sequence. , , ..., } and audio feature sequences { , , ..., }

[0068] S203: The third segment sequence is processed by a multimodal model to obtain a first processing result. The first processing result includes the first prediction information corresponding to each segment in the third segment sequence and the second prediction information corresponding to the masked segment in the third segment sequence. The first prediction information is used to indicate the prediction time position of the corresponding segment in the first segment sequence. The second prediction information includes the prediction content obtained by the multimodal model after restoring the masked segment.

[0069] In one scenario, the multimodal model includes a multimodal encoder, a first prediction head, and a second prediction head. The multimodal model processes a third segment sequence to obtain a first processing result, including: extracting features from the third segment sequence using the multimodal encoder to obtain first multimodal features corresponding to each segment in the third segment sequence; performing content prediction using the second prediction head based on the first multimodal features corresponding to the masked segments in the third segment sequence to obtain second prediction information; and performing temporal prediction using the first prediction head based on the first multimodal features and second prediction information corresponding to each segment in the third segment sequence to obtain the first prediction information.

[0070] For example, continuing with multimodal data as audio and video data, such as... Figure 3 As shown, the multimodal encoder performs forward propagation to extract high-level semantic features across segments and modalities from the masked visual and audio feature sequences. The first prediction head, the cross-modal temporal reconstruction prediction head, feeds the multimodal features of T segments into it. This prediction head contains a self-attention sequence model, outputting the predicted probability distribution for the original position of each segment. The second prediction head, the masked modality reconstruction prediction head, feeds the multimodal features of a masked segment into it to generate a prediction of the masked modality content. If the segment's visual aspect is masked, its visual features are reconstructed; if the audio aspect is masked, its audio features are reconstructed.

[0071] It's important to note that the prediction process of the cross-modal temporal reconstruction prediction head relies on the prediction information from the masked modality reconstruction prediction head. The masked modality reconstruction prediction head allows a multimodal model to predict missing content using information from another modality when data is masked. For example, when vision is masked, the multimodal model infers image information from audio, prompting it to fuse multimodal features rather than relying solely on vision for temporal reconstruction. The temporal reconstruction task is responsible for learning the sequential logic between segments, while the masked reconstruction task is responsible for learning complementary mappings between modalities. Their collaboration reduces dependence on a single modality.

[0072] In one scenario, the third segment sequence includes a first modal segment sequence corresponding to the first modality and a second modal segment sequence corresponding to the second modality. The multimodal encoder includes a first modal encoder corresponding to the first modality, a second modal encoder corresponding to the second modality, and a cross-modal attention module. Feature extraction is performed on the third segment sequence using the multimodal encoder to obtain the first multimodal feature corresponding to each segment in the third segment sequence. This includes: encoding the first modal segment sequence using the first modal encoder to obtain the first modal feature corresponding to each segment in the third segment sequence; and extracting the first modal segment sequence using the second modal encoder to obtain the second modal feature corresponding to each segment in the third segment sequence. For each segment in the third segment sequence, the cross-modal attention module is used to perform cross-modal feature fusion on the first and second modal features corresponding to the segment to obtain the first multimodal feature corresponding to the segment.

[0073] For example, continuing with multimodal data as audio and video data, the masked visual feature sequence and audio feature sequence are processed by a visual encoder and an audio encoder respectively to extract features. Then, the audiovisual features are fused through a cross-modal attention module, and the fused features of each segment are output, which is the first multimodal feature mentioned above. This enables the acquisition of high-level semantic features across segments and modalities, which facilitates subsequent temporal reconstruction and masked modality reconstruction.

[0074] S204: Calculate the loss based on the first processing result and the first segment sequence, and update the model parameters of the multimodal model according to the loss calculation result.

[0075] For example, the training objective of the cross-modal temporal reconstruction prediction head is to minimize the cross-entropy loss L1 between the predicted temporal position and the true temporal position, and the training objective of the masked modality reconstruction prediction head is to minimize the mean squared error L2 between the predicted content and the original content. The total loss is calculated. a and b represent weight coefficients, which are used by L3 to balance cross-modal response capability and single-modal preservation capability, and to collaboratively optimize the multimodal feature extraction capability of the multimodal encoder, the temporal prediction capability of the cross-modal temporal reconstruction prediction head, and the content prediction capability of the masked modality reconstruction prediction head. In addition, the loss calculation function can be set according to requirements and there are no restrictions on it.

[0076] By employing the above method, the corresponding masking strategy can be dynamically determined based on the temporal reconstruction difficulty of multimodal data and the temporal reconstruction accuracy of the multimodal model. This allows for dynamic adjustment of the masking intensity according to the sample difficulty and model capability, achieving adaptive matching between sample difficulty and model capability. For example, it can guide the multimodal model to learn progressively from easy to difficult, effectively improving the optimization efficiency and cross-modal fusion depth of the multimodal model and further overcoming the shortcut effect. Furthermore, it can fully utilize multimodal data in AI data lakes for model optimization training, effectively improving the data utilization efficiency of AI data lakes.

[0077] In one scenario, the first masking strategy includes at least one of the following parameters: a masking modality selection parameter, used to indicate that at least one modality among text modality, visual modality, and audio modality is masked; a masking segment index parameter, used to indicate the temporal position of the masked segment in the second segment sequence; a masking intensity level parameter, used to indicate the masking intensity of the masked segment; and a masking type parameter, used to indicate the masking method of the modality corresponding to the masking modality selection parameter.

[0078] For example, a masking policy space C, which is a mixture of discrete and continuous masking policies, can be defined, representing a complete masking policy. It may include at least one of the following adjustable parameters: Masking modality selection parameters: can include multiple mutually exclusive options such as text only, visual only, audio only, and simultaneous multimodal masking; Masking segment index parameter: an integer vector of length K, representing the segment number selected for masking (K is dynamically determined by the policy network, ranging from [0, T / 2], and can be adjusted according to requirements); Masking strength level parameters: can include options such as complete masking, high noise replacement, and partial area masking, each corresponding to different cross-modal inference difficulty; Masking type parameter: When visual masking is selected, the visual masking type can be black frame replacement, Gaussian noise replacement, adjacent frame interpolation replacement, etc. When audio masking is selected, the audio masking type can be silence replacement, white noise replacement, frequency band selective masking. The specific settings are based on the requirements and there are no restrictions on this.

[0079] The above parameters can be set according to requirements and are not restricted. This allows the policy network to flexibly generate corresponding adaptive masking policies based on the sample difficulty and the current capabilities of the multimodal model, effectively improving the accuracy and efficiency of model optimization training.

[0080] In one scenario, the reinforcement learning model includes a first network and a second network, wherein the first network outputs a masking policy and the second network outputs a predicted reward score. The method further includes: determining a first true reward score based on a preset reward function, and obtaining a first predicted reward score through the second network based on the first score, the second score, and the first masking policy; updating the parameters of the first network and the second network based on the first true reward score and the first predicted reward score; wherein the preset reward function includes at least one of the following: a first function term, representing a first positive reward when the temporal reconstruction accuracy of the first predicted information is higher than the temporal reconstruction accuracy of the third predicted information, otherwise a first negative reward is assigned; wherein the third predicted information is obtained by a multimodal model performing temporal prediction on a first segment sequence; a second function term, representing a second negative reward when the temporal reconstruction accuracy of the first predicted information is less than a first preset threshold, otherwise a second positive reward is assigned; a third function term, representing a third positive reward when the variance between the temporal reconstruction accuracy of the first predicted information and the temporal reconstruction accuracy of historical predicted information is less than a second preset threshold, otherwise a third negative reward is assigned, wherein the historical predicted information is obtained by a multimodal model performing temporal prediction on a randomly adjusted historical segment sequence.

[0081] For example, the first and second preset thresholds can be the same or different. The first network is a policy network and the second network is a value network. After each training iteration of the multimodal model, a composite reward is calculated based on the performance of the multimodal model on the temporal reconstruction task and the masked modality reconstruction task.

[0082] Where R represents the compound reward. Represents the first function term. Represents the second function term. Represents the third function term. This represents the weighting coefficient, which can be adjusted as needed.

[0083] For example, the first function term can be understood as a difficulty gain reward: if the multimodal model performs better (or improves more significantly) with masking than the baseline without masking, a positive reward is given; otherwise, a negative or zero reward is given. The difficulty gain reward can be calculated using the temporal reconstruction accuracy p1 of the unmasked data and the temporal reconstruction accuracy p2 of the data with strategic masking.

[0084] in, This indicates a difficulty-increasing reward. This indicates a minimum value and prevents division by zero. A positive value indicates that masking actually improves performance, suggesting that the multimodal model uses more robust features, and is therefore rewarded positively. Negative values ​​or 0 indicate that the masking strategy is too disruptive, leading to a decline in the performance of the multimodal model. Negative or zero rewards are then given. In addition to performing an unmasked temporal reconstruction task at each training iteration, an unmasked temporal reconstruction task can be performed at preset intervals. The corresponding temporal reconstruction accuracy is then calculated as a baseline value. Alternatively, historical accuracy from the validation set can be used. Interpolation can be obtained, and there are no restrictions on this.

[0085] For example, the second function term can be understood as an efficiency loss penalty: if the masking intensity is too high and the multimodal model cannot complete the reconstruction task (accuracy is lower than the preset threshold), a strong negative reward is given to prevent excessive masking from causing ineffective learning; otherwise, a negative reward is given.

[0086] For example, the third function term can be understood as a stability reward: if the multimodal model performs well in a predetermined number of training rounds ( If the variance of a policy network is below a preset threshold, it indicates that the current difficulty is appropriate. A positive reward is given to encourage the network to maintain the current policy difficulty range. Otherwise, a negative reward is given to encourage the policy network to reduce the magnitude of difficulty changes or reduce the average difficulty.

[0087] It should be understood that, through extensive training and iteration, the policy network can learn an implicit "difficulty-performance" mapping, thus enabling it to... At lower levels, choosing simple masking (such as unimodal, low masking ratio) yields higher long-term rewards because it provides high stability rewards without incurring significant efficiency penalties. When the difficulty level is high, challenging masking (such as multimodal or high masking ratio) is chosen because this yields a positive reward in terms of difficulty gain. This allows the policy network to learn to output a masking difficulty policy that matches the current model's capabilities, thus achieving a dynamic equilibrium.

[0088] In one scenario, updating the first network based on a first true reward score and a first predicted reward score includes: determining a first loss value as the difference between the first true reward score and the first predicted reward score; determining a third loss value based on the first loss value and a second loss value corresponding to a preset regularization term; and updating the first network based on the third loss value. The preset regularization term is used to maximize the policy entropy of the first network. Alternatively, the second network can also be updated based on the difference between the first predicted reward score and the first true reward score.

[0089] For example, the complete empirical tuples generated in each training iteration ( , R Deposit Figure 3 The experience replay pool shown, in which, This indicates the masking strategy for sampling. The feedback environment state. A batch of empirical data can be sampled from the experience replay pool, and the policy network and value network can be updated separately using a preset algorithm. The policy network update objective can be to maximize the expected cumulative reward evaluated by the value network, while limiting the policy update magnitude through a pruning mechanism to maintain training stability. The value network update objective can be to more accurately estimate the expected cumulative reward in a given state. The preset algorithm can be a proximal policy optimization (PPO) algorithm, a soft actor-critic (SAC) algorithm, a trust region policy optimization (TRPO) algorithm, a deep deterministic policy gradient (DDPG) algorithm, etc., and there are no restrictions on the specific algorithm.

[0090] Additionally, an entropy regularization term can be added to the policy network loss function:

[0091] Where L4 represents the third loss value, L5 represents the first loss value, H represents the information entropy of the policy network output action distribution, and c represents any possible action in the policy space.

[0092] By incorporating entropy regularization, the policy network is prevented from outputting a fixed masking strategy in the early stages of training, which could lead to insufficient exploration and missing out on better masking solutions. This ensures that even when the multimodal model performs well, the policy network will still try masks of varying difficulty, thereby discovering more efficient learning paths. Furthermore, other regularization terms can be selected as needed, without any restrictions.

[0093] In one scenario, the multimodal model includes a discriminator used to determine the probability that each segment in the input segment sequence is in a temporally normal position; the method further includes: applying a transformation that preserves temporal logic to the first segment sequence to obtain a first sample, and applying a perturbation that disrupts temporal logic to the first segment sequence to obtain a second sample; using the first sample as a positive sample, the second sample as a negative sample, and the first prediction information as an anchor sample, to perform comparative training on the discriminator.

[0094] The discriminator is trained by using the first sample as a positive sample, the second sample as a negative sample, and the first predicted information as an anchor sample. This includes: inputting the first sample, the second sample, and the first predicted information into the discriminator to obtain a first similarity and a second similarity. The first similarity represents the similarity between the first sample and the first predicted information, and the second similarity represents the similarity between the second sample and the first predicted information. A contrastive loss is obtained based on the first similarity and the second similarity, and the parameters of the discriminator are updated based on the contrastive loss. The contrastive loss is used to increase the first similarity and decrease the second similarity.

[0095] For example, continue to refer to Figure 3 The same batch of training data can be used, but independent data augmentation paths are employed for training the discriminator. In the weak augmentation branch, standard data augmentation (such as slight cropping and horizontal flipping) is performed on the original segment sequence to preserve temporal information; the extracted features are then used as positive samples. In the strong augmentation branch, stronger temporal perturbations are performed on the same segment sequence (such as local segment swapping, replacing random segments with irrelevant video segments, and pairing audio with asynchronous video) to simulate samples that may contain logical fallacies; the extracted features are then used as negative samples. Additionally, the predicted temporal sequence output from the cross-modal temporal reconstruction prediction head is used as anchor samples.

[0096] Positive samples, negative samples, and anchor point samples are then input into the discriminator. Through a contrastive learning mechanism, the discriminator is trained to distinguish between normal segments with consistent temporal logic and abnormal segments with contradictory temporal logic. The contrastive loss function is used to calculate the contrastive loss before backpropagation and parameter updates, ensuring that the feature embeddings of logically normal segments maintain high similarity to the predicted temporal sequence, while the feature embeddings of logically abnormal segments maintain low similarity to the predicted temporal sequence. The contrastive loss function can be set according to requirements and is not restricted.

[0097] In one scenario, the method further includes: acquiring second multimodal data, splitting the second multimodal data into multiple segments to obtain a fourth segment sequence; determining a third score corresponding to each segment in the fourth segment sequence using a discriminator, and labeling segments in the fourth segment whose third score is less than a third preset threshold with a first label, where the third score represents the probability that the corresponding segment is in a normal temporal position, and the first label represents the risk of temporal anomalies in the corresponding segment; and sending the third score and the fourth segment sequence labeled with the first label to a data detection system, which is used to perform data anomaly detection on the fourth segment sequence.

[0098] In practical applications, the second type of multimodal data can be obtained from the AI ​​data lake for processing, or it can be data that needs to be stored in the AI ​​data lake after processing by the discriminator and labeler. There are no restrictions on this. Furthermore, the third preset threshold can be set according to specific requirements, and there are no restrictions on this either.

[0099] For example, such as Figure 3 As shown, for any input multimodal data, the preprocessing module performs segmentation and feature extraction to obtain a feature sequence. The feature sequence is then passed through a multimodal encoder and a cross-modal temporal reconstruction prediction head path to obtain temporal prediction information. The discriminator calculates a temporal consistency score for each segment based on the feature sequence and the temporal prediction information. Segments with consistency scores below the threshold are automatically marked as segments that may have temporal logic anomalies.

[0100] This allows for the generation of a structured report containing the identifiers, consistency scores, and recommended detection priorities for the tagged segments. This enables downstream data detection systems to perform further anomaly detection based on the report, achieving automated fact detection of multimodal content.

[0101] It should be noted that after training, a multimodal model that can be directly deployed is obtained, including a multimodal encoder, a first prediction head, a second prediction head, a discriminator, and a labeler. Different modules can be organized and called according to different data processing tasks.

[0102] In addition, the reinforcement learning-based adaptive multimodal self-supervised learning system provided in this content can be trained online in an infinite loop. After processing each batch of multimodal data, the policy network and multimodal encoder are updated. As the capabilities of the multimodal encoder improve, the policy network automatically adjusts the masking strategy, always maintaining the adaptive training optimization of the multimodal model. The performance of the temporal consistency detection plugin also continuously improves with the enhancement of the multimodal encoder's capabilities, forming a positive flywheel effect.

[0103] In one scenario, the method further includes: acquiring intermediate data of the multimodal model during training, the intermediate data including feature vectors corresponding to the third segment sequence; and storing the intermediate data in an AI data lake.

[0104] For example, training intermediates (feature vectors, consistency labels, data time-series reconstruction difficulty, etc.) can be automatically written back to the AI ​​data lake, and data lineage can be recorded for easy data traceability. This achieves a closed-loop architecture that uses the AI ​​data lake as a unified data foundation to realize the entire lifecycle of intelligent training data ingestion, automatic writing back of training intermediates, and automatic tracking of data lineage.

[0105] like Figure 4 As shown, the training coordinator uses difficulty scores and the recent performance of current multimodal models on the validation set. As joint state input The input is a policy network, and the policy network outputs a masking policy. The training coordinator applies a masking strategy to the disordered feature sequences corresponding to the preprocessed multimodal data. Subsequently, the masked out-of-order feature sequence is obtained and input into a multimodal encoder. The multimodal encoder extracts multimodal features and inputs them into a cross-modal temporal reconstruction prediction head to obtain the predicted temporal sequence. The predicted temporal sequence is then fed back to the training coordinator for reward calculation. Additionally, the discriminator can be trained in parallel by inputting the weak enhancement feature sequence of the weak enhancement branch, the strong enhancement feature sequence of the strong enhancement branch, and the aforementioned predicted temporal sequence for comparative training. In practical applications, segments with consistency scores below a preset value can be marked for further detection processing by downstream data detection systems.

[0106] It should be noted that this content provides a reinforcement learning scheduling layer composed of a policy network, a value network, and a difficulty estimator. This scheduling layer dynamically determines the modality masking decision in multimodal self-supervised learning based on the difficulty characteristics of the current training samples and the model's recent performance. Furthermore, it provides a self-supervised temporal consistency detection module that trains in parallel with multimodal self-supervised learning. This module trains a temporal consistency discriminator by combining a two-branch data augmentation mechanism (a weak augmentation branch maintains temporal integrity, while a strong augmentation branch introduces temporal perturbations) with a contrastive learning mechanism. This discriminator can score the temporal logical consistency of any multimodal data without manual annotation and automatically mark segments that may contain temporal logical anomalies.

[0107] It is worth noting that this content also provides a joint training method that integrates multimodal jigsaw puzzle self-supervised learning, reinforcement learning dynamic masking scheduling, and temporal consistency detection. In this method, self-supervised learning is the main training task, the reinforcement learning scheduling layer provides adaptive difficulty adjustment, and temporal consistency detection shares encoder features with the main training task and is optimized in parallel.

[0108] For example, the dynamic masking strategy based on reinforcement learning allows the difficulty of training samples to be adapted to the current capabilities of the multimodal model, avoiding the problems of wasted computing power on simple samples and insufficient training on difficult samples under a fixed strategy, thus achieving a dual improvement in training efficiency and effectiveness. The policy network continuously interacts with the environment (model performance feedback during training), automatically learning an automated, progressive learning path from "masking a small number of segments of a single modality" to "high-proportion masking of two modalities." Furthermore, a temporal consistency verification plugin endows the multimodal model with the ability to automatically detect temporal logical anomalies in multimodal data, making it directly usable for downstream applications (such as fake video detection and news fact-checking). This plugin is trained entirely in a self-supervised manner, requiring no manual labeling of negative samples, greatly reducing the difficulty of acquiring training data. In addition, it does not depend on a specific multimodal encoder structure and can be used as a plug-and-play training enhancement module, adaptable to various audio and video understanding models, including but not limited to different network architectures based on Transformer, convolutional neural networks, etc.

[0109] Furthermore, the reinforcement learning scheduling layer and the multimodal encoder form a co-evolutionary relationship. The stronger the multimodal encoder, the more complex the decision-making environment faced by the policy network (it needs to find more difficult masking strategies to maintain learning gains). As a result, the policy network continues to evolve, forming a sustainable performance improvement flywheel.

[0110] It should be noted that the entire dynamic decision-making task can be modeled as a decision problem model, and the masking strategy can be selected using Thompson sampling or UCB (Upper Confidence Bound) algorithms. The discriminator can also adopt a generative adversarial network-based training method, that is, training a generator to actively generate temporally contradictory samples, and the discriminator is responsible for distinguishing between true and false samples. Alternatively, the reconstruction error of the variational autoencoder can be used to detect abnormal temporal segments.

[0111] Furthermore, the difficulty estimator can be simplified from a neural network model to an evaluation function based on heuristic rules (such as the mean cosine similarity of feature vectors between segments). The masking strategy space can also be pruned according to scene requirements, such as disabling audio masking (in pure video analysis scenarios) or fixing the masking strength to adjust the number of masked segments, etc., without any restrictions.

[0112] In one scenario, this content provides a model optimization method based on an AI data lake, comprising: acquiring first multimodal data from the AI ​​data lake; splitting the first multimodal data into multiple segments to obtain a first segment sequence; randomly adjusting the segment order of the first segment sequence to obtain a second segment sequence; generating a first masking strategy for the second segment sequence based on a first score and a second score using a reinforcement learning model; and masking the second segment sequence based on the first masking strategy to obtain a third segment sequence. The first score is used to characterize the temporal sequence reconstruction difficulty of the first multimodal data, and the first score is based on modal content similarity, modal content change frequency, and cross-modal content association. At least one of the scores is determined, and the second score is used to characterize the temporal reconstruction accuracy of the multimodal model before this training. The third segment sequence is processed by the multimodal model to obtain a first processing result. The first processing result includes the first prediction information corresponding to each segment in the third segment sequence and the second prediction information corresponding to the masked segment in the third segment sequence. The first prediction information is used to indicate the predicted temporal position of the corresponding segment in the first segment sequence, and the second prediction information includes the prediction content obtained by the multimodal model after reconstructing the masked segment. The loss is calculated based on the first processing result and the first segment sequence, and the model parameters of the multimodal model are updated according to the loss calculation result.

[0113] By employing the above method, the corresponding masking strategy can be dynamically determined based on the temporal reconstruction difficulty of quantized multimodal data and the temporal reconstruction accuracy of the multimodal model. This allows for dynamic adjustment of the masking intensity according to the sample difficulty and model capability, achieving adaptive matching between sample difficulty and model capability. For example, it can guide the multimodal model to learn progressively from easy to difficult, effectively improving the optimization efficiency and cross-modal fusion depth of the multimodal model and further overcoming the shortcut effect. Furthermore, it can fully utilize multimodal data in AI data lakes for model optimization training, effectively improving the data utilization efficiency of AI data lakes.

[0114] Figure 5 This is a schematic diagram of the structure of an exemplary model optimization device based on an AI data lake. Figure 5 As shown, the AI ​​data lake-based model optimization device 500 includes: The acquisition module 501 is used to acquire first multimodal data from the AI ​​data lake, split the first multimodal data into multiple segments to obtain a first segment sequence, and randomly adjust the segment order of the first segment sequence to obtain a second segment sequence; The masking module 502 is used to generate a first masking strategy for the second segment sequence based on a first score and a second score through a reinforcement learning model, and to perform masking processing on the second segment sequence based on the first masking strategy to obtain a third segment sequence. The first score is used to characterize the temporal reconstruction difficulty of the first multimodal data, and the second score is used to characterize the temporal reconstruction accuracy of the multimodal model before this training. Processing module 503 is used to process the third segment sequence through the multimodal model to obtain a first processing result. The first processing result includes first prediction information corresponding to each segment in the third segment sequence and second prediction information corresponding to the masked segment in the third segment sequence. The first prediction information is used to indicate the prediction temporal position of the corresponding segment in the first segment sequence. The second prediction information includes the prediction content obtained by the multimodal model after restoring the masked segment. The update module 504 is used to perform loss calculation based on the first processing result and the first segment sequence, and update the model parameters of the multimodal model according to the loss calculation result.

[0115] Using the aforementioned device, a corresponding masking strategy can be dynamically determined based on the temporal reconstruction difficulty of multimodal data and the temporal reconstruction accuracy of the multimodal model through reinforcement learning. This allows for dynamic adjustment of the masking intensity according to the sample difficulty and model capability, achieving adaptive matching between sample difficulty and model capability. For example, it can guide the multimodal model to learn progressively from easy to difficult, effectively improving the optimization efficiency and cross-modal fusion depth of the multimodal model and further overcoming the shortcut effect. Furthermore, it can fully utilize multimodal data in the AI ​​data lake for model optimization training, effectively improving the data utilization efficiency of the AI ​​data lake.

[0116] Optionally, the first multimodal data includes data corresponding to the first mode and data corresponding to the second mode; the acquisition module 501 is used for: The first multimodal data is split into multiple segments to obtain an initial segment sequence; The initial segment sequence is subjected to feature extraction by the feature extractor corresponding to the first modality to obtain a first modality segment sequence corresponding to the first modality. The initial segment sequence is subjected to feature extraction by the feature extractor corresponding to the second modality to obtain a second modality segment sequence corresponding to the second modality. The first modality segment sequence includes the feature vector of the first modality corresponding to each segment in the initial segment sequence, and the second modality segment sequence includes the feature vector of the second modality corresponding to each segment in the initial segment sequence. The first segment includes the first modality segment sequence and the second modality segment sequence.

[0117] Optionally, the multimodal model includes a multimodal encoder, a first prediction head, and a second prediction head, and the processing module 503 is used for: The third segment sequence is processed by a multimodal encoder to extract features, thereby obtaining the first multimodal feature corresponding to each segment in the third segment sequence. The second prediction information is obtained by performing content prediction based on the first multimodal features corresponding to the masked segments in the third segment sequence using the second prediction head, and the first prediction information is obtained by performing temporal prediction based on the first multimodal features corresponding to each segment in the third segment sequence and the second prediction information using the first prediction head.

[0118] Optionally, the third segment sequence includes a first modal segment sequence corresponding to the first modality and a second modal segment sequence corresponding to the second modality; the multimodal encoder includes a first modal encoder corresponding to the first modality, a second modal encoder corresponding to the second modality, and a cross-modal attention module; the processing module 503 is used for: The first modal segment sequence is feature-encoded by the first modal encoder to obtain the first modal feature corresponding to each segment in the third segment sequence, and the first modal segment sequence is feature-extracted by the second modal encoder to obtain the second modal feature corresponding to each segment in the third segment sequence. For each segment in the third segment sequence, the cross-modal attention module performs cross-modal feature fusion on the first modal feature and the second modal feature corresponding to the segment to obtain the first multimodal feature corresponding to the segment.

[0119] Optionally, the first score is determined based on at least one of the following: Modal content similarity is used to measure the degree of content similarity between different segments in the first segment sequence under the same modality. Modal content change frequency is used to measure the degree of drastic change in content within the same segment of the first segment sequence under the same modality. Cross-modal content correlation is used to measure the degree of content matching between different modalities within the same segment in the first segment sequence.

[0120] Optionally, the second score is determined as follows: The fifth segment sequence is obtained from the validation set. The fifth segment sequence is obtained by randomly adjusting the segment attributes of the sixth segment sequence, which is obtained by splitting the third multimodal data. The fifth segment sequence is temporally predicted using the multimodal model to obtain fourth prediction information, which is used to indicate the predicted temporal position of the corresponding segment in the sixth segment sequence. Based on the difference between the fourth prediction information and the sixth segment sequence, a fourth score is determined, which characterizes the temporal reconstruction accuracy of the multimodal model; The second score is determined based on the fourth score.

[0121] Optionally, the first masking strategy includes at least one of the following parameters: Masking modality selection parameters are used to indicate whether to mask at least one of the text modality, visual modality, and audio modality; The masked segment index parameter is used to indicate the temporal position of the masked segment in the second segment sequence; The masking strength level parameter indicates the masking strength of the masked segment; The masking type parameter indicates the masking method for the mode corresponding to the masking mode selection parameter.

[0122] Optionally, the reinforcement learning model includes a first network and a second network, wherein the first network is used to output a masking policy, the second network is used to output a predicted reward score, and the model optimization device 500 based on the AI ​​data lake further includes a reward module for: The first true reward score is determined based on a preset reward function, and the first predicted reward score is obtained through the second network based on the first score, the second score, and the first masking strategy. The parameters of the first network and the second network are updated based on the first actual reward score and the first predicted reward score; The preset reward function includes at least one of the following: The first function term represents that a first positive reward is given when the temporal reconstruction accuracy of the first predicted information is higher than that of the third predicted information, and a first negative reward is given otherwise; wherein, the third predicted information is obtained by the multimodal model performing temporal prediction on the first segment sequence; The second function term represents that when the temporal reconstruction accuracy of the first prediction information is less than the first preset threshold, a second negative reward is given; otherwise, a second positive reward is given. The third function term represents a third positive reward when the variance between the temporal reconstruction accuracy of the first prediction information and the temporal reconstruction accuracy of the historical prediction information is less than a second preset threshold; otherwise, a third negative reward is given. The historical prediction information is obtained by the multimodal model through temporal prediction of randomly adjusted historical fragment sequences.

[0123] Optionally, the reward module is used for: The difference between the first actual reward score and the first predicted reward score is determined as the first loss value; A third loss value is determined based on the first loss value and the second loss value corresponding to the preset regularization term, and the first network is updated based on the third loss value. The preset regularization term is used to maximize the policy entropy of the first network.

[0124] Optionally, the multimodal model includes a discriminator used to determine the probability of each segment in the input segment sequence being in a temporally normal position; the AI ​​data lake-based model optimization device 500 further includes a training module used for: A transformation that preserves the timing logic is applied to the first segment sequence to obtain a first sample, and a perturbation that disrupts the timing logic is applied to the first segment sequence to obtain a second sample; The discriminator is trained by comparing the first sample as a positive sample, the second sample as a negative sample, and the first prediction information as an anchor sample.

[0125] Optionally, the training module is used for: The first sample, the second sample, and the first prediction information are input into the discriminator to obtain a first similarity and a second similarity. The first similarity represents the similarity between the first sample and the first prediction information, and the second similarity represents the similarity between the second sample and the first prediction information. A contrast loss is obtained based on the first similarity and the second similarity, and the parameters of the discriminator are updated based on the contrast loss. The contrast loss is used to increase the first similarity and decrease the second similarity.

[0126] Optionally, the model optimization device 500 based on the AI ​​data lake further includes a labeling module for: Acquire the second multimodal data, split the second multimodal data into multiple segments, and obtain the fourth segment sequence; The discriminator determines the third score corresponding to each segment in the fourth segment sequence, and marks the segments in the fourth segment whose third score is less than a third preset threshold with a first label. The third score represents the probability that the corresponding segment is in a normal temporal position, and the first label represents the risk of temporal anomaly in the corresponding segment. The third score and the fourth segment sequence labeled with the first tag are sent to the data detection system, which is used to detect data anomalies in the fourth segment sequence.

[0127] Optionally, each segment in the second segment sequence is associated with a first position component and a second position component, wherein the first position component represents the temporal position of the corresponding segment in the first segment sequence, and the second position component represents the temporal position of the corresponding segment in the second segment sequence.

[0128] Optionally, the model optimization device 500 based on the AI ​​data lake further includes an acquisition submodule, used for: Obtain intermediate data during the training process of the multimodal model, the intermediate data including the feature vector corresponding to the third segment sequence; The intermediate data is stored in the AI ​​data lake.

[0129] Regarding the AI ​​data lake-based model optimization device 500 mentioned above, the execution logic of each functional module has been explained in detail in the section on methods, and will not be repeated here.

[0130] Based on the same concept, a computer-readable medium is also provided, on which a computer program is stored, which, when executed by a processing device, implements the steps of any of the above-described AI data lake-based model optimization methods.

[0131] This allows for the dynamic determination of appropriate masking strategies based on the temporal reconstruction difficulty of multimodal data and the temporal reconstruction accuracy of the multimodal model through reinforcement learning. The masking intensity is then dynamically adjusted according to the sample difficulty and model capability, achieving adaptive matching between sample difficulty and model capability. For example, it can guide multimodal models to learn progressively from easy to difficult, effectively improving the optimization efficiency and cross-modal fusion depth of multimodal models and further overcoming the shortcut effect. Furthermore, it allows for full utilization of multimodal data in AI data lakes for model optimization training, effectively improving the data utilization efficiency of AI data lakes.

[0132] Based on the same concept, an electronic device is also provided, which may include: A storage device on which computer programs are stored; A processing device for executing a computer program stored in a storage device to implement the steps of any of the above-described AI data lake-based model optimization methods.

[0133] This allows for the dynamic determination of appropriate masking strategies based on the temporal reconstruction difficulty of multimodal data and the temporal reconstruction accuracy of the multimodal model through reinforcement learning. The masking intensity is then dynamically adjusted according to the sample difficulty and model capability, achieving adaptive matching between sample difficulty and model capability. For example, it can guide multimodal models to learn progressively from easy to difficult, effectively improving the optimization efficiency and cross-modal fusion depth of multimodal models and further overcoming the shortcut effect. Furthermore, it allows for full utilization of multimodal data in AI data lakes for model optimization training, effectively improving the data utilization efficiency of AI data lakes.

[0134] Based on the same concept, a computer program product is also provided, including a computer program that, when executed by a processor, implements the steps of any of the above-described AI data lake-based model optimization methods.

[0135] This allows for the dynamic determination of appropriate masking strategies based on the temporal reconstruction difficulty of multimodal data and the temporal reconstruction accuracy of the multimodal model through reinforcement learning. The masking intensity is then dynamically adjusted according to the sample difficulty and model capability, achieving adaptive matching between sample difficulty and model capability. For example, it can guide multimodal models to learn progressively from easy to difficult, effectively improving the optimization efficiency and cross-modal fusion depth of multimodal models and further overcoming the shortcut effect. Furthermore, it allows for full utilization of multimodal data in AI data lakes for model optimization training, effectively improving the data utilization efficiency of AI data lakes.

[0136] The following is for reference. Figure 6 The diagram illustrates a structural schematic of an electronic device 600 suitable for implementing the above-described technical solution. The terminal device may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Tablet Personal Computers), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs (Televisions), desktop computers, etc. Figure 6 The electronic device shown is merely an example and should not be construed as limiting its functionality or scope of use.

[0137] like Figure 6 As shown, electronic device 600 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 602 or a program loaded from storage device 608 into random access memory (RAM) 603. The random access memory 603 also stores various programs and data required for the operation of electronic device 600. The processing unit 601, ROM 602, and RAM 603 are interconnected via bus 604. An input / output (I / O) interface 605 is also connected to bus 604.

[0138] Typically, the following devices can be connected to the input / output interface 605: input devices 606 including, for example, a touchscreen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 607 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 608 including, for example, magnetic tape, hard disk, etc.; and communication devices 609. Communication device 609 allows electronic device 600 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 6 An electronic device 600 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0139] In particular, depending on certain circumstances, the processes described in the flowchart above can be implemented as computer software programs. For example, a computer program product is provided, comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowchart. This computer program can be downloaded and installed from a network via communication device 609, or installed from storage device 608, or installed from read-only memory 602. When the computer program is executed by processing device 601, it performs the functions defined in the above-described methods.

[0140] It should be noted that the aforementioned computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM, or flash memory), optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In one case, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In another case, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. The transmitted data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (Radio Frequency), etc., or any suitable combination thereof.

[0141] In some implementations, communication can be conducted using any currently known or future-developed network protocol, such as HTTP (Hypertext Transfer Protocol), and can be interconnected with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (LANs), wide area networks (WANs), the internet (e.g., the Internet), and end-to-end networks (e.g., ad-hoc end-to-end networks), as well as any currently known or future-developed networks.

[0142] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0143] The aforementioned computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device causes the following: it acquires first multimodal data from an AI data lake; it splits the first multimodal data into multiple segments to obtain a first segment sequence; and it randomly adjusts the segment order of the first segment sequence to obtain a second segment sequence; it generates a first masking strategy for the second segment sequence based on a first score and a second score using a reinforcement learning model; and it performs masking processing on the second segment sequence based on the first masking strategy to obtain a third segment sequence. The first score is used to characterize the temporal reconstruction difficulty of the first multimodal data, and the second score is used to... The multimodal model is characterized by its temporal reconstruction accuracy prior to this training. The multimodal model is then used to process the third segment sequence to obtain a first processing result. This first processing result includes first prediction information corresponding to each segment in the third segment sequence and second prediction information corresponding to the masked segments in the third segment sequence. The first prediction information indicates the predicted temporal position of the corresponding segment in the first segment sequence, and the second prediction information includes the prediction content obtained by the multimodal model after reconstructing the masked segments. Based on the first processing result and the first segment sequence, a loss is calculated, and the model parameters of the multimodal model are updated according to the loss calculation result.

[0144] Computer program code for performing the above operations can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include, but are not limited to, object-oriented programming languages, as well as conventional procedural programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0145] The flowcharts and block diagrams in the accompanying figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products under various scenarios. In this respect, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the figures. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0146] The modules mentioned above can be implemented in software or hardware. In some cases, the name of a module does not necessarily limit the functionality of that module.

[0147] The functions described above can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field-Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application-Specific Standard Parts (ASSPs), Systems on Chips (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0148] In this context, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0149] The above description is merely illustrative and explains the technical principles employed. Those skilled in the art should understand that the scope of the technical solution is not limited to specific combinations of the above-described technical features, but also includes other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features provided herein that have similar functions.

[0150] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. Multitasking and parallel processing may be advantageous in certain environments. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limitations on the scope of the technical solution. Certain features described in the context of a single example can also be implemented in combination in a single example. Conversely, various features described in the context of a single example can also be implemented individually or in any suitable sub-combination in multiple examples.

[0151] Although the technical solution has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims. Regarding the aforementioned apparatus, the specific manner in which each module performs its operation has already been described in detail in the section concerning the method, and will not be elaborated upon here.

Claims

1. A model optimization method based on AI data lake, comprising: First multimodal data is obtained from the AI ​​data lake, the first multimodal data is split into multiple segments to obtain a first segment sequence, and the segment order of the first segment sequence is randomly adjusted to obtain a second segment sequence; A first masking strategy is generated for the second segment sequence based on the first score and the second score by a reinforcement learning model, and the second segment sequence is masked based on the first masking strategy to obtain the third segment sequence. The first score is used to characterize the temporal reconstruction difficulty of the first multimodal data, and the second score is used to characterize the temporal reconstruction accuracy of the multimodal model before this training. The third segment sequence is processed by the multimodal model to obtain a first processing result. The first processing result includes first prediction information corresponding to each segment in the third segment sequence and second prediction information corresponding to the masked segment in the third segment sequence. The first prediction information is used to indicate the prediction temporal position of the corresponding segment in the first segment sequence. The second prediction information includes the prediction content obtained by the multimodal model after restoring the masked segment. Loss is calculated based on the first processing result and the first segment sequence, and the model parameters of the multimodal model are updated according to the loss calculation result; The first score is determined based on at least one of modal content similarity, modal content change frequency, and cross-modal content correlation. Modal content similarity measures the degree of content similarity between different segments in the first segment sequence under the same modality. Modal content change frequency measures the degree of content change within the same segment under the same modality in the first segment sequence. Cross-modal content correlation measures the degree of content matching between different modalities within the same segment in the first segment sequence. The second score is determined as follows: The fifth segment sequence is obtained from the validation set. The fifth segment sequence is obtained by randomly adjusting the segment order of the sixth segment sequence, which is obtained by splitting the third multimodal data. The fifth segment sequence is temporally predicted using the multimodal model to obtain fourth prediction information, which is used to indicate the predicted temporal position of the corresponding segment in the sixth segment sequence. Based on the difference between the fourth prediction information and the sixth segment sequence, a fourth score is determined, which characterizes the temporal reconstruction accuracy of the multimodal model; The second score is determined based on the fourth score.

2. The method according to claim 1, wherein the first masking strategy includes at least one of the following parameters: Masking modality selection parameters are used to indicate whether to mask at least one of the text modality, visual modality, and audio modality; The masked segment index parameter is used to indicate the temporal position of the masked segment in the second segment sequence; The masking strength level parameter indicates the masking strength of the masked segment; The masking type parameter indicates the masking method for the mode corresponding to the masking mode selection parameter.

3. The method according to claim 1, wherein the reinforcement learning model comprises a first network and a second network, wherein, The first network is used to output a masking strategy, the second network is used to output a predicted reward score, and the method further includes: The first true reward score is determined based on a preset reward function, and the first predicted reward score is obtained through the second network based on the first score, the second score, and the first masking strategy. The parameters of the first network and the second network are updated based on the first actual reward score and the first predicted reward score; The preset reward function includes at least one of the following: The first function term represents that a first positive reward is given when the temporal reconstruction accuracy of the first predicted information is higher than that of the third predicted information, and a first negative reward is given otherwise; wherein, the third predicted information is obtained by the multimodal model performing temporal prediction on the first segment sequence; The second function term represents that when the temporal reconstruction accuracy of the first prediction information is less than the first preset threshold, a second negative reward is given; otherwise, a second positive reward is given. The third function term represents a third positive reward when the variance between the temporal reconstruction accuracy of the first prediction information and the temporal reconstruction accuracy of the historical prediction information is less than a second preset threshold; otherwise, a third negative reward is given. The historical prediction information is obtained by the multimodal model through temporal prediction of randomly adjusted historical fragment sequences.

4. The method according to any one of claims 1-3, wherein the multimodal model includes a discriminator, the discriminator being used to determine the probability that each segment in the input segment sequence is in a temporally normal position; The method further includes: A transformation that preserves the timing logic is applied to the first segment sequence to obtain a first sample, and a perturbation that disrupts the timing logic is applied to the first segment sequence to obtain a second sample; The discriminator is trained by comparing the first sample as a positive sample, the second sample as a negative sample, and the first prediction information as an anchor sample.

5. The method according to claim 4, further comprising: Acquire the second multimodal data, split the second multimodal data into multiple segments, and obtain the fourth segment sequence; The discriminator determines the third score corresponding to each segment in the fourth segment sequence, and marks the segments in the fourth segment whose third score is less than a third preset threshold with a first label. The third score represents the probability that the corresponding segment is in a normal temporal position, and the first label represents the risk of temporal anomaly in the corresponding segment. The third score and the fourth segment sequence labeled with the first tag are sent to the data detection system, which is used to detect data anomalies in the fourth segment sequence.

6. The method according to any one of claims 1-3, wherein each segment in the second segment sequence is associated with a first position component and a second position component, the first position component representing the temporal position of the corresponding segment in the first segment sequence, and the second position component representing the temporal position of the corresponding segment in the second segment sequence.

7. The method according to any one of claims 1-3, further comprising: Obtain intermediate data during the training process of the multimodal model, the intermediate data including the feature vector corresponding to the third segment sequence; The intermediate data is stored in the AI ​​data lake.

8. A model optimization device based on an AI data lake, comprising: The acquisition module is used to acquire first multimodal data from the AI ​​data lake, split the first multimodal data into multiple segments to obtain a first segment sequence, and randomly adjust the segment order of the first segment sequence to obtain a second segment sequence; The masking module is used to generate a first masking strategy for the second segment sequence based on a first score and a second score through a reinforcement learning model, and to mask the second segment sequence based on the first masking strategy to obtain a third segment sequence. The first score is used to characterize the temporal reconstruction difficulty of the first multimodal data, and the second score is used to characterize the temporal reconstruction accuracy of the multimodal model before this training. The processing module is used to process the third segment sequence through the multimodal model to obtain a first processing result. The first processing result includes first prediction information corresponding to each segment in the third segment sequence and second prediction information corresponding to the masked segment in the third segment sequence. The first prediction information is used to indicate the prediction temporal position of the corresponding segment in the first segment sequence. The second prediction information includes the prediction content obtained by the multimodal model after restoring the masked segment. An update module is used to perform loss calculation based on the first processing result and the first segment sequence, and update the model parameters of the multimodal model according to the loss calculation result; The first score is determined based on at least one of modal content similarity, modal content change frequency, and cross-modal content correlation. Modal content similarity measures the degree of content similarity between different segments in the first segment sequence under the same modality. Modal content change frequency measures the degree of content change within the same segment under the same modality in the first segment sequence. Cross-modal content correlation measures the degree of content matching between different modalities within the same segment in the first segment sequence. The second score is determined as follows: The fifth segment sequence is obtained from the validation set. The fifth segment sequence is obtained by randomly adjusting the segment order of the sixth segment sequence, which is obtained by splitting the third multimodal data. The fifth segment sequence is temporally predicted using the multimodal model to obtain fourth prediction information, which is used to indicate the predicted temporal position of the corresponding segment in the sixth segment sequence. Based on the difference between the fourth prediction information and the sixth segment sequence, a fourth score is determined, which characterizes the temporal reconstruction accuracy of the multimodal model; The second score is determined based on the fourth score.

9. A computer-readable medium having a computer program stored thereon, wherein, When executed by a processing device, the computer program performs the steps of the method according to any one of claims 1-7.

10. An electronic device, comprising: A storage device on which computer programs are stored; A processing device for executing the computer program in the storage device to implement the steps of the method according to any one of claims 1-7.

Images