Data processing method and device, equipment and medium

Abstract

The invention discloses a data processing method and device, equipment and a medium. Comprising the steps that a training sample set is obtained, the training sample set comprises a plurality of sample pairs, each sample pair comprises an image sample and a text sample, and through a to-be-trained model, prediction is carried out based on the image samples and the text samples to obtain an output lexical element sample sequence, determining a first target lexical element with relatively high confidence and a second target lexical element with relatively low confidence from a plurality of output lexical elements contained in the output lexical element sample sequence; filtering the loss value corresponding to the first target lexical element to obtain a first target loss value; performing upper limit constraint on the loss value corresponding to the second target lexical element to obtain a second target loss value; and training based on the first target loss value and the second target loss value to obtain a visual language model. According to the technical scheme, the reliability of data processing in a visual language model scene is improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically, to a data processing method, a data processing apparatus, an electronic device, and a computer-readable medium. Background Technology

[0002] Understandably, in the context of Vision-Language Models (VLMs), both the question image and the question text are input into the VLM, which then outputs the corresponding response to achieve complex tasks such as visual question answering, image caption generation, and text-based image retrieval. During the training process of VLMs, standard cross-entropy loss is typically used. This leads to the VLM rapidly overfitting to frequently occurring head data in the early stages of training, while underfitting to the scarce tail data. Furthermore, as training progresses, the VLM becomes overconfident in the head data, resulting in a decline in performance on tail data and a catastrophic forgetting phenomenon.

[0003] Currently, most methods for dealing with long-tail data distribution rely on prior knowledge of the data distribution. However, in the complex scenario of visual language models, long-tail biases are intertwined in multiple dimensions, making prior knowledge of the data distribution unsuitable for training visual language models, thereby reducing the reliability of data processing in visual language model scenarios.

[0004] Therefore, improving the reliability of data processing in visual language model scenarios is an urgent problem to be solved. Summary of the Invention

[0005] The embodiments of this application provide a data processing method, apparatus, device, and medium that improve the reliability of data processing in visual language model scenarios.

[0006] In a first aspect, embodiments of this application provide a data processing method, the method comprising: acquiring a training sample set, the training sample set including multiple sample pairs, the sample pairs including image samples and text samples, and inputting the training sample set into a model to be trained, and performing the following steps through the model to be trained: predicting based on the image samples and the text samples to obtain an output word sample sequence, the output word sample sequence including multiple output words, and determining a first target word with a confidence greater than a preset confidence level and a second target word with a confidence level less than or equal to the preset confidence level from the multiple output words; filtering the loss value corresponding to the first target word to obtain a first target loss value; applying an upper limit constraint to the loss value corresponding to the second target word to obtain a second target loss value; and training the model to be trained based on the first target loss value and the second target loss value to obtain a visual language model, the visual language model being used to generate response content based on the input question image and question text.

[0007] Secondly, embodiments of this application provide a data processing apparatus, the apparatus comprising: an acquisition module configured to acquire a training sample set, the training sample set including multiple sample pairs, the sample pairs including image samples and text samples, and input the training sample set into a model to be trained, through which the model to be trained performs the following steps: a prediction and determination module configured to perform prediction based on the image samples and the text samples to obtain an output word sample sequence, the output word sample sequence including multiple output words, and determine a first target word with a confidence greater than a preset confidence level and a second target word with a confidence level less than or equal to the preset confidence level from the multiple output words; a filtering module configured to filter the loss value corresponding to the first target word to obtain a first target loss value; a constraint module configured to impose an upper limit constraint on the loss value corresponding to the second target word to obtain a second target loss value; and a training module configured to train the model to be trained based on the first target loss value and the second target loss value to obtain a visual language model, the visual language model being used to generate response content based on the input question image and question text.

[0008] In one embodiment of this application, based on the aforementioned scheme, the training of the model to be trained includes multiple training batches, and each training batch is used to update the model parameters of the model to be trained once; correspondingly, the device further includes: a threshold update module, configured to: calculate a confidence increment based on the confidence of the first target word corresponding to the current training batch; sum the confidence increment and the preset confidence corresponding to the current training batch to obtain an updated preset confidence, and the updated preset confidence is used to re-determine the first target word and the second target word in the next training batch adjacent to the current training batch.

[0009] In one embodiment of this application, based on the aforementioned scheme, the first target word corresponding to the current training batch includes multiple words; correspondingly, the threshold update module is specifically configured as follows: for each first target word, the confidence level corresponding to the first target word and the preset confidence level corresponding to the current training batch are compared to obtain a confidence ratio, and the confidence ratio is determined as the base, the first preset adjustment factor is determined as the exponent, and an exponentiation operation is performed to obtain the exponentiation result, wherein the first preset adjustment factor is a positive number; based on the preset incremental update rate and the exponentiation results corresponding to the multiple first target words, the confidence increment is obtained, wherein the preset incremental update rate is a positive number.

[0010] In one embodiment of this application, based on the foregoing scheme, the threshold update module is further configured to: sum the exponentiation results corresponding to the plurality of first target words to obtain a total value; and multiply the preset incremental update rate and the total value to obtain the confidence increment.

[0011] In one embodiment of this application, based on the aforementioned scheme, the constraint module is specifically configured to: calculate the weight of the second target word based on the confidence level of the second target word; perform a weighted operation on the loss value corresponding to the second target word based on the weight to obtain the second target loss value, wherein the second target loss value is less than or equal to a preset maximum loss value.

[0012] In one embodiment of this application, based on the foregoing scheme, the constraint module is further configured to: calculate the loss value corresponding to the second target word based on the confidence level of the second target word; perform a ratio operation on the preset maximum loss value and the loss value corresponding to the second target word to obtain a loss ratio; if the loss ratio is greater than or equal to the standard weight threshold, then determine the weight of the second target word as the standard weight threshold; if the loss ratio is less than the standard weight threshold, then determine the weight of the second target word as the loss ratio.

[0013] In one embodiment of this application, based on the foregoing scheme, the constraint module is further specifically configured to: sum the loss value corresponding to the second target word and the second preset adjustment factor to obtain an intermediate loss value, wherein the second preset adjustment factor is a positive number; and perform a ratio calculation on the preset maximum loss value and the intermediate loss value to obtain the loss ratio.

[0014] In one embodiment of this application, based on the aforementioned scheme, the prediction and determination module is specifically configured as follows: obtaining a visual word sample sequence corresponding to the image sample, the visual word sample sequence including multiple visual words; determining visual words to be masked from the multiple visual words whose word entropy is less than a preset word entropy; performing masking processing on the visual words to be masked to obtain a target visual word sample sequence; and performing prediction based on the target visual word sample sequence and the text sample to obtain the output word sample sequence.

[0015] In one embodiment of this application, based on the foregoing scheme, the prediction and determination module is further configured to: sort the plurality of visual lexical units in descending order of lexical entropy to obtain a sorted sequence; determine the lexical entropy of the visual lexical unit corresponding to the preset quantile in the sorted sequence as the preset lexical entropy, and determine the visual lexical units arranged after the visual lexical unit corresponding to the preset quantile as candidate mask visual lexical units; and randomly sample the candidate mask visual lexical units to obtain the visual lexical units to be masked.

[0016] In one embodiment of this application, based on the foregoing scheme, the prediction and determination module is further configured to: obtain the text lexical sample sequence corresponding to the text sample; concatenate the target visual lexical sample sequence and the text lexical sample sequence to obtain the input lexical sample sequence; and perform prediction based on the input lexical sample sequence to obtain the output lexical sample sequence. In one embodiment of this application, based on the foregoing scheme, the prediction and determination module is further specifically configured as follows: the lexical entropy of any visual lexical is calculated by: obtaining the probability distribution on the vocabulary based on the prediction of the visual lexical; calculating the entropy value corresponding to the visual lexical for representing the amount of information based on the degree of dispersion of the probability distribution, wherein the degree of dispersion is proportional to the entropy value; and determining the entropy value as the lexical entropy of the visual lexical.

[0017] In one embodiment of this application, based on the foregoing scheme, the training module is specifically configured to: sum the first target loss value and the second target loss value to obtain the total sample loss of the sample pair; average the total sample losses corresponding to the plurality of sample pairs to obtain the model loss; and update the model parameters of the model to be trained based on the model loss to obtain the visual language model.

[0018] In one embodiment of this application, based on the foregoing scheme, the device further includes a reasoning module configured to: receive the input question image and the question text; input the question image and the question text into the visual language model, and perform the following steps through the visual language model: obtain the visual word sequence corresponding to the question image, and obtain the text word sequence corresponding to the question text; concatenate the visual word sequence and the text word sequence to obtain the input word sequence; perform prediction based on the input word sequence to obtain the output word sequence, and determine the output word sequence as the response content.

[0019] Thirdly, embodiments of this application provide an electronic device, including one or more processors; and a memory for storing one or more computer programs, which, when executed by the one or more processors, cause the electronic device to implement the data processing method described above.

[0020] Fourthly, embodiments of this application provide a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the data processing method described above.

[0021] Fifthly, embodiments of this application provide a computer program product, including computer instructions, which, when executed by a processor, implement the data processing method described above.

[0022] In the technical solutions provided by the embodiments of this application: By identifying high-confidence terms and filtering their loss values, high-confidence terms are filtered out. This frees up training resources from learning from the head data, leaving more room for learning from the tail data. It avoids the phenomenon that the model under training overlearns from the head data in the later stages of training, leading to irreversible overfitting of the head data and impaired learning ability of the tail data.

[0023] By identifying low-confidence terms and imposing upper limits on their loss values, the upper limits of the loss values ​​for low-confidence terms are controlled. This allows for appropriate attention to tail data, reasonable constraints on optimization intensity, and avoids the phenomenon that the model to be trained may perform sharp optimization on low-confidence terms during training, leading to bias in the virtual mode and a high risk of overfitting.

[0024] Furthermore, training is performed based on the first target loss value of high-confidence words and the second target loss value of low-confidence words. This achieves a combination of high-confidence word filtering and low-confidence word adaptive reweighting, which together constitute bounded adaptive loss training. This improves the training effect of the visual language model, and the generalization performance and robustness of the visual language model are both high, thereby improving the reliability of data processing in the visual language model scenario.

[0025] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0026] Figure 1 This is a schematic diagram illustrating an exemplary implementation environment in which the technical solutions of the embodiments of this application can be applied.

[0027] Figure 2 This is a flowchart illustrating a data processing method in an exemplary embodiment of this application.

[0028] Figure 3 This is a schematic diagram illustrating a data processing method in an exemplary embodiment of this application.

[0029] Figure 4 This is a schematic diagram illustrating a data processing method in another exemplary embodiment of this application.

[0030] Figure 5 This is a flowchart illustrating a data processing method in another exemplary embodiment of this application.

[0031] Figure 6 This is a schematic diagram illustrating a data processing method in another exemplary embodiment of this application.

[0032] Figure 7 This is a block diagram illustrating a data processing apparatus according to an exemplary embodiment of this application.

[0033] Figure 8 This is a schematic diagram of the structure of a computer system suitable for implementing the electronic devices of the present application embodiments. Detailed Implementation

[0034] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments identical to those described in this application. Rather, they are merely examples of apparatuses and methods identical to some aspects of this application as detailed in the appended claims.

[0035] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0036] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0037] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0038] It should be noted that "multiple" as mentioned in this application refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0039] Before detailing the technical solutions of the embodiments of this application, let's first introduce the technical terms used in the embodiments of this application.

[0040] Visual language models are cross-modal artificial intelligence models capable of simultaneously understanding and processing visual (image / video) and linguistic (text) information. Their core objective is to establish deep semantic alignment and association between vision and language, enabling complex tasks such as visual question answering, image description generation, and text-based image retrieval. Currently, mainstream visual language models are typically built on a Transformer architecture, pre-trained with large-scale image-text sample pairs to learn how to map visual features to a linguistic semantic space, or vice versa, thereby achieving joint representation and reasoning across modalities. For example, contrastive language-image pre-training (CLIP) models can learn universal representations of images and text, while bootstrapping language-image pre-training (BLIP) models focus on generative visual language understanding and interaction.

[0041] Long-tailed data distribution refers to the imbalanced statistical phenomenon in real-world datasets where a few categories (the head) have a large number of samples, while most categories (the tail) have only a very small number of samples. The distribution curve exhibits a very high "head" and a gradually declining, but very long, "tail." In machine learning, long-tailed distribution is a prevalent and extremely challenging problem because models are easily dominated by the data-rich head categories, leading to severely insufficient learning and poor generalization performance for the tail categories (which, although having fewer individuals, represent a large total number of categories).

[0042] Parameter-Efficient Fine-Tuning (PEFT) is a fine-tuning strategy that adapts a large pre-trained model to downstream tasks by updating or introducing only a very small number of additional model parameters while freezing the vast majority of the original parameters. Its core objective is to efficiently adapt the model to a specific task with extremely low computational and storage costs, while maintaining the powerful general knowledge of the pre-trained model. Common techniques include adapters, prefix-tuning, and low-rank adaptation (LoRA). PEFT solves the problems of huge resource consumption and catastrophic forgetting associated with full-parameter fine-tuning, making it a key technology for achieving lightweight and personalized deployment in the model era.

[0043] Understandably, in the context of visual language models, both the question image and the question text are input into the model, which then outputs the corresponding response to achieve complex tasks such as visual question answering, image description generation, and text-based image retrieval. However, during the training process, visual language models typically use standard cross-entropy loss. This leads to rapid overfitting of frequently occurring head data in the early stages of training, while underfitting the scarce tail data. Furthermore, as training progresses, the visual language model becomes overconfident in the head data, resulting in a decline in performance on tail data and a catastrophic forgetting phenomenon.

[0044] Currently, most methods for dealing with long-tail data distribution rely on prior knowledge of the data distribution. However, in the complex scenario of visual language models, long-tail biases are intertwined in multiple dimensions, making prior knowledge of the data distribution unsuitable for training visual language models, thereby reducing the reliability of data processing in visual language model scenarios.

[0045] Therefore, to improve the reliability of data processing in visual language model scenarios, this application provides a data processing scheme. Please refer to... Figure 1 , Figure 1 This is a schematic diagram of an implementation environment related to this application. The implementation environment mainly includes a terminal device 101 and a server 102; wherein: Terminal devices 101 include, but are not limited to, mobile phones, computers (such as tablets, laptops, desktop computers, etc.), smart home devices (such as televisions, refrigerators, air conditioners, washing machines, robot vacuums, etc.), smart wearable devices (such as wristbands, watches, etc.), and extended reality devices (such as virtual reality devices, augmented reality devices, mixed reality devices, etc.).

[0046] Server 102 can be a standalone physical server, or a server cluster or distributed system consisting of multiple physical servers. The server cluster or distributed system includes cloud servers used to provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0047] It is understood that terminal device 101 and server 102 establish a communication connection via a wired or wireless network. Exemplarily, the wireless or wired network uses standard communication technologies and / or protocols. The network is typically the Internet, but can also be any other network, including but not limited to a Local Area Network (LAN), Metropolitan Area Network (MAN), Wide Area Network (WAN), mobile, wired or wireless networks, private networks, or any combination of virtual private networks.

[0048] In one embodiment of this application, the data processing method can be executed by server 102. Specifically, the following steps are performed: First, a training sample set is acquired, which includes multiple sample pairs, including image samples and text samples. The training sample set is then input into the model to be trained. The model to be trained performs the following steps: Based on the image samples and text samples, prediction is performed to obtain an output word sample sequence, which includes multiple output words. From the multiple output words, a first target word with a confidence level greater than a preset confidence level and a second target word with a confidence level less than or equal to the preset confidence level are determined. For the first target word, the loss value corresponding to the first target word is filtered to obtain a first target loss value. For the second target word, an upper limit constraint is applied to the loss value corresponding to the second target word to obtain a second target loss value. Finally, based on the first target loss value and the second target loss value, the model to be trained is trained to obtain a visual language model. The visual language model is used to generate response content based on the input question image and question text. It should be clarified that the data processing solution provided in this application can be executed by server 102 alone, by terminal device 101 alone, or by terminal device 101 and server 102 together. In practical applications, the executing entity of the data processing solution can be flexibly adjusted according to the specific application scenario. It is understood that... Figure 1 The number of terminal devices 101 and servers 102 shown is merely illustrative; any number of terminal devices 101 and servers 102 can be used as needed.

[0049] It should be noted that in the specific implementation of this application, user-related data is involved. When the embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0050] The following details the various implementation details of the technical solutions in the embodiments of this application: Please see Figure 2 , Figure 2 This is a flowchart illustrating a data processing method according to an embodiment of this application. Figure 2 As shown, this data processing method includes at least S201 to S205, which are described in detail below: S201, Obtain the training sample set, which includes multiple sample pairs, including image samples and text samples, and input the training sample set into the model to be trained, and execute S202 to S205 through the model to be trained.

[0051] In this embodiment, the training sample set consists of multiple sample pairs. Each sample pair includes an image sample and its associated text sample. The association can be reflected in the text sample being a description of the image sample, a question posed to the image sample, or an instruction related to the content of the image sample. It is understood that the data in the training sample set often exhibits long-tail data distribution characteristics. For example, the number of sample pairs corresponding to common objects, scenes, and concepts (such as "person," "car," and "sky") in the training sample set is large, forming the "head" of the distribution; while the number of sample pairs corresponding to a large number of uncommon, fine-grained, and professional semantic categories (such as specific species, rare objects, and complex relationships) is small, but the total number of categories is large, forming the "long tail" of the distribution.

[0052] After obtaining the training sample set in this embodiment, the training sample set can be input into the model to be trained, and then the model to be trained can execute S202 to S205 to train the model to be trained, thereby obtaining a visual language model.

[0053] S202, based on image samples and text samples, prediction is performed to obtain an output word sample sequence. The output word sample sequence includes multiple output words, and from the multiple output words, a first target word with a confidence level greater than a preset confidence level and a second target word with a confidence level less than or equal to the preset confidence level are determined.

[0054] For ease of explanation, this application embodiment takes the processing of a single sample pair as an example. That is, the model to be trained can predict the image samples and text samples included in the sample pair to obtain an output word sample sequence with multiple output words, and obtain the confidence of each output word. Then, from the multiple output words, it determines the first target word with a confidence greater than the preset confidence and the second target word with a confidence less than or equal to the preset confidence.

[0055] In the embodiments of this application, the output word sample sequence refers to the discretized representation of the text sequence corresponding to the task objective generated by the model to be trained for the sample pair, including image samples and text samples. For example, in a visual question answering task, if the text sample is "What's in the image?", and the image sample contains a cat, then the model to be trained should reply "a cat". At this time, the word list obtained after the word segmenter (e.g., ["one", "only", "cat"]) is the output word sample sequence, where each output word is arranged in the order of generation in the output word sample sequence.

[0056] In this embodiment, the confidence level of the output word refers to the probability value calculated by the model under test, given image samples, text samples, and previously generated output words as context, when predicting the output word at any specific position (e.g., the t-th position) in the output word sample sequence. This probability value reflects the degree of confidence the model under test has in the correctness of its prediction of the output word, and typically ranges from 0 to 1. A higher confidence level indicates that the model under test believes the output word is more likely to appear in the current context, and its knowledge is more robust; conversely, a lower confidence level indicates that the model under test believes the output word is less likely to appear in the current context, and its knowledge is less robust.

[0057] In the embodiments of this application, the first target word refers to the output word with higher confidence among multiple output words, also known as a high-confidence word. These output words usually correspond to head concepts, simple patterns, etc., that the model to be trained has fully learned and that frequently appear in the training sample set. Because of their high confidence, it indicates that the model to be trained is "overconfident" or "has mastered" them well.

[0058] In this embodiment, the second target word refers to the output word with lower confidence among multiple output words, also known as low-confidence word. These output words usually correspond to knowledge points that the model to be trained has not yet mastered and cannot predict with sufficient certainty. They are often associated with tail concepts and complex patterns that appear less frequently in the training sample set. Because of their low confidence, it indicates that the model to be trained "lacks confidence" or "under-learns" them.

[0059] In this embodiment, the pre-set confidence level refers to the scalar threshold used to classify output lexical units (i.e., classify them as the first target lexical unit or the second target lexical unit). It can be static or dynamic. When it is dynamic, it can be set to a relatively low value in the early stage of training so as to focus on quickly improving the overall model capability in the early stage of training. As training progresses, the model under training becomes highly confident in its prediction of certain concepts, and the confidence level can be appropriately increased to continuously raise the standard of "mastery" and drive the model under training to continuously allocate optimization resources to more challenging parts.

[0060] In one embodiment of this application, the process of predicting based on image samples and text samples to obtain an output word sample sequence in S202 may include: obtaining a visual word sample sequence corresponding to an image sample, wherein the visual word sample sequence includes multiple visual words; determining visual words to be masked from the multiple visual words whose word entropy is less than a preset word entropy; performing masking processing on the visual words to be masked to obtain a target visual word sample sequence; and predicting based on the target visual word sample sequence and text samples to obtain an output word sample sequence.

[0061] That is, in the optional embodiment, the image sample is first segmented and encoded to obtain a visual word sample sequence with multiple visual words corresponding to the image sample, and the word entropy of each visual word is obtained. Then, the visual words to be masked are determined from the multiple visual words whose word entropy is less than the preset word entropy. After that, the visual words to be masked are masked to obtain the target visual word sample sequence. Finally, prediction is performed based on the target visual word sample sequence and the text sample to obtain the output word sample sequence.

[0062] In the optional embodiment, the visual word sample sequence refers to the vector sequence used to represent the local region features of the image after the visual encoder (e.g., the Vision Transformer (ViT)) in the model to be trained segments and encodes the image sample. It includes multiple visual words, and each visual word can correspond to an image patch in the image sample.

[0063] In one optional embodiment, lexical entropy refers to a metric used to quantify the uncertainty or richness of information carried by a visual lexical unit. A higher lexical entropy indicates more dispersed and uncertain predictions of the visual region, typically corresponding to information-rich and semantically crucial objects or detailed areas; conversely, a lower lexical entropy indicates more concentrated and certain predictions of the visual region, typically corresponding to information-redundant or simple background areas.

[0064] In an optional embodiment, the target visual word sample sequence refers to a new sequence obtained by performing masking processing on the visual word sample sequence (i.e., the original sequence). Specifically, the new sequence is obtained by replacing the feature vectors of the "visual words to be masked" in the original sequence with preset masking markers (such as zero vectors or learnable masking embedding vectors). It is understood that the new sequence retains the same length and structure as the original sequence, but information at some positions is masked, thus constituting an enhanced and more challenging visual input.

[0065] By implementing optional embodiments, simple visual cues with low information content and prone to over-reliance (i.e., low-entropy visual lexics) are adaptively masked, forcing the model to focus more on and utilize complex visual cues (i.e., high-entropy visual lexics) in image samples that are richer in information and more discriminative during feature extraction and context understanding. Thus, by using visual lexicographic masking, the overfitting tendency of the model to the head data in long-tail data distributions can be effectively mitigated, and the ability to discriminate the tail data is enhanced, thereby improving the overall model's generalization performance and robustness.

[0066] In some embodiments, the lexical entropy of any visual lexical unit is calculated as follows: obtaining the probability distribution on the vocabulary based on the prediction of the visual lexical unit; calculating the entropy value corresponding to the visual lexical unit for representing the amount of information based on the degree of dispersion of the probability distribution, wherein the degree of dispersion is proportional to the entropy value; and determining the entropy value as the lexical entropy of the visual lexical unit.

[0067] That is, in the optional embodiment, the probability distribution on the vocabulary based on visual word prediction is first obtained, and then the entropy value corresponding to the visual word, which is used to represent the amount of information, is calculated based on the degree of dispersion of the probability distribution. The degree of dispersion is proportional to the entropy value, and the entropy value is the word entropy of the visual word.

[0068] In an optional embodiment, the probability distribution on the vocabulary refers to the set of probabilities predicted by the model to be trained for each possible visual word in the vocabulary as the next generation target, given image samples, text samples, and previously processed visual words as context. This probability distribution is a vector with a dimension equal to the size of the vocabulary, where the value of each element represents the probability that the corresponding visual word will be associated with or predicted by the model to be trained, and the sum of all elements is 1.

[0069] In the optional embodiment, the dispersion of the probability distribution refers to a metric used to quantify whether the probabilities in the probability distribution are concentrated on a few visual words or relatively evenly distributed across multiple visual words. If the distribution is highly concentrated (e.g., the probability of a certain visual word is close to 1, while the others are close to 0), the dispersion is low, and correspondingly, the entropy value is low; if the distribution is uniform and flat (e.g., multiple visual words have similar non-zero probabilities), the dispersion is high, and correspondingly, the entropy value is high. Optionally, the dispersion of the probability distribution can be calculated using Shannon entropy, Gini coefficient, variance, etc.

[0070] By implementing optional embodiments, the information value carried by visual lexical units can be objectively and quantitatively evaluated by using prediction uncertainty. The lexical entropy of visual lexical units can be calculated easily and accurately. The entire process is adaptively matched with the signals generated internally in specific tasks, so that the masking process is closely aligned with the actual learning needs, improving the accuracy of masking process, thereby further improving the generalization performance and robustness of the overall model.

[0071] In some embodiments, the process of determining visual words to be masked from a visual word sample sequence whose word entropy is less than a preset word entropy may include: sorting multiple visual words in descending order of word entropy to obtain a sorted sequence; determining the word entropy of the visual words corresponding to a preset quantile in the sorted sequence as the preset word entropy, and determining the visual words arranged after the visual words corresponding to the preset quantile as candidate mask visual words; and randomly sampling the candidate mask visual words to obtain the visual words to be masked.

[0072] That is, in the optional embodiment, a dynamic preset word entropy related to the image content is first determined by sorting multiple visual words and a preset quantile (e.g., the kth percentile). Then, candidate mask visual words with relatively low information content relative to the preset word entropy are identified. Finally, the final mask visual words are obtained by randomly sampling from the candidate mask visual words.

[0073] By implementing optional embodiments and using preset quantiles, the algorithm can automatically adapt to the lexical entropy distribution of the image content itself. This allows the masking process to be dynamically adjusted according to the complexity of the image content (e.g., the overall lexical entropy of simple image content may be low, while the overall lexical entropy of complex image content may be high), resulting in high universality. At the same time, random sampling of candidate mask visual lexical units, rather than masking all of them, ensures the diversity of data augmentation and prevents the model to be trained from learning a certain fixed masking pattern during training, thus preventing dependence or bias. This encourages the model to learn more robust and generalized feature representations, thereby further improving the generalization performance and robustness of the overall model.

[0074] In some embodiments, the process of predicting based on the target visual word sample sequence and text samples to obtain the output word sample sequence may include: obtaining the text word sample sequence corresponding to the text sample; concatenating the target visual word sample sequence and the text word sample sequence to obtain the input word sample sequence; and predicting based on the input word sample sequence to obtain the output word sample sequence.

[0075] That is, in the optional embodiment, the text sample is first segmented and encoded to obtain a text word sample sequence with multiple text words corresponding to the text sample. Then, the visual words contained in the target visual word sample sequence and the text words contained in the text word sample sequence are concatenated to obtain the input word sample sequence. Finally, prediction is performed based on the input word sample sequence to obtain the output word sample sequence.

[0076] In the optional embodiment, the text word sample sequence refers to the vector sequence used to represent the basic units of language after the text sample is segmented and encoded by the word segmenter in the model to be trained. It includes multiple text words, each of which can be a word, a sub-word unit, a special symbol, etc.

[0077] By implementing optional embodiments, the masked visual feature sequence (i.e., the target visual word sample sequence) and the text feature sequence (i.e., the text word sample sequence) are concatenated, which can easily and accurately obtain an input word sequence that contains both visual and linguistic context, thus providing strong support for the accurate acquisition of the output word sequence.

[0078] In one embodiment of this application, the training of the model to be trained includes multiple training batches, and each training batch is used to update the model parameters of the model to be trained once. Accordingly, it may also include: calculating the confidence increment based on the confidence of the first target word corresponding to the current training batch; summing the confidence increment and the preset confidence corresponding to the current training batch to obtain the updated preset confidence, and the updated preset confidence is used to re-determine the first target word and the second target word in the next training batch adjacent to the current training batch.

[0079] That is, in the optional embodiment, the preset confidence is dynamically variable, wherein the preset confidence will continuously increase as training progresses, and the update of the preset confidence is aligned with the training pace of stochastic gradient descent. Specifically, an update of the preset confidence will be triggered after a batch of training is completed (i.e., a model parameter update).

[0080] In one optional embodiment, the confidence increment is first calculated based on the confidence level of the first target word corresponding to the current training batch. Then, the confidence increment is summed with the preset confidence level corresponding to the current training batch to obtain the updated preset confidence level. It can be understood that the updated preset confidence level is used to re-determine the first and second target words in the next training batch adjacent to the current training batch.

[0081] By implementing the optional embodiments, synchronous adaptation and closed-loop feedback of pre-set reliability updates and model parameter updates are achieved. The knowledge changes learned by the model under training in each training batch are fed back to the pre-set reliability updates in real time. This allows the determination of the first target word and the second target word to closely match the ability evolution of the model under training in the micro-training steps. This enables resources to be continuously and effectively guided to the tail data that the model under training needs to improve the most, thereby improving the efficiency of continuous optimization learning.

[0082] In some embodiments, the first target word corresponding to the current training batch includes multiple words; correspondingly, the process of calculating the confidence increment based on the confidence of the first target word corresponding to the current training batch may include: for each first target word, performing a ratio operation on the confidence of the first target word and the preset confidence of the current training batch to obtain a confidence ratio, and determining the confidence ratio as the base and the first preset adjustment factor as the exponent, performing a power operation to obtain the power operation result, wherein the first preset adjustment factor is a positive number; and obtaining the confidence increment based on the preset incremental update rate and the power operation results corresponding to the multiple first target words respectively, wherein the preset incremental update rate is a positive number.

[0083] That is, in the optional embodiment, firstly, for each first target word in the current training batch, the ratio of the confidence level of the first target word to the preset confidence level corresponding to the current training batch is calculated to obtain the confidence ratio. Based on the confidence ratio, the degree to which the first target word "exceeds" the preset confidence level corresponding to the current training batch can be determined. Then, a power operation is performed with the confidence ratio as the base and the first preset adjustment factor as the exponent to obtain the power operation result. Finally, the power operation results of all first target words are aggregated based on the preset incremental update rate to obtain the confidence increment.

[0084] By implementing the optional embodiment, the noise interference of individual first target words is avoided by aggregating all first target words in the current training batch, and update overshoot is prevented by non-linear adjustment (i.e., exponentiation). This allows the preset reliability to increase smoothly and robustly throughout the training process, thereby improving the accuracy of preset reliability updates.

[0085] Optionally, the process of obtaining the confidence increment based on the preset incremental update rate and the power operation results corresponding to the multiple first target words may include: summing the power operation results corresponding to the multiple first target words to obtain a total value; and multiplying the preset incremental update rate and the total value to obtain the confidence increment.

[0086] That is, in the optional embodiment, the power operation results corresponding to the multiple first target words are first summed to obtain a total value, and then the preset incremental update rate and the total value are multiplied to obtain the confidence increment.

[0087] By implementing the optional embodiments, the confidence increment can be calculated easily and accurately.

[0088] S203, filter the loss value corresponding to the first target word to obtain the first target loss value.

[0089] In this embodiment, after determining the first target word, the loss value corresponding to the first target word can be filtered to obtain the first target loss value. It can be understood that filtering the loss value corresponding to the first target word means ignoring the loss value of the first target word; correspondingly, the first target loss value is 0. That is, this embodiment implements the filtering of high-confidence words to free up training resources from learning from the head data, leaving more learning space for the tail data. This avoids the phenomenon that the model to be trained overlearns from the head data in the later stages of training, leading to irreversible overfitting of the head data and impaired learning ability of the tail data.

[0090] S204, apply an upper limit constraint to the loss value corresponding to the second target word to obtain the second target loss value.

[0091] In this embodiment, after determining the second target word, an upper limit constraint can be applied to the loss value corresponding to the second target word to obtain the second target loss value. That is, this embodiment implements upper limit control of the loss value of low-confidence words, so as to give appropriate attention to the tail data, and the optimization intensity is reasonably constrained. This avoids the phenomenon that the model to be trained will perform sharp optimization on low-confidence words during the training process, resulting in bias in the virtual mode and high risk of overfitting.

[0092] In one embodiment of this application, the process of calculating the weight of the second target word based on the confidence level of the second target word in S204 may include: calculating the weight of the second target word based on the confidence level of the second target word; performing a weighted operation on the loss value corresponding to the second target word based on the weight to obtain the second target loss value, wherein the second target loss value is less than or equal to a preset maximum loss value.

[0093] That is, in the optional embodiment, the weight of the second target word is first calculated based on the confidence level of the second target word, and then the loss value corresponding to the second target word is weighted based on the weight to obtain the second target loss value, wherein the second target loss value is less than or equal to the preset maximum loss value.

[0094] By implementing optional embodiments, adaptive reweighting of low-confidence terms can be used to easily and accurately constrain the upper limit of the loss value of low-confidence terms, thereby improving the accuracy and efficiency of controlling the upper limit of the loss value of low-confidence terms.

[0095] In some embodiments, the process of calculating the weight of the second target word based on the confidence level of the second target word may include: calculating the loss value corresponding to the second target word based on the confidence level of the second target word; performing a ratio operation on the preset maximum loss value and the loss value corresponding to the second target word to obtain a loss ratio; if the loss ratio is greater than or equal to the standard weight threshold, then determining the weight of the second target word as the standard weight threshold; if the loss ratio is less than the standard weight threshold, then determining the weight of the second target word as the loss ratio.

[0096] In other words, in an optional embodiment, the loss value corresponding to the second target word is first calculated based on the confidence level of the second target word. Then, the ratio of the preset maximum loss value to the loss value corresponding to the second target word is calculated to obtain the loss ratio. If the loss ratio is greater than or equal to the standard weight threshold (usually set to 1), it indicates that the second target word belongs to the "normal difficulty" level, and it is given full weight (i.e., the weight is the standard weight threshold), and normal intensity optimization can be performed. If the loss ratio is less than the standard weight threshold, it indicates that the second target word belongs to the "extreme difficulty" or "abnormal" level, and it is given a weight less than the standard weight threshold (i.e., the weight is the loss ratio) to limit the corresponding second target loss value within the preset maximum loss value.

[0097] In the optional embodiment, the preset maximum loss value refers to the scalar threshold used to constrain the upper limit of the contribution of a single low-confidence word to the total loss, also known as the maximum allowable loss value. It defines the maximum penalty intensity that the model to be trained can accept when optimizing any difficult sample.

[0098] By implementing optional embodiments, a "mild yet firm" and secure adaptive reweighting of low-confidence terms is achieved. Specifically, while incentivizing the model to be trained to strengthen its learning of low-confidence terms, it proactively sets an insurmountable strength upper limit for the optimization process, avoiding many risks that may be caused by unrestricted sharp optimization, such as instability in the training process due to huge gradients, overfitting of the model to data noise or spurious patterns, and destruction of robust general knowledge learned in the pre-training stage.

[0099] Optionally, the process of calculating the ratio between the preset maximum loss value and the loss value corresponding to the second target word to obtain the loss ratio may include: summing the loss value corresponding to the second target word and the second preset adjustment factor to obtain the intermediate loss value, wherein the second preset adjustment factor is a positive number; and calculating the ratio between the preset maximum loss value and the intermediate loss value to obtain the loss ratio.

[0100] That is, in the optional embodiment, the loss value corresponding to the second target word and the second preset adjustment factor are first summed to obtain the intermediate loss value. The second preset adjustment factor is a positive number. Then, the ratio of the preset maximum loss value and the intermediate loss value is calculated to obtain the loss ratio.

[0101] By implementing the optional embodiment, and with the help of a positive second preset adjustment factor, the phenomenon that the loss value corresponding to the high confidence of the second target word tends to be close to 0, which leads to an unstable loss ratio, is avoided. This ensures that the ratio calculation between the preset maximum loss value and the loss value corresponding to the second target word can be performed stably.

[0102] S205, Based on the first target loss value and the second target loss value, train the model to be trained to obtain a visual language model. The visual language model is used to generate response content based on the input question image and question text.

[0103] In this embodiment of the application, after obtaining the first target loss value and the second target loss value, the model to be trained can be trained based on the first target loss value and the second target loss value to obtain the visual language model.

[0104] In one embodiment of this application, the process of training the model to be trained based on the first target loss value and the second target loss value in S205 to obtain a visual language model may include: summing the first target loss value and the second target loss value to obtain the total sample loss of the sample pair; averaging the total sample loss corresponding to multiple sample pairs to obtain the model loss; and updating the model parameters of the model to be trained based on the model loss to obtain the visual language model.

[0105] That is, in the optional embodiment, the first target loss value and the second target loss value are summed to obtain the total sample loss of the sample pair. Then, the total sample loss corresponding to multiple sample pairs is averaged to obtain the model loss. Finally, the model parameters of the model to be trained are updated based on the model loss. This update can correspond to the update of a training batch. The cycle is iterated until the cycle termination condition is met (e.g., the training reaches a preset total number of iterations, the model loss no longer decreases significantly on the validation set, the performance index on the validation set reaches a predetermined threshold, etc.) to end the training process, thereby obtaining the visual language model.

[0106] By implementing optional embodiments, the final effects of word-level operations (i.e., filtering of high-confidence words and adaptive reweighting of low-confidence words) are aggregated into a standardized scalar loss value (i.e., model loss), which is directly compatible with existing gradient-based optimizers and training loops without changing the existing basic training architecture. At the same time, by averaging, the optimization direction is determined by the overall performance of a batch of training samples, smoothing out the noise that may be introduced by a single training sample and improving the stability of training.

[0107] For ease of understanding, such as Figure 3 As shown, in some embodiments, the training of the model to be trained can be achieved based on the filtering of high-confidence words and the adaptive reweighting of low-confidence words, thereby obtaining a visual language model.

[0108] For ease of understanding, such as Figure 4 As shown, in some embodiments, the training of the model to be trained can be achieved based on the filtering of high-confidence words and the adaptive reweighting of low-confidence words, as well as visual word masks, thereby obtaining a visual language model. In one embodiment of this application, after training the model to be trained based on the first target loss value and the second target loss value in S205 to obtain the visual language model, the process may further include: receiving an input question image and question text; inputting the question image and question text into the visual language model, and performing the following steps through the visual language model: obtaining the visual word sequence corresponding to the question image and obtaining the text word sequence corresponding to the question text; concatenating the visual word sequence and the text word sequence to obtain the input word sequence; predicting based on the input word sequence to obtain the output word sequence, and determining the output word sequence as the response content.

[0109] That is, in the optional embodiment, after receiving the input question image and question text, the question image and question text can be input into the visual language model, and then the visual language model can obtain the visual word sequence corresponding to the question image and the text word sequence corresponding to the question text. Then, the visual word sequence and the text word sequence are concatenated to obtain the input word sequence. Finally, prediction is performed based on the input word sequence to obtain the output word sequence. At this time, the output word sequence is the answer content.

[0110] By implementing the optional embodiments, since the visual language model achieves high-confidence word filtering and low-confidence word adaptive reweighting during the training phase, the visual language model has significantly enhanced long-tail scene understanding ability and overall generalization performance during the inference phase, thereby improving the reliability of data processing in visual language model scenarios.

[0111] It should be noted that the aforementioned embodiments mainly involve the training phase, while the embodiments described here mainly involve the inference phase. To distinguish between the two different phases, the aforementioned embodiments used the word "sample" to identify relevant terms (such as image samples, text samples, output word sequence samples, etc.) when describing the training process, while the embodiments described here use more general terms (such as question images, question text, output word sequence, etc.) when describing the inference process.

[0112] Meanwhile, "training" in the training phase refers to the model optimization process in a broad sense, covering all learning behaviors of the model from the initial state to the final state, including but not limited to full fine-tuning of the pre-trained model, efficient parameter fine-tuning (such as using adapters, prefix fine-tuning, low-rank adaptation, etc. to update only a small number of parameters), or continuous pre-training on a specific downstream task training sample set.

[0113] This embodiment of the application achieves high-confidence word filtering by determining high-confidence words and filtering their loss values. This frees up training resources from learning from the head data, leaving more learning space for the tail data. This avoids the phenomenon of the model overlearning the head data in the later stages of training, leading to irreversible overfitting of the head data and impaired learning ability of the tail data. Simultaneously, by determining low-confidence words and constraining the upper limit of their loss values, the upper limit of the loss values ​​for low-confidence words is controlled. This allows for appropriate attention to the tail data, with reasonable constraints on the optimization intensity. This avoids the model from sharply optimizing low-confidence words during training, which could lead to bias in the virtual mode and a high risk of overfitting. Furthermore, training is performed based on the first target loss value of high-confidence words and the second target loss value of low-confidence words. This achieves a combination of high-confidence word filtering and low-confidence word adaptive reweighting, which together constitute bounded adaptive loss training. This improves the training effect of the visual language model, and the generalization performance and robustness of the visual language model are both high, thereby improving the reliability of data processing in the visual language model scenario.

[0114] The following provides a detailed description of specific scenarios in the embodiments of this application: Please see Figure 5 , Figure 5 This is a flowchart illustrating a data processing method according to an embodiment of this application. Figure 5 As shown, this data processing method includes at least S501 to S509, which are described in detail below: S501, Obtain the training sample set, which includes multiple sample pairs, including image samples and text samples, and input the training sample set into the model to be trained. Then, execute S501 to S505 through the model to be trained.

[0115] Optionally, such as Figure 6 As shown, the model to be trained includes a visual encoder and a large language model backbone network. The visual encoder is mainly used to extract image features and project them into a sequence of visual word samples, while the large language model backbone network is mainly used to understand and process visual and linguistic information.

[0116] S502, based on image samples and text samples, prediction is performed to obtain an output word sample sequence. The output word sample sequence includes multiple output words, and from the multiple output words, a first target word with a confidence level greater than a preset confidence level and a second target word with a confidence level less than or equal to the preset confidence level are determined.

[0117] In some embodiments, the process of predicting based on image samples and text samples to obtain an output word sample sequence may include: First, a visual word sample sequence corresponding to the image sample is obtained, which includes multiple visual words. Then, the multiple visual words are sorted in descending order of word entropy to obtain a sorted sequence. The word entropy of the visual words corresponding to the preset quantile in the sorted sequence is determined as the preset word entropy. The visual words following the visual words corresponding to the preset quantile are determined as candidate mask visual words. The candidate mask visual words are randomly sampled to obtain the visual words to be masked. Then, the visual words to be masked are masked to obtain the target visual word sample sequence.

[0118] Optionally, such as Figure 6 As shown, image samples can be input into a visual encoder, which performs masking processing as shown above to obtain a target visual word sample sequence.

[0119] As an example, the following formula can be used for calculation:

[0120]

[0121] In this formula, This refers to the time step Predicted vocabulary list The Middle each word element The probability distribution of the next generated target; It refers to a sequence of text word samples; It refers to the visual word sample sequence. , This refers to the number of visual lexical units. This refers to the processing of visual lexical units. Previously processed visual lexical sample sequences; Refers to visual lexical units lexical entropy.

[0122] Following the previous example, the following formula can be used for calculation:

[0123]

[0124] In this formula, It refers to the set of visual lexical units to be masked; Refers to visual word sample sequences The first of all visual lexical meta-entropy in the text Percentile (e.g., the 25th percentile); It refers to probability A successful (returning 1) Bernoulli random trial introduces randomness, ensuring that only a subset of eligible low-entropy visual terms are selected. This refers to the target visual word sample sequence; other parameters are explained in the previous example and will not be repeated here.

[0125] Next, the target visual word sample sequence and the corresponding text word sample sequence are concatenated to obtain the input word sample sequence. Then, prediction is performed based on the input word sample sequence to obtain the output word sample sequence.

[0126] Optionally, such as Figure 6 As shown, the input word sample sequence, obtained by concatenating the target visual word sample sequence and the text word sample sequence, can be input into the backbone network of a large language model. The backbone network of the large language model then makes predictions based on the input word sample sequence to obtain the output word sample sequence.

[0127] S503, filter the loss value corresponding to the first target word to obtain the first target loss value.

[0128] Following the previous example, the following formula can be used for calculation:

[0129] In this formula, This refers to reliability based on pre-set parameters. The indicator function is a binary function used to determine whether the output word should be filtered out of the loss calculation, that is, whether the loss value corresponding to the output word should be set to zero. This refers to the time step Output the confidence level of the term; The model can be updated adaptively based on its performance during training. When an update is needed, it refers to the pre-set reliability corresponding to the current training batch. For other parameters, please refer to the example above, which will not be repeated here.

[0130] The output logic of the indicator function is as follows: when hour, This indicates that the output term has a high confidence level, is the first target term, and the model is "overconfident" or "has a good grasp" of it. Therefore, the indicator function outputs 0, that is, the loss contribution of the first target term is set to zero, in order to achieve filtering. Otherwise, in other cases (e.g.) ), This indicates that the confidence level of the output term is low, and it is the second target term. The model "lacks confidence" or "underlearns" about it. Therefore, the indicator function outputs 1 to retain the loss contribution of the second target term and allow it to participate in the subsequent weighted calculation.

[0131] Following the previous example, the following formula can be used for calculation:

[0132] In this formula, This refers to the updated pre-set reliability; This refers to the preset incremental update rate, which is a positive number. This refers to the first preset adjustment factor, which is a positive number; for other parameters, please refer to the previous example description, which will not be repeated here.

[0133] S504, calculate the weight of the second target word based on the confidence level of the second target word, and perform a weighted operation on the loss value corresponding to the second target word based on the weight to obtain the second target loss value, which is less than or equal to the preset maximum loss value.

[0134] In some embodiments, the process of calculating the weight of the second target word based on the confidence level of the second target word may include: calculating the loss value corresponding to the second target word based on the confidence level of the second target word; performing a ratio operation on the preset maximum loss value and the loss value corresponding to the second target word to obtain a loss ratio; if the loss ratio is greater than or equal to the standard weight threshold (usually set to 1), then the weight of the second target word is determined to be the standard weight threshold; if the loss ratio is less than the standard weight threshold, then the weight of the second target word is determined to be the loss ratio.

[0135] Following the previous example, the following formula can be used for calculation:

[0136] In this formula, This refers to the weight of the second target word; This refers to the preset maximum loss value; This refers to the second preset adjustment factor, which is a positive number; for other parameters, please refer to the previous example description, which will not be repeated here.

[0137] S505 trains the model to be trained based on the first target loss value and the second target loss value to obtain the visual language model.

[0138] In some embodiments, the process of training the model to be trained based on the first target loss value and the second target loss value to obtain a visual language model may include: summing the first target loss value and the second target loss value to obtain the total sample loss of the sample pair; averaging the total sample losses corresponding to multiple sample pairs to obtain the model loss; and updating the model parameters of the model to be trained based on the model loss to obtain the visual language model.

[0139] Following the previous example, the following formula can be used for calculation:

[0140] In this formula, This refers to model loss; This refers to the training sample set; This refers to the length of the output word sample sequence; other parameters are explained in the previous example and will not be repeated here.

[0141] As an example, the following formula can also be used for calculation:

[0142] For the parameters in this formula, please refer to the previous example; they will not be repeated here.

[0143] For example Figure 6 As shown, the left coordinate system illustrates the distribution of output words in the probability space in the traditional scheme, while the right coordinate system illustrates the distribution of output words in the probability space after adopting bounded adaptive loss (dynamic filtering of high-confidence words and adaptive reweighting of low-confidence words) in this scheme.

[0144] This completes the training of the model to be trained. It should be noted that the training in this embodiment can be long-tail fine-tuning of adaptive lexical optimization for the visual language model, thereby improving the training efficiency of the visual language model.

[0145] S506, if the input problem image and problem text are received, the problem image and problem text are input into the visual language model, and S507 to S509 are executed through the visual language model.

[0146] S507, obtain the visual word sequence corresponding to the problem image, and obtain the text word sequence corresponding to the problem text.

[0147] S508 concatenates the visual lexical sequence and the text lexical sequence to obtain the input lexical sequence.

[0148] S509, based on the input word sequence, predict to obtain the output word sequence, and determine the output word sequence as the response content.

[0149] In this embodiment, a visual lexical masking strategy is employed at the input end to adaptively mask simple visual cues (i.e., low-entropy visual lexical units) that are low in information and prone to over-reliance. This forces the model under test to focus more on and utilize complex visual cues (i.e., high-entropy visual lexical units) in image samples that are richer in information and more discriminative during feature extraction and context understanding. Thus, by using visual lexical masking, the overfitting tendency of the model under test to the head data in long-tail data distributions can be effectively alleviated, and the ability to distinguish the tail data is enhanced, thereby improving the overall model's generalization performance and robustness.

[0150] In this embodiment, a bounded adaptive loss strategy is adopted at the output end. This strategy not only filters high-confidence words, freeing up training resources from learning the head data and leaving more learning space for the tail data, thus avoiding the phenomenon of the model overlearning the head data in the later stages of training, leading to irreversible overfitting of the head data and impaired learning ability of the tail data, but also achieves adaptive reweighting of low-confidence words, giving appropriate attention to the tail data and reasonably constraining the optimization intensity. This avoids the phenomenon of the model sharply optimizing low-confidence words during training, leading to bias in the virtual mode and high risk of overfitting.

[0151] In the process of determining high-confidence and low-confidence terms in this embodiment, a pre-set reliability dynamic update strategy is also adopted. This achieves synchronous adaptation and closed-loop feedback between the pre-set reliability update and the model parameter update. The knowledge changes learned by the model in each training batch are fed back to the pre-set reliability update in real time. This allows the determination of the first and second target terms to closely match the ability evolution of the model in the micro-training steps, so that resources can be continuously and effectively guided to the tail data that the model needs to improve the most, thereby improving the efficiency of continuous optimization learning.

[0152] Therefore, the embodiments of this application improve the training effect of visual language models. Without requiring any prior knowledge of data distribution, visual language models can effectively balance the learning of head and tail data in various long-tail scenarios. The overall performance of visual language models is high, thus enabling them to perform better reasoning and ensuring high reliability of data processing in visual language model scenarios.

[0153] Figure 7 This is a block diagram illustrating a data processing apparatus according to an embodiment of this application. Figure 7 As shown, the device includes: The acquisition module 701 is configured to acquire a training sample set, which includes multiple sample pairs, including image samples and text samples. The training sample set is then input into the model to be trained, and the model to be trained performs the following steps: The prediction and determination module 702 is configured to make predictions based on the image samples and the text samples to obtain an output word sample sequence, the output word sample sequence including multiple output words, and to determine a first target word with a confidence level greater than a preset confidence level and a second target word with a confidence level less than or equal to the preset confidence level from the multiple output words. The filtering module 703 is configured to filter the loss value corresponding to the first target word to obtain the first target loss value; The constraint module 704 is configured to impose an upper limit constraint on the loss value corresponding to the second target word to obtain the second target loss value; The training module 705 is configured to train the model to be trained based on the first target loss value and the second target loss value to obtain a visual language model. The visual language model is used to generate response content based on the input question image and question text.

[0154] In one embodiment of this application, based on the aforementioned scheme, the training of the model to be trained includes multiple training batches, and each training batch is used to update the model parameters of the model to be trained once; correspondingly, the device further includes: a threshold update module, configured to: calculate a confidence increment based on the confidence of the first target word corresponding to the current training batch; sum the confidence increment and the preset confidence corresponding to the current training batch to obtain an updated preset confidence, and the updated preset confidence is used to re-determine the first target word and the second target word in the next training batch adjacent to the current training batch.

[0155] In one embodiment of this application, based on the aforementioned scheme, the first target word corresponding to the current training batch includes multiple words; correspondingly, the threshold update module is specifically configured as follows: for each first target word, the confidence level corresponding to the first target word and the preset confidence level corresponding to the current training batch are compared to obtain a confidence ratio, and the confidence ratio is determined as the base, the first preset adjustment factor is determined as the exponent, and an exponentiation operation is performed to obtain the exponentiation result, wherein the first preset adjustment factor is a positive number; based on the preset incremental update rate and the exponentiation results corresponding to the multiple first target words, the confidence increment is obtained, wherein the preset incremental update rate is a positive number.

[0156] In one embodiment of this application, based on the foregoing scheme, the threshold update module is further configured to: sum the exponentiation results corresponding to the plurality of first target words to obtain a total value; and multiply the preset incremental update rate and the total value to obtain the confidence increment.

[0157] In one embodiment of this application, based on the aforementioned scheme, the constraint module 704 is specifically configured to: calculate the weight of the second target word based on the confidence level of the second target word; perform a weighted operation on the loss value corresponding to the second target word based on the weight to obtain the second target loss value, wherein the second target loss value is less than or equal to a preset maximum loss value.

[0158] In one embodiment of this application, based on the aforementioned scheme, the constraint module 704 is further specifically configured to: calculate the loss value corresponding to the second target word based on the confidence level of the second target word; perform a ratio operation on the preset maximum loss value and the loss value corresponding to the second target word to obtain a loss ratio; if the loss ratio is greater than or equal to the standard weight threshold, then determine the weight of the second target word as the standard weight threshold; if the loss ratio is less than the standard weight threshold, then determine the weight of the second target word as the loss ratio.

[0159] In one embodiment of this application, based on the aforementioned scheme, the constraint module 704 is further specifically configured to: sum the loss value corresponding to the second target word and the second preset adjustment factor to obtain a loss median value, wherein the second preset adjustment factor is a positive number; and perform a ratio calculation on the preset maximum loss value and the loss median value to obtain the loss ratio.

[0160] In one embodiment of this application, based on the aforementioned scheme, the prediction and determination module 702 is specifically configured as follows: obtaining a visual word sample sequence corresponding to the image sample, the visual word sample sequence including multiple visual words; determining visual words to be masked from the multiple visual words whose word entropy is less than a preset word entropy; performing masking processing on the visual words to be masked to obtain a target visual word sample sequence; and performing prediction based on the target visual word sample sequence and the text sample to obtain the output word sample sequence.

[0161] In one embodiment of this application, based on the aforementioned scheme, the prediction and determination module 702 is further configured to: sort the plurality of visual lexical units in descending order of lexical entropy to obtain a sorted sequence; determine the lexical entropy of the visual lexical unit corresponding to the preset quantile in the sorted sequence as the preset lexical entropy, and determine the visual lexical units arranged after the visual lexical unit corresponding to the preset quantile as candidate mask visual lexical units; and randomly sample the candidate mask visual lexical units to obtain the visual lexical units to be masked.

[0162] In one embodiment of this application, based on the foregoing scheme, the prediction and determination module 702 is further configured to: obtain the text word sample sequence corresponding to the text sample; concatenate the target visual word sample sequence and the text word sample sequence to obtain the input word sample sequence; and perform prediction based on the input word sample sequence to obtain the output word sample sequence. In one embodiment of this application, based on the aforementioned scheme, the prediction and determination module 702 is further configured as follows: the lexical entropy of any visual lexical is calculated by: obtaining the probability distribution on the vocabulary based on the prediction of the visual lexical; calculating the entropy value corresponding to the visual lexical for representing the amount of information based on the degree of dispersion of the probability distribution, wherein the degree of dispersion is proportional to the entropy value; and determining the entropy value as the lexical entropy of the visual lexical.

[0163] In one embodiment of this application, based on the aforementioned scheme, the training module 705 is specifically configured to: sum the first target loss value and the second target loss value to obtain the total sample loss of the sample pair; average the total sample losses corresponding to the plurality of sample pairs to obtain the model loss; and update the model parameters of the model to be trained based on the model loss to obtain the visual language model.

[0164] In one embodiment of this application, based on the foregoing scheme, the device further includes a reasoning module configured to: receive the input question image and the question text; input the question image and the question text into the visual language model, and perform the following steps through the visual language model: obtain the visual word sequence corresponding to the question image, and obtain the text word sequence corresponding to the question text; concatenate the visual word sequence and the text word sequence to obtain the input word sequence; perform prediction based on the input word sequence to obtain the output word sequence, and determine the output word sequence as the response content.

[0165] It should be noted that the apparatus provided in the foregoing embodiments and the method provided in the foregoing embodiments belong to the same concept, and the specific way in which each module and unit performs operations has been described in detail in the method embodiments.

[0166] Embodiments of this application also provide an electronic device, including: one or more processors; and a memory for storing one or more computer programs, which, when executed by one or more processors, cause the electronic device to perform the aforementioned data processing method.

[0167] Figure 8 It is an electronic device suitable for implementing the embodiments of this application (e.g. Figure 1 The diagram shows the structure of a computer system (terminal device or server).

[0168] It should be noted that, Figure 8 The computer system 800 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0169] like Figure 8 As shown, the computer system 800 includes a Central Processing Unit (CPU) 801, which can perform various appropriate actions and processes, such as executing the methods described in the above embodiments, based on a computer program stored in a Read-Only Memory (ROM) 802 or a computer program loaded from a storage portion 808 into a Random Access Memory (RAM) 803. The RAM 803 also stores various computer programs and data required for system operation. The CPU 801, ROM 802, and RAM 803 are interconnected via a bus 804. An Input / Output (I / O) interface 805 is also connected to the bus 804.

[0170] The following components are connected to I / O interface 805: an input section 806 including a keyboard, mouse, etc.; an output section 807 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to I / O interface 805 as needed. A removable medium 811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 810 as needed so that computer programs read from it can be installed into storage section 808 as needed.

[0171] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing computer instructions for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 809, and / or installed from removable medium 811. When the computer program is executed by central processing unit (CPU) 801, it performs various functions defined in the system of this application.

[0172] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium, a computer-readable storage medium, or any combination thereof. For example, a computer-readable medium can be an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable medium can be any tangible medium containing or storing a computer program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. Such transmitted data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a computer program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0173] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0174] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0175] Another aspect of this application provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the data processing method as described above. This computer-readable medium may be included in the electronic device described in the above embodiments, or it may exist independently and not assembled into the electronic device.

[0176] Another aspect of this application provides a computer program product or computer program including computer instructions stored in a computer-readable medium. A processor of an electronic device reads the computer instructions from the computer-readable medium and executes the computer instructions, causing the electronic device to perform the data processing methods provided in the various embodiments described above.

[0177] The above description is merely a preferred exemplary embodiment of this application and is not intended to limit the implementation of this application. Those skilled in the art can easily make corresponding modifications or alterations based on the main concept and spirit of this application. Therefore, the scope of protection of this application should be determined by the scope of protection claimed in the claims.

Claims

1. A data processing method, characterized in that, include: Obtain a training sample set, which includes multiple sample pairs, including image samples and text samples, and input the training sample set into the model to be trained. Then, perform the following steps through the model to be trained: Based on the image samples and the text samples, a prediction is made to obtain an output word sample sequence, which includes multiple output words. From the multiple output words, a first target word with a confidence level greater than a preset confidence level and a second target word with a confidence level less than or equal to the preset confidence level are determined. The loss value corresponding to the first target word is filtered to obtain the first target loss value; An upper limit constraint is applied to the loss value corresponding to the second target word to obtain the second target loss value; Based on the first target loss value and the second target loss value, the model to be trained is trained to obtain a visual language model, which is used to generate response content based on the input question image and question text.

2. The method according to claim 1, characterized in that, The training of the model to be trained includes multiple training batches, with each training batch used to update the model parameters of the model to be trained once; the method further includes: Calculate the confidence increment based on the confidence of the first target word corresponding to the current training batch; The confidence increment and the preset confidence corresponding to the current training batch are summed to obtain the updated preset confidence. The updated preset confidence is used to re-determine the first target word and the second target word in the next training batch adjacent to the current training batch.

3. The method according to claim 2, characterized in that, The first target word corresponding to the current training batch includes multiple terms; the step of calculating the confidence increment based on the confidence of the first target word corresponding to the current training batch includes: For each first target word, the confidence level corresponding to the first target word is calculated as a ratio to the preset confidence level corresponding to the current training batch to obtain a confidence ratio. The confidence ratio is then used as the base, and the first preset adjustment factor is used as the exponent. An exponentiation operation is then performed to obtain the exponentiation result. The first preset adjustment factor is a positive number. The confidence increment is obtained based on the preset incremental update rate and the power operation results corresponding to the multiple first target words, wherein the preset incremental update rate is a positive number.

4. The method according to claim 3, characterized in that, The confidence increment is obtained based on a preset incremental update rate and the power operation results corresponding to multiple first target words, including: The summation operation is performed on the exponentiation results corresponding to the plurality of first target words to obtain a total value; The confidence increment is obtained by multiplying the preset incremental update rate and the sum value.

5. The method according to claim 1, characterized in that, The step of applying an upper limit constraint to the loss value corresponding to the second target word to obtain the second target loss value includes: The weight of the second target word is calculated based on its confidence level. The loss value corresponding to the second target word is weighted based on the weight to obtain the second target loss value, which is less than or equal to the preset maximum loss value.

6. The method according to claim 5, characterized in that, The calculation of the weight of the second target word based on the confidence level of the second target word includes: Calculate the loss value corresponding to the second target word based on the confidence level of the second target word; The ratio of the preset maximum loss value to the loss value corresponding to the second target word is calculated to obtain the loss ratio. If the loss ratio is greater than or equal to the standard weight threshold, then the weight of the second target word is determined to be the standard weight threshold. If the loss ratio is less than the standard weight threshold, then the weight of the second target word is determined to be the loss ratio.

7. The method according to claim 6, characterized in that, The step of calculating the ratio between the preset maximum loss value and the loss value corresponding to the second target word to obtain the loss ratio includes: The loss value corresponding to the second target word is summed with the second preset adjustment factor to obtain the median loss value. The second preset adjustment factor is a positive number. The ratio of the preset maximum loss value to the intermediate loss value is calculated to obtain the loss ratio.

8. The method according to claim 1, characterized in that, The prediction based on the image samples and the text samples to obtain the output word sample sequence includes: Obtain the visual word sample sequence corresponding to the image sample, wherein the visual word sample sequence includes multiple visual words; From the plurality of visual lexical elements, determine the visual lexical elements to be masked whose lexical entropy is less than the preset lexical entropy; The visual words to be masked are masked to obtain a target visual word sample sequence; Based on the target visual word sample sequence and the text sample, prediction is performed to obtain the output word sample sequence.

9. The method according to claim 8, characterized in that, The step of determining visual words to be masked from the visual word sample sequence whose word entropy is less than a preset word entropy includes: The multiple visual lexical units are sorted in descending order of lexical entropy to obtain a sorted sequence. The lexical entropy of the visual lexical corresponding to the preset quantile in the sorted sequence is determined as the preset lexical entropy, and the visual lexical arranged after the visual lexical corresponding to the preset quantile is determined as the candidate mask visual lexical. Randomly sample the candidate mask visual words to obtain the visual words to be masked.

10. The method according to claim 8, characterized in that, The step of predicting based on the target visual word sample sequence and the text sample to obtain the output word sample sequence includes: Obtain the text word sample sequence corresponding to the text sample; The target visual word sample sequence and the text word sample sequence are concatenated to obtain the input word sample sequence. Based on the input word sample sequence, prediction is performed to obtain the output word sample sequence.

11. The method according to claim 8, characterized in that, The lexical entropy of any visual lexical unit is calculated as follows: Obtain the probability distribution on the vocabulary based on the visual lexical predictions; Based on the degree of dispersion of the probability distribution, the entropy value corresponding to the visual word is calculated to represent the amount of information, and the degree of dispersion is proportional to the entropy value. The entropy value is determined as the lexical entropy of the visual lexical.

12. The method according to claim 1, characterized in that, The step of training the model to be trained based on the first target loss value and the second target loss value to obtain a visual language model includes: The total sample loss of the sample pair is obtained by summing the first target loss value and the second target loss value. The model loss is obtained by averaging the total sample loss for each of the multiple sample pairs. The model parameters of the model to be trained are updated based on the model loss to obtain the visual language model.

13. The method according to any one of claims 1 to 12, characterized in that, After training the model to be trained based on the first target loss value and the second target loss value to obtain a visual language model, the method further includes: Receive the input of the problem image and the problem text; The problem image and the problem text are input into the visual language model, and the following steps are performed through the visual language model: Obtain the visual word sequence corresponding to the problem image, and obtain the text word sequence corresponding to the problem text; The visual word sequence and the text word sequence are concatenated to obtain the input word sequence. Based on the input word sequence, a prediction is performed to obtain the output word sequence, and the output word sequence is determined as the response content.

14. A data processing apparatus, characterized in that, include: The acquisition module is configured to acquire a training sample set, which includes multiple sample pairs, including image samples and text samples. The training sample set is then input into the model to be trained, and the model performs the following steps: The prediction and determination module is configured to make predictions based on the image samples and the text samples to obtain an output word sample sequence, the output word sample sequence including multiple output words, and to determine a first target word with a confidence level greater than a preset confidence level and a second target word with a confidence level less than or equal to the preset confidence level from the multiple output words. The filtering module is configured to filter the loss value corresponding to the first target word to obtain the first target loss value; The constraint module is configured to impose an upper limit constraint on the loss value corresponding to the second target word to obtain the second target loss value. The training module is configured to train the model to be trained based on the first target loss value and the second target loss value to obtain a visual language model. The visual language model is used to generate response content based on the input question image and question text.

15. An electronic device, characterized in that, include: One or more processors; A memory for storing one or more computer programs that, when executed by one or more processors, cause the electronic device to perform the data processing method according to any one of claims 1 to 13.

16. A computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the data processing method according to any one of claims 1 to 13.

17. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the data processing method according to any one of claims 1 to 13.

Images