Training method and device of visual language model, electronic equipment, computer readable storage medium and computer program product

Abstract

The application provides a training method and device of a visual language model, electronic equipment, a computer readable storage medium and a computer program product; the method comprises the following steps: calling an n-th visual language model, rewriting first summary information of an image sample to obtain a plurality of second summary information; constructing a plurality of partial order data based on the second summary information, one partial order data comprising one positive sample and one negative sample; generating a prediction probability of illusion information corresponding to the plurality of partial order data through the n-th visual language model; determining a loss value of the n-th visual language model according to the prediction probability; updating the n-th visual language model according to the loss value to obtain an n+1-th visual language model; when the iteration value of n is N, the obtained N+1-th visual language model is used as a visual language model for generating summary information of a to-be-processed image. Through the application, more detailed summary information of an image can be generated, and illusion information in the summary information is reduced.

Description

Technical Field

[0001] This application relates to artificial intelligence technology, and more particularly to a method, apparatus, electronic device, computer-readable storage medium, and computer program product for training a visual language model. Background Technology

[0002] In related technologies, the image summary information generated by visual language models is usually short in text description, which cannot fully and accurately convey the key information in the image, resulting in low information value; or there is too much stylistic description, while neglecting the accurate expression of the core content of the image, which is not suitable for training multimodal large models; or the text description of the summary information does not match the content of the image, resulting in poor data quality. Summary of the Invention

[0003] This application provides a method, apparatus, electronic device, computer-readable storage medium, and computer program product for training a visual language model, which can generate more detailed summary information of images and reduce illusory information in the summary information.

[0004] The technical solution of this application embodiment is implemented as follows:

[0005] This application provides a method for training a visual language model, the method comprising:

[0006] The nth visual language model is called to rewrite the first summary information of the image sample to obtain multiple second summary information; where n is an integer variable that increments from 1, and the value of n satisfies N, where N is a positive integer greater than or equal to 2. When n is 1, the nth visual language model is a pre-trained visual language model. When n is 1, the nth visual language model is trained on the basis of the (n-1)th visual language model.

[0007] The plurality of second summary information is classified into positive samples and negative samples, wherein the number of hallucination information included in the positive samples is less than the number of hallucination information included in the negative samples;

[0008] Multiple partial order data are constructed based on the multiple second summary information, wherein each of the partial order data includes a positive sample and a negative sample;

[0009] The nth visual language model generates the predicted probabilities of the multiple partially ordered data corresponding to the hallucination information.

[0010] The loss value of the nth visual language model is determined based on the predicted probabilities corresponding to the multiple partially ordered data.

[0011] The nth visual language model is updated based on the loss value to obtain the (n+1)th visual language model;

[0012] When the iteration reaches the value of N, the obtained (N+1)th visual language model is used as the visual language model for generating summary information of the image to be processed.

[0013] This application provides a training device for a visual language model, the device comprising:

[0014] The processing module is used to call the nth visual language model to rewrite the first summary information of the image sample to obtain multiple second summary information; where n is an integer variable that increments from 1, and the value of n satisfies 1≤n≤N, and N is a positive integer greater than or equal to 2. When n is 1, the nth visual language model is a pre-trained visual language model, and when 2≤n, the nth visual language model is trained based on the (n-1)th visual language model.

[0015] A classification module is used to classify the plurality of second summary information into positive samples and negative samples, wherein the number of hallucination information included in the positive samples is less than the number of hallucination information included in the negative samples;

[0016] A construction module is used to construct multiple partial-order data based on the multiple second summary information, wherein one of the partial-order data includes one positive sample and one negative sample;

[0017] The prediction module is used to generate the predicted probabilities of the multiple partially ordered data corresponding to the hallucination information through the nth visual language model;

[0018] The determination module is used to determine the loss value of the nth visual language model based on the prediction probabilities corresponding to the multiple partially ordered data respectively.

[0019] The update module is used to update the nth visual language model according to the loss value to obtain the (n+1)th visual language model; when the iteration reaches the value of N, the obtained (N+1)th visual language model is used as the visual language model for generating summary information of the image to be processed.

[0020] This application provides an electronic device, the electronic device comprising:

[0021] Memory is used to store executable instructions or computer programs.

[0022] The processor, when executing computer-executable instructions or computer programs stored in the memory, implements the training method for the visual language model provided in the embodiments of this application.

[0023] This application provides a computer-readable storage medium storing a computer program or computer-executable instructions, which, when executed by a processor, implements the training method for the visual language model provided in this application.

[0024] This application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, they implement the training method for the visual language model provided in this application.

[0025] The embodiments of this application have the following beneficial effects:

[0026] The nth visual language model is invoked to rewrite the first summary information of the image sample, obtain multiple second summary information, and further classify them into positive and negative samples to construct partial order data. The nth visual language model generates the prediction probability of the partial order data including hallucination information. This allows the visual language model to learn which summary information is reasonable and consistent with the image content, and which summary information contains hallucinations, by processing the positive and negative samples in the partial order data during the training process.

[0027] Based on the predicted probability, the loss value is determined, and the nth visual language model is updated to obtain the (n+1)th visual language model. Starting from the pre-trained visual language model, and finally obtaining the Nth visual language model used to generate summary information of the image to be processed, the parameters of the visual language model are continuously updated according to the loss value through the above iterative training. This allows the visual language model to gradually adapt to the task requirements, learn the difference between real information and hallucinatory information, improve the performance of generating summary information, make the generated summary information more likely to be detailed information that matches the actual content of the image, reduce the occurrence of hallucinatory information, and improve the accuracy of the visual language model in generating summary information based on the image. Attached Figure Description

[0028] Figure 1 This is a schematic diagram of the architecture of the visual language model training system 100 provided in the embodiments of this application;

[0029] Figure 2 This is a schematic diagram of the structure of the server 200 provided in the embodiments of this application;

[0030] Figure 3A This is a schematic diagram of the first process of the training method for the visual language model provided in the embodiments of this application;

[0031] Figure 3B This is a schematic diagram of the second process of the training method for the visual language model provided in the embodiments of this application;

[0032] Figure 3CThis is a schematic diagram of the third process of the training method for the visual language model provided in the embodiments of this application;

[0033] Figure 3D This is a schematic diagram of the fourth process of the training method for the visual language model provided in the embodiments of this application;

[0034] Figure 3E This is a schematic diagram of the fifth step of the training method for the visual language model provided in the embodiments of this application;

[0035] Figure 3F This is a schematic diagram of the sixth step in the training method of the visual language model provided in the embodiments of this application;

[0036] Figure 3G This is a schematic diagram of the seventh process of the training method for the visual language model provided in the embodiments of this application;

[0037] Figure 3H This is a schematic diagram of the eighth process of the training method for the visual language model provided in the embodiments of this application;

[0038] Figure 3I This is a schematic diagram of the ninth process of the training method for the visual language model provided in the embodiments of this application;

[0039] Figure 3J This is a schematic diagram of the tenth process of the training method for the visual language model provided in the embodiments of this application;

[0040] Figure 3K This is a schematic diagram of the eleventh step of the training method for the visual language model provided in the embodiments of this application;

[0041] Figure 3L This is a schematic diagram of the twelfth step of the training method for the visual language model provided in the embodiments of this application;

[0042] Figure 3M This is a schematic diagram of the thirteenth step of the training method for the visual language model provided in the embodiments of this application;

[0043] Figure 4 This is a first schematic diagram of an image sample provided in an embodiment of this application;

[0044] Figure 5 This is a flowchart of the training method for the visual language model provided in the embodiments of this application;

[0045] Figure 6 This is a second schematic diagram of the image sample provided in the embodiments of this application;

[0046] Figure 7 This is a schematic diagram of the structure of the visual language model provided in the embodiments of this application.

[0047] It should be noted that the terms "first" and "second" mentioned above are only used to distinguish between different options and do not represent the degree of superiority or inferiority of the options or their priority in the implementation process. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0049] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0050] In the following description, the terms "first / second / third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first / second / third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0051] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0052] Unless otherwise specified, "at least one" as used below refers to one or more cases, and "multiple" can refer to two or more cases.

[0053] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.

[0054] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0055] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0056] 1) Vision-Language Models (VLMs) are a type of artificial intelligence model capable of understanding and processing visual information (such as images or videos) and linguistic information (such as text or speech). These models combine techniques from both computer vision and natural language processing to achieve the understanding and generation of multimodal data.

[0057] 2) Summary information: This refers to descriptive information about the content of an image, reflecting its core content and key information. For example, if the image is a picture of a dog, the summary information might be "A black dog is running in a park."

[0058] 3) Hallucinatory information: This refers to information where the generated descriptive content does not match the actual content of the original image. For example, an image shows a cat playing in the grass, but the generated description might be "a dog running in the snow." This description is completely inconsistent with the actual content of the image and is considered hallucinatory information.

[0059] 4) Partially ordered data: This refers to data that exhibits partial ordering relationships across certain dimensions. In visual language models, partially ordered data may involve the correlation between images and text, but this correlation may not be entirely deterministic; rather, it may be partially ordered. For example, in terms of the level of detail, accuracy, or relevance of image content, positive samples in partially ordered data may provide better descriptive information than negative samples.

[0060] 5) Image-text pair: This refers to a set of images and their corresponding descriptive text. In visual language models, image-text pairs are the basic units for model training and testing; the model needs to learn the relationships between images and text.

[0061] 6) Information density: refers to the amount of information contained per unit length or unit space. In text generation and image description, information density measures the compactness and richness of the generated summary information.

[0062] 7) Temperature coefficient: This is a hyperparameter used to adjust the randomness and diversity of the generated text. In the process of generating text based on probability distribution, the model assigns a probability value to each possible next word (or token). By adjusting the temperature coefficient, the probability distribution of different words can be changed, thereby affecting the randomness and determinism of the final generated text.

[0063] 8) Responding to: used to indicate the conditions or states on which the operation is performed. When the conditions or states on which the operation is performed are met, one or more operations may be performed in real time or with a set delay. Unless otherwise specified, there is no restriction on the order in which the multiple operations are performed.

[0064] 9) Graphical interface, used to display images and image summary information.

[0065] For example, graphical user interfaces (GUIs) include augmented reality (AR) interfaces, virtual reality (VR) interfaces, voice user interfaces (VUIs), interactive projection interfaces (using projection technology to display information on a flat surface), eye-tracking interfaces (interfaces controlled by detecting the user's gaze), holographic interfaces (three-dimensional holograms formed by projecting images using holographic projection technology, allowing users to see stereoscopic images without wearing special glasses), multimodal interfaces (interfaces that combine multiple interaction methods, such as tactile, visual, and auditory interaction), and brain-machine interfaces (BMIs).

[0066] In related technologies, the image summary information generated by visual language models is usually short in text description, which cannot fully and accurately convey the key information in the image, resulting in low information value; or there is too much stylistic description, while neglecting the accurate expression of the core content of the image, which is not suitable for training multimodal large models; or the text description of the summary information does not match the content of the image, resulting in poor data quality.

[0067] Based on the above analysis, the applicant found that the training methods of visual language models in related technologies cannot generate summary information of images that contains detailed descriptions and less illusory information. In response to the above problems, embodiments of this application provide a training method, apparatus, electronic device, computer-readable storage medium, and computer program product for a visual language model, which can generate more detailed summary information of images and reduce illusory information in the summary information.

[0068] The following describes exemplary applications of the electronic devices provided in the embodiments of this application. These electronic devices can be implemented as various types of terminals such as laptops, tablets, desktop computers, set-top boxes, smartphones, smart speakers, smartwatches, smart TVs, and in-vehicle terminals, or as servers. The following will describe exemplary applications when the electronic device is implemented as a server.

[0069] See Figure 1 , Figure 1 This is a schematic diagram of the architecture of the visual language model training system 100 provided in the embodiments of this application. In order to support the training application of a visual language model, the terminal 400 connects to the server 200 through the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0070] Terminal 400 is used to acquire image samples and send them to server 200. Server 200 calls a pre-trained visual language model to rewrite the first summary information of the image samples to obtain the second summary information. Based on the positive and negative samples in the second summary information, it constructs partial order data and uses the partial order data to train the pre-trained visual language model to obtain the trained visual language model. The above training process is iteratively executed to obtain the final visual language model. The visual language model is then used to generate summary information of the image to be processed, which is sent to terminal 400 and displayed on the graphical interface 411 of terminal 400.

[0071] Taking the education scenario as an example, the terminal 400 is used to acquire images from students' paper or electronic textbooks and send them to the server 200. The server 200 calls the visual language model to generate image summary information, sends it to the terminal 400, and displays it on the display interface 411 of the terminal 400 used by the student, so that the student can understand the content of the image more quickly.

[0072] Taking a medical imaging diagnosis scenario as an example, the terminal 400 is used to acquire medical images of patients from imaging equipment such as X-rays and CT scans in hospitals, and send them to the server 200. The server 200 calls a trained visual language model to generate image summary information, such as "This lung CT image shows a nodule with a diameter of about 2 cm in the upper lobe of the right lung, with slightly blurred edges and a few linear shadows around it," and sends it to the terminal 400 to be displayed on the graphical interface 411 to assist doctors in making a preliminary diagnosis of the disease.

[0073] Taking autonomous driving scenarios as an example, terminal 400 is used to acquire road images from multiple cameras on the car during the vehicle's driving process and send them to server 200. Server 200 calls a trained visual language model to generate image summary information, such as "There is a crossroads ahead in the image, the green light is on, there is a white car in the left lane preparing to turn left, and there are pedestrians crossing the zebra crossing directly ahead," and sends it to terminal 400, which displays it on the graphical interface 411 of terminal 400, helping the autonomous driving system to better understand the surrounding environment.

[0074] Taking the cultural relic appraisal scenario as an example, the terminal 400 is used to acquire multi-angle images of cultural relics from the museum's high-definition imaging equipment and send them to the server 200. The server 200 calls the trained visual language model to generate image summary information, such as "The image of this bronze artifact shows that the artifact is generally square, with a taotie pattern on the belly, obvious wear marks on the feet, and a slight crack on the body," and sends it to the terminal 400, which displays it on the graphical interface 411, providing detailed image information for cultural relic experts to assist in the cultural relic appraisal work.

[0075] Taking agricultural production scenarios as an example, terminal 400 is used to acquire images of crop growth from drones or fixed cameras installed in farmland and send them to server 200. Server 200 calls a trained visual language model to generate summary information of the images, such as "The image shows a cornfield. Most corn plants are about 1.5 meters tall and the leaves are bright green, but some areas have yellowing leaves, which may indicate pests and diseases." This information is then sent to terminal 400 and displayed on the graphical interface 411 to help farmers understand the growth status of their crops in a timely manner.

[0076] In some embodiments, server 200 may be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Terminals and servers can be connected directly or indirectly via wired or wireless communication, which is not limited in this embodiment.

[0077] See Figure 2 , Figure 2 This is a schematic diagram of the structure of the server 200 provided in the embodiments of this application. Figure 2 The server 200 shown includes at least one processor 210, memory 230, and at least one network interface 220. The various components of server 200 are coupled together via a bus system 240. It is understood that the bus system 240 is used to implement communication between these components. In addition to a data bus, the bus system 240 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2 The general labeled all buses as Bus System 240.

[0078] The processor 210 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0079] The memory 230 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 230 may optionally include one or more storage devices physically located away from the processor 210.

[0080] The memory 230 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 230 described in this application embodiment is intended to include any suitable type of memory.

[0081] In some embodiments, memory 230 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0082] Operating system 231 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0083] The network communication module 232 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 220, exemplary network interfaces 220 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

[0084] In some embodiments, the apparatus provided in this application can be implemented in software. Figure 2 A training device 233 for a visual language model stored in memory 230 is shown. This device can be software in the form of programs and plugins, and includes the following software modules: a processing module 2331, a classification module 2332, a construction module 2333, a prediction module 2334, a determination module 2335, and an update module 2336. These modules are logically connected and can therefore be arbitrarily combined or further split according to their implemented functions. The functions of each module will be described below.

[0085] In some embodiments, the terminal or server can implement the visual language model training method provided in this application by running various computer-executable instructions or computer programs. For example, computer-executable instructions can be microprogram-level commands, machine instructions, or software instructions. Computer programs can be native programs or software modules in an operating system; they can be native applications (APPs), i.e., programs that need to be installed in the operating system to run; or they can be applets that can be embedded in any APP, i.e., programs that only need to be downloaded to a browser environment to run. In summary, the aforementioned computer-executable instructions can be any form of instruction, and the aforementioned computer programs can be any form of application, module, or plugin.

[0086] The training method for the visual language model provided in this application will be described in conjunction with exemplary applications and implementations of the server provided in the embodiments of this application.

[0087] See Figure 3A , Figure 3A This is a schematic diagram of the first process of the training method for the visual language model provided in this application embodiment. Taking the server as the execution entity as an example, it will be combined with... Figure 3A The steps shown are explained.

[0088] In step 101, the nth visual language model is invoked to rewrite the first summary information of the image sample to obtain multiple second summary information; where n is an integer variable that increments from 1, and the value of n satisfies 1≤n≤N, and N is a positive integer greater than or equal to 2. When n is 1, the nth visual language model is a pre-trained visual language model, and when 2≤n, the nth visual language model is trained based on the (n-1)th visual language model.

[0089] Here, a visual language model is an artificial intelligence model capable of understanding and processing visual information (such as images or videos) and linguistic information (such as text or speech). These models combine techniques from both computer vision and natural language processing to achieve the understanding and generation of multimodal data.

[0090] It should be noted that the first summary information of an image sample refers to the original summary information of the image sample, while the second summary information refers to the summary information obtained by rewriting the first summary information and containing more descriptive details. For example, for an image of a long-haired longhorn beetle, the first summary information is "long-haired longhorn beetle", and the second summary information is "The image shows a long-haired longhorn beetle with a predominantly black body covered with multiple spots".

[0091] In some embodiments, see Figure 3B , Figure 3BThis is a schematic diagram of the second process of the training method for the visual language model provided in the embodiments of this application. Figure 3A Step 101 can be done Figure 3B Steps 1011 to 1013 are implemented, and the details are explained below.

[0092] In step 1011, the nth visual language model is called to encode the image sample to obtain image features, and the first summary information of the image sample is encoded to obtain text features.

[0093] In some embodiments, image features can be obtained by calling the visual encoder of the nth visual language model to encode the image sample; and text features can be obtained by calling the text encoder of the nth visual language model to encode the first summary information of the image sample.

[0094] As an example, see Figure 7 , Figure 7 This is a schematic diagram of the structure of the visual language model provided in the embodiments of this application, such as... Figure 7 As shown, image features can be obtained by calling the visual encoder of the visual language model illustrated in the figure; text features can be obtained by calling the text encoder to encode the first summary information of the image samples.

[0095] As an example, the nth visual language model could be Contrastive Language-Image Pre-training (CLIP), Fusion of multi-Modal representations with Vision and Language (FLAVA), Generalized Image-Text (GIT), BridgeTower, Bootstrapping Language-Image Pre-training, Open-World Learning Vision Transformer (OWL-ViT), Extended CLIP (X-CLIP), or Transformer-based Optical Character Recognition (TrOCR). Visual encoders can be convolutional neural networks (CNNs), residual networks (ResNets), or vision transformers (ViTs). Text encoders can be recurrent neural networks (RNNs), gated recurrent units (GRUs), transformers, or bidirectional encoder representations from transformers (BERTs).

[0096] In step 1012, the image features and text features are fused to obtain fused features.

[0097] In some embodiments, image features and text features can be fused by concatenation to obtain fused features. Alternatively, weights can be configured for image features and text features separately, and the image features and text features can be weighted and summed according to their weights to obtain fused features. Furthermore, a multilayer perceptron (MLP) can be used to perform nonlinear transformations on image features and text features to generate fused features. For example, it can be done through... Figure 7 The fusion layer shown fuses image features and text features to obtain fused features.

[0098] As an example, if the image feature is a and the text feature is b, then the fused feature obtained by concatenating the image feature and the text feature as described above can be represented as [a; b].

[0099] In step 1013, the fused features are decoded to obtain multiple second summary information.

[0100] In some embodiments, the fused features can be decoded by calling the decoder of the nth visual language model to obtain multiple second summary information. See also, as an example. Figure 7 It can be done Figure 7 The decoder shown decodes the fused features to obtain multiple second summary information.

[0101] Here, during the decoding of fused features by the decoder of the nth visual language model, the randomness of the generated second summary information can be controlled by adjusting the temperature coefficient. For example, in a coordinate system with candidate words as the x-axis and their corresponding probabilities as the y-axis, the temperature coefficient is used to adjust the shape of the probability distribution. A larger temperature coefficient results in a smoother probability distribution, indicating smaller differences in probability values ​​among multiple candidate words and higher randomness in the generated result. Conversely, a smaller temperature coefficient results in a sharper probability distribution, with the generated result tending to select the word with the highest probability and lower randomness. For instance, if only one candidate word has a probability close to 1, while other candidate words have probabilities almost 0, this means that when generating summary information, the visual language model will directly select the candidate word with a probability close to 1. The temperature coefficient can range from 0.1 to 2.0. This will be explained in detail below.

[0102] In some embodiments, see Figure 3C , Figure 3C This is a schematic diagram of the third process of the training method for the visual language model provided in the embodiments of this application. Figure 3B Step 1013 can be achieved through Figure 3C Steps 10131 to 10133 are implemented, and the details are explained below.

[0103] In step 10131, multiple candidate temperature coefficients are generated, wherein the candidate temperature coefficients are used to control the randomness of the nth visual language model in decoding the fused features to obtain candidate summary information.

[0104] In some embodiments, during the decoding of fused features by the decoder of the nth visual language model, multiple candidate temperature coefficients can be generated at uniform intervals within a defined temperature coefficient range. This method is simple and intuitive, and can cover the entire range.

[0105] As an example, if the determined temperature coefficient range is [0.2, 1.8] and we want to generate 5 candidate temperature coefficients, we can determine the interval as (1.8-0.2) / (5-1)=0.4. Then the generated candidate temperature coefficients are 0.2, 0.6, 1.0, 1.4, and 1.8 respectively.

[0106] In step 10132, in response to the selection operation for multiple candidate temperature coefficients, at least one selected candidate temperature coefficient is taken as the target temperature coefficient.

[0107] In some embodiments, the selection of multiple candidate temperature coefficients can be random or can be performed by the user in response to a selection operation in the graphical interface.

[0108] As an example, in response to the selection operation for the above 5 candidate temperature coefficients, 1.4 and 1.8 can be used as target temperature coefficients.

[0109] In step 10133, the fusion features are decoded based on at least one target temperature coefficient to obtain multiple summary information corresponding to each target temperature coefficient, which are used as second summary information.

[0110] As an example, the fused features can be decoded based on target temperature coefficients of 1.4 and 1.8 respectively to obtain multiple summary information corresponding to each target temperature coefficient, which can be used as the second summary information.

[0111] This application generates multiple candidate temperature coefficients and allows selection of a target temperature coefficient, enabling control over the randomness of the decoding process to varying degrees, resulting in more diverse generated summary information. Since a single temperature coefficient may cause the model to get stuck in a local optimum, leading to an incomplete or inaccurate summary, using multiple temperature coefficients helps the model escape local optima, explore more possible summary expressions, and thus improve the overall quality of the summary. Allowing selection of multiple candidate temperature coefficients gives users more control and a greater sense of participation. Users can flexibly adjust the randomness of the generated summary according to their needs and preferences, thereby obtaining summary information that better meets their expectations. The requirements for summary information vary across different application fields. For example, in news reporting, accurate and objective summaries may be more necessary; while in artistic creation, creativity and imagination in the summary may be more important. Setting multiple candidate temperature coefficients allows the system to flexibly adapt to the needs of different fields, generating summary information that meets the requirements of specific fields by adjusting the temperature coefficients.

[0112] In some embodiments, see Figure 3D , Figure 3D This is a schematic diagram of the fourth process of the training method for the visual language model provided in the embodiments of this application. Figure 3C Step 10133 can be achieved through Figure 3D Steps 201 to 203 are implemented, and the details are explained below.

[0113] In step 201, the fused features are decoded based on at least one target temperature coefficient to obtain the probability distribution corresponding to each target temperature coefficient, wherein the probability distribution represents the probability of multiple candidate words being output respectively.

[0114] In some embodiments, a low temperature coefficient (e.g., less than or equal to 1) makes the probability distribution of multiple candidate words more concentrated, causing the visual language model to be more inclined to select the word with the highest probability. A high temperature coefficient (e.g., greater than 1) makes the probability distribution of multiple candidate words smoother, causing the visual language model to be more inclined to select the word with the lower probability, thereby increasing the diversity of the generated summary information.

[0115] As an example, if the temperature coefficient is 0.5, the probability distribution of candidate words is: P(walk) = 0.7, P(run) = 0.2, P(swim) = 0.1; while when the temperature coefficient is 1.8, the probability distribution of candidate words is: P(enjoy) = 0.3, P(play) = 0.25, P(walk) = 0.25, P(run) = 0.2.

[0116] In step 202, the target word corresponding to each target temperature coefficient is determined according to the probability distribution corresponding to each target temperature coefficient.

[0117] In some embodiments, based on the probability distribution of multiple candidate words corresponding to each target temperature coefficient, the candidate word with the highest probability among the multiple candidate words can be used as the target word corresponding to the target temperature coefficient.

[0118] As an example, if the temperature coefficient is 0.5, the probability distribution of candidate words is: P(walk) = 0.7, P(run) = 0.2, P(swim) = 0.1. In this case, the visual language model will choose the word "walk" with the highest probability. However, when the temperature coefficient is 1.8, the probability distribution of candidate words is: P(enjoy) = 0.3, P(play) = 0.25, P(walk) = 0.25, P(run) = 0.2. In this case, the visual language model may choose the word "enjoy" with a relatively high probability.

[0119] In step 203, the target words corresponding to each target temperature coefficient are combined to obtain multiple summary information corresponding to each target temperature coefficient.

[0120] Continuing with the example above, if the temperature coefficient is 0.5, the visual language model might select "beach," "on," "people," and "walking" in sequence, generating the summary information "There are people walking on the beach." If the temperature coefficient is 1.8, the visual language model might select "people," "on," "golden," "sand," "on," "enjoy," and "sunshine" in sequence, generating the summary information "People are enjoying the sunshine on the golden sand."

[0121] This application embodiment decodes the fused features based on at least one target temperature coefficient to obtain a probability distribution corresponding to each target temperature coefficient. The probability distribution represents the probability of multiple candidate words being output. Based on the probability distribution corresponding to each target temperature coefficient, the target word corresponding to each target temperature coefficient is determined. The target words corresponding to each target temperature coefficient are combined to obtain multiple summary information corresponding to each target temperature coefficient.

[0122] In some embodiments, see Figure 3E , Figure 3E This is a schematic diagram of the fifth step of the training method for the visual language model provided in this application embodiment. Step 101 above, "calling the nth visual language model to rewrite the first summary information of the image sample to obtain multiple second summary information," can be achieved through... Figure 3E Steps 1014 and 1015 are implemented, and the details are explained below.

[0123] In step 1014, the nth image sample is extracted from the image sample dataset, wherein the nth image sample is different from the (n-1)th image sample.

[0124] In some embodiments, the nth image sample can be obtained from public datasets or online resources, or it can be an image sample taken by the user himself through the camera of the terminal device.

[0125] In some embodiments, the nth image sample processed by the nth visual language model may be the same as the (n-1)th image sample processed by the (n-1)th visual language model. For example, the (n-1)th visual language model processes image sample A, and the nth visual language model processes image sample B. Alternatively, the nth image sample processed by the nth visual language model may be different from the (n-1)th image sample processed by the (n-1)th visual language model. For example, the (n-1)th visual language model processes image sample A, and the nth visual language model processes image sample B.

[0126] In step 1015, the nth visual language model is invoked to rewrite the first summary information of the nth image sample, resulting in multiple second summary information.

[0127] Here, the implementation method of calling the nth visual language model to rewrite the first summary information of the nth image sample to obtain multiple second summary information is described in the specific implementation of step 101 above, and will not be repeated here.

[0128] In this embodiment, the image samples processed by the nth visual language model and the (n-1)th visual language model can be the same or different. When the image samples processed by the nth visual language model and the (n-1)th visual language model are different, the visual language model can better learn the correlation between the image and the summary information when processing different images and summary information, thereby improving the generalization ability of the visual language system. Even when processing the same image samples, based on the randomness of the summary information generated by the visual language model, the rewriting results of the visual language model at different stages can complement each other, making the final summary information more comprehensive and accurate. This helps to further optimize the training and parameter adjustment of the visual language model and improve the overall performance of the visual language model.

[0129] In some embodiments, see Figure 3F , Figure 3F This is a schematic diagram of the sixth step in the training method for the visual language model provided in this application embodiment. In step 101 above, "When n is 1, the nth visual language model is a pre-trained visual language model", the pre-trained visual language model can be trained by executing... Figure 3F The training was completed in steps 1016 to 1019, and the details are explained below.

[0130] In step 1016, a third prompt word is constructed for the first summary information of the image sample, wherein the third prompt word is used to instruct the generation of a third summary information containing more descriptive details based on the first summary information.

[0131] In some embodiments, the third cue word can guide the visual language model to add descriptive details from multiple dimensions based on the first summary information, such as the object's appearance features, its environment, and the specific manner of its actions. Guiding expressions such as "detailed description," "add more details," and "specific explanation" can be used.

[0132] As an example, if the image sample is a picture of a cat and the first summary information is "A cat in the yard," then the constructed third prompt could be "Based on 'A cat in the yard,' please supplement the details such as the cat's appearance, actions, the layout of the yard, and the environment to generate a richer summary information."

[0133] In some embodiments, see Figure 3G , Figure 3G This is a schematic diagram of the seventh step in the training method of the visual language model provided in this application embodiment. Before step 1016 above, in order to filter the first summary information, the following can be performed: Figure 3G Steps 301 to 302 are explained in detail below.

[0134] In step 301, the filtering indicators of the first summary information are extracted, wherein the filtering indicators include at least one of the following: the number of entity words included in the first summary information, the type of entity words, the information density included in the first summary information, and the matching degree between the first summary information and the image sample.

[0135] Here, entity words refer to words in the text that have clear semantics and refer to specific objects. Types of entity words include: people, places, plants and animals, organizations, and products. Information density refers to the amount of information contained within a unit length or unit space. In text generation and image description, information density measures the compactness and richness of the generated summary information.

[0136] In some embodiments, the information density of the first summary information can be determined as follows: First, the summary information is segmented into individual words. Second, each word is tagged with its part of speech to distinguish between content words and function words. Then, the number of content words and the total number of words are counted, and the proportion of content words is obtained by dividing the number of content words by the total number of words, which is used as the information density of the first summary information.

[0137] As an example, the first summary information is "There is a cute little dog running happily in a beautiful garden". Segmenting the first summary information yields 14 words, including "in", "beautiful", "of", "garden", "in", "have", "a", "cute", "of", "little dog", "in", "happy", "ground", and "running". After tagging the parts of speech, the remaining 7 content words are "beautiful", "garden", "a", "cute", "little dog", "happy", and "running". The total number of words is 14, so the information density is approximately 7 ÷ 14 = 0.5.

[0138] In some embodiments, the matching degree between the first summary information and the image sample can be comprehensively considered from multiple levels such as semantics, visual elements and logical coherence, for example, key entity matching, action and state matching, color and appearance feature matching, and spatial relationship matching.

[0139] As an example, suppose an image sample presents the following: "There is a cup on the table and a book next to it." The corresponding first summary information is "The cup is on the table, and the book is next to the cup." The first summary information accurately describes the spatial relationship between objects in the image sample; that is, the spatial relationship between the image sample and the first summary information matches.

[0140] In some embodiments, the matching degree between the first summary information and the image sample can be determined by performing the following process: first, the first summary information is encoded to obtain a text encoding vector, the image sample is encoded to obtain an image encoding vector, and then the similarity between the text encoding vector and the image encoding vector is determined by a similarity algorithm as the matching degree between the first summary information and the image sample.

[0141] For example, similarity algorithms can be cosine similarity, Euclidean distance, Manhattan distance, Chebyshev distance, edit distance, etc.

[0142] Taking cosine similarity as an example, if the text encoding vector is m and the image encoding vector is n, we determine the vector product of the text and image encoding vectors, the length product of the lengths of the text and image encoding vectors, and the ratio of the vector product to the length product, which is used as the similarity. For example, the vector product of the text and image encoding vectors is m*n. The length of the text encoding vector is |m|, the length of the image encoding vector is |n|, and the length product is |m|*|n|, then the similarity is (m*n) / (|m|*|n|).

[0143] In step 302, in response to the screening index being less than the screening index threshold, the first summary information corresponding to the screening index is filtered out to obtain the filtered first summary information.

[0144] As an example, if the number of entity words included in the first summary information A is 2, the number of entity words included in the first summary information B is 5, the number of entity words included in the first summary information C is 4, and the threshold for the number of entity words is 3, then the first summary information A will be filtered out, and the first summary information B and the first summary information C will be retained.

[0145] This application embodiment filters the first summary information based on screening criteria. By focusing on the entity words included in the first summary information, it is possible to filter summary information containing key information. Information density criteria help filter out rich yet concise summaries. Excessively high information density may make the summary difficult to understand, while excessively low information density renders the summary worthless. Through screening, summary information with moderate information density can be found. When processing a large amount of first summary information, screening criteria can help quickly identify and eliminate redundant or duplicate summary information. The screening process allows the visual language model to concentrate more resources on high-quality summary information. For summaries that do not meet the screening criteria, no further complex processing and analysis is required, saving computational resources and time, and improving the data processing efficiency of the visual language model.

[0146] See also Figure 3F The following will be explained following step 1016 above.

[0147] In step 1017, based on the third prompt word, the first summary information is rewritten into third summary information using the visual language model to be trained.

[0148] As an example, if the first summary information is "a cat is in the yard," and the third prompt is "Based on 'a cat is in the yard,' please supplement the details of the cat's appearance, actions, yard layout, and environment to generate a richer summary information," then the third summary information obtained by rewriting it through the visual language model to be trained based on the third prompt could be: "In a spacious yard, there is a black and white cat lying on a stone bench in the corner of the yard, and there are several tall osmanthus trees planted in the yard."

[0149] In step 1018, the image sample and the third summary information are used to construct an image-text pair.

[0150] As an example, the above example, which includes an image sample of a cat and the third summary information "In a spacious courtyard, there is a black and white cat lying on a stone bench in the corner of the courtyard, and there are several tall osmanthus trees in the courtyard," can be constructed into an image-text pair.

[0151] In step 1019, the visual language model to be trained is trained using image-text pairs to obtain a pre-trained visual language model.

[0152] In some embodiments, see Figure 3H , Figure 3H This is a schematic diagram of the eighth step in the training method of the visual language model provided in the embodiments of this application. Figure 3F Step 1019 can be achieved through Figure 3H Steps 10191 to 10194 are implemented, and the details are explained below.

[0153] In step 10191, the visual language model to be trained is invoked to encode the image samples in the image-text pair to obtain image features.

[0154] In some embodiments, the image samples and third-party summary information in the constructed image-text pair can be used as input to the visual language model to be trained. The image samples in the image-text pair are then encoded by the visual encoder of the visual language model to be trained to obtain image features.

[0155] As an example, the visual encoder of a trained visual language model can be a convolutional neural network, a residual network, a visual converter, etc.

[0156] In step 10192, the image features are decoded to obtain the predicted summary information.

[0157] In some embodiments, image features can be decoded by invoking the decoder of the visual language model to be trained to obtain predicted summary information. When the decoder structure includes a multi-head self-attention layer, a feed-forward network layer, and a layer normalization layer, the image features can be decoded to obtain predicted summary information by performing the following processing:

[0158] First, image features are input into a multi-head attention layer. Three different linear transformation matrices are used to obtain three matrices: query, key, and value. For each head, the dot product of the query and key matrices is determined, and then a normalization function is applied to obtain attention weights. These weights are multiplied by the value matrix and summed to obtain the output of each head. The outputs of multiple heads are concatenated and transformed using a linear transformation matrix to obtain the final multi-head self-attention features. Second, a feedforward neural network is used to perform a nonlinear transformation on the multi-head self-attention features to obtain nonlinear features. Then, a normalization layer is used to normalize the nonlinear features to obtain normalized features. Finally, a linear layer maps the normalized features to obtain the predicted summary information.

[0159] In step 10193, the loss value is determined based on the difference between the third summary information and the predicted summary information in the image-text pair.

[0160] In some embodiments, the loss value can be determined by a preset loss function based on the difference between the third summary information and the predicted summary information in the image-text pair.

[0161] As an example, the loss function can be a loss function based on the difference in probability distribution, such as Cross-Entropy Loss or Kullback-Leibler Divergence; it can be a loss function based on distance metrics, such as Mean Squared Error Loss (MSE), Root Mean Squared Error (RMSE), or Mean Absolute Error Loss (MAE); or it can be a loss function based on sequence matching, such as Edit Distance Loss.

[0162] Taking edit distance loss as an example, the edit distance between the third summary information and the predicted summary information can be determined, and the edit distance can be used as the loss value. For example, if the third summary information is "a black dog is running in the park" and the predicted summary information is "a white dog is lying in the park", then the edit distance between the third summary information and the predicted summary information is 3.

[0163] In step 10194, the parameters of the visual language model to be trained are updated according to the loss value to obtain the pre-trained visual language model.

[0164] In some embodiments, based on the loss value, the backpropagation algorithm can be used to treat the loss value as error information and backpropagate from the output layer of the visual language model. Using the chain rule, the gradient of the loss function at each layer is calculated layer by layer. Based on the magnitude and direction of the obtained gradient, an optimization algorithm (such as stochastic gradient descent) is used to determine the step size and direction of parameter adjustment. Based on the step size and direction of parameter adjustment, the parameters of the visual language model to be trained at each layer are updated to obtain the pre-trained visual language model.

[0165] This application embodiment encodes image samples by calling the visual language model to be trained. It learns how to extract key information from images and decodes image features to obtain predicted summary information. During this process, the visual language model continuously learns how to transform the extracted image features into natural, fluent, accurate, and appropriate text descriptions. By comparing and learning from the third-party summary information in the image-text pairs, the visual language model can gradually improve the quality of text generation, making the generated predicted summary information more consistent with the image content, and the language expression richer and more accurate. During training, a large number of image-text pairs are used, allowing the visual language model to encounter various images and corresponding summary information, which helps improve the model's generalization ability.

[0166] See also Figure 3AThe following will be an explanation following step 101 above.

[0167] In step 102, multiple second summary information is classified into positive samples and negative samples, wherein the number of hallucination information included in the positive samples is less than the number of hallucination information included in the negative samples.

[0168] In some embodiments, see Figure 3I , Figure 3I This is a ninth flowchart illustrating the training method for the visual language model provided in this application embodiment. Figure 3A Step 102 can be achieved through Figure 3I Steps 1021 to 1024 are implemented, and the details are explained below.

[0169] In step 1021, each second summary information is divided into multiple summary segments.

[0170] In some embodiments, each second summary information can be divided into multiple summary segments according to the position of punctuation marks in each second summary information. Alternatively, each second summary information can be divided into multiple summary segments according to the position of entity words in the second summary information.

[0171] As an example, the content of the image sample is "a cat lying on the sofa", and the second summary information 1 is "a cat running on the grass". According to the position of the entity words, the second summary information 1 can be divided into 3 summary segments, namely "a cat", "on the grass" and "running".

[0172] In step 1022, hallucination recognition is performed on each summary segment to obtain the number of summary segments that include hallucination information in each second summary information, wherein the hallucination information is information that is unrelated to the content of the image sample.

[0173] In some embodiments, a pre-trained hallucination recognition model can be used to perform hallucination recognition on each summary segment in the second summary information. Summary segments that match the content of the image sample are considered as segments that do not contain hallucination information, while summary segments that do not match the content of the image sample are considered as segments that include hallucination information.

[0174] As an example, the content of the image sample is "a cat lying on the sofa". The three summary fragments in the second summary information 1 are "a cat", "on the grass" and "running". The summary fragments that include hallucination information are "on the grass" and "running", that is, the number of summary fragments that include hallucination information in the second summary information is 2.

[0175] In step 1023, the second summary information is sorted in ascending order according to the quantity to obtain the sorting result.

[0176] As an example, if the second summary information 1 includes a summary fragment of hallucination information as 3, the second summary information 2 includes a summary fragment of hallucination information as 5, the second summary information 3 includes a summary fragment of hallucination information as 4, and the second summary information 4 includes a summary fragment of hallucination information as 6, then the second summary information is sorted in ascending order according to the quantity, and the sorted result is second summary information 1, second summary information 3, second summary information 2, and second summary information 4.

[0177] In step 1024, at least one second summary information from the head of the sorting result is taken as a positive sample, and at least one second summary information from the tail of the sorting result is taken as a negative sample.

[0178] As an example, the second summary information 1 at the head of the sorting result can be used as a positive sample, or the second summary information 1 and the second summary information 3 starting from the head can be used as positive samples; the second summary information 4 at the tail of the sorting result can be used as a negative sample, or the second summary information 4 and the second summary information 2 starting from the tail can be used as negative samples.

[0179] See also Figure 3A The following will be an explanation following step 102 above.

[0180] In step 103, multiple partial order data are constructed based on multiple second summary information, wherein each partial order data includes a positive sample and a negative sample.

[0181] As an example, based on the second summary information 4 in the example above, a partially ordered data A is constructed, which includes a positive sample (i.e., second summary information 1) and a negative sample (i.e., second summary information 4).

[0182] In step 104, multiple partially ordered data are generated using the nth visual language model to predict the probabilities of hallucination information.

[0183] In some embodiments, step 104 above can be implemented by performing the following processing: generating a first cue word, wherein the first cue word is used to indicate a first predicted probability for each positive sample and a second predicted probability for each negative sample in the partially ordered data generated based on the image samples, wherein the first predicted probability represents the probability that the positive sample includes hallucination information and the second predicted probability represents the probability that the negative sample includes hallucination information; calling the nth visual language model model to generate the first predicted probability and the second predicted probability based on the first cue word.

[0184] As an example, the image sample is a city street scene photo (without any animal elements). The positive sample is "There is a yellow taxi on the left side of the image" (actually exists), and the negative sample is "There is a guide dog walking on the sidewalk" (completely fictional). The generated first prompt could be: "The image sample is a city street scene photo, with no animal elements. Please generate predicted probabilities for the following positive and negative samples: Positive sample: There is a yellow taxi on the left side of the image (actually exists). Please predict the probability that this positive sample contains hallucinatory information. Negative sample: There is a guide dog walking on the sidewalk (completely fictional). Please predict the probability that this negative sample contains hallucinatory information."

[0185] The nth visual language model is invoked. Based on the first prompt word mentioned above, the probability of generating a positive sample containing hallucination information is 0.15, and the probability of generating a negative sample containing hallucination information is 0.85. That is, the first prediction probability is 0.15 and the second prediction probability is 0.85.

[0186] This application embodiment generates a first prompt word, calls the nth visual language model, and generates a first predicted probability for each positive sample and a second predicted probability for each negative sample in the partially ordered data based on the first prompt word. The first prompt word clearly specifies the specific content of the image sample and the description of the positive and negative samples. Through the precise prompt word, the visual language model can better understand the task requirements, clearly distinguish the features of positive and negative samples, and thus more accurately calculate the probability that they contain hallucination information. This avoids prediction errors caused by vague task descriptions and improves the accuracy and reliability of hallucination information prediction.

[0187] In other embodiments, step 104 above can also be implemented by performing the following processing: generating a second cue word, wherein the second cue word is used to indicate the generation of a third predicted probability of a positive sample in the partially ordered data based on the image samples and the negative samples in the partially ordered data, and a fourth predicted probability of a negative sample in the partially ordered data based on the image samples and the positive samples in the partially ordered data, wherein the third predicted probability represents the probability that the positive sample includes hallucination information, and the fourth predicted probability represents the probability that the negative sample includes hallucination information; invoking the nth visual language model model to generate the third predicted probability and the fourth predicted probability based on the second cue word.

[0188] As an example, the second prompt is: "Based on image sample A and negative sample B in the partially ordered data, generate a third predicted probability of positive sample C in the partially ordered data; based on image sample A and positive sample C in the partially ordered data, generate a fourth predicted probability of negative sample B in the partially ordered data. The third predicted probability represents the probability that positive sample C contains hallucination information, and the fourth predicted probability represents the probability that negative sample B contains hallucination information."

[0189] If image sample A is a picture of a cat, negative sample B in the partially ordered data is "This is a dog", and positive sample C is "This is a cat", then based on the second cue word, the nth visual language model is invoked. The nth visual language model identifies image sample A and negative sample B and finds that image sample A is identified as a cat, while negative sample B describes a dog. Therefore, it judges that negative sample B has a higher probability of containing hallucination information, and the generated third prediction probability may be 0.9. Thus, the visual language model believes that the probability of positive sample C containing hallucination information is 0.1.

[0190] Next, the visual language model analyzes image sample A and positive sample C. It finds that the descriptions of image A and positive sample C are consistent—both are cats. Therefore, it considers positive sample C to be accurate and not containing hallucination information, generating a fourth prediction probability of 0.1. Thus, the visual language model considers the probability that negative sample B contains hallucination information to be 0.9. In other words, the probability of generating a third prediction based on the second cue word is 0.1, and the probability of generating a fourth prediction is 0.9.

[0191] This application's embodiments, through the use of a second cue word, enable the visual language model to more accurately identify illusory information contained in positive samples, thereby improving the accuracy of the third prediction probability; simultaneously, it helps the visual language model identify illusory information contained in negative samples, improving the accuracy of the fourth prediction probability. The generation of the third and fourth prediction probabilities helps the visual language model reduce misjudgments caused by illusory information when recognizing images, making the visual language model more robust to noisy data and outliers, and reducing the risk of incorrect predictions.

[0192] In some embodiments, before performing step 104 above, the partially ordered data can be filtered in the following way: for each partially ordered data, the following processing is performed: determine the difference between the lengths of positive and negative samples in the partially ordered data; if the difference is greater than the difference threshold, the partially ordered data is deleted from the multiple partially ordered data.

[0193] Here, the length of a sample (including positive and negative samples) refers to the number of characters contained in the text description of the sample. These characters can be letters, Chinese characters, punctuation marks, etc. For example, if the text description of a sample is "This is a beautiful landscape picture.", then the character length of the sample is 12 (including punctuation).

[0194] As an example, if the length of the positive sample in the partially ordered data A is 5 and the length of the negative sample is 17, then the difference between the lengths of the positive and negative samples in the partially ordered data is 12. If the difference threshold is 10, then the partially ordered data A will be deleted from the multiple partially ordered data.

[0195] This application embodiment compares the lengths of positive and negative samples in partially ordered data. When the difference between their lengths exceeds a threshold, the corresponding partially ordered data is deleted from the partially ordered data. By deleting partially ordered data with excessively large length differences, the size of the dataset can be reduced, thereby saving computational resources and storage space. It can also reduce noise and outliers in the dataset, thereby improving the quality of partially ordered data. This helps to prevent the visual language model from being misled during training and improves the accuracy and stability of the image summarization information generated by the visual language model.

[0196] In step 105, the loss value of the nth visual language model is determined based on the prediction probabilities corresponding to the multiple partially ordered data.

[0197] In some embodiments, see Figure 3J , Figure 3J This is a schematic diagram of the tenth step of the training method for the visual language model provided in this application embodiment. When executing the above-described step of "calling the nth visual language model and generating a first prediction probability and a second prediction probability based on the first prompt word," Figure 3A Step 105 can be achieved through Figure 3J Steps 1051A to 1055A are implemented, and the details are explained below.

[0198] In step 1051A, a first label pre-set for positive samples in the partially ordered data and a second label pre-set for negative samples in the partially ordered data are obtained.

[0199] In some embodiments, the first label pre-set for positive samples in the partially ordered data can be 0, indicating that it does not contain hallucination information; the second label pre-set for negative samples in the partially ordered data can be 1, indicating that it contains hallucination information.

[0200] In step 1052A, based on the first predicted probability and the first label, the first loss for the positive sample in each partially ordered data is determined.

[0201] In some embodiments, based on a first predicted probability and a first label, a first loss for positive samples in each partially ordered data can be determined using a preset first loss function.

[0202] As an example, the first loss function can be as shown in Equation (1).

[0203] L pos1 =-y pos log(p pos1 )-(1-y pos log(1-p) pos1 (1)

[0204] Among them, y posp represents the first label corresponding to the positive sample. pos1 1-y represents the first predicted probability corresponding to the positive sample. pos This represents the complement of the first label corresponding to the positive sample, 1-p. pos1 L represents the difference between the first predicted probability of a 1 and the probability of a positive sample. pos1 This represents the first loss for positive samples.

[0205] As an example, the first predicted probability of a positive sample in partially ordered data A is 0.1, and the second predicted probability of a negative sample is 0.8; the first predicted probability of a positive sample in partially ordered data B is 0.2, and the second predicted probability of a negative sample is 0.9. Then, when the first label is 0, the first loss L for the positive samples of partially ordered data A is... pos1 = -log(1-p pos1 )=-log0.9≈0.1054, the first loss L for positive samples of partially ordered data B pos1 = -log(1-p pos1 = -log0.8 ≈ 0.2231.

[0206] In step 1053A, a second loss for negative samples in each partially ordered data is determined based on the second predicted probability and the second label.

[0207] In some embodiments, based on the second predicted probability and the second label, a second loss for negative samples in each partially ordered data can be determined using a preset second loss function.

[0208] As an example, the second loss function can be as shown in Equation (2).

[0209] L neg1 =-y neg log(p neg1 )-(1-y neg log(1-p) neg1 (2)

[0210] Among them, y neg p represents the second label corresponding to the negative sample. neg1 1-y represents the second predicted probability corresponding to the negative sample. neg This represents the complement of the second label corresponding to the negative sample, 1-p neg1 L represents the difference between the second predicted probability of a 1 and the probability of a negative sample. neg1 This represents the second loss for negative samples.

[0211] As an example, the first predicted probability of a positive sample in partially ordered data A is 0.1, and the second predicted probability of a negative sample is 0.8; the first predicted probability of a positive sample in partially ordered data B is 0.2, and the second predicted probability of a negative sample is 0.9. Then, when the second label y... negWhen the value is 1, the second loss L for negative samples of partially ordered data A neg1 =-log(p neg1 )=-log0.8≈0.2231, the second loss L for negative samples of partially ordered data B neg1 =-log(p neg1 = -log0.9 ≈ 0.1054.

[0212] In step 1054A, the first sum of the first loss and the second loss for each partially ordered data is determined.

[0213] As an example, the first loss L of the positive samples of the partially ordered data A above pos1 = -log(1-p pos1 The second loss L for negative samples of partially ordered data A is -log0.9 ≈ 0.1054. neg1 =-log(p neg1 If )=-log0.8≈0.2231, then the first sum of the first and second losses for the partially ordered data A is 0.3285. The first loss L for the positive samples of the partially ordered data B is... pos1 = -log(1-p pos1 )=-log0.8≈0.2231, the second loss L for negative samples of partially ordered data B neg1 =-log(p neg1 If )=-log0.9≈0.1054, then the first sum of the first loss and the second loss of the partially ordered data B is 0.3285.

[0214] In step 1055A, the average of the first sums corresponding to multiple partially ordered data is used as the loss value of the nth visual language model.

[0215] As an example, the sum of the first and second losses of the partially ordered data A is 0.3285, and the sum of the first and second losses of the partially ordered data B is 0.3285. Therefore, the loss value of the nth visual language model is 0.3285.

[0216] In other embodiments, see Figure 3K , Figure 3K This is a schematic diagram of the eleventh step in the training method of the visual language model provided in this application embodiment. When executing the above-described step of "calling the nth visual language model and generating the third and fourth prediction probabilities based on the second prompt word," Figure 3A Step 105 can be achieved through Figure 3K Steps 1051B to 1055B are implemented, and the details are explained below.

[0217] In step 1051B, a first label pre-set for positive samples in the partially ordered data and a second label pre-set for negative samples in the partially ordered data are obtained.

[0218] In some embodiments, the first label pre-set for positive samples in partially ordered data can be 0, indicating that it does not contain hallucination information; the second label pre-set for negative samples in partially ordered data can be 1, indicating that it contains hallucination information. Here, the value of the label represents the probability that the sample contains hallucination information; the larger the value of the label, the greater the probability that the sample corresponding to the label contains hallucination information.

[0219] In step 1052B, a third loss is determined for positive samples in each partially ordered data based on the third predicted probability and the first label.

[0220] In some embodiments, based on the third predicted probability and the first label, a preset third loss function can be used to determine the third loss for positive samples in each partially ordered data.

[0221] As an example, the third loss function can be as shown in Equation (3).

[0222] L pos2 =-y pos log(p pos2 )-(1-y pos log(1-p) pos2 (3)

[0223] Among them, y pos p represents the first label corresponding to the positive sample. pos2 1-y represents the third predicted probability corresponding to the positive sample. pos This represents the complement of the first label corresponding to the positive sample, 1-p. pos2 L represents the difference between the third predicted probability of 1 and the probability of a positive sample. pos2 This represents the third loss for positive samples.

[0224] As an example, the third prediction probability for positive samples in partially ordered data A is 0.1, and the fourth prediction probability for negative samples is 0.8; the third prediction probability for positive samples in partially ordered data B is 0.2, and the fourth prediction probability for negative samples is 0.8. Then, when the first label is 0, the third loss L for positive samples in partially ordered data A is... pos2 = -log(1-p pos2 The third loss L for the positive samples of partially ordered data B is approximately -log0.9 ≈ 0.1054. pos2 = -log(1-p pos2 = -log0.8 ≈ 0.2231.

[0225] In step 1053B, a fourth loss is determined for negative samples in each partially ordered data based on the fourth predicted probability and the second label.

[0226] In some embodiments, based on the fourth predicted probability and the second label, a preset fourth loss function can be used to determine the fourth loss for negative samples in each partially ordered data.

[0227] As an example, the fourth loss function can be as shown in Equation (4).

[0228] L neg2 =-y neg log(p neg2 )-(1-y neg log(1-p) neg2 (4)

[0229] Among them, y neg p represents the second label corresponding to the negative sample. neg2 This represents the fourth predicted probability for the negative sample, 1-y. neg This represents the complement of the second label corresponding to the negative sample, 1-p neg2 L represents the difference between the fourth predicted probability corresponding to 1 and the negative sample. neg2 This represents the fourth loss for negative samples.

[0230] As an example, the third predicted probability of a positive sample in partially ordered data A is 0.1, and the fourth predicted probability of a negative sample is 0.8; the third predicted probability of a positive sample in partially ordered data B is 0.2, and the fourth predicted probability of a negative sample is 0.9. Then, when the second label y... neg When the value is 1, the fourth loss L for the negative samples of the partially ordered data A is... neg1 =-log(p neg1 )=-log0.8≈0.2231, the fourth loss L for negative samples of partially ordered data B. neg1 =-log(p neg1 = -log0.9 ≈ 0.1054.

[0231] In step 1054B, the second sum of the third loss and the fourth loss for each partially ordered data is determined.

[0232] As an example, the third loss L of the positive samples of the partially ordered data A above. pos2 = -log(1-p pos1 The fourth loss L for negative samples of partially ordered data A is -log0.9 ≈ 0.1054. neg2 =-log(p neg1 Since ) = -log0.8 ≈ 0.2231, the second sum of the third and fourth losses for partially ordered data A is 0.3285. The third loss L for the positive samples of partially ordered data B...pos1 = -log(1-p pos1 )=-log0.8≈0.2231, the fourth loss L for negative samples of partially ordered data B. neg1 =-log(p neg1 If )=-log0.9≈0.1054, then the second sum of the third and fourth losses of the partially ordered data B is 0.3285.

[0233] In step 1055B, the average of the second sums corresponding to multiple partially ordered data is used as the loss value of the nth visual language model.

[0234] As an example, the second sum of the third and fourth losses of the partially ordered data A is 0.3285, and the first sum of the third and fourth losses of the partially ordered data B is 0.3285. Therefore, the loss value of the nth visual language model is 0.3285.

[0235] In this embodiment, based on a first label pre-set for positive samples in partially ordered data and a second label pre-set for negative samples, the loss of each positive sample in partially ordered data is determined based on the predicted probability of the positive sample and the first label; the loss of each negative sample in partially ordered data is determined based on the predicted probability of the negative sample and the second label. The average of the sum of the losses of positive samples and negative samples is used as the loss value of the nth visual language model. The above scheme, by pre-setting labels for positive and negative samples and calculating the loss based on these labels and the predicted probability, enables the visual language model to better adapt to the characteristics of partially ordered data during training, better capture the patterns and regularities in the partially ordered data, improve the accuracy and stability of prediction, and thus optimize the performance of the visual language model.

[0236] In step 106, the nth visual language model is updated according to the loss value to obtain the (n+1)th visual language model; when the iteration reaches the value of N, the obtained (N+1)th visual language model is used as the visual language model for generating summary information of the image to be processed.

[0237] In some embodiments, the backpropagation algorithm can be used to update the nth visual language model based on the loss value to obtain the (n+1)th visual language model.

[0238] Here, the value of N is dynamically determined after n is incremented and the (n+1)th visual language model is trained; see [link to relevant documentation]. Figure 3L , Figure 3L This is a schematic diagram of the twelfth step of the training method for the visual language model provided in this application embodiment. After performing the above step 106, it can be executed... Figure 3L Steps 401 to 402 determine the value of N, which are explained in detail below.

[0239] In step 401, the performance of the (n+1)th visual language model is tested to obtain performance parameter values.

[0240] Here, the performance of the (n+1)th visual language model can be determined by comparing the predicted summary information generated by the (n+1)th visual language model for the target image with the target summary information of the target image. This will be explained in detail below.

[0241] In some embodiments, see Figure 3M , Figure 3M This is a schematic diagram of the thirteenth step of the training method for the visual language model provided in the embodiments of this application. Figure 3L Step 401 can be achieved through Figure 3M Steps 4011 to 4013 are implemented, and the details are explained below.

[0242] In step 4011, training data is obtained, wherein the training data includes multiple target images and target summary information for each target image.

[0243] In some embodiments, target images and corresponding target summary information can be extracted directly from publicly available image-text multimodal datasets. For example, publicly available datasets may include Conceptual Captions or Large-scale AIOpenNetwork. Alternatively, target images can be obtained from publicly available datasets, online resources, or self-captured images, and corresponding target summary information can be prepared for each image. This summary information can be manually annotated or generated through a visual language model.

[0244] As an example, the training data contains three target images and target summary information for each target image. Target image 1 is a picture of a cat, and the target summary information 1 corresponding to target image 1 is "an orange cat playing on the grass"; target image 2 is a picture of a dog, and the target summary information 2 corresponding to target image 2 is "a black dog running in the park"; target image 3 is a picture of a bird, and the target summary information 3 corresponding to target image 3 is "a blue bird resting on a branch".

[0245] In step 4012, the (n+1)th visual language model is called to generate prediction summary information corresponding to multiple target images.

[0246] In some embodiments, the (n+1)th visual language model can be invoked to encode the target image to obtain image features, and the image features can be decoded to obtain the predicted summary information corresponding to each target image.

[0247] As an example, for target image 1 in the above example, the generated prediction summary information 1 is "an orange cat playing on the grass"; for target image 2 in the above example, the generated prediction summary information 2 is "a white dog lying in the park"; for target image 3 in the above example, the generated prediction summary information 3 is "a blue bird resting on a tree branch".

[0248] In step 4013, the ratio of the number of predicted summary information matching the target summary information to the number of multiple target images is determined as a performance parameter value.

[0249] In some embodiments, the matching of predicted summary information and target summary information can be determined in the following ways: 1) Keyword matching: Extract keywords from the predicted summary information and target summary information respectively. When the keywords of the predicted summary information and target summary information are the same, the predicted summary information and target summary information are considered to match; 2) Determine the edit distance between the predicted summary information and target summary information. If the edit distance is less than the edit distance threshold (e.g., the edit distance between the two is 1 and the edit distance threshold is 2), the predicted summary information and target summary information are considered to match; 3) Determine the cosine similarity between the text vector corresponding to the predicted summary information and the text vector corresponding to the target summary information. If the cosine similarity is higher than the similarity threshold, the predicted summary information and target summary information are considered to match.

[0250] As an example, the target summary information 1 is "an orange cat is playing on the grass", and the predicted summary information 1 is "an orange cat is playing on the grass". It can be seen that the target summary information 1 and the predicted summary information 1 are exactly the same. Therefore, the target summary information 1 and the predicted summary information 1 match.

[0251] The target summary information 2 is "a black dog is running in the park", and the predicted summary information 2 is "a white dog is lying in the park". It can be seen that the edit distance between the target summary information 2 and the predicted summary information 2 is 3. When the edit distance is 2, the target summary information 1 and the predicted summary information 1 do not match.

[0252] The target summary information 3 is "a blue bird is resting on a branch", and the predicted summary information 3 is "a blue bird is resting on a tree branch". It can be seen that the edit distance between the target summary information 3 and the predicted summary information 3 is 1. When the edit distance is 2, the target summary information 3 and the predicted summary information 3 match.

[0253] Therefore, if the number of predicted summary information matching the target summary information is 2 and the number of multiple target images is 3, the ratio is 0.67, that is, the performance parameter value is 0.67.

[0254] See also Figure 3L The following will be an explanation following step 401 above.

[0255] In step 402, in response to the performance parameter value being greater than the performance parameter threshold, the current value of n is taken as N.

[0256] As an example, if the performance parameter threshold is 0.65 and the performance parameter value of the (n+1)th visual language model is 0.67, then the current value of n is taken as N.

[0257] This embodiment of the application dynamically determines the value of N after each increment of n and training to obtain the (n+1)th visual language model. This allows for flexible determination of the termination condition for model training, ensuring that training stops when the model performance reaches a preset threshold, avoiding unnecessary waste of computational resources, and ensuring that the model performance meets requirements. The performance is quantified by using the ratio of the number of predicted summary information matching the target summary information to the number of multiple target images as a performance parameter. When the performance parameter value is greater than the performance parameter threshold, the current value of n is taken as N, indicating that the model has reached the expected performance level. This mechanism ensures that the model is adopted when performance requirements are met, thereby improving the model's practicality and reliability.

[0258] In some embodiments, the value of N can also be preset. For example, if N is 3, when n is 1, the pre-trained visual language model is updated according to the loss value to obtain the second visual language model; when n is 2, the obtained third visual language model is used as the visual language model for generating summary information of the image to be processed.

[0259] By pre-setting the number of iterations for training a visual language model, resource waste can be avoided. Too few iterations may prevent the model from fully learning the data's features, resulting in suboptimal performance; while too many iterations not only consume significant computational resources and time but can also lead to overfitting. By appropriately pre-setting the number of iterations, computational resources can be utilized efficiently while ensuring the model's performance. Furthermore, pre-setting the number of iterations clarifies the training timeframe, facilitating the rational planning of the training schedule.

[0260] In summary, by calling the nth visual language model through the embodiments of this application, the first summary information of the image sample is rewritten to obtain multiple second summary information, and positive and negative samples are further classified to construct partial order data. The prediction probability of the partial order data including hallucination information is generated by the nth visual language model. This allows the visual language model to learn which summary information is reasonable and consistent with the image content, and which summary information contains hallucinations, by processing the positive and negative samples in the partial order data during the training process.

[0261] Based on the predicted probability, the loss value is determined, and the nth visual language model is updated to obtain the (n+1)th visual language model. Starting from the pre-trained visual language model, and finally obtaining the Nth visual language model used to generate summary information of the image to be processed, the parameters of the visual language model are continuously updated according to the loss value through the above iterative training. This allows the visual language model to gradually adapt to the task requirements, learn the difference between real information and hallucinatory information, improve the performance of generating summary information, make the generated summary information more likely to be detailed information that matches the actual content of the image, reduce the occurrence of hallucinatory information, and improve the accuracy of the visual language model in generating summary information based on the image.

[0262] The following will describe exemplary applications of the embodiments of this application in educational scenarios.

[0263] In educational settings, students exhibit varying learning abilities, interests, and knowledge bases, making it difficult for traditional educational models to meet individualized needs. The education sector possesses a wealth of image and text resources, such as books, courseware, and online courses. The visual language model training method provided in this application allows the trained model to effectively integrate and utilize these multimodal resources, uncovering their potential information. By combining image (e.g., charts and experimental images in textbooks) and text (e.g., course explanations and problem descriptions) information, the model provides students with image-based summary information, thus offering personalized learning content.

[0264] With the rapid development of large model technology, adding visual perception capabilities to large models is an important new direction, giving rise to the training and application of Large Multimodal Models (LMMs). However, the training of LMMs heavily relies on high-quality image-text pairs. Yet, naturally occurring high-quality image-text pairs are relatively limited. Lowering quality standards and forcibly expanding the scale of image-text pairs will introduce a large amount of low-quality, dirty data, causing the marginal effect of increasing the scale of multimodal data to continuously diminish.

[0265] In related technologies, the image summary information generated by visual language models is either short in text length and low in information value, or has a high degree of image-generated stylistic description (referring to the generation of natural language descriptions that are semantically related to the given image and consistent with a specific language style). This type of data is mainly geared towards text-generated image scenarios. The excessive stylistic descriptions are not suitable for training large multimodal models. In addition, some summary information includes a lot of hallucinatory information (referring to the description of the summary information generated by the model not matching the content of the input image), resulting in poor data quality.

[0266] To address the aforementioned issues, this application's embodiments optimize the process in two ways: first, by reducing illusory information in the summary information; and second, by enhancing the richness of the summary information's content.

[0267] The training method for the visual language model provided in this application can reduce illusory information in the generated summary information and improve the richness of the summary information. Specifically, this application proposes a Continuous Direct Preference Optimization (CDPO) algorithm, which can significantly reduce illusory information in the image summary information, and proposes Knowledge-enriching Supervised Fine-Tuning (KSFT) to enhance the content richness of the summary information, thereby generating large-scale, high-quality image summary information to aid in the training of multimodal large models and further improve the visual perception capabilities of multimodal large models.

[0268] As an example, see Figure 4 , Figure 4 This is a first schematic diagram of an image sample provided in an embodiment of this application. Figure 4 Four image samples are shown, namely image 461, image 462, image 463 and image 464. The following describes the summary information generated for the four images in the embodiments of this application and related technologies.

[0269] For image 461, the abstract information generated by method 1 is "oxalic acid", while the abstract information generated by the embodiment of this application is "The image shows the chemical molecular structure of oxalic acid, with the chemical formula C2H2O4. The oxalic acid molecule is composed of two carbon atoms, two hydrogen atoms, and four oxygen atoms. In the structural formula, the two carbon atoms are connected by a single bond, and each carbon atom is connected to a hydroxyl group (OH) and a carbonyl group (C=O), which is a typical structural feature of oxalic acid."

[0270] For image 462, the summary information generated by method 1 is "long-haired longhorn beetle", while the summary information generated by this embodiment is "The image shows a long-haired longhorn beetle. Its body is mainly black and covered with multiple spots. This insect has particularly long antennae that are serrated and forked, six slender legs with small hooks at the ends. Its exoskeleton is smooth, its body is flat, and its head is relatively small."

[0271] For image 463, the abstract information generated by method 1 is "Original new green connector dx5 printhead for Mutoh VJ1204 VJ1304 VJ1604 Mimaki JV33 JV5 CJV30 printer dx5 printhead solvent new and original dx4 printhead eco-solvent dx4 printhead for Epson Roland VP 540 for Mimaki JV2 JV4 printer." The abstract information generated by method 2 is "Close-up image of a laptop screen with black edges and a white backlit background. The screen has been removed from the laptop and placed on a white background. Two screws are visible on the edge of the screen, and the screen is slightly tilted upwards." The abstract information generated by the embodiments of this application is "The image shows a printhead assembly, which is a component used in a printer. The printhead has a central portion of metal or similar metal, square or rectangular in shape, mounted on a black or dark base. Screws are visible securing the printhead to the base. On the left side of the printhead, there is a green connector or interface, typically used for electrical connections in printer hardware. The overall design indicates that it is a printer component for printing, possibly suitable for Mutoh." VJ1204, VJ1304, VJ1604, Mimaki JV33, JV5, CJV30 or similar printers.

[0272] For image 462, the summary information generated by method 1 is "Changes in head weight sharing. We calculate the proportion of shared features within the structure in every 50 samples as the search progresses." The summary information generated by this embodiment is "The image is a line graph showing how the percentage of shared operations changes with the number of samples. The x-axis represents the number of samples, ranging from 0 to 600, while the y-axis represents the percentage of shared operations, ranging from 0 to 100%. The graph starts at approximately 20% with 50 samples and shows an overall upward trend as the number of samples increases. There are some significant peaks and troughs, with the highest percentage reaching 95% at approximately 600 samples. Data points are marked with dots and connected by lines."

[0273] The training method of the visual language model provided in the embodiments of this application is described below. See also... Figure 5 , Figure 5 This is a flowchart of the training method for the visual language model provided in the embodiments of this application, which will be described in detail below.

[0274] like Figure 5 As shown, firstly, knowledge-rich training data is constructed, and then the constructed training data is used to perform knowledge-enhanced supervised fine-tuning on the visual language model to be trained, resulting in a pre-trained visual language model.

[0275] If the image summary information simply describes the content of the image, such as "a man is taking a picture," the amount of descriptive information is very low, and its help in training multimodal large models is limited. Therefore, embodiments of this application propose knowledge-enhanced supervised fine-tuning, the purpose of which is to inject as much knowledge as possible into the image summary information.

[0276] In some embodiments, a pre-trained visual language model can be obtained by performing the following processes:

[0277] First, obtain the original image-text pair training data; then filter the original image-text pair training data to select some data with more entity-related knowledge, such as animal names, plant names, celebrity names, movie names, etc., and at the same time filter some general image-text data, such as the wukong and CC3M datasets.

[0278] Then, fine-tuned data containing as much knowledge detail as possible is constructed using a large model. The input is an image and the original text (corresponding to the first summary information above), and the output is the summary information after re-summarization (corresponding to the third summary information obtained by rewriting the first summary information above), such as "The long-haired longhorn beetle has a black body covered with bright yellow spots. This insect has particularly long antennae, which are...".

[0279] Finally, the constructed training data can be further refined through manual annotation to enrich the description of the summary information. The aim is to make the target's re-summary information (i.e., the third summary information after manual annotation) as rich as possible, containing more descriptive details and less illusory information. The training data obtained in the above way is used to train the visual language model to be trained, so that the pre-trained visual language model can generate rich re-summary information based on the input image and the original summary information of the image.

[0280] As an example, the pre-trained visual language model can be the Qwen2-Vision-Language (Qwen2-VL).

[0281] After obtaining the pre-trained visual language model, the pre-trained visual language model is called to perform partial order data sampling and partial order data equalization to obtain the first partial order data after equalization. The pre-trained visual language model is then trained using the first partial order data after equalization to obtain the second visual language model.

[0282] In some embodiments, a pre-trained visual language model can be invoked to sample re-summarized data. A higher temperature coefficient is used to sample the same image-text pair 8 times to obtain re-summarized data corresponding to each sampling (corresponding to the second summary information mentioned above). Then, a hallucination judgment model is used to perform hallucination discrimination on the re-summarized data. For each re-summarized data segment, the number of summary information including hallucination information is determined. The multiple re-summarized data are sorted in ascending order according to the number of summary information including hallucination information. At least one re-summarized data starting from the head in the sorting result is taken as a positive sample, and at least one re-summarized data starting from the tail in the sorting result is taken as a negative sample. The positive and negative samples are constructed as partially ordered data.

[0283] As an example, see Figure 6 , Figure 6 This is a second schematic diagram of an image sample provided in an embodiment of this application. Figure 6 Two image samples are shown, namely image 601 and image 602.

[0284] For image 601, the description text for the positive sample is: "The image shows a traditional Chinese-style archway with a plaque hanging on it, above which are written Chinese characters. There is a stone lion statue on each side of the archway. The design of the archway has a strong classical Chinese style. In front of the archway is a wide road with sparse trees and buildings on both sides. The horizon in the distance meets the sky."

[0285] The negative sample description is: "This is a photo of a traditional Chinese archway. The archway consists of five layers. The top layer is the eaves with yellow Chinese characters. The second layer has a gate with a blue plaque. The third and fourth layers each have an arch, and the fifth layer is a crossbeam with inscriptions. There is a stone lion on each side of the archway. There is a road around the archway with palm trees and streetlights. In the background is a large, bare forest."

[0286] For image 602, the description text for the positive sample is: "In the image, a mother wearing a coat is kneeling in a pile of rubble, next to the debris of a collapsed building, including bricks, tiles and steel bars. Her right hand is holding onto a concrete wall, and her eyes are fixed on the wall. More scattered building materials can be seen in the background."

[0287] The negative sample description is: "In the picture, a woman wearing a coat sits amidst rubble and ruins, crying with her hand over her mouth, looking at a collapsed wall. Scattered bricks, rubble, and other building debris indicate that this is a post-earthquake scene."

[0288] In some embodiments, class equalization is performed on the partially ordered data. First, partially ordered data with significant length differences between positive and negative samples are filtered out. The goal is to keep the lengths of positive and negative samples similar, preventing the model from tending to output short, repetitive summaries that could create illusions. Furthermore, partially ordered data with significant feature differences between positive and negative samples are filtered out; that is, samples that are difficult to distinguish are removed, while samples with obvious feature differences are retained. Sample data obtained in this way allows the trained visual language model to learn the key features of different categories of samples more quickly, thereby improving the training efficiency and accuracy of the visual language model.

[0289] In some embodiments, a pre-trained visual language model is trained using equalized first partially ordered data to obtain a second visual language model. Then, the second visual language model can be called to perform partially ordered data sampling and partially ordered data equalization to obtain equalized second partially ordered data. The equalized second partially ordered data is used to train the second visual language model to obtain a third visual language model. The third visual language model is then called to perform inference on large-scale re-summary data.

[0290] As an example, Figure 5 The input image-text pair shown is image 501 and its summary information 502. Summary information 502 is "long-haired longhorn beetle." After rewriting using a third-party visual language model, the resulting summary information 503 for image 501 is: "The image shows a long-haired longhorn beetle. Its body is mainly black and covered with multiple spots. This insect has particularly long, serrated antennae, six slender legs with small hooks at the ends, a smooth exoskeleton, a flat body, and a relatively small head."

[0291] The following describes the experimental comparison results between the summary information generated by the above technical solutions of this application and the summary information in related technologies in terms of the length, detail, number of hallucinations and hallucination rate of the descriptive text. A total of 1,000 data samples were extracted for the experiment, and the experimental results are shown in Table 1 below.

[0292] Table 1

[0293]

[0294] As shown in Table 1, the summary information (Ours) generated by the embodiments of this application differs from the summary information of the synthetic caption datasets Capfusion-120M, Recap-DataComp-1B, and Hunyuan-Recap100M in terms of text length, detail, number of illusions, and illusion rate. Here, length represents the average length of the caption, detail represents the average number of visual details appearing in each caption, no illusion represents the proportion of non-illusion data in the overall data, and illusion rate represents the proportion of illusions among all details. It is evident that the summary information generated by the embodiments of this application has a significant advantage over the summary information in the original text-image pair data.

[0295] To verify the impact of different caption datasets on the training of visual language models, this application provides experimental results in four aspects: scientific question answering, benchmark question answering, optical character recognition question answering, and illusion tasks. The experimental results are shown in Table 2 (Tables 2.1-2.2).

[0296] Table 2.1

[0297]

[0298] Table 2.2

[0299]

[0300] Table 2 shows the experimental results of training the visual language model on three different caption datasets (Alt-text, Recap-DataComp-1B, and Hunyuan-Recap100M) in four aspects: scientific question answering, benchmark question answering, optical character recognition question answering, and hallucination tasks. A total of 15 visual language tasks are included: for scientific question answering, mathematical visual test, multimodal matching unit verification, AI zero-shot detection test, and scientific question answering test; for benchmark question answering, multimodal benchmark English development, multimodal star rating test, real-world question answering test, and multimodal veterinary test; for optical character recognition question answering, optical character recognition benchmark test, document visual question answering verification, text visual question answering verification, infographic visual question answering verification, and chart question answering test; and for hallucination tasks, hallucination benchmark test and no-hallucination rate. This application provides the accuracy of three caption datasets for the aforementioned 15 visual language tasks under a fixed data setting (20 MB). Based on the experimental data, the average accuracy of the three caption datasets (Alt-text, Recap-DataComp-1B, and Hunyuan-Recap100M) for the 15 visual language tasks is determined to be 53.12, 52.07, and 59.35, respectively. It is evident that among the three datasets, Hunyuan-Recap100M's accuracy is only 0.45 lower than Recap-DataComp-1B in the scientific question answering test task; its accuracy in the other 14 tasks is superior to the other two datasets. Therefore, visual language models can be trained based on Hunyuan-Recap100M.

[0301] This application proposes a continuous direct preference optimization algorithm that can significantly reduce illusory information in image summary information. It also proposes knowledge-enhanced supervised fine-tuning to strengthen the content richness of the summary information, thereby generating large-scale, high-quality image summary information to better train the visual language model and further improve the visual perception ability of the visual language model. This enables the trained visual language model to generate summary information with rich descriptions and less illusory information for images (such as charts in textbooks, experimental pictures, etc.), providing students with personalized learning content.

[0302] The following description continues to illustrate the exemplary structure of the visual language model training device 233 provided in the embodiments of this application as a software module. In some embodiments, such as Figure 2 As shown, the software modules in the training device 233 for the visual language model stored in the memory 230 may include:

[0303] Processing module 2331 is used to call the nth visual language model to rewrite the first summary information of the image sample to obtain multiple second summary information; where n is an integer variable that increments from 1, and the value of n satisfies 1≤n≤N, and N is a positive integer greater than or equal to 2. When n is 1, the nth visual language model is a pre-trained visual language model, and when 2≤n, the nth visual language model is trained on the basis of the (n-1)th visual language model.

[0304] The classification module 2332 is used to classify multiple second summary information into positive samples and negative samples, wherein the number of hallucination information included in the positive samples is less than the number of hallucination information included in the negative samples.

[0305] The construction module 2333 is used to construct multiple partial order data based on multiple second summary information, wherein a partial order data includes a positive sample and a negative sample.

[0306] The prediction module 2334 is used to generate prediction probabilities of multiple partially ordered data corresponding to hallucination information through the nth visual language model.

[0307] The determination module 2335 is used to determine the loss value of the nth visual language model based on the prediction probabilities corresponding to multiple partially ordered data.

[0308] The update module 2336 is used to update the nth visual language model based on the loss value to obtain the (n+1)th visual language model. When the iteration reaches the value of N, the obtained (N+1)th visual language model is used as the visual language model for generating summary information of the image to be processed.

[0309] In some embodiments, the processing module 2331 is further configured to call the nth visual language model to encode the image sample to obtain image features, and to encode the first summary information of the image sample to obtain text features; to fuse the image features and text features to obtain fused features; and to decode the fused features to obtain multiple second summary information.

[0310] In some embodiments, the processing module 2331 is further configured to generate a plurality of candidate temperature coefficients, wherein the candidate temperature coefficients are used to control the randomness of the nth visual language model in decoding the fusion features to obtain candidate summary information; in response to the selection operation for the plurality of candidate temperature coefficients, at least one selected candidate temperature coefficient is used as a target temperature coefficient; and the fusion features are decoded based on at least one target temperature coefficient to obtain a plurality of summary information corresponding to each target temperature coefficient, which is used as second summary information.

[0311] In some embodiments, the processing module 2331 is further configured to decode the fused features based on at least one target temperature coefficient to obtain a probability distribution corresponding to each target temperature coefficient, wherein the probability distribution represents the probability of multiple candidate words being output respectively; determine the target word corresponding to each target temperature coefficient according to the probability distribution corresponding to each target temperature coefficient; and combine the target words corresponding to each target temperature coefficient to obtain multiple summary information corresponding to each target temperature coefficient.

[0312] In some embodiments, the classification module 2332 is further configured to divide each second summary information into multiple summary segments; perform hallucination recognition on each summary segment to obtain the number of summary segments in each second summary information that include hallucination information, wherein the hallucination information is information unrelated to the content of the image sample; sort the second summary information in ascending order according to the number to obtain a sorting result; take at least one second summary information starting from the head in the sorting result as a positive sample, and take at least one second summary information starting from the tail in the sorting result as a negative sample.

[0313] In some embodiments, the prediction module 2334 is further configured to perform the following processing for each partially ordered data: determine the difference in length between the positive and negative samples in the partially ordered data; if the difference is greater than a difference threshold, delete the partially ordered data from the multiple partially ordered data.

[0314] In some embodiments, the prediction module 2334 is further configured to generate a first cue word, wherein the first cue word is used to indicate a first prediction probability for each positive sample and a second prediction probability for each negative sample in the partially ordered data generated based on the image samples, wherein the first prediction probability represents the probability that the positive sample includes hallucination information and the second prediction probability represents the probability that the negative sample includes hallucination information; and to call the nth visual language model to generate the first prediction probability and the second prediction probability based on the first cue word.

[0315] In some embodiments, the determining module 2335 is further configured to obtain a first label pre-set for positive samples in the partially ordered data and a second label pre-set for negative samples in the partially ordered data; determine a first loss for positive samples in each partially ordered data based on a first prediction probability and a first label; determine a second loss for negative samples in each partially ordered data based on a second prediction probability and a second label; determine a first sum of the first loss and the second loss for each partially ordered data; and take the average of the first sums corresponding to multiple partially ordered data as the loss value of the nth visual language model.

[0316] In some embodiments, the prediction module 2334 is further configured to generate a second cue word, wherein the second cue word is used to indicate a third prediction probability of generating a positive sample in the partially ordered data based on the negative sample in the image sample and the partially ordered data, and a fourth prediction probability of generating a negative sample in the partially ordered data based on the positive sample in the image sample and the partially ordered data, wherein the third prediction probability represents the probability that the positive sample includes hallucination information, and the fourth prediction probability represents the probability that the negative sample includes hallucination information; and to call the nth visual language model to generate the third prediction probability and the fourth prediction probability based on the second cue word.

[0317] In some embodiments, the determining module 2335 is further configured to obtain a first label pre-set for positive samples in the partially ordered data and a second label pre-set for negative samples in the partially ordered data; determine a third loss for positive samples in each partially ordered data based on a third prediction probability and the first label; determine a fourth loss for negative samples in each partially ordered data based on a fourth prediction probability and the second label; determine a second sum of the third loss and the fourth loss for each partially ordered data; and take the average of the second sums corresponding to multiple partially ordered data as the loss value of the nth visual language model.

[0318] In some embodiments, the processing module 2331 is further configured to extract the nth image sample from the image sample dataset, wherein the nth image sample is different from the (n-1)th image sample; and call the nth visual language model to rewrite the first summary information of the nth image sample to obtain multiple second summary information.

[0319] In some embodiments, the processing module 2331 is further configured to construct a third prompt word for the first summary information of the image sample, wherein the third prompt word is used to indicate the generation of third summary information containing more descriptive details based on the first summary information; rewrite the first summary information into third summary information based on the third prompt word using the visual language model to be trained; construct an image-text pair between the image sample and the third summary information; and train the visual language model to be trained using the image-text pair to obtain a pre-trained visual language model.

[0320] In some embodiments, the processing module 2331 is further configured to extract screening indicators for the first summary information, wherein the screening indicators include at least one of the following: the number of entity words included in the first summary information, the type of entity words, the information density included in the first summary information, and the matching degree between the first summary information and the image sample; in response to the screening indicator being less than the screening indicator threshold, the first summary information corresponding to the screening indicator is screened out to obtain the screened first summary information.

[0321] In some embodiments, the processing module 2331 is further configured to call the visual language model to be trained to encode the image samples in the image-text pair to obtain image features; decode the image features to obtain predicted summary information; determine the loss value based on the difference between the third summary information in the image-text pair and the predicted summary information; and update the parameters of the visual language model to be trained based on the loss value to obtain a pre-trained visual language model.

[0322] In some embodiments, the value of N is dynamically determined after n is incremented and the (n+1)th visual language model is trained; the update module 2336 is also used to detect the performance of the (n+1)th visual language model and obtain the performance parameter value; in response to the performance parameter value being greater than the performance parameter threshold, the current value of n is used as N.

[0323] In some embodiments, the update module 2336 is further configured to acquire training data, wherein the training data includes multiple target images and target summary information for each target image; in some embodiments, the update module 2336 is further configured to call the (n+1)th visual language model to generate predicted summary information corresponding to the multiple target images respectively; in some embodiments, the update module 2336 is further configured to determine the ratio of the number of predicted summary information matching the target summary information to the number of multiple target images, as a performance parameter value.

[0324] This application provides a computer program product comprising a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the visual language model training method described above in this application.

[0325] This application provides a computer-readable storage medium storing computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the processor will execute the training method for the visual language model provided in this application. For example, ... Figure 3A The training method for the visual language model is shown.

[0326] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0327] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0328] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0329] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0330] In summary, by calling the nth visual language model through the embodiments of this application, the first summary information of the image sample is rewritten to obtain multiple second summary information, and positive and negative samples are further classified to construct partial order data. The prediction probability of the partial order data including hallucination information is generated by the nth visual language model. This allows the visual language model to learn which summary information is reasonable and consistent with the image content, and which summary information contains hallucinations, by processing the positive and negative samples in the partial order data during the training process.

[0331] Based on the predicted probability, the loss value is determined, and the nth visual language model is updated to obtain the (n+1)th visual language model. Starting from the pre-trained visual language model, and finally obtaining the Nth visual language model used to generate summary information of the image to be processed, the parameters of the visual language model are continuously updated according to the loss value through the above iterative training. This allows the visual language model to gradually adapt to the task requirements, learn the difference between real information and hallucinatory information, improve the performance of generating summary information, make the generated summary information more likely to be detailed information that matches the actual content of the image, reduce the occurrence of hallucinatory information, and improve the accuracy of the visual language model in generating summary information based on the image.

[0332] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A training method for a visual language model, characterized in that, The method includes: The nth visual language model is invoked to rewrite the first summary information of the image sample, resulting in multiple second summary information. Here, n is an integer variable that increments from 1, and the value of n satisfies 1≤n≤N. N is a positive integer greater than or equal to 2. When n is 1, the nth visual language model is a pre-trained visual language model. When 2≤n, the nth visual language model is trained based on the (n-1)th visual language model. The plurality of second summary information is classified into positive samples and negative samples, wherein the number of hallucination information included in the positive samples is less than the number of hallucination information included in the negative samples; Multiple partial order data are constructed based on the multiple second summary information, wherein each of the partial order data includes a positive sample and a negative sample; The nth visual language model generates the predicted probabilities of the multiple partially ordered data corresponding to the hallucination information. The loss value of the nth visual language model is determined based on the predicted probabilities corresponding to the multiple partially ordered data. The nth visual language model is updated based on the loss value to obtain the (n+1)th visual language model; When the iteration reaches the value of N, the obtained (N+1)th visual language model is used as the visual language model for generating summary information of the image to be processed.

2. The method according to claim 1, characterized in that, The process of calling the nth visual language model to rewrite the first summary information of the image sample yields multiple second summary information items, including: The nth visual language model is invoked to encode the image sample to obtain image features, and the first summary information of the image sample is encoded to obtain text features; The image features and the text features are fused to obtain the fused features; The fusion features are decoded to obtain multiple second summary information.

3. The method according to claim 2, characterized in that, Decoding the fused features yields multiple second summary pieces of information, including: Multiple candidate temperature coefficients are generated, wherein the candidate temperature coefficients are used to control the randomness of the nth visual language model in decoding the fusion features to obtain candidate summary information; In response to the selection operation for the plurality of candidate temperature coefficients, at least one of the selected candidate temperature coefficients is taken as the target temperature coefficient. The fusion features are decoded based on at least one of the target temperature coefficients to obtain multiple summary information corresponding to each target temperature coefficient, which serves as the second summary information.

4. The method according to claim 3, characterized in that, The fusion features are decoded based on at least one of the target temperature coefficients to obtain multiple summary information corresponding to each target temperature coefficient, including: The fused features are decoded based on at least one of the target temperature coefficients to obtain a probability distribution corresponding to each target temperature coefficient, wherein the probability distribution represents the probability of multiple candidate words being output respectively; Based on the probability distribution corresponding to each target temperature coefficient, determine the target word corresponding to each target temperature coefficient; The target words corresponding to each target temperature coefficient are combined to obtain multiple summary information corresponding to each target temperature coefficient.

5. The method according to claim 1, characterized in that, The process of classifying the plurality of second summary information into positive and negative samples includes: Each second summary information is divided into multiple summary segments; Hallucination recognition is performed on each of the above-mentioned summary segments to obtain the number of summary segments that include hallucination information in each of the second summary information, wherein the hallucination information is information unrelated to the content of the image sample; The second summary information is sorted in ascending order according to the stated quantity to obtain the sorting result; At least one of the second summary information from the head in the sorting result is taken as a positive sample, and at least one of the second summary information from the tail in the sorting result is taken as a negative sample.

6. The method according to claim 1, characterized in that, Before generating the predicted probabilities of the hallucination information corresponding to the plurality of partially ordered data through the nth visual language model, the method further includes: Perform the following processing for each of the partially ordered data: Determine the difference in length between positive and negative samples in the partially ordered data; If the difference is greater than the difference threshold, the partially ordered data will be deleted from the plurality of partially ordered data.

7. The method according to claim 1, characterized in that, The step of generating the predicted probabilities of the multiple partially ordered data corresponding to the hallucination information through the nth visual language model includes: Generate a first cue word, wherein the first cue word is used to indicate the generation of a first predicted probability for each positive sample in the partial order data based on the image samples, and a second predicted probability for each negative sample, wherein the first predicted probability characterizes the probability that the positive sample includes the hallucination information, and the second predicted probability characterizes the probability that the negative sample includes the hallucination information; The nth visual language model is invoked to generate the first prediction probability and the second prediction probability based on the first prompt word.

8. The method according to claim 7, characterized in that, Determining the loss value of the nth visual language model based on the predicted probabilities corresponding to the multiple partially ordered data includes: Obtain a first label pre-set for the positive samples in the partially ordered data, and a second label pre-set for the negative samples in the partially ordered data; Based on the first predicted probability and the first label, determine the first loss of the positive sample in each of the partially ordered data; Based on the second predicted probability and the second label, a second loss is determined for each negative sample in the partially ordered data; Determine a first sum of the first loss and the second loss for each of the partially ordered data; The average of the first sums corresponding to the multiple partially ordered data is used as the loss value of the nth visual language model.

9. The method according to claim 1, characterized in that, The step of generating the predicted probabilities of the multiple partially ordered data corresponding to the hallucination information through the nth visual language model includes: A second prompt word is generated, wherein the second prompt word is used to indicate a third predicted probability of generating the positive sample in the partially ordered data based on the image sample and the negative sample in the partially ordered data, and a fourth predicted probability of generating the negative sample in the partially ordered data based on the image sample and the positive sample in the partially ordered data, wherein the third predicted probability represents the probability that the positive sample includes the hallucination information, and the fourth predicted probability represents the probability that the negative sample includes the hallucination information is high; The nth visual language model is invoked to generate the third and fourth prediction probabilities based on the second prompt word.

10. The method according to claim 9, characterized in that, Determining the loss value of the nth visual language model based on the predicted probabilities corresponding to the multiple partially ordered data includes: Obtain a pre-set first label for the positive samples in the partially ordered data, and a label for the... Based on the third predicted probability and the first label, a third loss is determined for each positive sample in the partially ordered data. Based on the fourth predicted probability and the second label, a fourth loss is determined for each negative sample in the partially ordered data. Determine a second sum of the third loss and the fourth loss for each of the partially ordered data; The average of the second summation corresponding to multiple partially ordered data is used as the loss value of the nth visual language model.

11. The method according to claim 1, characterized in that, The process of calling the nth visual language model to rewrite the first summary information of the image sample yields multiple second summary information items, including: Extract the nth image sample from the image sample dataset, wherein the nth image sample is different from the (n-1)th image sample; The first summary information of the nth image sample is rewritten by calling the nth visual language model to obtain multiple second summary information.

12. The method according to claim 1, characterized in that, The pre-trained visual language model is trained in the following way: For the first summary information of the image sample, a third prompt word is constructed, wherein the third prompt word is used to indicate the generation of third summary information containing more descriptive details based on the first summary information; Based on the third prompt word, the first summary information is rewritten into the third summary information using the visual language model to be trained; Construct image-text pairs using the image samples and the third summary information; The image-text pair is used to train the visual language model to be trained, resulting in the pre-trained visual language model.

13. The method according to claim 12, characterized in that, Before constructing the third cue word based on the first summary information of the image sample, the method further includes: Extract the filtering indicators of the first summary information, wherein the filtering indicators include at least one of the following: the number of entity words included in the first summary information, the type of the entity words, the information density included in the first summary information, and the matching degree between the first summary information and the image sample; In response to the filtering index being less than the filtering index threshold, the first summary information corresponding to the filtering index is filtered out to obtain the filtered first summary information.

14. The method according to claim 12, characterized in that, The step of training the visual language model to be trained using the image-text pair to obtain the pre-trained visual language model includes: The visual language model to be trained is invoked to encode the image samples in the image-text pair to obtain image features; The image features are decoded to obtain the predicted summary information; The loss value is determined based on the difference between the third summary information and the predicted summary information in the image-text pair; Based on the loss value, the parameters of the visual language model to be trained are updated to obtain the pre-trained visual language model.

15. The method according to claim 1, characterized in that, The value of N is dynamically determined after n is incremented each time and the (n+1)th visual language model is trained. After updating the nth visual language model based on the loss value to obtain the (n+1)th visual language model, the method further includes: The performance of the (n+1)th visual language model is tested, and the performance parameter values ​​are obtained; In response to the performance parameter value being greater than the performance parameter threshold, the current value of n is taken as N.

16. The method according to claim 15, characterized in that, The performance of the (n+1)th visual language model is tested to obtain performance parameter values, including: Acquire training data, wherein the training data includes multiple target images and target summary information for each target image; The (n+1)th visual language model is invoked to generate predicted summary information corresponding to the multiple target images respectively; The ratio of the number of predicted summary information matching the target summary information to the number of target images is determined and used as a performance parameter value.

17. A training device for a visual language model, characterized in that, The device includes: The processing module is used to call the nth visual language model to rewrite the first summary information of the image sample to obtain multiple second summary information; where n is an integer variable that increments from 1, and the value of n satisfies 1≤n≤N, and N is a positive integer greater than or equal to 2. When n is 1, the nth visual language model is a pre-trained visual language model, and when 2≤n, the nth visual language model is trained based on the (n-1)th visual language model. A classification module is used to classify the plurality of second summary information into positive samples and negative samples, wherein the number of hallucination information included in the positive samples is less than the number of hallucination information included in the negative samples; A construction module is used to construct multiple partial-order data based on the multiple second summary information, wherein one of the partial-order data includes one positive sample and one negative sample; The prediction module is used to generate the predicted probabilities of the multiple partially ordered data corresponding to the hallucination information through the nth visual language model; The determination module is used to determine the loss value of the nth visual language model based on the prediction probabilities corresponding to the multiple partially ordered data respectively. The update module is used to update the nth visual language model according to the loss value to obtain the (n+1)th visual language model; when the iteration reaches the value of N, the obtained (N+1)th visual language model is used as the visual language model for generating summary information of the image to be processed.

18. An electronic device, characterized in that, The electronic device includes: Memory is used to store executable instructions or computer programs. A processor, when executing computer-executable instructions or computer programs stored in the memory, implements the training method for the visual language model according to any one of claims 1 to 16.

19. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, they implement the training method of the visual language model according to any one of claims 1 to 16.

20. A computer program product comprising computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, they implement the training method of the visual language model according to any one of claims 1 to 16.

Images