Model training method and related device, image processing method and related device

Abstract

Embodiments of the present application disclose a model training method and related device, an image processing method and related device. Embodiments of the present application fuse the knowledge of multiple visual models, can significantly improve the visual perception ability and language generation ability of the visual language model, so that the visual language model has better model performance, and further migrates the integrated expert knowledge to a lightweight student visual model through knowledge distillation, which can reduce the demand for large-scale training data and computing resources, thereby improving the inference efficiency; the student visual model obtained by final training and the large language model constitute a visual language model for inference, and the lighter student visual model can efficiently run on a terminal device with limited resources, thereby improving the actual deployment and application ability of the visual language model.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, specifically to a model training method and related apparatus, and an image processing method and related apparatus. Background Technology

[0002] Visual language models (VLMs) are multimodal models that combine visual and language processing capabilities. Also known as visual-language models, they are used to process the relationship between images and text. They are increasingly being used in areas such as content moderation, including tasks such as image moderation, video moderation, and text generation.

[0003] In current visual language models, visual perception, as the core of the task, typically relies on large-scale training data and complex visual encoders to achieve good performance. However, these visual encoders usually require significant computational resources, have slow inference speeds, and are difficult to deploy efficiently on resource-constrained devices. This makes improving inference efficiency while ensuring model performance a pressing issue in content moderation. Summary of the Invention

[0004] To address the aforementioned technical problems, embodiments of this application provide a model training method, a model training apparatus, an image processing method, an image processing apparatus, an electronic device, a computer-readable storage medium, and a computer program product. Embodiments of this application can provide visual language models with strong visual perception capabilities while significantly reducing training data and computing resources, thereby achieving efficient and accurate content moderation. This makes them more suitable for deployment on resource-constrained terminal devices, facilitating the better application of visual language models in real-world business environments and improving the efficiency and accuracy of content moderation.

[0005] One aspect of this application provides a model training method, comprising: extracting visual features from a first training image using a base visual model and an expert visual model respectively, and fusing the visual features extracted by each visual model to obtain a first fused visual feature; performing initial training on a large language model and the base visual model based on the first fused visual feature and a first training text corresponding to the first training image to obtain an initially trained large language model and an initially trained base visual model; extracting visual features from a second training image using the initially trained base visual model and the expert visual model, and fusing the visual features extracted by each visual model to obtain a second fused visual feature; extracting visual features from the second training image based on a student visual model, and training the student visual model and the initially trained large language model according to the second fused visual feature, the visual features extracted by the student visual model, and the second training text corresponding to the second training image to generate a visual language model based on the trained student visual model and the large language model. The visual language model is formed by combining the trained student visual model and the large language model.

[0006] In another aspect of this application, a model training apparatus is provided, comprising: a first fusion module configured to extract visual features of a first training image using a base visual model and an expert visual model respectively, and to fuse the visual features extracted by each visual model to obtain a first fused visual feature; a first training module configured to perform initial training on a large language model and the base visual model based on the first fused visual feature and training text corresponding to the first training image to obtain an initially trained large language model and an initially trained base visual model; a second fusion module configured to extract visual features of a second training image using the initially trained base visual model and the expert visual model, and to fuse the visual features extracted by each visual model to obtain a second fused visual feature; and a second training module configured to extract visual features of the second training image based on a student visual model, and to train the student visual model and the initially trained large language model according to the second fused visual feature, the visual features extracted by the student visual model, and the second training text corresponding to the second training image to generate a visual language model based on the trained student visual model and the large language model.

[0007] In another exemplary embodiment, the first fusion module includes: a determining unit configured to determine the visual fusion contribution of each expert visual model to the first training image based on a preset mapping vector and visual features extracted by the basic visual model; wherein the dimension of the mapping vector is represented as C×K, C represents the length of the feature vector, and K represents the number of expert visual models; and a fusion unit configured to fuse the visual features extracted by each visual model according to the visual fusion contribution to obtain the first fused visual feature.

[0008] In another exemplary embodiment, the determining unit is further configured to: calculate the product and norm between the visual features extracted by the base visual model and the mapping vector; and calculate the quotient of the product and the norm to obtain the visual fusion contribution of each expert visual model to the first training image.

[0009] In another exemplary embodiment, the fusion unit is further configured to: calculate the product of the visual fusion contribution and the visual features extracted by the corresponding expert visual model; sum the products corresponding to each expert visual model and perform feature space mapping to obtain expert fusion features adapted to the input feature space corresponding to the large language model; and add the expert fusion features to the visual features extracted by the basic visual model to obtain the first fused visual feature.

[0010] In another exemplary embodiment, the visual features extracted by each visual model include visual features corresponding to at least two image patches respectively; the apparatus further includes: a downsampling module configured to downsample the visual features of the image patches extracted by the expert visual model, so as to perform the fusion processing based on the visual features of the downsampled image patches; wherein the downsampled visual features maintain the same dimension as the visual features extracted by the basic visual model.

[0011] In another exemplary embodiment, the first training module is further configured to: input the first fused visual features and the text features corresponding to the prompt text into the large language model to obtain the predicted text output by the large language model; determine an initial training loss value based on the predicted text output by the large language model and the training text corresponding to the first training image, and optimize the parameters of the basic visual model and the large language model based on the initial training loss value to obtain the initially trained large language model and the initially trained basic visual model.

[0012] In another exemplary embodiment, the first training module is further configured to optimize the mapping vector used to compute the first fused visual features during the initial training of the large language model and the basic visual model.

[0013] In another exemplary embodiment, the second training module is further configured to: input the visual features extracted by the student visual model and the text features corresponding to the prompt text into the initially trained large language model to obtain the predicted text output by the initially trained large language model; determine a target training loss value based on the predicted text output by the initially trained large language model, the second training text, the second fused visual features, and the visual features extracted by the student visual model; and optimize the parameters of the student visual model and the initially trained large language model based on the target training loss value to generate the visual language model based on the trained student visual model and large language model.

[0014] In another exemplary embodiment, the second training module is further configured to: determine a first partial loss based on the predicted text output by the large language model after initial training and the second training text, and determine a second partial loss based on the second fused visual features and the visual features extracted by the student visual model; and determine the target training loss value based on the first partial loss and the second partial loss.

[0015] In another aspect of this application, an image processing method is provided, comprising: acquiring an image to be processed; inputting the image to be processed into a visual language model trained based on the method mentioned in the foregoing embodiments to obtain predicted text output by the visual language model; wherein the visual language model consists of a student visual model and a large language model, the student visual model being used to extract visual features of the image to be processed, and the large language model being used to output the predicted text based on the visual features of the image to be processed.

[0016] In another aspect of this application, an image processing apparatus is provided, comprising: an image acquisition module configured to acquire an image to be processed; and a model processing module configured to input the image to be processed into a visual language model trained based on the method mentioned in the foregoing embodiments, so as to obtain predicted text output by the visual language model; wherein the visual language model consists of a student visual model and a large language model, the student visual model being used to extract visual features of the image to be processed, and the large language model being used to output the predicted text based on the visual features of the image to be processed.

[0017] Another aspect of this application provides an electronic device, including: one or more processors; and a memory for storing one or more computer programs, which, when executed by the one or more processors, cause the electronic device to implement the model training method or image processing method as described above.

[0018] Another aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor of an electronic device, causes the electronic device to perform the model training method or image processing method as described above.

[0019] Another aspect of this application provides a computer program product, including a computer program that, when executed by a processor of an electronic device, implements the model training method or image processing method described above.

[0020] The technical solution provided in the embodiments of this application fuses the visual features extracted from the first training image by the basic visual model and the expert visual model respectively. The obtained first visual fusion feature and the training text corresponding to the first training image are used to initially train the large language model and the basic visual model to learn to integrate the visual knowledge of different visual models into expert knowledge, and enable the large language model to understand visual information. The second fused visual feature, which is the unified integrated expert knowledge, is obtained by using the initially trained basic visual model and the expert visual model. Based on the second fused visual feature, the visual features of the second training image extracted by the student visual model and the second training text, the student visual model and the initially trained large language model are trained to use the trained student visual model and the large language model as the final visual language model. This not only enables the student visual model to learn rich visual perception capabilities by imitating the integrated expert knowledge, but also enhances the adaptability of the student visual model and the large language model.

[0021] Therefore, the embodiments of this application integrate knowledge from multiple visual models, which can significantly improve the visual perception and language generation capabilities of the visual language model, resulting in better model performance. Furthermore, by using knowledge distillation, the integrated expert knowledge is transferred to a lightweight student visual model, which reduces the need for large-scale training data and computing resources, thereby improving inference efficiency. The final trained student visual model and the large language model together constitute a visual language model for inference. The lighter student visual model can run efficiently on resource-constrained terminal devices, thereby improving the practical deployment and application capabilities of the visual language model.

[0022] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. Attached Figure Description

[0023] Figure 1 This illustrates the inference result output by an exemplary visual language model for an image to be inferred;

[0024] Figure 2 This is a schematic diagram of an exemplary implementation environment involved in this application;

[0025] Figure 3 An exemplary visual language model pre-training architecture is illustrated.

[0026] Figure 4 This is a flowchart illustrating a model training method in an exemplary embodiment of this application;

[0027] Figure 5 This illustrates the pre-training process included in the model training scheme disclosed in this embodiment;

[0028] Figure 6 Based on Figure 4 The flowchart of the model training method further proposed in the illustrated embodiment is shown.

[0029] Figure 7 This illustrates an exemplary process for obtaining a training dataset;

[0030] Figure 8 This is a flowchart illustrating an exemplary image processing method;

[0031] Figure 9 The diagram illustrates the model training architecture and inference schematics used in the embodiments of this application;

[0032] Figure 10 This is a block diagram illustrating a model training apparatus according to an exemplary embodiment of this application;

[0033] Figure 11 This is a block diagram illustrating an image processing apparatus according to an exemplary embodiment of this application;

[0034] Figure 12 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown. Detailed Implementation

[0035] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0036] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0037] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0038] In this application, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0039] The terms "first," "second," "third," and "fourth," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. The terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0040] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0041] First, it's important to clarify that large models, also known as pre-training models (PTMs) or foundational models, refer to deep neural networks (DNNs) with a large number of parameters. These DNNs are trained on massive amounts of unlabeled data, leveraging the function approximation capabilities of large-parameter DNNs to extract common features from the data. Through fine-tuning, efficient parameter fine-tuning (PEFT), prompt-tuning, and other techniques, they are suitable for downstream tasks. Therefore, large models can achieve ideal results in scenarios with few or zero samples.

[0042] Large models can be categorized based on the data modalities they process, including language models (such as ELMO, BERT, GPT), visual models (such as Swin-Transformer, ViT, V-MOE), speech models (such as VALL-E), and multimodal models (such as ViBERT, CLIP, Flamingo, Gato). Multimodal models, in particular, are models that build feature representations for two or more data modalities. Large models are crucial tools for outputting AI-generated content (AIGC) and can also serve as a universal interface connecting multiple task-specific models.

[0043] The Vision Language Models (VLMs) mentioned in this application are multimodal models that combine image and natural language processing techniques; they can also be called visual-language models. VLMs aim to understand and interpret the relationship between images and text, and generate natural language descriptions based on images. This type of model uses deep learning technology to combine image and text information, constructing a bridge capable of understanding and generating the relationship between images and text.

[0044] For example, in a typical content moderation scenario, a visual language model is used to analyze the provided image and determine whether it contains any inappropriate content, and to provide a clear assessment indicating whether the image is safe or contains elements that are considered unsuitable for all viewers. Figure 1 This illustrates an exemplary visual language model's output of inference results for an image to be inferred, such as... Figure 1 As shown, the visual language model's inference result for this exemplary image output is: "The image you provided appears to be a picturesque scene, with a dock extending into a body of water and mountains and trees in the background. There are no visible people or objects in the image that would suggest inappropriate content. The image is safe and does not contain any elements that would be considered unsuitable for all viewers."

[0045] It should also be noted that visual language models can be trained to adapt to different application scenarios as multimodal models. Therefore, the visual language model shown in Figure 1 is only an example of an application scenario and does not represent a limitation on visual language models.

[0046] Please see Figure 2 , Figure 2 This is a schematic diagram of an exemplary implementation environment related to this application. The implementation environment includes a terminal 210 and a server 220, which communicate with each other via wired or wireless means.

[0047] Terminal 210 can be a smartphone, tablet, laptop, computer, smart home appliance, smart terminal, or other device, without limitation. Server 120 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services, without limitation.

[0048] like Figure 2 As shown, terminal 210 is used to acquire the image to be reasoned and upload the image to server 220 so that the image can be applied to reasoning through the visual language model deployed in server 220 and the reasoning result returned by server 220 can be obtained.

[0049] It should be understood that, in other embodiments, the visual language model can also be deployed on terminal 110, so that after terminal 110 obtains the image to be inferred, it can directly obtain the inference result based on the locally deployed visual language model. The visual language model can also be deployed on both terminal 110 and server 120, and this implementation environment is not limited to this.

[0050] In the embodiments of this application, the visual language model consists of two parts: a visual model and a language model. The visual model is used to extract the visual features of the image, and the visual features extracted by the visual model and the text features corresponding to the prompt text are respectively input into the language model, so that the language model outputs a text description about the image.

[0051] The visual model portion of the visual language model includes a lighter-weight student visual model to reduce the need for large-scale training data and computing resources, thereby achieving efficient inference. Furthermore, the lightweight student visual model can run efficiently on resource-constrained terminal devices, thus improving the model's practical deployment and application capabilities. For example, knowledge integration can be achieved using visual knowledge from multiple visual models (including a base visual model and expert visual models), and through knowledge distillation (KD), the integrated expert knowledge can be transferred to the student visual model to significantly improve the visual perception and language generation capabilities of the visual language model, making it perform better in multimodal tasks. It should also be noted that the student visual model mentioned in the embodiments of this application is a lighter-weight visual model compared to deploying multiple visual models. Since the student visual model is obtained through knowledge distillation of integrated expert knowledge, it can possess visual perception capabilities similar to multiple visual models, thus replacing multiple visual models for visual perception; therefore, the student visual model is more lightweight.

[0052] The language model part can use a large language model. As can be understood, a large language model refers to a deep learning model trained on a large amount of text data that can generate natural language text or understand the meaning of language text, and can handle complex natural language tasks such as text generation, question answering, and dialogue.

[0053] Please see Figure 3 , Figure 3 An exemplary visual language model pre-training architecture is illustrated. As mentioned earlier, the training of large language models typically involves two stages: pre-training and instruction fine-tuning. Figure 3 The illustrated training architecture corresponds to the image and text pre-training stage of the visual language model.

[0054] like Figure 3 As shown, this pre-trained architecture includes an expert visual model, a base visual model, a student visual model, a language segmenter, and a large language model. The expert visual model is a model pre-trained on a specific visual task and possesses specialized knowledge. Therefore, the expert visual model is typically frozen during the training of the visual language model and its parameters are not optimized as the visual language model trains. The base visual model provides basic visual knowledge and usually participates in parameter optimization during the training of the visual language model. The student visual model is a lighter-weight visual model. For example, the student visual model can initially be the base visual model, meaning it uses the base visual model as its initial parameters for updates. However, with gradual training, the student visual model develops stronger visual perception capabilities and can eventually replace multiple visual models.

[0055] For ease of understanding, Figure 3 The illustrated visual language model pre-training architecture can be divided into three parts: visual feature fusion, visual language pre-training, and expert knowledge integration.

[0056] (1) The explanation of the visual feature fusion part is as follows:

[0057] A visual model consists of a visual encoder and a feature projector. The feature projector projects the visual knowledge extracted by the visual encoder onto a unified representation space. For example, an expert visual model may contain a visual encoder such as DINOv2 or SAM (Segment Anything Model), while a basic visual model may contain a visual encoder such as CLIP (Contrastive Language-Image Pre-training); there are no restrictions on this. The feature projector may contain a two-layer MLP (Multi-layer Perceptron) network. An MLP is a neural network composed of multiple fully connected neurons used for mapping the feature space.

[0058] The visual encoder included in the expert vision model is usually frozen during the training of the visual language model, but the feature projector included in the expert vision model usually needs to be optimized in terms of parameters during the training of the visual language model.

[0059] In order to enable the visual projectors of different visual models to project the visual knowledge extracted by different visual models onto a unified representation space, the visual knowledge extracted by the visual encoders of each visual model is passed through their respective visual projectors and then through a unified visual representation mapping head, such as an MLP network layer, thereby obtaining visual features corresponding to the same representation space.

[0060] It should be noted that, generally speaking, visual models perform image feature extraction on a per-patch basis. Since different visual models may use different patch sizes for the input image, the visual features of multiple image patches output by each expert visual model can be downsampled to ensure that the downsampled visual features maintain the same dimensionality as the visual features output by the base visual model. For example, the visual features output by the base visual model can be represented as N×C, where N represents the number of image patches and C represents the vector length corresponding to the visual features of each image patch.

[0061] Next, the visual language model itself contains predefined mapping vectors. When the visual model performs image feature extraction on a patch-by-pattern basis, the dimension of the mapping vector can be represented as N×C×K, where N represents the number of patch images, C represents the length of each feature vector, and K represents the number of expert visual models. When the visual model performs image feature extraction on an image-by-image basis, the dimension of the mapping vector can be represented as C×K.

[0062] Since the pre-defined mapping vectors can establish a connection with expert visual models, when the visual model performs image feature extraction on an image-by-image basis, the visual fusion contribution of each visual expert model to the image can be determined based on the pre-defined mapping vectors and the visual features extracted by the base visual model. When the visual model performs image feature extraction on an image patch-by-patch basis, the visual fusion contribution of each visual expert model to each image patch can be determined based on the pre-defined mapping vectors and the visual features extracted by the base visual model. Therefore, by fusing the visual features extracted by each visual model based on their visual fusion contributions, the corresponding fused visual features can be obtained, which can also be called unified and integrated expert knowledge.

[0063] (2) The explanation of the visual language pre-training part is as follows:

[0064] The input spaces of visual and text modalities are aligned using an image-text pre-training method, enabling large language models to understand visual information. Large language models can be models such as LLAMA (Large Language Model Meta AI), GPT (Generative Pre-trained Transformer), and Qwen, and are not limited here.

[0065] By inputting integrated expert knowledge and text features extracted from the prompt text via a language segmenter into a large language model, both the basic visual model and the large language model are trained. The prompt text is used to indicate the function to be performed by the large language model. Assuming that image-text pairs {I,T}∈D are used for training the visual language model, where I represents the training image, T represents the training text (which can be the actual description of the training image I), and D represents the training dataset, the corresponding training loss value can be calculated based on the text content output by the large language model for the training image I and the training text T. The parameters of both the basic visual model and the large language model are then optimized based on the obtained training loss value.

[0066] The training process described above can be called the initial training process or the first stage training process. This not only allows the learning to integrate the visual knowledge of different visual models into expert knowledge, but also enables the large language model to understand visual information.

[0067] (3) The explanation of the expert knowledge integration part is as follows:

[0068] By using knowledge distillation, integrated expert knowledge is transferred to a single, lighter student visual model, enabling the student visual model to imitate the output after the integration of expert knowledge, thereby learning rich visual perception capabilities and achieving a balance between computing power and ability.

[0069] To further enhance the adaptability and training efficiency of the student visual model and the large language model, the student visual model and the large language model are trained simultaneously. This not only reduces the computational overhead caused by multiple forward and backward propagations, but also enables the student visual model to dynamically adapt to different datasets and task requirements.

[0070] It can be seen that, based on Figure 3 The training architecture shown can provide strong visual perception capabilities for visual language models while significantly reducing training data and computing resources, thereby achieving efficient and accurate content moderation. This makes it more suitable for deployment on resource-constrained terminal devices, which is conducive to better applying visual language models to real business environments and improving the efficiency and accuracy of content moderation.

[0071] Please see Figure 4 , Figure 4 This is a flowchart illustrating a model training method in an exemplary embodiment of this application. This method can be applied to... Figure 2 The implementation environment shown can be executed by server 220, terminal 210, or both terminal 210 and server 220. Of course, this method can also be applied to other implementation environments and executed by terminals or servers in other implementation environments, or by both terminals and servers in other implementation environments; this embodiment does not impose any limitations on this.

[0072] like Figure 4 As shown, in an exemplary embodiment, the model training method includes steps S410-S450, which are described in detail below:

[0073] S410: Visual features of the first training image are extracted by the basic visual model and the expert visual model respectively, and the visual features extracted by each visual model are fused to obtain the first fused visual features.

[0074] As mentioned earlier, a basic vision model refers to an artificial intelligence model that provides basic visual knowledge and typically participates in model parameter optimization during the pre-training process of the visual language model. An expert vision model, on the other hand, is an artificial intelligence model pre-trained on a specific visual task and possesses specialized knowledge. Because expert vision models have already been pre-trained for a specific visual task, they are usually in a frozen state during the pre-training process of the visual language model and do not optimize their model parameters as the visual language model is trained.

[0075] In some exemplary embodiments, the visual model consists of a visual encoder and a feature projector. The visual encoder is used to extract visual knowledge from the input image, and the feature projector is used to project the visual knowledge extracted by the visual encoder onto a unified representation space. During the pre-training process of the visual language model, the visual encoder contained in the expert visual model is usually in a frozen state, while the feature projector contained in the expert visual model needs to be optimized for parameters as the visual language model is trained.

[0076] This embodiment does not impose any restrictions on the number or type of the base visual model and the expert visual models. For example, typically one base visual model is used to provide basic visual knowledge for the pre-training of the visual language model, and at least two expert visual models are used to provide richer expert visual knowledge for the pre-training of the visual language model. It is understood that the more expert visual models there are, the better the visual perception capability of the final trained visual language model will be, but this also undoubtedly increases the demand for computing resources. Therefore, in practical application scenarios, the number of expert visual models used can be determined based on the resource availability of the terminal device.

[0077] In some exemplary embodiments, multiple expert vision models can be pre-set, and the most suitable expert vision model can be adaptively selected for pre-training of the visual language model based on specific task requirements and the feature information of the input image. For example, the multiple pre-set expert vision models can include vision models with different architectures, such as Convolutional Neural Networks (CNN), Transformers, and Hybrid Attention Networks. They can also include vision models of different sizes and complexities, such as lightweight, medium-sized, and heavyweight vision models. They can also include vision models optimized for specific tasks, such as object detection, image segmentation, and pose estimation. The specific model structure of the multiple expert vision models is not limited here. Before pre-training begins, a detailed analysis of the requirements of the specific task can be performed, which may include parsing the task type (such as classification, detection, generation, etc.), the features of the input image (such as color, texture, shape, etc.), and the output requirements (such as accuracy, speed, etc.). Based on the parsing results, one or more expert vision models can be matched. This embodiment also does not limit the specific matching rules.

[0078] The visual features of the first training image extracted by the basic visual model and the expert visual model respectively correspond to a unified representation space, facilitating subsequent fusion processing of the visual features extracted by each visual model. For example, the output information of the basic visual model and the expert visual model for the first training image can be processed by a unified visual representation mapping head and used as the visual features of the first training image extracted by the basic visual model and the expert visual model respectively. It can be understood that the unified visual representation mapping head can refer to an MLP network that operates on the mapping of the feature space.

[0079] In some exemplary embodiments, the basic vision model and the expert vision model perform image feature extraction on an image-by-image basis. Therefore, after the visual features of the first training image are extracted by the basic vision model and the expert vision model respectively, the visual features extracted by each vision model can be directly fused.

[0080] In some exemplary embodiments, the base vision model and the expert vision model perform image feature extraction on a per-image-patch basis. Therefore, the visual features of the first training image extracted by the base vision model and the expert vision model respectively include the visual features corresponding to each image patch contained in the first training image. Since different vision models use different patch sizes, the number of visual features output by different vision models for the same training image may differ. Therefore, the visual fusion features of the image patches extracted by the expert vision model can also be downsampled. The downsampled visual features corresponding to each expert vision model should maintain the same dimension as the visual features extracted by the base vision model to facilitate subsequent fusion processing of the downsampled visual features corresponding to each expert vision model with the visual features extracted by the base vision model. For example, the visual features extracted by the base vision model can be represented as N×C, where N represents the number of image patches and C represents the vector length corresponding to the visual features of each image patch.

[0081] It should be understood that the various visual models mentioned in this embodiment include basic visual models and expert visual models. This embodiment does not limit the specific method of fusing the visual features extracted by each visual model. For example, it can involve adding the visual features extracted by each visual model, calculating the average, or first determining the visual fusion contribution of each expert visual model to the same image, and then fusing the visual features extracted by each visual model according to the determined visual fusion contribution to obtain the first fused visual feature. The visual fusion contribution of each expert visual model to the same image can be preset vector information or vector information determined based on specific rules, and is not limited here.

[0082] Therefore, the fused visual features obtained in this embodiment cover the visual knowledge perceived by the basic visual model and the expert visual model, and thus this fused visual feature can be called the integrated expert knowledge.

[0083] S420, based on the first fused visual features and the first training text corresponding to the first training image, performs initial training on the large language model and the basic visual model to obtain the initially trained large language model and the initially trained basic visual model.

[0084] The first training text corresponding to the first training image refers to the text content of the image-text pair formed with the first training image in the training dataset. If we assume that the image-text pair {I,T}∈D is used for visual language model training, then I represents the training image, T represents the training text, and D represents the training dataset. The training text T can be understood as the actual descriptive text of the training image I. For example, if... Figure 1 The image shown is used as a training image, and the corresponding training text could be: "The image you provided appears to be a picturesque scene, with a dock extending into a body of water and mountains and trees in the background. There are no visible people or objects in the image that would evoke inappropriate content. The image is safe and does not contain any elements that are considered unsuitable for all viewers."

[0085] This embodiment performs initial training on the large language model and the basic visual model based on the first fused visual features and the training text corresponding to the first training image. This not only enables the learning to unify and integrate the visual knowledge of different visual models into expert knowledge, but also enables the large language model to understand visual information.

[0086] For example, the first fused visual features and the text features corresponding to the prompt text can be input into a large language model to obtain the predicted text output by the large language model. Then, based on the predicted text output by the large language model and the training text corresponding to the first training image, an initial training loss value is determined, and the parameters of the base visual model and the large language model are optimized based on this initial training loss value. The initial training can stop when the initial training value obtained based on the optimized base visual model and the large language model is less than a preset initial loss threshold, or when a preset number of iterations is reached; there are no restrictions here. If the initial training stops when the preset number of iterations is reached, the model stages in the iteration process can be evaluated using a validation dataset, and the model stage with the best performance can be selected to obtain the initially trained base visual model and the large language model.

[0087] Among them, the prompt text acts as a prompt for the large language model to achieve corresponding functions, such as in Figure 1In the application scenario shown, the prompt text could be, "Analyze the provided image and determine whether it contains any inappropriate content, providing a clear assessment indicating whether the image is safe or contains elements deemed unsuitable for all viewers." Under the prompt text's guidance, the large language model, based on its understanding of the first fused visual features, outputs a descriptive text about the image content of the first training image. This descriptive text serves as the predicted text for the first training image by the prediction large language model. The text features corresponding to the prompt text can be extracted using a language segmenter. This language segmenter is essentially a neural network used to segment continuous text into independent words or phrases, and then extract text feature vectors based on these segments.

[0088] The initial training loss value can be determined by the following formula:

[0089]

[0090] Among them, L lm I1 represents the initial training values, T1 represents the first training image, D represents the training dataset, l represents the standard cross-entropy loss function, LLM represents the large language model, and H represents the initial training values. vis1 H represents the first fused visual feature. lang This indicates the text features corresponding to the prompt text. This represents the calculation of the expected value of a text-image pair (I1, T1) randomly selected from the training dataset D.

[0091] The purpose of initial training is to minimize the initial training values. For example, gradient descent or other methods can be used to optimize the parameters of the basic visual model and the large language model, and this is not restricted here. Initial training and parameter optimization are performed iteratively until the initial training values ​​obtained based on the optimized basic visual model and the large language model are less than a preset initial loss threshold, or until a preset number of iterations is reached. At this point, initial training stops, and the initially trained basic visual model and the initially trained large language model are obtained accordingly.

[0092] It should also be noted that a visual model typically consists of a visual encoder and a feature projector. The feature projector is usually frozen during the pre-training process. Therefore, parameter optimization for the basic visual model usually refers to parameter optimization for the visual encoder.

[0093] S430 extracts visual features from the second training image using the initial trained basic visual model and expert visual model, and then fuses the visual features extracted by each visual model to obtain the second fused visual features.

[0094] In this embodiment, the second training image and the first training image can be the same image or different images, and there is no limitation.

[0095] It should also be noted that the process of extracting visual features from the second training image through the initial trained basic visual model and expert visual model, and fusing the visual features extracted by each visual model to obtain the second fused visual features, is the same as the process of obtaining the first fused visual features, and will not be described in detail here.

[0096] S440: Based on the student visual model, extract the visual features of the second training image, and train the student visual model and the initially trained large language model according to the second fused visual features, the visual features of the second training image extracted by the student visual model, and the second training text corresponding to the second training image, so as to generate a visual language model based on the trained student visual model and large language model.

[0097] To reduce computational resource requirements, this embodiment transfers the integrated expert knowledge to a lighter student vision model. It is understood that knowledge distillation (KD) is a model compression technique in deep learning. Its core idea is to train a smaller model (student model) to mimic a large, pre-trained model (teacher model), thereby reducing model complexity and computational cost while maintaining performance.

[0098] In this embodiment, the student visual model is used to mimic the output of the second fused visual features, enabling the student visual model to integrate the visual perception capabilities of different visual models. During the application inference phase, deploying a lighter student visual model replaces the deployment of multiple visual models, reducing the need for large-scale training data and computational resources, while still obtaining rich visual perception capabilities through the student visual model. Therefore, the visual features of the second training image extracted based on the student visual model can also be understood as mimicked fused visual features.

[0099] In some exemplary embodiments, the student visual model can be the base visual model when it is first trained, but as training progresses, the model parameters of the student visual model become more lightweight compared to the base visual model.

[0100] This embodiment describes the training process of the student visual model and the initially trained large language model based on the second fused visual features, the visual features of the second training images extracted by the student visual model, and the second training text corresponding to the second training images. This process can be understood as comprising two parts. One part is the knowledge distillation process based on the second fused visual features, which gradually approximates the visual features of the second training images extracted by the student visual model with the second fused visual features. This allows the student visual model to learn the visual perception capabilities of multiple visual models (including the initially trained basic visual model and the expert visual model). The other part is the process of enhancing the adaptability between the student visual model and the initially trained large language model.

[0101] Therefore, the visual features extracted by the student visual model and the text features corresponding to the prompt text can be input into the large language model after initial training to obtain the predicted text output by the large language model after initial training. Then, based on the predicted text output by the large language model after initial training, the second training text, the second fused visual features, and the visual features extracted by the student visual model, the target training loss value is determined, and the parameters of the student visual model and the large language model after initial training are optimized based on the target training loss value.

[0102] Similar to the initial training process, the conditions for stopping training can be that the training loss value obtained based on the optimized student visual model and the large language model is less than the preset target loss threshold, or that the preset number of iterations has been reached. The preset number of iterations in the training phases of both can be the same or different, and there is no restriction on this.

[0103] The training process for the student visual model and the initially trained large language model in this embodiment includes two parts, and the target training loss value calculated in this embodiment also includes two parts of loss. Specifically, the first part of the loss is determined based on the predicted text output by the initially trained large language model and the second training text, and the second part of the loss is determined based on the second fused visual features and the visual features extracted by the student visual model. Thus, the target training loss value is determined based on the first part of the loss and the second part of the loss.

[0104] It is understandable that the first part of the loss refers to the loss caused by knowledge distillation, and the second part of the loss refers to the adaptation loss between the student's visual model and the initially trained large language model. The sum of the first and second parts of the loss can be used as the target training loss value, or a weighted sum of the first and second parts of the loss can be performed to obtain the target training loss value.

[0105] For example, the target training loss value can be determined based on the following formula:

[0106]

[0107] Among them, L kd This represents the target training loss value. This indicates the first part of the loss. This indicates the second part of the loss. I² represents the squared L2 (Euclidean) norm, I² represents the second training image, T² represents the second training text, D represents the training dataset, l represents the standard cross-entropy loss function, LLM represents the large language model, and H represents the large language model. vis2 H represents the second fused visual feature. stu H represents the visual features extracted from the second training image by the student's visual model. lang This indicates the text features corresponding to the prompt text. This represents calculating the expected value of a randomly selected image-text pair (I2, T2) from the training dataset D. This indicates that the expected value is calculated for the second training image, and α represents the loss coefficient corresponding to the second part of the loss.

[0108] The process of optimizing the parameters of the student visual model and the initially trained large language model based on the target training loss value is also the process of minimizing the target training loss value. The training is iterated until the training loss value obtained based on the optimized student visual model and large language model is less than the preset target loss threshold, or the preset number of iterations is reached, thereby obtaining the final student visual model and large language model.

[0109] Please see Figure 5 , Figure 5 This illustration shows the pre-training process included in the model training scheme disclosed in this embodiment. In summary, the model pre-training process disclosed in this embodiment includes two training stages. The first training stage is the initial training process of the basic visual model and the large language model based on the first visual fusion feature obtained by fusing the expert visual model and the basic visual model, and the first training text, resulting in the initially trained basic visual model and the initially trained large language model. The second training stage is the training process of the student visual model and the initially trained large language model based on the second visual fusion feature obtained by fusing the expert visual model and the initially trained basic visual model, the second training text, and the visual features of the second training image extracted by the student visual model, resulting in the trained student visual model and the trained large language model.

[0110] The trained student visual model and the trained large language model together constitute a visual language model, which can be used for application reasoning. That is, the trained student visual model and the large language model are deployed in the application reasoning stage, and together they form the final visual language model. Specifically, the visual language model includes a student visual model part and a large language model part, and each model part is trained based on the model training method mentioned in this embodiment. The student visual model part is used to extract image visual features and input the extracted image visual features into the large language model part, which then outputs the corresponding predicted text. Therefore, the final visual language model has at least the following beneficial effects:

[0111] (1) High model performance: Since the visual language model integrates the visual knowledge of multiple visual encoders, it can significantly improve the visual perception and language generation capabilities of the visual language model, making it perform better in multimodal tasks.

[0112] (2) It can reduce the demand for computing resources during the training process: Through knowledge distillation, the integrated expert knowledge is transferred to the lightweight student vision model, thereby reducing the demand for large-scale training data and computing resources.

[0113] (3) Improved reasoning efficiency: The visual language model uses a lighter student visual model for application reasoning, which can run efficiently on resource-constrained terminal devices, thereby improving the actual deployment and application capabilities of the visual language model.

[0114] (4) Enhanced model adaptability: The model training scheme disclosed in this embodiment is applicable to various content review scenarios, including image review, video review and text generation tasks, and can provide efficient and accurate review results in different application environments.

[0115] Please continue reading. Figure 6 , Figure 6 Based on Figure 4 The flowchart of the model training method further proposed in the illustrated embodiment is shown. Figure 6 The process of fusing the visual features extracted from the first training image by the basic visual model and the expert visual model is illustrated in steps S610-S620, which are detailed below:

[0116] S610, based on the preset mapping vector and the visual features extracted by the basic visual model, determines the visual fusion contribution of each expert visual model to the first training image.

[0117] The pre-defined mapping vector is used to establish a connection with the expert visual model. The dimension of the mapping vector can be represented as C×K, where C represents the length of the feature vector and K represents the number of expert visual models.

[0118] In some exemplary embodiments, the visual fusion contribution of each expert visual model to the first training image is obtained by calculating the product and norm between the visual features extracted by the base visual model and the mapping vector, and then calculating the quotient of the product and the norm.

[0119] The following example uses a visual model to perform visual feature extraction on an image patch basis. The above calculation process can be expressed as the following formula:

[0120]

[0121] Among them, w k (n) represents the contribution of the k-th expert visual model to the visual information fusion of the image patch at position n. Q represents the visual features of image patch n extracted by the basic vision model. k (n) represents the feature vector at position n that corresponds to the k-th expert vision model.

[0122] In some exemplary embodiments, other methods may also be used to determine the visual fusion contribution of each expert visual model to the first training image. For example, for each expert visual model, the product between its extracted visual features and the corresponding feature vector in the mapping vector is calculated, and then the ratio of its own product to the sum of all products is calculated. The obtained ratio is used as its own visual fusion contribution to the training image.

[0123] S620: Based on the visual fusion contribution of each expert visual model to the first training image, the visual features extracted by each visual model are fused to obtain the first fused visual features.

[0124] For example, the product of the visual fusion contribution and the visual features extracted by the corresponding expert visual model can be calculated. Then, the products corresponding to each expert visual model are added together and the feature space is mapped to obtain the expert fusion features that are adapted to the input feature space corresponding to the large language model. Then, the expert fusion features are added to the visual features extracted by the basic visual model to obtain the first fused visual feature.

[0125] The above fusion process can be expressed as the following formula:

[0126]

[0127] Among them, H vis1 (n) represents the first fused visual feature. w represents the visual features extracted from the first training image by the base vision model. k (n) represents the contribution of the k-th expert visual model to the visual information fusion of the image patch at position n. Let M represent the visual features extracted from the first training image by the k-th expert visual model, and let M represent the feature space mapping of the fused visual tokens so that they can be adapted to the input space corresponding to the large language model.

[0128] As can be seen, the visual feature fusion process disclosed in this embodiment first determines the visual fusion contribution of each expert visual model to the first training image, and then fuses the visual features extracted by each visual model based on the determined visual fusion contribution, which can fully fuse the visual knowledge of different visual models.

[0129] It should be noted that, in some exemplary embodiments, during the initial training of the large language model and the basic visual model, the mapping vector used to calculate the first fused visual features is also optimized. That is, the mapping vector also needs to be pre-trained to learn to more accurately represent the correlation with each expert visual model, thereby improving the reliability of the unified integrated expert knowledge and promoting the reliability of the student visual model obtained by knowledge distillation.

[0130] In another exemplary embodiment, the model training method further includes a process of fine-tuning the trained visual language model with instructions, so that the fine-tuned visual language model is more adapted to real-world application scenarios. The instruction fine-tuning dataset is typically derived from or related to real-world application scenarios.

[0131] To enable visual language models to effectively handle multimodal tasks involving images and text, a large amount of image-text pairs is typically required for pre-training and fine-tuning. However, when dealing with specific domains, the amount of image-text pairs obtained from publicly available resources may be insufficient, especially when dealing with sensitive content. The scarcity of data becomes a significant obstacle to training the model.

[0132] To address the above issues, the model training method provided in the embodiments of this application further includes a process for acquiring a training dataset. This embodiment also proposes several schemes for acquiring the training dataset to ensure that the visual language model can be trained and fine-tuned with instructions efficiently and accurately. Please refer to... Figure 7 , Figure 7 This illustrates an exemplary process for obtaining a training dataset, applicable to image content moderation scenarios.

[0133] like Figure 7As shown, in order to ensure that the visual language model can accurately follow instructions and make reasonable judgments when dealing with sensitive content (referring to image content that cannot pass the content review), at least one of the following methods can be used to expand and enrich the training dataset during the image and text pre-training stage.

[0134] (1) Sensitive Data Generation: Generative models, such as GAN (Generative Adversarial Networks) and Stable Diffusion, are used to automatically generate image-text pairs containing sensitive content. First, the generative model is trained using a small amount of labeled sensitive content data. Then, the trained generative model is used to generate new sensitive content images and corresponding text descriptions. This generated data can also undergo rigorous manual review to ensure its authenticity and diversity, making it suitable for the training needs of visual language models.

[0135] (2) Context Reconstruction and Editing: Sensitive content is extracted from existing image-text pairs, and new image-text pairs are generated through context reconstruction techniques. For example, sensitive areas in an image are modified, replaced, or repainted to generate a new image, and corresponding text descriptions are written. This approach not only expands the dataset but also enhances the model's ability to identify sensitive content in different contexts.

[0136] (3) Cross-domain sensitive data integration: Integrate sensitive content data from different domains and fuse this data with data from the target domain using existing large language models. For example, combine sensitive data from the medical field with sensitive content from social media to enhance the model's ability to understand and process various types of sensitive content.

[0137] (4) Real-world scenario generation: Using simulation technology to generate sensitive content data in real-world scenarios. For example, simulating communication scenarios on social media platforms to generate dialogues and images containing sensitive information. This data undergoes manual review and filtering to ensure its authenticity and diversity, which helps the model perform well in practical applications.

[0138] During the instruction fine-tuning phase, the focus is on enhancing the model's ability to follow instructions, particularly those related to content moderation. To ensure the model can accurately follow instructions and make reasonable judgments when dealing with sensitive content, at least one of the following methods can be used to expand and enrich the instruction fine-tuning dataset:

[0139] (1) Refine instruction scenarios: Content moderation-related instructions are refined into multiple specific scenarios, such as identifying inappropriate content and detecting harmful speech. The dataset for each instruction scenario includes detailed instruction descriptions and corresponding image-text pairs to ensure that the model can accurately execute instructions in various situations.

[0140] (2) Multi-turn dialogue instructions: Design a dataset of instructions containing multi-turn dialogues to enable the model to understand and execute complex instructions in continuous interactions. For example, users may provide cues of sensitive content step by step, and the model needs to identify and process these cues step by step during the dialogue. This multi-turn dialogue dataset helps the model better understand user intent and contextual relationships.

[0141] (3) Dynamic instruction adjustment: A mechanism for dynamically adjusting instructions is introduced. By adding instances containing changed instructions or priority adjustments to the training dataset, the model can flexibly respond to changes in instructions. For example, when the priority of a certain type of sensitive content changes, the model can adjust its processing strategy in a timely manner.

[0142] (4) Combining multimodal instructions: Combining multimodal instructions involving images and text enables the model to process both visual and textual information simultaneously. For example, given an image with accompanying text description, the model is required to simultaneously identify sensitive content in the image and analyze sensitive information in the text. This can improve the model's performance in complex multimodal tasks.

[0143] (5) Simulate real-world application scenarios: By simulating content moderation scenarios in real-world applications, a complex instruction dataset containing real-world scenarios is generated. For example, simulating content moderation tasks on social media platforms generates instruction data containing diverse content posted by users and platform moderation rules. This ensures that the model can efficiently execute instructions in real-world applications.

[0144] It should be noted that in practical application scenarios, training datasets can be obtained through one or more of the methods shown in the examples above, or through other methods. This does not imply any restriction on the method of obtaining training datasets.

[0145] Therefore, the training dataset acquisition scheme provided above effectively expands and enriches the training dataset, which not only improves the accuracy and reliability of the visual language model when dealing with sensitive content, but also enhances the adaptability of the visual language model in various practical application scenarios.

[0146] Please see Figure 8 , Figure 8 This is a flowchart illustrating an exemplary image processing method. This method can be applied to... Figure 2The implementation environment shown can be executed by server 220, terminal 210, or both terminal 210 and server 220. Of course, this method can also be applied to other implementation environments and executed by terminals or servers in other implementation environments, or by both terminals and servers in other implementation environments; this embodiment does not limit this.

[0147] like Figure 8 As shown, the image processing methods include S810-S820, which are described in detail below:

[0148] S810, acquire the image to be processed;

[0149] S820: The image to be processed is input into the trained visual language model to obtain the predicted text output by the visual language model. The visual language model consists of a student visual model and a large language model. The student visual model is used to extract the visual features of the image to be processed, and the large language model is used to output the corresponding predicted text based on the visual features of the image to be processed.

[0150] It should be noted that the visual language model obtained through training mentioned in the image processing method refers to the model training method described in the foregoing embodiments. For detailed training procedures, please refer to the descriptions in the foregoing embodiments; they will not be repeated here.

[0151] This embodiment constructs a visual language model for reasoning based on a trained student visual model and a trained large language model. Because the student visual model possesses rich visual perception capabilities and is more lightweight compared to other visual models, it reduces the demand for computing resources, enabling the visual language model to be deployed and run on resource-constrained terminal devices, thus improving its practical deployment and application capabilities. Furthermore, due to the rapid responsiveness and high accuracy of the visual language model during training, it can accurately predict image content, thereby improving the efficiency and reliability of image content review.

[0152] Furthermore, to better understand the visual language model training and inference architecture proposed in this application, Figure 9 The diagram illustrates the model training architecture and inference schematics used in the embodiments of this application. For example... Figure 9 As shown, the visual language model training and inference architecture is divided into a text-image pre-training stage, an instruction fine-tuning stage, and an inference stage.

[0153] During the image-text pre-training phase, multiple visual models need to be deployed simultaneously, including a base visual model, an expert visual model, a student visual model, and a large language model. The visual encoder contained in the expert visual model does not require training, while the feature projectors contained in the base visual model, the student visual model, and the expert visual model do require training.

[0154] During the instruction fine-tuning phase, all expert vision models and basic vision models are removed, leaving only the student vision model and the large language model. At this point, all model parts are released for instruction fine-tuning.

[0155] In the inference phase, the trained model parameters (student visual model and large language model) are used to input the text features of the image to be inferred and the prompt text after passing through the language segmenter into the model to obtain the final output predicted text.

[0156] Please see Figure 10 , Figure 10 This is a block diagram illustrating a model training apparatus according to an exemplary embodiment of this application. The apparatus can be applied to… Figure 2 The implementation environment shown can be deployed on server 220, terminal 210, or both terminal 210 and server 220. Of course, this device can also be applied to other implementation environments and deployed on terminals or servers in other implementation environments, or deployed on terminals and servers in other implementation environments; this embodiment does not limit this.

[0157] like Figure 10 As shown, in an exemplary embodiment, the model training apparatus includes:

[0158] The first fusion module 1010 is configured to extract visual features from the first training image through the basic visual model and the expert visual model respectively, and to fuse the visual features extracted by each visual model to obtain the first fused visual features.

[0159] The first training module 1020 is configured to perform initial training on the large language model and the basic visual model based on the training text corresponding to the first fused visual features and the first training image, so as to obtain the initially trained large language model and the initially trained basic visual model.

[0160] The second fusion module 1030 is configured to extract visual features of the second training image through the initial trained basic visual model and expert visual model, and to fuse the visual features extracted by each visual model to obtain the second fused visual features.

[0161] The second training module 1040 is configured to extract visual features of the second training image based on the student visual model, and train the student visual model and the initially trained large language model based on the second fused visual features, the visual features of the second training image extracted by the student visual model, and the second training text corresponding to the second training image.

[0162] In another exemplary embodiment, the first fusion module 1010 includes:

[0163] The unit is configured to determine the visual fusion contribution of each expert visual model to the first training image based on a preset mapping vector and visual features extracted from the basic visual model; wherein, the dimension of the mapping vector is represented as C×K, where C represents the length of the feature vector and K represents the number of expert visual models.

[0164] The fusion unit is configured to fuse the visual features extracted from each visual model based on the visual fusion contribution to obtain the first fused visual feature.

[0165] In another exemplary embodiment, the determining unit is further configured as follows:

[0166] Calculate the product and norm between the visual features extracted from the basic visual model and the mapping vector;

[0167] Calculate the quotient of the product and the norm to obtain the visual fusion contribution of each expert visual model to the first training image.

[0168] In another exemplary embodiment, the fusion unit is further configured as follows:

[0169] The product of the visual fusion contribution and the visual features extracted by the corresponding expert visual model is calculated.

[0170] The products of each expert visual model are added together and then mapped to the feature space to obtain expert fusion features that are adapted to the input feature space of the large language model.

[0171] The expert-fused features are added to the visual features extracted from the basic visual model to obtain the first fused visual feature.

[0172] In another exemplary embodiment, the visual features extracted by each visual model include visual features corresponding to at least two image patches; the model training device further includes:

[0173] The downsampling module is configured to downsample the visual features of the image patches extracted by the expert visual model, and then perform fusion processing based on the visual features of the downsampled image patches; wherein the visual features after downsampling are consistent in dimension with the visual features extracted by the basic visual model.

[0174] In another exemplary embodiment, the first training module 1020 is further configured as follows:

[0175] The first fused visual features and the text features corresponding to the prompt text are input into the large language model to obtain the predicted text output by the large language model;

[0176] Based on the predicted text output by the large language model and the training text corresponding to the first training image, the initial training loss value is determined, and the parameters of the basic visual model and the large language model are optimized based on the initial training loss value to obtain the initially trained large language model and the initially trained basic visual model.

[0177] In another exemplary embodiment, the first training module 1020 is further configured to:

[0178] During the initial training of the large language model and the basic vision model, the mapping vector used to compute the first fused visual features was also optimized.

[0179] In another exemplary embodiment, the second training module 1040 is further configured as follows:

[0180] The visual features extracted from the student's visual model and the text features corresponding to the prompt text are input into the large language model after initial training to obtain the predicted text output by the large language model after initial training.

[0181] The target training loss value is determined based on the predicted text output by the large language model after initial training, the second training text, the second fused visual features, and the visual features extracted by the student visual model.

[0182] The parameters of the student visual model and the initially trained large language model are optimized based on the target training loss value, so as to generate a visual language model based on the trained student visual model and large language model.

[0183] In another exemplary embodiment, the second training module 1040 is further configured to:

[0184] The first part of the loss is determined based on the predicted text output by the large language model after initial training and the second training text, and the second part of the loss is determined based on the second fused visual features and the visual features extracted by the student visual model.

[0185] The target training loss value is determined based on the first part of the loss and the second part of the loss.

[0186] Please see Figure 11 , Figure 11 This is a block diagram illustrating an image processing apparatus according to an exemplary embodiment of this application. The apparatus can be applied to... Figure 2The implementation environment shown can be deployed on server 220, terminal 210, or both terminal 210 and server 220. Of course, this device can also be applied to other implementation environments and deployed on terminals or servers in other implementation environments, or deployed on terminals and servers in other implementation environments; this embodiment does not limit this.

[0187] like Figure 11 As shown, in an exemplary embodiment, the image processing apparatus includes:

[0188] Image acquisition module 1110 is configured to acquire an image to be processed;

[0189] The model processing module 1120 is configured to input the image to be processed into a visual language model trained based on the method mentioned in the foregoing embodiments, so as to obtain the predicted text output by the visual language model; wherein, the visual language model consists of a student visual model and a large language model, the student visual model is used to extract the visual features of the image to be processed, and the large language model is used to output the predicted text based on the visual features of the image to be processed.

[0190] It should be noted that the apparatus and method provided in the above embodiments belong to the same concept, and the specific ways in which each module and unit performs operations have been described in detail in the method embodiments, and will not be repeated here. In practical applications, the model training apparatus or image processing apparatus provided in the above embodiments can be assigned to different functional modules as needed, that is, the internal structure of the apparatus can be divided into different functional modules to complete all or part of the functions described above, and this is not a limitation here.

[0191] Embodiments of this application also provide an electronic device, including: one or more processors; and a memory for storing one or more computer programs, which, when executed by the one or more processors, cause the electronic device to implement the model training methods provided in the above embodiments.

[0192] Figure 12 A schematic diagram of a computer system suitable for implementing an electronic device according to embodiments of this application is shown. It should be noted that the electronic device can be... Figure 2 The terminal 210 or server 220 in the illustrated implementation environment can also be a terminal or server in other implementation environments; no restrictions are imposed here. It should also be noted that... Figure 12 The computer system 1200 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0193] like Figure 12 As shown, the computer system 1200 includes a Central Processing Unit (CPU) 1201, which can perform various appropriate actions and processes based on a computer program stored in Read-Only Memory (ROM) 1202 or a computer program loaded from storage portion 1208 into Random Access Memory (RAM) 1203, such as performing the methods described in the above embodiments. Various computer programs and data required for system operation are also stored in RAM 1203. The CPU 1201, ROM 1202, and RAM 1203 are interconnected via bus 1204. An Input / Output (I / O) interface 1205 is also connected to bus 1204.

[0194] The following components are connected to I / O interface 1205: an input section 1206 including a keyboard, mouse, etc.; an output section 1207 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1208 including a hard disk, etc.; and a communication section 1209 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 1209 performs communication processing via a network such as the Internet. A drive 1210 is also connected to I / O interface 1205 as needed. Removable media 1211, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 1210 as needed so that computer programs read from them can be installed into storage section 1208 as needed.

[0195] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including a computer program for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1209, and / or installed from removable medium 1211. When the computer program is executed by central processing unit (CPU) 1201, it performs various functions defined in the system of this application.

[0196] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium, a computer-readable storage medium, or any combination of the two. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0197] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0198] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0199] Another aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor of an electronic device, implements the model training method as described above. This computer-readable storage medium may be included in the electronic device described in the above embodiments, or it may exist independently and not assembled into the electronic device.

[0200] Another aspect of this application provides a computer program product comprising a computer program stored in a computer-readable storage medium. A processor of an electronic device reads the computer program from the computer-readable storage medium and executes the computer program, causing the electronic device to perform the model training method or image processing method provided in the various embodiments described above.

[0201] The above description is merely a preferred exemplary embodiment of this application and is not intended to limit the implementation of this application. Those skilled in the art can easily make corresponding modifications or alterations based on the main concept and spirit of this application. Therefore, the scope of protection of this application should be determined by the scope of protection claimed in the claims.

[0202] It is understood that in the specific embodiments of this application, data such as images and text are involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

Claims

1. A model training method, characterized in that, The method includes: Visual features of the first training image are extracted by the basic visual model and the expert visual model respectively, and the visual features extracted by each visual model are fused to obtain the first fused visual features. Based on the first fused visual features and the first training text corresponding to the first training image, the large language model and the basic visual model are initially trained to obtain the initially trained large language model and the initially trained basic visual model. Visual features of the second training image are extracted using the initial trained basic visual model and the expert visual model, and the visual features extracted by each visual model are fused to obtain the second fused visual features. Visual features of the second training image are extracted based on the student visual model. The student visual model and the initially trained large language model are then trained based on the second fused visual features, the visual features extracted by the student visual model, and the second training text corresponding to the second training image, so as to generate a visual language model based on the trained student visual model and large language model.

2. The method according to claim 1, characterized in that, The step of extracting visual features from the first training image using a basic visual model and an expert visual model respectively, and then fusing the visual features extracted by each visual model to obtain the first fused visual feature, includes: Based on the preset mapping vector and the visual features extracted by the basic visual model, the visual fusion contribution of each expert visual model to the first training image is determined; wherein, the dimension of the mapping vector is represented as C×K, C represents the length of the feature vector, and K represents the number of expert visual models; the visual features extracted by each visual model are fused according to the visual fusion contribution to obtain the first fused visual feature.

3. The method according to claim 2, characterized in that, The determination of the visual fusion contribution of each expert visual model to the first training image, based on the preset mapping vector and the visual features extracted by the basic visual model, includes: Calculate the product and norm between the visual features extracted by the basic visual model and the mapping vector; The quotient of the product and the norm is calculated to obtain the visual fusion contribution of each expert visual model to the first training image.

4. The method according to claim 2, characterized in that, The step of fusing the visual features extracted from each visual model based on the visual fusion contribution to obtain the first fused visual feature includes: Calculate the product of the visual fusion contribution and the visual features extracted by the corresponding expert visual model; After summing the products of the various expert visual models, feature space mapping is performed to obtain expert fusion features that are adapted to the input feature space corresponding to the large language model. The expert fusion feature is added to the visual feature extracted from the basic visual model to obtain the first fused visual feature.

5. The method according to any one of claims 2-4, characterized in that, The visual features extracted by each visual model include visual features corresponding to at least two image patches; the method further includes: The visual features of the image patches extracted by the expert visual model are downsampled, and the fusion process is performed based on the visual features of the downsampled image patches; wherein the downsampled visual features maintain the same dimension as the visual features extracted by the basic visual model.

6. The method according to claim 1, characterized in that, The initial training of the large language model and the basic visual model based on the training text corresponding to the first fused visual features and the first training image, to obtain the initially trained large language model and the initially trained basic visual model, includes: The first fused visual features and the text features corresponding to the prompt text are input into the large language model to obtain the predicted text output by the large language model; Based on the predicted text output by the large language model and the training text corresponding to the first training image, an initial training loss value is determined, and the parameters of the basic visual model and the large language model are optimized based on the initial training loss value to obtain the initially trained large language model and the initially trained basic visual model.

7. The method according to claim 6, characterized in that, The method further includes: During the initial training of the large language model and the basic visual model, the mapping vector used to calculate the first fused visual feature is also optimized.

8. The method according to claim 1, characterized in that, The step of extracting visual features from the second training image based on the student visual model, and training the student visual model and the initially trained large language model based on the second fused visual features, the visual features extracted by the student visual model, and the second training text corresponding to the second training image, to generate a visual language model based on the trained student visual model and large language model, includes: The visual features extracted from the student visual model and the text features corresponding to the prompt text are input into the initially trained large language model to obtain the predicted text output by the initially trained large language model. The target training loss value is determined based on the predicted text output by the large language model after initial training, the second training text, the second fused visual features, and the visual features extracted by the student visual model. Based on the target training loss value, the parameters of the student visual model and the initially trained large language model are optimized to generate the visual language model based on the trained student visual model and large language model.

9. The method according to claim 8, characterized in that, The determination of the target training loss value based on the predicted text output by the initially trained large language model, the second training text, the second fused visual features, and the visual features extracted by the student visual model includes: The first part of the loss is determined based on the predicted text output by the large language model after the initial training and the second training text, and the second part of the loss is determined based on the second fused visual features and the visual features extracted by the student visual model. The target training loss value is determined based on the first part of the loss and the second part of the loss.

10. An image processing method, characterized in that, The method includes: Obtain the image to be processed; The image to be processed is input into a visual language model trained according to any one of claims 1-9 to obtain the predicted text output by the visual language model; wherein the visual language model consists of a student visual model and a large language model, the student visual model is used to extract the visual features of the image to be processed, and the large language model is used to output the predicted text based on the visual features of the image to be processed.

11. A model training device, characterized in that, The device includes: The first fusion module is configured to extract visual features from the first training image through the basic visual model and the expert visual model respectively, and to fuse the visual features extracted by each visual model to obtain the first fused visual features. The first training module is configured to perform initial training on the large language model and the basic visual model based on the training text corresponding to the first fused visual features and the first training image, so as to obtain the initially trained large language model and the initially trained basic visual model. The second fusion module is configured to extract visual features of the second training image through the initial trained basic visual model and the expert visual model, and to fuse the visual features extracted by each visual model to obtain the second fused visual features. The second training module is configured to extract visual features of the second training image based on the student visual model, and train the student visual model and the initially trained large language model based on the second fused visual features, the visual features extracted by the student visual model, and the second training text corresponding to the second training image, so as to generate a visual language model based on the trained student visual model and large language model.

12. An image processing apparatus, characterized in that, The device includes: The image acquisition module is configured to acquire the image to be processed. The model processing module is configured to input the image to be processed into a visual language model trained based on the method of any one of claims 1-9, so as to obtain the predicted text output by the visual language model; wherein the visual language model consists of a student visual model and a large language model, the student visual model is used to extract the visual features of the image to be processed, and the large language model is used to output the predicted text based on the visual features of the image to be processed.

13. An electronic device, characterized in that, include: One or more processors; A memory for storing one or more computer programs that, when executed by one or more processors, cause the electronic device to perform the method as described in any one of claims 1-10.

14. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by the processor of the electronic device, causes the electronic device to perform the method of any one of claims 1-10.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor of the electronic device, it implements the method as described in any one of claims 1-10.

Images