Image processing method and device, equipment, storage medium and program product

Abstract

The invention provides an image processing method and device, equipment, a storage medium and a program product. The method comprises the following steps: generating description information matched with a plurality of sample images by utilizing a trained first content generation model based on a first image set comprising a plurality of first sample images and first cue word information; training a second content generation model based on the plurality of first sample images and the description information respectively matched with the plurality of sample images, the second content generation model being configured to generate description information matched with the input image; and generating description information respectively matched with a plurality of second sample images based on a second image set input comprising the plurality of second sample images by using the trained second content generation model.

Description

Technical Field

[0001] Example embodiments of the present disclosure generally relate to the field of computers, and in particular, to methods, apparatuses, electronic devices, computer-readable storage media, and computer program products for image processing. Background Art

[0002] With the development of machine learning technology, machine learning models can be used to perform tasks in a variety of application environments. Multimodal language models have attracted much attention because of their ability to understand and generate language guided by visual input. The model can process and fuse multiple types of data inputs and support applications across various fields, including image captioning, visual question answering, and video analysis. It is usually necessary to use a large amount of multimodal data to train a multimodal language model, and the model performance of the multimodal language model will be affected by the quality of the multimodal data used to train the model. Summary of the invention

[0003] In a first aspect of the present disclosure, a method for image processing is provided. The method includes: using a trained first content generation model, based on a first image set including a plurality of first sample images and first prompt word information, generating description information matching each of the plurality of sample images; based on the plurality of first sample images and the description information matching each of the plurality of sample images, training a second content generation model, the second content generation model being configured to generate description information matching an input image; and using a trained second content generation model, based on a second image set input including a plurality of second sample images, generating description information matching each of the plurality of second sample images.

[0004] In a second aspect of the present disclosure, a device for image processing is provided. The device includes: a first description information generation module, configured to generate description information matching each of a plurality of sample images based on a first image set including a plurality of first sample images and first prompt word information using a trained first content generation model; a content generation model training module, configured to train a second content generation model based on the plurality of first sample images and the description information matching each of the plurality of sample images, the second content generation model being configured to generate description information matching an input image; and a second description information generation module, configured to generate description information matching each of the plurality of second sample images based on a second image set input including a plurality of second sample images using a trained second content generation model.

[0005] In a third aspect of the present disclosure, an electronic device is provided. The electronic device includes: at least one processor; and at least one memory, the at least one memory is coupled to the at least one processor and stores instructions for execution by the at least one processor, and when the instructions are executed by the at least one processor, the electronic device executes the method according to the first aspect of the present disclosure.

[0006] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, the processor executes the method according to the first aspect of the present disclosure.

[0007] In a fifth aspect of the present disclosure, a computer program product is provided, which is tangibly stored in a computer storage medium and includes computer executable instructions, which, when executed by a device, cause the device to perform the method of the first aspect.

[0008] It should be understood that the contents described in this content section are not intended to limit the key features or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0010] Figure 1 A schematic diagram showing an example environment for model training and application according to some embodiments of the present disclosure is shown;

[0011] Figure 2 A schematic diagram showing an example application environment of a machine learning model according to some embodiments of the present disclosure;

[0012] Figure 3 An example process for image processing according to some embodiments of the present disclosure is shown;

[0013] Figure 4 An example for image processing according to some embodiments of the present disclosure is shown;

[0014] Figure 5 A flowchart of a method for image processing according to some embodiments of the present disclosure is shown;

[0015] Figure 6 An exemplary structural block diagram of an apparatus for image processing according to some embodiments of the present disclosure is shown; and

[0016] Figure 7 A block diagram of an electronic device in which one or more embodiments of the present disclosure may be implemented is shown. DETAILED DESCRIPTION

[0017] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0018] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below.

[0019] Herein, unless explicitly stated, executing a step “in response to A” does not mean executing the step immediately after “A” but may include one or more intermediate steps.

[0020] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition, use, storage or deletion of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0021] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0022] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information, so that the user can independently choose whether to provide personal information to software or hardware such as electronic devices, applications, servers or storage media that execute operations of the technical solution of the present disclosure based on the prompt message.

[0023] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information is sent to the user in a manner such as a pop-up window, in which the prompt information can be presented in text form. In addition, the pop-up window can also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0024] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0025] As used herein, the term "model" can learn the association between the corresponding input and output from the training data, so that after the training is completed, the corresponding output can be generated for a given input. The generation of the model can be based on machine learning technology. Deep learning is a machine learning algorithm that processes inputs and provides corresponding outputs by using multi-layer processing units. A neural network model is an example of a model based on deep learning. In this article, "model" may also be referred to as "machine learning model", "learning model", "machine learning network" or "learning network", and these terms are used interchangeably in this article.

[0026] A "neural network" is a machine learning network based on deep learning. A neural network is capable of processing inputs and providing corresponding outputs, and typically includes an input layer and an output layer and one or more hidden layers between the input layer and the output layer. Neural networks used in deep learning applications typically include many hidden layers, thereby increasing the depth of the network. The layers of a neural network are connected in sequence so that the output of the previous layer is provided as input to the next layer, where the input layer receives the input of the neural network and the output of the output layer serves as the final output of the neural network. Each layer of a neural network includes one or more nodes (also called processing nodes or neurons), each of which processes input from the previous layer.

[0027] Generally, machine learning can be roughly divided into three stages, namely the training stage, the testing stage, and the application stage (also called the inference stage). In the training stage, a given model can be trained using a large amount of training data, and the parameter values ​​are continuously updated iteratively until the model can obtain consistent inferences that meet the expected goals from the training data. Through training, the model can be considered to be able to learn the association between input and output (also called input-to-output mapping) from the training data. The parameter values ​​of the trained model are determined. In the testing stage, the test input is applied to the trained model to test whether the model can provide the correct output, thereby determining the performance of the model. The testing stage can sometimes be integrated into the training stage. In the application or inference stage, the trained model can be used to process the actual model input based on the parameter values ​​obtained through training to determine the corresponding model output.

[0028] In the process of training a multimodal language model, in order to achieve accurate and effective alignment between features of different modalities, it is often necessary to use a very large amount of multimodal data to carry out multi-stage training on the model. This is because data of different modalities usually include different information and correspond to different features (for example, images have visual features, videos also include temporal features compared to images, and texts are semantic symbol features). In order to make them adapt to each other and work together in the model, a large number of samples and multi-stage training strategies are required to allow the model to fully learn the association patterns between different modalities. In order for the multimodal language model to understand the image more accurately, the training data for the multimodal language model usually includes at least the image and the annotation information for the image. The annotation information for the image can be used to describe the image. The training data can also include query text for the image, in which case the annotation information can also indicate the response to the query text.

[0029] Traditionally, it is usually necessary to manually annotate the training data (that is, determine the annotation information for the image). Traditionally, the labeler is usually asked some questions about the image, and the answer voice of the qualified personnel is collected. In order to control the quality of the answer, it is usually required that the duration of the answer voice be controlled above a certain length. Traditionally, speech recognition (ASR) is performed on the collected answer voice from the labeler to determine the corresponding text, and a model is used to generate annotation information corresponding to the image based on the text. This method of determining annotation information requires a lot of manpower costs and is inefficient. In addition, due to the subjective limitations of the qualified personnel, the accuracy of the annotation information may be insufficient.

[0030] In view of this, according to an embodiment of the present disclosure, an improved solution for image processing is provided. According to the solution of the embodiment of the present disclosure, based on a first image set including a plurality of first sample images and first prompt word information, a first content generation model that has been trained is used to generate description information that matches each of the plurality of sample images. Based on the plurality of first sample images and the description information that matches each of the plurality of sample images, a second content generation model is trained, and the second content generation model is configured to generate description information that matches an input image. Based on a second image set input including a plurality of second sample images, a second content generation model that has been trained is used to generate description information that matches each of the plurality of second sample images.

[0031] In this way, a more complex and more powerful content generation model can be used to generate high-quality description information, and the generated description information and its images can be used as training data for training a lighter content generation model. The lightweight content generation model trained in this way can generate description information for a large number of images. In this way, a large amount of description information that meets the requirements can be obtained as annotation data for the image, which is used to train the machine learning model for multimodal response. In this way, training data for the machine learning model for multimodal response can be generated quickly and conveniently, which can help improve the efficiency of determining training data, ensure the quality of training data, and further help improve the training efficiency of the machine learning model and improve the model performance of the machine learning model.

[0032] Figure 1 A schematic diagram of an example environment 100 for model training and application according to some embodiments of the present disclosure is shown. Figure 1 The example environment 100 of FIG. 1 shows three different stages of the model, including a pre-training stage 102, a fine-tuning stage 104, and an application stage 106. After the pre-training stage 102 or the fine-tuning stage 104 is completed, there may also be a testing stage, which is not shown in the figure.

[0033] The data collection system 110 can collect data from at least one data source and generate a pre-training data set 112 and / or a training data set 114 for training the machine learning model 105 based on the collected data.

[0034] In the pre-training stage 102, the model pre-training system 120 is configured to perform pre-training of the machine learning model 105 using the pre-training data set 112. At the beginning of pre-training, each component in the machine learning model 105 may have an initial parameter value. The pre-training process is to update the parameter values ​​of the machine learning model 105 to the expected value based on the data in the pre-training data set 112. The pre-training task is used to help update the parameters of the machine learning model 105.

[0035] In the pre-training stage 102, the machine learning model 105 can learn powerful generalization capabilities through a pre-training data set 112 including a large amount of data. After the pre-training is completed, the parameter values ​​of the machine learning model 105 have been updated with the pre-trained parameter values. In some embodiments, the pre-trained machine learning model 105 can more accurately extract the feature representation of the image and more accurately map the feature representation to a part of the input feature of the language model.

[0036] The pre-trained machine learning model 105 can be provided to the fine-tuning stage 104, and the model fine-tuning system 130 fine-tunes it for different downstream tasks. In the fine-tuning stage 104, the training data set 114 is further used to adjust the parameter values ​​of the machine learning model 105. In some embodiments, for the scenario of multimodal response, the pre-trained machine learning model 105 can be continued to be updated using a data set constructed according to an embodiment of the present disclosure (including multiple images, query texts for images, and responses to query texts). As an example, the training data set 114 may include images, specific questions for images, and responses to specific questions. The specific questions may be complex questions that users may actually ask, which may include, for example, "How many XX objects does this image include?", "What is the positional relationship between object A and object B in the image?", etc. In general, the response to the image may correspond to the descriptive information annotated with the image.

[0037] During fine-tuning, the corresponding training algorithm is also used to update and adjust the parameters of the machine learning model 105. Since the model has learned a lot of knowledge from the training data in the pre-training stage, a small amount of training data can be used in the fine-tuning stage 104 to obtain a downstream task model that meets the expectations.

[0038] In some embodiments, a testing phase may be included after the fine-tuning phase 104, and the model performance of the machine learning model 105 may continue to be tested using a test data set. The data set used in the testing phase is of the same type as that used in the fine-tuning phase. In some embodiments, for scenarios involving multimodal responses, a test data set constructed according to an embodiment of the present disclosure (including multiple images, query texts for images, and responses to query texts) may be used to test the model performance of the trained machine learning model 105.

[0039] Of course, in addition to supporting scenarios of multimodal responses, the machine learning model 105 can also support scenarios of unimodal responses, such as scenarios where the query request includes only text or only images. Or in addition to image data and text, a multimodal query request can also include data in other modalities, such as audio. In this case, during the pre-training and fine-tuning stages, the training data set can include query requests of the corresponding modality, and the corresponding responses. Similarly, during the testing stage, a test data set including corresponding query requests and their responses can also be constructed to test the model performance of the machine learning model 105 in the corresponding scenario.

[0040] In the application stage 106, the obtained machine learning model 105 has trained parameter values ​​and can be provided to the model application system 140 for use. In the application stage 106, the machine learning model 105 can be used to process the corresponding query request 142 in the actual scenario and provide a corresponding response 145. In some embodiments, for a multimodal response scenario, the query request 142 includes an image and a corresponding query text.

[0041] exist Figure 1 In the invention, the data acquisition system 110, the model pre-training system 120, the model fine-tuning system 130 and the model application system 140 can be deployed at any appropriate electronic device. The electronic device can be any type of device with computing capabilities, including a terminal device or a server device. The terminal device can be any type of mobile terminal, fixed terminal or portable terminal, including a mobile phone, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a media computer, a multimedia tablet, a personal communication system (PCS) device, a personal navigation device, a personal digital assistant (PDA), an audio / video player, a digital camera / camcorder, a positioning device, a television receiver, a radio broadcast receiver, an e-book device, a gaming device or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof.

[0042] The server device can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks, and big data and artificial intelligence platforms. The server device can include, for example, computing systems / servers, such as mainframes, edge computing nodes, computing devices in cloud environments, and the like. It can be understood that the data acquisition system 110, the model pre-training system 120, the model fine-tuning system 130, and the model application system 140 can be deployed on the same electronic device or on different electronic devices, and the present disclosure does not limit this.

[0043] It should be understood that Figure 1The components and arrangements in the example environment 100 shown are merely examples, and a computing system suitable for implementing the exemplary implementations described in the present disclosure may include one or more different components, other components, and / or different arrangements. For example, although shown as separate, the data acquisition system 110, the model pre-training system 120, the model fine-tuning system 130, and the model application system 140 may be integrated into the same system or device. For example, at least the model pre-training system 120 and the model fine-tuning system 130 may be integrated into a model training system or device. Implementations of the present disclosure are not limited in this respect.

[0044] In some embodiments, via Figure 1 The trained machine learning model 105 may be configured to process multimodal inputs and provide corresponding responses as outputs. Figure 2 A schematic diagram of an example application environment 200 of a machine learning model according to some embodiments of the present disclosure is shown. In the example application environment 200, an electronic device 220 may receive a query request 142 from a user 210 and generate a response 145 to the query request 142 using a machine learning model 105.

[0045] The query request 142 may include content of at least one modality, and the at least one modality may include any appropriate modality such as video, image, text, audio, etc. As an example, the query request 142 may include an image 202 and a query text 204 for the image 202. When including data of multiple different modalities, the query request 142 may also be referred to as a multimodal query request. The image 202 may be an image that the user 210 attempts to capture via his or her corresponding image capture equipment, or may be an image that is captured in advance and uploaded in response to receiving a user selection.

[0046] The trained machine learning model 105 can be based on any appropriate model structure, including but not limited to a Transformer model, a convolutional neural network (CNN), a recurrent neural network (RNN), a deep neural network (DNN), etc. In some embodiments, the machine learning model 105 can be based on a language model (LM). The machine learning model 105 can also be based on other appropriate models.

[0047] The machine learning model 105 may include one or more machine learning models. It should be noted that if the machine learning model 105 includes multiple machine learning models, these multiple machine learning models may have different structures, uses and functions, and the present disclosure does not limit this. As an example only, the machine learning model 105 may include a multimodal language model (MLLM), a visual language model (VLM), and the like.

[0048] The machine learning model 105 may include at least a visual encoder model 232 (Visual encoder), an adapter model 234 (Adapter), and a language model 236. The visual encoder model 232 is configured to extract visual features from a received image. The adapter model 234 is configured to map the visual features extracted by the visual encoder model 232 into a portion of the input features of the language model 236, so that the language model 236 can understand and process the visual features. The language model 236 can have question-answering capabilities by learning from a large amount of corpus, and can determine specific requirements in the input based on the received input, and generate corresponding responses to meet the determined requirements.

[0049] In some embodiments, the machine learning model 105 may also include a text encoder model (not shown in the figure), which is configured to extract text features of text data. In some embodiments, the text encoder model and the language model 236 can be regarded as a whole, so the text features extracted by the text encoder do not need to be feature mapped by the corresponding adapter model. The input of the language model 236 may include the feature portion mapped from the visual feature by the adapter model 234, and the text feature portion mapped by the text encoder.

[0050] The electronic device 220 may be any type of device with computing capabilities, including a terminal device or a server device. Figure 1 At least one of the data collection system 110, the model pre-training system 120, the model fine-tuning system 130, and the model application system 140 in the environment 100. It should be understood that the structure and function of each element in the environment 100 are described only for exemplary purposes and do not imply any limitation on the scope of the present disclosure.

[0051] For training Figure 1 and Figure 2 The training data required for the machine learning model 105 mentioned above includes at least images and annotation data for the images, that is, description information of the images. The description information of the images can be in the form of text. Generally, images can be collected from multiple data sources, but accurate description information of the images also needs to be obtained through automatic means to improve the efficiency of generating training data for the model.

[0052] Some example embodiments of the present disclosure will be described below with reference to the accompanying drawings. The image processing process involved in the present disclosure can be implemented to obtain image and its matching description information. The obtained image and its description information can be used for model training as needed.

[0053] refer to Figure 3 , Figure 3An example process 300 for image processing according to some embodiments of the present disclosure is shown. The process 300 can be implemented in any appropriate electronic device or system. In the following description, an example description is given by taking the process 300 being implemented in the data acquisition system 110 as an example.

[0054] like Figure 3 As shown, the example process 300 may involve four stages from stage 301 to stage 304, and these four stages may be executed sequentially. Stage 301 may also be referred to as a data selection stage, a data filtering stage, an image selection stage, etc. In stage 301, the data acquisition system 110 may obtain access information corresponding to a candidate image in at least one data source, where the data source may include any appropriate data source such as a website, a database, etc., to which the data acquisition system 110 has access rights. The data acquisition system 110 may parse the access information to obtain a parsed access address, which may be a uniform resource locator (URL), which may identify the location of the candidate image.

[0055] The data collection system 110 can extract multiple candidate images in the image library from the resolved access address to obtain a candidate image set 310 including multiple candidate images. The data collection system 110 can filter the multiple candidate images in the candidate image set 310 to remove images with poor quality from the multiple candidate images. The data selection unit 320 in the data collection system 110 can determine the first image set and the second image set for image processing from the filtered candidate image set 310 (at this time, the candidate image set only includes multiple candidate images with good quality).

[0056] In some embodiments, the number of sample images in the second image set may be greater than the number of sample images in the first image set. As an example, the electronic device 110 may select a small portion from a plurality of candidate images, and determine the small portion of candidate images as the first image set, and determine the remaining portion of the plurality of candidate images (that is, most of the plurality of candidate images) as the second image set. The candidate images in the first image set may be referred to as first sample images, and the candidate images in the second image set may be referred to as second sample images. As an example, the data selection unit 320 may select a small portion of images (e.g., 30% of the plurality of candidate images) from the image set 330 to determine the image set 330 (which may correspond to the first image set).

[0057] Stage 302 may also be referred to as a description information determination stage. In stage 302, the data collection system 110 may obtain prompt word information 342 (also referred to as first prompt word information) for a content generation model 340 (also referred to as a first content generation model). The prompt word information 342 may be predetermined prompt word information used to guide the content generation model 340 to generate description information that meets quality requirements. The content generation model 340 may be based on any appropriate model structure. In some embodiments, the content generation model 340 may be based on a language model. As an example only, the content generation model 340 may be a multimodal language model (MLLM).

[0058] The data collection system 110 can use the trained content generation model 340 (i.e., the first content generation model) to generate description information 350 that matches each of the multiple sample images in the image set 330 based on the image set 330 including the multiple first sample images and the prompt word information 342. Specifically, the data collection system 110 can determine the model input for the content generation model 340 based on the multiple first sample images in the image set 330 and the prompt word information 342, and obtain the model output of the content generation model 340 for the model input by providing the model input to the content generation model 340.

[0059] Based on the model output, description information 350 for each of the plurality of first sample images may be determined. As an example, if 10 first sample images are included, the content generation model 340 may be used to output description information (e.g., 10 paragraphs of description text) corresponding to each of the 10 first sample images. The description information for each image may indicate which objects the corresponding image includes, the number of objects included in the corresponding object, the positional relationship of the objects included in the corresponding image, the color style of the corresponding image, which texts are included in the corresponding image, the background of the corresponding image, etc. The present disclosure does not limit the content of the specific description information. The description information 350 may, for example, include any appropriate number of description texts.

[0060] Further, in stage 303 (which may also be referred to as the training stage of the second content generation model), the data acquisition system 110 may train a content generation model 360 (which may also be referred to as the second content generation model) based on multiple first sample images in the image set 330 and the description information 350 for the multiple first sample images in the image set 330 determined in stage 302. The content generation model 360 is configured to generate description information that matches the input image. The content generation model 360 may also be based on any appropriate model structure, which is not limited by the present disclosure. As an example only, the content generation model 360 may also be based on a language model. In some embodiments, the model scale of the content generation model 360 may be smaller than the model scale of the content generation model 340.

[0061] The model scale (also called model size) of each machine learning model is associated with the parameter scale, model structure complexity, number of model layers, model parameter amount, model training data scale, etc. of the machine learning model. Generally, the larger the parameter scale of the machine learning model, the more complex the structure, the more layers of the model, the larger the model parameter amount, or the larger the training data scale, the larger the model scale of the machine learning model. The larger the model scale, the greater the resource overhead required for the machine learning model, including but not limited to computing resources, memory resources, time consumption, etc.

[0062] The reason why the content generation model 340 with a larger model scale is not used to generate the description information of the entire image set is that the use of a more complex and larger model usually results in higher resource overhead. Therefore, in the embodiment of the present disclosure, a larger content generation model 340 is used to generate high-quality description information for a portion of the images, and then the portion of the images and their description information are used to train a new content generation model 360. In this way, the content generation model 360 can be trained to obtain the image annotation capability of the content generation model 340, and can also run with lower resource overhead. The content generation model 360 trained in this way is more suitable for performing image annotation on large-scale images.

[0063] In some embodiments, the data collection system 110 may also obtain prompt word information for the content generation model 360, and the prompt word information may be configured to guide the content generation model 360 to generate description information for the received image. The data collection system 110 may determine the model input for the content generation model 360 based on the plurality of first sample images and the prompt word information for the content generation model 360, and obtain the model output for the model input output by the content generation model 360 by providing the model input to the content generation model 360. The predicted description information 370 for each of the plurality of first sample images may be determined based on the model output. The content generation model 360 may be trained based on the difference between the description information 350 and the predicted description information 370 for each of the plurality of first sample images. In response to determining that the difference is less than a threshold, it may be determined that the training of the content generation model 360 is completed.

[0064] After completing the training of the content generation model 360, the trained content generation model 360 can be used to generate description information that matches each of the multiple second sample images based on the second image set input including the multiple second sample images. In some embodiments, after completing the training of the content generation model 360, the content generation model 360 can also be used to verify the model performance using a predetermined test set, and can only be put into use after the model performance verification passes. The test set also includes sample images and description information that matches the sample images. If the description information output by the trained content generation model 360 is very similar to the description information in the test set, then it is considered that the performance test of the model has passed. That is, the data acquisition system 110 can use the trained and verified content generation model 360 to generate description information that matches each of the multiple second sample images based on the second image set input including the multiple second sample images.

[0065] The data collection system 110 may, for example, determine the remaining portion of the candidate image set 310 after removing the image set 330 as the second image set, that is, the remaining portion of the images in the candidate image set 310 may be determined as a plurality of second sample images. In stage 304, the data collection system 110 may generate a model input for the trained content generation model 360 based at least on the plurality of second sample images in the candidate image set 310 (for example, it may also be based on the prompt word information for the content generation model 360), and provide the model input to the trained content generation model 360. The data collection system 110 may obtain the model output of the trained content generation model 360, and determine the description information 380 for the plurality of second sample images based on the model output.

[0066] In some embodiments, a parallel scheduling architecture may be used to determine description information 380 for the plurality of second sample images using the trained content generation model 360. In some embodiments, the data collection system 110 may divide the plurality of second sample images into a plurality of image subsets, each image subset including at least one second sample image. Figure 4 , Figure 4 An example 400 for image processing according to some embodiments of the present disclosure is shown. The data acquisition system 110 may, for example, divide a plurality of second sample images based on a plurality of tasks 410 (e.g., may include task 410-1, task 410-2, ..., task 410-K, where K is a positive integer, and for ease of description, one or more tasks may be collectively referred to as task 410), and each of the divided tasks 410 corresponds to an image subset. Different tasks may correspond to different task types, and different task types may correspond to different categories. Task types may, for example, include counting, mathematical reasoning, task understanding, task planning, and the like.

[0067] The data acquisition system 110 can use the trained content generation model 360 to process multiple image subsets in parallel according to the parallel scheduling mechanism of the content generation model 360 to obtain description information that matches each of the multiple second sample images. Specifically, for each image subset in the multiple image subsets, the data acquisition system 110 can assign the second sample images in the image subset to multiple work nodes. For example, the image subset corresponding to task 410-2 can be assigned to multiple work nodes 420 corresponding to task 410-2 (for example, it can include work node 420-1, work node 420-2, ..., work node 420-M, where M is a positive integer, and for the convenience of description, one or more work nodes can be collectively referred to as work node 420).

[0068] At least a content generation model 360 is deployed on each work node. In some embodiments, each work node may also be deployed with multiple processing units to provide computing resources required for model reasoning of the content generation model 360. The processing unit may be, for example, a graphics processing unit (GPU). For example, the work node 420-2 may be deployed with multiple processing units 430 (for example, it may include processing unit 430-1, processing unit 430-2, ..., processing unit 430-N, where N is a positive integer, and for the convenience of description, one or more processing units may be collectively referred to as processing unit 430).

[0069] The assigned second sample images may be processed in parallel by multiple working nodes using the content generation model 360 to obtain description information matching each second sample image. For example, for task 410-2, the assigned second sample images may be processed by multiple working nodes 420 using the content generation model 360 deployed at the respective working nodes. It is understood that the second sample images assigned to different working nodes may be different. Thus, it is possible to process multiple second sample images simultaneously, and the efficiency of determining image description information may be improved.

[0070] In some embodiments, the electronic device 110 may determine a training data set 390 for the machine learning model 105 based at least on the plurality of second sample images and the description information 380 that matches each of the plurality of second sample images, and the training data set 390 is used to train the machine learning model 105. As mentioned above, the machine learning model 105 is configured to process a multimodal query request and generate a response to the multimodal query request, and the multimodal query request includes at least image data and text data. The machine learning model 105 includes at least: a visual encoder model (e.g., the visual encoder model 232), an adapter model (e.g., the adapter model 234), and a language model (e.g., the language model 236).

[0071] The training data set 390 may be used to pre-train the machine learning model 105 and / or fine-tune the machine learning model 105. That is, the pre-training data set 112 and / or the training data set 114 for the machine learning model 105 may be determined based on the training data set 309. In the pre-training stage for the machine learning model 105, the model pre-training system 120 may first use a portion of the pre-training data set 112 to adjust the parameters of the adapter model 234, and keep the parameters of the visual encoder model 232 and the language model 236 unchanged. After the parameters of the adapter model 234 are adjusted, the other portions of the pre-training data set 112 may be used to adjust the parameters of the visual encoder model 232 and the adapter model 234, and keep the parameters of the language model 236 unchanged.

[0072] In the fine-tuning stage for the machine learning model 105, the model fine-tuning system 130 can use the training data set 114 to perform at least one stage of fine-tuning on the machine learning model 105, and the training data corresponding to different fine-tuning stages can be different. As an example only, if three fine-tuning stages are included, the training data set 114 can be divided into three training data subsets based on the training purpose of each fine-tuning stage. It can be understood that at least one fine-tuning stage can include any other appropriate number of fine-tuning stages.

[0073] In some embodiments, in the first fine-tuning stage of at least one fine-tuning stage, the model fine-tuning system 130 may divide the training data subset corresponding to the fine-tuning stage into multiple sets, each set including a sample image, a sample text, and a response label corresponding to the sample text. In a multimodal response scenario, the sample text is also a query text, and the response label may indicate a response to the query text. Each sample image in the multiple sets is divided into multiple image blocks of different sizes. The different sizes may include any appropriate sizes such as 48×48, 124×124, etc., and the sizes in actual applications may be flexibly set based on user needs. The visual encoder model 232 is configured to extract visual feature units one by one according to the divided image blocks, and the visual feature unit corresponding to each image block may be mapped by the adapter model 124 into an input unit (also called a token) of the language model 236.

[0074] The model fine-tuning system 130 can use multiple sets to fine-tune the pre-trained machine learning model 105 to obtain an updated machine learning model 105. For the training samples in each set, the model fine-tuning system 130 can use the pre-trained machine learning model 105 to extract visual features from multiple image blocks divided from the sample image in the training sample and extract text features from the sample text in the training sample. The text features can be extracted from the sample text by a separate text encoder model in the machine learning model 105, or can be extracted from the sample text by a text encoder in the language model 105. The model fine-tuning system 130 can use the pre-trained machine learning model 105 to generate a predicted response based on visual features and text features, and fine-tune the pre-trained machine learning model 105 based on the difference between the predicted response and the response label in the training sample.

[0075] In some embodiments, the data quality of the training samples corresponding to the fine-tuning stages in at least one fine-tuning stage will be successively higher than the data quality of the training samples corresponding to the previous fine-tuning stages. For example, the data quality of the training samples corresponding to the second fine-tuning stage will be higher than the data quality of the training samples corresponding to the first fine-tuning stage, the data quality of the training samples corresponding to the third fine-tuning stage will be higher than the data quality of the training samples corresponding to the second fine-tuning stage, and so on. Thus, in at least one fine-tuning stage, the data quality of the training data used can be successively improved, so that the machine learning model 105 can gradually master the model inputs in a rich form, and gradually improve the quality of the machine learning model 105, so that the fine-tuned machine learning model 105 can output complex and logical responses, which helps to improve the accuracy of the responses output by the machine learning model 105.

[0076] In some embodiments, in each pre-training stage and each fine-tuning stage for the machine learning model 105, an autoregressive method can be used to update the machine learning model 105. The language model 236 in the machine learning model 105 can predict the first N tokens of the input (for example, the tokens shown in the figure) each time. <user>To Token <am>) predicts the next N+1th token (e.g. the token shown in the figure <fine>). During training, in order to complete the training of the entire text at one time, an attention mask can be used to constrain the calculation of the self-attention mechanism of the language model 236 to ensure that the network can only use the previous features at each token. For example, the following formula can be used to determine the cross entropy loss predicted using each token, and the model loss can be optimized based on the cross entropy loss:

[0077]

[0078] In the scenario of multimodal input, text tokens can be replaced with text tokens corresponding to the question (i.e., text features) and visual tokens corresponding to the image (i.e., visual feature comments mapped via the adapter model 124). The output of the language model 236 includes predictions of each token corresponding to the response. The predicted tokens can be converted into corresponding text and / or images (depending on the language model 236's understanding of the input requirements and content generation). In pre-training or fine-tuning, the parameters of one or more components in the model can be continuously updated based on the difference between the response predicted by the language model 236 and the actual response given in the pre-training dataset or the training dataset (depending on the different stages, the parameters to be updated are also different). Ultimately, the machine learning model 105 as a whole is trained to minimize the difference between the predicted response and the actual response (i.e., the training loss) or achieve the expected goal.

[0079] In pre-training or subsequent fine-tuning, the model parameters are updated by minimizing or reducing the training loss at each stage. In the training algorithm, the gradient of the model loss for the model parameters to be updated can be calculated, and the model parameters can be updated by the stochastic gradient descent method, so as to minimize the loss value or continuously reduce it to the expected target.

[0080] In summary, according to the embodiments of the present disclosure, a more complex and more powerful content generation model can be used to generate high-quality description information, and the generated description information and its images can be used as training data for training a lighter content generation model. The lightweight content generation model trained in this way can generate description information for a large number of images. In this way, a large amount of description information that meets the requirements can be obtained as annotation data for the image, which is used to train the machine learning model for multimodal response. In this way, training data for the machine learning model for multimodal response can be generated quickly and conveniently, which can help improve the efficiency of determining training data, ensure the quality of training data, and further help improve the training efficiency of the machine learning model and improve the model performance of the machine learning model.

[0081] Figure 5 1 is a flowchart of a method 500 for image processing according to some embodiments of the present disclosure. The method 500 may be implemented in Figure 2 The data acquisition system 110 will refer to Figure 2 The method 500 is described with reference to the environment 200 .

[0082] In block 510 , the data collection system 110 generates description information matching each of the plurality of sample images based on a first image set including a plurality of first sample images and first cue word information by using the trained first content generation model.

[0083] In block 520 , the data collection system 110 trains a second content generation model based on the plurality of first sample images and the description information that matches each of the plurality of sample images, where the second content generation model is configured to generate description information that matches the input image.

[0084] In block 530 , the data collection system 110 generates description information matching each of the plurality of second sample images based on a second image set input including the plurality of second sample images, using the trained second content generation model.

[0085] In some embodiments, the number of sample images in the second image set is greater than the number of sample images in the first image set.

[0086] In some embodiments, the model scale of the first content generation model is greater than the model scale of the second content generation model.

[0087] In some embodiments, method 500 also includes: obtaining access information corresponding to candidate images in at least one data source; parsing the access information to obtain a parsed access address; extracting multiple candidate images in the image library from the parsed access address; and selecting a first image set and a second image set from the multiple candidate images.

[0088] In some embodiments, generating description information that matches each of the multiple second sample images includes: dividing the multiple second sample images into multiple image subsets, each image subset including at least one second sample image; and according to a parallel scheduling mechanism for the second content generation model, using a trained second content generation model to process the multiple image subsets in parallel to obtain description information that matches each of the multiple second sample images.

[0089] In some embodiments, using a trained second content generation model to process multiple image subsets in parallel includes: for each image subset in the multiple image subsets, assigning the second sample images in the image subset to multiple working nodes, each working node having a second content generation model deployed thereon; and using the second content generation model by the multiple working nodes to process the assigned second sample images to obtain description information matching each second sample image.

[0090] In some embodiments, each of the multiple working nodes is deployed with multiple processing units for providing computing resources required for model inference of the second content generation model.

[0091] In some embodiments, the first prompt word information is configured to guide the first content generation model to generate description information that meets quality requirements.

[0092] In some embodiments, method 500 also includes: training a machine learning model based at least on multiple second sample images and descriptive information matching each of the multiple second sample images, the machine learning model being configured to process a multimodal query request and generate a response to the multimodal query request, the multimodal query request including at least image data and text data.

[0093] The embodiments of the present disclosure also provide corresponding devices for implementing the above methods or processes. Figure 6 An exemplary structural block diagram of an apparatus 600 for image processing according to some embodiments of the present disclosure is shown. The apparatus 600 may be implemented as or included in the data acquisition system 110. Each module / component in the apparatus 600 may be implemented by hardware, software, firmware, or any combination thereof.

[0094] like Figure 6 As shown, the device 600 includes a first description information generation module 610, which is configured to use a trained first content generation model to generate description information that matches each of the multiple sample images based on a first image set including multiple first sample images and first prompt word information. The device 600 also includes a content generation model training module 620, which is configured to train a second content generation model based on the multiple first sample images and the description information that matches each of the multiple sample images, and the second content generation model is configured to generate description information that matches the input image. The device 600 also includes a second description information generation module 630, which is configured to use a trained second content generation model to generate description information that matches each of the multiple second sample images based on a second image set input including multiple second sample images.

[0095] In some embodiments, the number of sample images in the second image set is greater than the number of sample images in the first image set.

[0096] In some embodiments, the model scale of the first content generation model is greater than the model scale of the second content generation model.

[0097] In some embodiments, the device 600 also includes: an access information acquisition module, configured to obtain access information corresponding to a candidate image in at least one data source; an access address acquisition module, configured to parse the access information to obtain a parsed access address; a candidate image extraction module, configured to extract multiple candidate images in an image library from the parsed access address; and an image set selection module, configured to select a first image set and a second image set from multiple candidate images.

[0098] In some embodiments, the second description information generation module 630 is further configured to: divide the multiple second sample images into multiple image subsets, each image subset including at least one second sample image; and according to a parallel scheduling mechanism for the second content generation model, use the trained second content generation model to process the multiple image subsets in parallel to obtain description information that matches each of the multiple second sample images.

[0099] In some embodiments, the second description information generation module 630 is further configured to: for each image subset in multiple image subsets, distribute the second sample images in the image subset to multiple working nodes, each working node having a second content generation model deployed thereon; and the multiple working nodes use the second content generation model to process the distributed second sample images to obtain description information matching each second sample image.

[0100] In some embodiments, each of the multiple working nodes is deployed with multiple processing units for providing computing resources required for model inference of the second content generation model.

[0101] In some embodiments, the first prompt word information is configured to guide the first content generation model to generate description information that meets quality requirements.

[0102] In some embodiments, the device 600 also includes: a model training module, configured to train a machine learning model based at least on multiple second sample images and description information matching each of the multiple second sample images, the machine learning model being configured to process a multimodal query request and generate a response to the multimodal query request, the multimodal query request including at least image data and text data.

[0103] The units and / or modules included in the device 600 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units and / or modules can be implemented using software and / or firmware, such as machine executable instructions stored on a storage medium. In addition to or as an alternative to machine executable instructions, some or all of the units and / or modules in the device 600 can be implemented at least in part by one or more hardware logic components. As an example and not limitation, exemplary types of hardware logic components that can be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0104] It should be understood that one or more steps in the above method can be performed by a suitable electronic device or a combination of electronic devices. Such an electronic device or a combination of electronic devices may include, for example, Figure 2 The electronic device 220 in.

[0105] Figure 7 1 shows a block diagram of an electronic device 700 in which one or more embodiments of the present disclosure may be implemented. It should be understood that Figure 7 The electronic device 700 shown is merely exemplary and should not constitute any limitation on the functionality and scope of the embodiments described herein. Figure 7 The electronic device 700 shown can be used to implement Figure 1 Data acquisition system 110 or Figure 6 Device 600.

[0106] like Figure 7 As shown, the electronic device 700 is in the form of a general electronic device. The components of the electronic device 700 may include, but are not limited to, one or more processors or processing units 710, a memory 720, a storage device 730, one or more communication units 740, one or more input devices 750, and one or more output devices 760. The processing unit 710 may be an actual or virtual processor and is capable of performing various processes according to a program stored in the memory 720. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to improve the parallel processing capability of the electronic device 700.

[0107] The electronic device 700 typically includes a plurality of computer storage media. Such media may be any accessible media that is accessible to the electronic device 700, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 720 may be a volatile memory (e.g., registers, caches, random access memory (RAM)), a non-volatile memory (e.g., a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 730 may be a removable or non-removable medium, and may include a machine-readable medium, such as a flash drive, a disk, or any other medium, which may be capable of being used to store information and / or data and may be accessed within the electronic device 700.

[0108] The electronic device 700 may further include additional removable / non-removable, volatile / non-volatile storage media. Figure 7 As shown in , a disk drive for reading or writing from a removable, non-volatile disk (e.g., a "floppy disk") and an optical drive for reading or writing from a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to the bus (not shown) by one or more data media interfaces. The memory 720 may include a computer program product 725 having one or more program modules that are configured to perform various methods or actions of various embodiments of the present disclosure.

[0109] The communication unit 740 implements communication with other electronic devices through a communication medium. Additionally, the functions of the components of the electronic device 700 can be implemented with a single computing cluster or multiple computing machines that can communicate through a communication connection. Therefore, the electronic device 700 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

[0110] The input device 750 may be one or more input devices, such as a mouse, a keyboard, a tracking ball, etc. The output device 760 may be one or more output devices, such as a display, a speaker, a printer, etc. The electronic device 700 may also communicate with one or more external devices (not shown) through the communication unit 740 as needed, such as a storage device, a display device, etc., communicate with one or more devices that allow a user to interact with the electronic device 700, or communicate with any device that allows the electronic device 700 to communicate with one or more other electronic devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0111] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above.

[0112] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices, equipment, and computer program products implemented according to the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0113] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0114] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0115] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple implementations of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of a module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some implementations as an update, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the function involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0116] The above descriptions of various implementations of the present disclosure are exemplary, non-exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described implementations. The selection of terms used herein is intended to best explain the principles of the implementations, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the various implementations disclosed herein.< / fine> < / am> < / user>

Claims

1. A method for image processing, comprising: Generate description information matching each of the plurality of sample images based on a first image set including a plurality of first sample images and first prompt word information by using the trained first content generation model; Based on the plurality of first sample images and the description information respectively matching the plurality of sample images, training a second content generation model, wherein the second content generation model is configured to generate description information matching the input image; as well as Using the trained second content generation model, based on a second image set input including a plurality of second sample images, description information matching each of the plurality of second sample images is generated. 2 . The method according to claim 1 , wherein the number of sample images in the second image set is greater than the number of sample images in the first image set. 3 . The method according to claim 1 , wherein a model scale of the first content generation model is greater than a model scale of the second content generation model.

4. The method according to claim 1, further comprising: Obtaining access information corresponding to the candidate image in at least one data source; Parsing the access information to obtain a parsed access address; Extracting a plurality of candidate images in the image library from the parsed access address; as well as The first set of images and the second set of images are selected from the plurality of candidate images.

5. The method according to claim 1, wherein generating description information matching each of the plurality of second sample images comprises: dividing the plurality of second sample images into a plurality of image subsets, each image subset including at least one second sample image; as well as According to the parallel scheduling mechanism for the second content generation model, the trained second content generation model is used to process the multiple image subsets in parallel to obtain the description information matching each of the multiple second sample images.

6. The method of claim 5, wherein processing the plurality of image subsets in parallel using the trained second content generation model comprises: For each image subset in the plurality of image subsets, Distributing the second sample image in the image subset to a plurality of working nodes, each working node being deployed with the second content generation model; as well as The plurality of working nodes use the second content generation model to process the allocated second sample images to obtain description information matching each second sample image.

7. The method according to claim 6, wherein each of the multiple working nodes is deployed with multiple processing units for providing computing resources required for model reasoning of the second content generation model. 8 . The method according to claim 1 , wherein the first prompt word information is configured to guide the first content generation model to generate description information that meets quality requirements.

9. The method according to claim 1, further comprising: A machine learning model is trained based at least on the multiple second sample images and the description information matching each of the multiple second sample images, and the machine learning model is configured to process a multimodal query request and generate a response to the multimodal query request, wherein the multimodal query request includes at least image data and text data.

10. An apparatus for image processing, comprising: A first description information generating module is configured to generate description information matching each of the plurality of sample images based on a first image set including a plurality of first sample images and first prompt word information by using a trained first content generation model; a content generation model training module, configured to train a second content generation model based on the plurality of first sample images and the description information that matches each of the plurality of sample images, wherein the second content generation model is configured to generate description information that matches an input image; as well as The second description information generation module is configured to use the trained second content generation model and, based on a second image set input including a plurality of second sample images, generate description information matching each of the plurality of second sample images.

11. An electronic device, comprising: at least one processor; as well as At least one memory, the at least one memory is coupled to the at least one processor and stores instructions for execution by the at least one processor, the instructions causing the electronic device to perform the method according to any one of claims 1 to 9 when executed by the at least one processor.

12. A computer-readable storage medium having computer-executable instructions stored thereon, wherein the computer-executable instructions can be executed by a processor to implement the method according to any one of claims 1 to 9.

13. A computer program product comprising computer executable instructions, wherein the computer executable instructions, when executed by a processor, implement the method according to any one of claims 1 to 9.

Images