Method and device for generating text description of image, electronic equipment and medium

Abstract

The embodiment of the invention relates to a method and device for generating text description of an image, electronic equipment and a medium. The method includes generating, by a visual encoder, a first feature of an image, a text encoder semantically aligned with the visual encoder for training a transformation model and a language model. The method further includes converting, by a conversion model, the first feature into a second feature, the first feature and the second feature corresponding to different feature spaces. In addition, the method includes generating, by the language model, a textual description of the image based on the second feature. According to the embodiment of the invention, the text encoder and the language model are subjected to semantic alignment in the training process, so that the text encoder is used as an intermediate medium to enable the visual encoder and the language model to be subjected to semantic alignment, and the influence of low-quality image text data on the model is avoided; and the performance of the model and the accuracy of text description of the image are improved.

Description

Technical Field

[0001] The present disclosure relates generally to the field of computers, and more particularly, to methods, apparatuses, electronic devices, and media for generating text descriptions of images. Background Art

[0002] With the development of technology, the application of neural network models such as large language models has become more and more widespread. Among them, large language models refer to neural network models trained with a large amount of text data, which have important application value in the fields of situational dialogue, text generation, language classification, etc.

[0003] With the increase in usage scenarios and usage requirements, multimodal large language models have gradually become an important development direction for natural language processing. In multimodal application scenarios, large language models need to process not only text input, but also voice, image, video and other inputs. Therefore, accurate and efficient input content understanding and learning capabilities are particularly important for multimodal large language models. Summary of the invention

[0004] Embodiments of the present disclosure provide a method, an apparatus, an electronic device, and a medium for generating a text description of an image.

[0005] In a first aspect of the present disclosure, a method for generating a text description of an image is provided. The method includes generating a first feature of the image by a visual encoder, and a text encoder semantically aligned with the visual encoder is used to train a conversion model and a language model. The method also includes converting the first feature into a second feature by the conversion model, and the first feature and the second feature correspond to different feature spaces. In addition, the method also includes generating a text description of the image based on the second feature by a language model.

[0006] In a second aspect of the present disclosure, a device for generating a text description of an image is provided. The device includes a first feature generation module configured to generate a first feature of an image by a visual encoder, and a text encoder semantically aligned with the visual encoder is used to train a conversion model and a language model. The device also includes a first feature conversion module configured to convert the first feature into a second feature by the conversion model, and the first feature and the second feature correspond to different feature spaces. In addition, the device also includes a text description generation module configured to generate a text description of the image based on the second feature by a language model.

[0007] In a third aspect of the present disclosure, an electronic device is provided, comprising a processor and a memory coupled to the processor, wherein the memory has instructions stored therein, and when the instructions are executed by the processor, the electronic device executes the method according to the first aspect.

[0008] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer-executable instructions, wherein the computer-executable instructions are executed by a processor to implement the method according to the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0010] Figure 1 A schematic diagram illustrating an example environment in which some embodiments of the present disclosure may be implemented;

[0011] Figure 2 A flowchart illustrating a method for generating a text description of an image according to some embodiments of the present disclosure is shown;

[0012] Figure 3 A schematic diagram showing a process of semantic alignment of a text encoder and an image encoder according to some embodiments of the present disclosure;

[0013] Figure 4 A schematic diagram showing a process of training a conversion model and a large language model according to some embodiments of the present disclosure is shown;

[0014] Figure 5 A schematic diagram showing a process of fine-tuning an image encoder according to some embodiments of the present disclosure;

[0015] Figure 6 A schematic diagram showing a process of generating a text description of an image according to some embodiments of the present disclosure;

[0016] Figure 7 A schematic diagram showing the process of training and reasoning of a model in some embodiments of the present disclosure is shown;

[0017] Figure 8 A block diagram showing an apparatus for generating a text description of an image according to some embodiments of the present disclosure;

[0018] Fig. 9 A block diagram of an electronic device according to some embodiments of the present disclosure is shown.

[0019] Throughout the drawings, the same or similar reference numbers denote the same or similar elements. DETAILED DESCRIPTION

[0020] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0021] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0022] For example, when receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0023] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0024] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0025] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0026] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc. may refer to different or the same objects, unless explicitly stated. Other explicit and implicit definitions may also be included below.

[0027] A multimodal large language model refers to a language model that generates text descriptions based on multimodal information, where the multimodal information mainly includes visual information and text information. Therefore, the multimodal large language model has important application value in the fields of intent understanding, summary extraction, and content creation. Traditionally, in order to obtain a high-quality multimodal large language model to generate a text description of an image, it is often necessary to train the multimodal large language model using an image-text dataset. However, since image-text datasets often contain a large amount of low-quality data (for example, images and texts have poor correlation), the performance of the trained multimodal large language model in the traditional scheme is low, which in turn leads to inaccurate text descriptions of the image.

[0028] In the embodiments of the present disclosure, a semantically aligned visual encoder and a text encoder are first determined, and the conversion model and the large language model are trained through the text encoder so that the text encoder and the large language model are semantically aligned, thereby achieving semantic alignment of the visual encoder and the large language model using the text encoder as an intermediate medium. Then, the features of the image can be extracted through the visual encoder (which can be called the first feature) and the features can be converted to the feature space corresponding to the large language model through the conversion model (the converted features can be called the second features), and the converted features are processed by the large language model to generate a text description of the image.

[0029] In this way, the visual encoder and the large language model can be quickly semantically aligned during the training process using the text encoder as an intermediate medium. Since the training process only involves adjusting the parameters in the conversion model and the large language model based on text data, the negative impact of low-quality data in the image text dataset during the training process can be avoided, thereby improving the accuracy of semantic alignment of the visual encoder and the large language model during the training process, thereby improving the overall model quality. During the inference process, since the visual encoder has been aligned with the large language model, the large language model can accurately generate text descriptions based on the converted image features.

[0030] Figure 1 1 is a schematic diagram of an example environment 100 in which some embodiments of the present disclosure may be implemented. Figure 1 As shown, the example environment 100 can be set as a computing device 102, wherein the computing device 102 can be set as a computing system, a single server, a distributed server, or a cloud-based server, etc., and can also be set as a user terminal, a mobile device, a computer, etc., and can also be set as a combination of the above devices. For example, the computing device 102 can be a combination of a server and a user terminal, and interaction with the user is achieved based on the combination.

[0031] refer to Figure 1, the computing device 102 stores a visual encoder 108, a text encoder 110, a conversion model 112, and a large language model 114. Among them, the visual encoder 108 and the text encoder 110 have achieved semantic alignment, that is, for the same image-text pair in the image-text data set, the image features output by the visual encoder 108 and the text features output by the text encoder 110 have a high similarity, and for different image-text pairs in the image-text data set, the image features output by the visual encoder 108 and the text features output by the text encoder 110 have a low similarity. It can be understood that the higher the similarity for the same image-text pair, and the lower the similarity for different image-text pairs, the better the semantic alignment effect of the visual encoder 108 and the text encoder 110.

[0032] In some embodiments, continue to refer to Figure 1 During the training process, the encoder 110 extracts the text features of the text 106, and the text features are input into the conversion model 112 for feature conversion, so that the text features meet the feature space requirements of the input features of the large language model 114, and the large language model 114 extracts the training text description 118 corresponding to the text 106 based on the converted text features. It should be noted that since only the text is used to train the conversion model 112 and the large language model 114 during the training process, there is no restriction on the source of the text 106 in this embodiment, that is, the text 106 can be text from the image text dataset or other text outside the image text dataset.

[0033] Then, after the large language model 114 generates a training text description 118 for the text 106, the loss 120 is determined based on the text 106 and the training text description 118, and the parameters in the conversion model 112 and the large language model 114 are trained by back propagation, and the parameters in the visual encoder 108 and the text encoder 110 are frozen during the training process. After multiple iterations, when the loss 120 meets the convergence condition, the training of the conversion model 112 and the large language model 114 can be completed. It can be understood that after the training is completed, the text 106 and the training text description 118 are aligned, so the text encoder 110 and the large language model 114 are semantically aligned, and then the visual encoder 108 and the large language model 114 are also semantically aligned.

[0034] In some embodiments, continue to refer to Figure 1In the inference phase, an image 104 is acquired and image features of the image 104 are extracted through a visual encoder 108, and then the image features are converted through a trained conversion model 112, and the converted image features are input into a trained large language model 114. The large language model 114 extracts a text description 116 of the image 104 based on the converted image features.

[0035] In this way, the visual encoder 108 and the large language model 114 can be quickly semantically aligned using the text encoder 110 as an intermediate medium during the training process, and since the training process only involves adjusting the parameters in the conversion model 112 and the large language model 114 based on the text data, the negative impact of low-quality data in the image text data set during the training process can be avoided, thereby improving the accuracy of semantic alignment of the visual encoder 108 and the large language model 114 during the training process, thereby improving the overall model quality. During the reasoning process, since the visual encoder 108 has been aligned with the large language model 114, the large language model 114 can accurately generate a text description based on the converted image features.

[0036] It should be understood that the architecture and functions in the example environment 100 are described for exemplary purposes only and do not imply any limitation on the scope of the present disclosure. Embodiments of the present disclosure may also be applied to other environments with different structures and / or functions.

[0037] The following will combine Figures 2 to 9 The process according to the embodiment of the present disclosure is described in detail. For ease of understanding, the specific data mentioned in the following description are exemplary and are not intended to limit the scope of protection of the present disclosure. It is understood that the embodiments described below may also include additional actions not shown and / or the actions shown may be omitted, and the scope of the present disclosure is not limited in this respect.

[0038] Figure 2 2 is a flowchart of a method 200 for generating a text description of an image according to some embodiments of the present disclosure. In some embodiments, Figure 1 The computing device 102 in can serve as the execution subject of the method 200.

[0039] In block 202, a first feature of an image is generated by a visual encoder, and a text encoder semantically aligned with the visual encoder is used to train a conversion model and a language model. Figure 1 In the environment 100 shown, an image 104 is obtained, and image features (which may be referred to as first features) of the image 104 are extracted by a visual encoder 108. The visual encoder 108 is semantically aligned with a text encoder 110, and the text encoder 110 is used to train a conversion model 112 and a large language model 114.

[0040] In some embodiments, the image 104 may be directly obtained from a database or received from other devices such as a user terminal. Alternatively or additionally, a video may be obtained from a database or other device, and the image 104 may be extracted from multiple video frames of the video. In some embodiments, multiple video frames of the video may be sampled at a certain time interval, and the sampled video frames may be used as the image 104.

[0041] In some embodiments, during the training process, the text features of the text 106 are extracted by the encoder 110, and the text features are input into the conversion model 112 for feature conversion, so that the text features meet the feature space requirements of the input features of the large language model 114, and the large language model 114 extracts the training text description corresponding to the text 106 based on the converted text features. It should be noted that since only the text is used to train the conversion model 112 and the large language model 114 during the training process, there is no restriction on the source of the text 106 in this embodiment, that is, the text 106 can be text from the image text dataset or other text outside the image text dataset.

[0042] Then, after the large language model 114 generates a training text description 118 for the text 106, the loss 120 is determined based on the text 106 and the training text description 118, and the parameters in the conversion model 112 and the large language model 114 are trained by back propagation, and the parameters in the visual encoder 108 and the text encoder 110 are frozen during the training process. After multiple iterations, when the loss 120 meets the convergence condition, the training of the conversion model 112 and the large language model 114 can be completed. It can be understood that after the training is completed, the text 106 and the training text description 118 are aligned, so the text encoder 110 and the large language model 114 are semantically aligned, and then the visual encoder 108 and the large language model 114 are also semantically aligned.

[0043] At block 204, the first feature is converted into a second feature by a conversion model, the first feature and the second feature corresponding to different feature spaces. Figure 1 In the illustrated environment 100, after the visual encoder 108 extracts the image features of the image 104, the trained conversion model 112 converts the image features (the converted image features may be referred to as second features) so that the feature space of the converted image features is adapted to the feature space of the input features of the large language model 114. In some embodiments, the feature space includes the size of the feature, such as the length of the feature. Alternatively or additionally, the feature space includes the spatial distribution of the feature.

[0044] At block 206, a text description of the image is generated by the language model based on the second feature. Figure 1 In the environment 100 shown, the converted image features are input into the large language model 114, and a text description 116 corresponding to the image 104 is generated by the large language model 114 based on the converted image features. In some embodiments, a prompt content associated with the image 104 can be set, and the large language model 114 generates a text description 116 corresponding to the image 104 based on the converted image features and the prompt content. The prompt content can be used to guide the large language model 114 to generate a corresponding type of text description 116. For example, the prompt content can be set to "Please describe this picture", and the type of the generated text description 116 is the description content of the image 104. In this way, a more accurate and demand-compliant text description 116 can be generated.

[0045] In the embodiments of the present disclosure, the visual encoder and the large language model can be quickly semantically aligned using the text encoder as an intermediate medium during the training process, and since the training process only involves adjusting the parameters in the conversion model and the large language model based on the text data, the negative impact of low-quality data in the image text data set during the training process can be avoided, thereby improving the accuracy of semantic alignment of the visual encoder and the large language model during the training process, thereby improving the overall quality of the model. During the reasoning process, since the visual encoder has been aligned with the large language model, the large language model can accurately generate a text description based on the converted image features.

[0046] Figure 3 A schematic diagram of a process 300 for semantically aligning a text encoder with an image encoder in some embodiments of the present disclosure is shown. As mentioned above, the text encoder 306 and the image encoder 308 are semantically aligned, and can be, for example, a pair of encoders obtained by contrastive learning. Process 300 shows a contrastive learning-based training process for the text encoder 306 and the image encoder 308. In some embodiments, the text encoder 306 can be, for example, a converter network with an attention head, and the image encoder 308 can be various residual networks. The present disclosure does not limit the structure of the text encoder 306 and the image encoder 308.

[0047] In some embodiments, the training data for the text encoder 306 and the image encoder 308 (which may be referred to as a training image-text pair) includes paired training text 302 and training image 304, for example, the training text 302 may be a classification label for the training image 304. Therefore, the training text 302 and the training image 304 as training data are semantically related. The text encoder 306 generates corresponding text features 310 based on the training text 302 in the training data. The image encoder 308 generates corresponding image features 312 based on the training image 304 in the training data. The text encoder 306 and the image encoder 308 are trained by constructing similarities 314 between positive and negative samples for contrastive learning.

[0048] In some embodiments, a text-image alignment model is obtained and trained by a text-image pair so that the visual encoding subnetwork and the text encoding subnetwork in the model can be semantically aligned. Then, the visual encoding subnetwork is set to the image encoder 308 herein, and the text encoding subnetwork is set to the text encoder 306 herein. In this way, the semantically aligned text encoder 306 and image encoder 308 can be quickly trained.

[0049] Figure 4 A schematic diagram of a process 400 for training a conversion model and a large language model in some embodiments of the present disclosure is shown. In some embodiments, during the forward propagation of process 400, text 402 is obtained and input into a text encoder 404, and text features (which may be referred to as first text features) of text 402 are extracted by text encoder 404. The text features are then input into a conversion model 406, and the text features are converted by the conversion model 406 (the converted text features may be referred to as second text features), so that the converted text features can meet the feature space requirements of the input features of the large language model 410.

[0050] Then, the converted text features are processed by the large language model 410 to generate a training text description 412 corresponding to the text 402. In some embodiments, the prompt content 408 and the text features can be processed based on the large language model 410 in combination with the prompt content 408 to generate a training text description 412 corresponding to the text 402. The prompt content 408 can be used to guide the large language model to generate a corresponding type of training text description 412.

[0051] Continue to refer Figure 4In the back propagation process of process 400, the loss 414 in the training process is determined by text 402 and training text description 412, and the loss 414 satisfies the convergence condition by adjusting the parameters in conversion model 406 and large language model 410. When the loss 414 satisfies the convergence condition, it can be determined that the training of conversion model 406 and large language model 410 is completed.

[0052] It can be understood that the training purpose of process 400 is to align the text encoder 404 with the large language model 410 semantically, thereby making the visual encoder (e.g. Figure 3 The image encoder 308 in the image encoder 308 can also be semantically aligned with the large language model 410, so during the training process, only the parameters in the conversion model 406 and the large language model 410 are adjusted, and the parameters in the text encoder 404 are not adjusted (otherwise, although the adjusted text encoder 404 can be aligned with the large language model 410, the visual encoder cannot be aligned with the large language model 410).

[0053] In this way, the visual encoder and the large language model 410 can be quickly semantically aligned using the text encoder 404 as an intermediate medium during the training process. Since the training process only involves adjusting the parameters in the conversion model 406 and the large language model 410 based on the text 402, the negative impact of low-quality image data in the image text dataset on the training process can be avoided, thereby improving the accuracy of the semantic alignment of the visual encoder and the large language model 410 during the training process, thereby improving the overall quality of the model.

[0054] Figure 5 A schematic diagram of a process 500 of fine-tuning an image encoder in some embodiments of the present disclosure is shown. After the conversion model 508 and the large language model 510 are trained in process 400, the image encoder 506 may be further fine-tuned. In some embodiments, an image-text data pair for training is obtained, the image-text data pair including a training image 502 and a training text 504. The image features (which may be referred to as first image features) of the training image 502 are extracted by the image encoder 506, wherein the image encoder 506 and the text encoder (e.g. Figure 4 The image features are then converted by the conversion model 508 (the converted image features may be referred to as second image features), so that the converted image features meet the feature space requirements of the input features of the large language model 510.

[0055] Then, the converted image features are processed by the large language model 510 to generate a fine-tuning text description 514. In some embodiments, a prompt content 512 (which may be referred to as a training prompt content) corresponding to the image-text pair may be obtained, and the prompt content 512 and the converted image features are processed by the large language model 510 to generate a fine-tuning text description 514. The prompt content 512 is used to guide the large language model 510 to generate a fine-tuning text description 514 of a corresponding type.

[0056] Continue to refer Figure 5 , after generating the fine-tuning text description 514, the loss 516 is determined by the training text 502 paired with the training image 502 and the fine-tuning text description 514, and the parameters in the image encoder 506 are fine-tuned by the loss 516. When the loss 516 meets the convergence condition, the trained image encoder 506 is used as the visual encoder 518. In some embodiments, after determining the loss 516, the image encoder 506, the conversion model 508, and the large language model 510 can also be fine-tuned at the same time by the loss 516, so as to improve the accuracy of the overall model.

[0057] In this way, the image encoder 506 can be fine-tuned after being aligned with the large language model 510, thereby further improving the image feature extraction capability of the image encoder 506 and the adaptability of the image encoder 506 to the large language model 510, thereby improving the accuracy of the text description of the image.

[0058] Figure 6 A schematic diagram of a process 600 for generating a text description of an image in some embodiments of the present disclosure is shown. In some embodiments, after the conversion model 606 and the large language model 608 are trained by the text encoder, a visual encoder 604 corresponding to the text encoder is determined, and the image 602 is processed by the visual encoder 604 to generate image features (which may be referred to as first features). The image features are converted by the conversion model 606 (the converted image features may be referred to as second features), so that the converted image features can conform to the feature space of the input features of the large language model 608.

[0059] Then, the converted image features are processed by the large language model 608 to generate a text description 610. In some embodiments, a prompt content 612 corresponding to the image can be obtained, and the prompt content 612 and the converted image features are processed by the large language model 608 to generate a text description 610 for the image 602. The prompt content 612 is used to guide the large language model 608 to generate a corresponding type of text description 610. In this way, during the reasoning process, since the visual encoder 604 has been aligned with the large language model 608, the large language model 608 can accurately generate a text description based on the converted image features.

[0060] Figure 7 A schematic diagram of a process 700 of training and reasoning of a model of some embodiments of the present disclosure is shown. In some embodiments, in the training phase, a text 702 is obtained and input into a text encoder 706, and text features of the text 702 are extracted by the text encoder 706. The text features are then input into a conversion model 710, and the text features are converted by the conversion model 710 so that the converted text features can meet the feature space requirements of the input features of the large language model 712. Then, the converted text features are processed by the large language model 712 to generate a training text description corresponding to the text 702 to train the conversion model 710 and the large language model 712.

[0061] In some embodiments, the loss in the training process is determined by the text 702 and the training text description, and the loss satisfies the convergence condition by adjusting the parameters in the conversion model 710 and the large language model 712. When the loss satisfies the convergence condition, it can be determined that the training of the conversion model 710 and the large language model 712 is completed. For example, when the text 702 is "a dog runs on the grass", and the training text description is "a dog runs on the grass", it can be considered that the loss satisfies the convergence condition.

[0062] In some embodiments, during the inference phase, after the conversion model 710 and the large language model 712 are trained by the text encoder 706, the visual encoder 708 corresponding to the text encoder 706 is determined, and the image 704 is processed by the visual encoder 708 to generate image features. The image features are converted by the conversion model 710 so that the converted image features can conform to the feature space of the input features of the large language model 710.

[0063] Then, the converted image features are processed by the large language model 710 to generate a text description 716. In some embodiments, a prompt content 714 corresponding to the image 704 may be obtained, and the prompt content 714 and the converted image features are processed by the large language model 712 to generate a text description 716 for the image 704. The prompt content 714 is used to guide the large language model 712 to generate a text description 716 of a corresponding type.

[0064] Figure 8 FIG. 8 is a block diagram of an apparatus 800 for generating a text description of an image according to some embodiments of the present disclosure. Figure 8 The device 800 includes a first feature generation module 802, which is configured to generate a first feature of an image by a visual encoder, and a text encoder semantically aligned with the visual encoder is used to train a conversion model and a language model. The device 800 also includes a first feature conversion module 804, which is configured to convert the first feature into a second feature by the conversion model, and the first feature and the second feature correspond to different feature spaces. In addition, the device 800 also includes a text description generation module 806, which is configured to generate a text description of the image based on the second feature by a language model.

[0065] Fig. 9 1 is a block diagram of an electronic device 900 according to some embodiments of the present disclosure. The device 900 may be a device or apparatus described in the embodiments of the present disclosure. Fig. 9 As shown, the device 900 includes a central processing unit (CPU) and / or a graphics processing unit (GPU) 902, which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 904 or computer program instructions loaded from a storage unit 916 to a random access memory (RAM) 906. In the RAM 906, various programs and data required for the operation of the device 900 can also be stored. The CPU / GPU 902, the ROM 904, and the RAM 906 are connected to each other via a bus 908. An input / output (I / O) interface 910 is also connected to the bus 908. Although not shown in FIG. Fig. 9 As shown in FIG. 9 , device 900 may further include a co-processor.

[0066] A number of components in the device 900 are connected to the I / O interface 910, including: an input unit 912, such as a keyboard, a mouse, etc.; an output unit 914, such as various types of displays, speakers, etc.; a storage unit 916, such as a disk, an optical disk, etc.; and a communication unit 918, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 918 allows the device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0067] The various methods or processes described above may be performed by the CPU / GPU 902. For example, in some embodiments, the methods may be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as a storage unit 916. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 900 via the ROM 904 and / or the communication unit 918. When the computer program is loaded into the RAM 906 and executed by the CPU / GPU 902, one or more steps or actions in the methods or processes described above may be performed.

[0068] In some embodiments, the methods and processes described above may be implemented as a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present disclosure.

[0069] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. The computer readable storage medium used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (for example, a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0070] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0071] The computer program instructions for performing the disclosed operation may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, programming languages ​​including object-oriented programming languages, and conventional procedural programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, executed as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., utilizing an Internet service provider to connect via the Internet). In certain embodiments, by utilizing the state information of a computer-readable program instruction to customize an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), the electronic circuit may execute a computer-readable program instruction, thereby realizing various aspects of the present disclosure.

[0072] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0073] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0074] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the equipment, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each frame in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the frame can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous frames can actually be executed substantially in parallel, and they can also be executed in the opposite order sometimes, depending on the functions involved. It should also be noted that each frame in the block diagram and / or flow chart, and the combination of frames in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0075] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The choice of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

[0076] Some example implementations of the present disclosure are listed below.

[0077] Example 1. A method for generating a text description of an image, comprising:

[0078] A first feature of an image is generated by a visual encoder, and a text encoder semantically aligned with the visual encoder is used to train a conversion model and a language model;

[0079] The first feature is converted into a second feature by the conversion model, the first feature and the second feature correspond to different feature spaces; and

[0080] A text description of the image is generated by the language model based on the second feature.

[0081] Example 2. The method of Example 1, wherein generating, by the language model, a text description of the image based on the second feature comprises:

[0082] Acquire prompt content for the image, where the prompt content is used to indicate the type of the text description; and

[0083] The text description of the image is generated by the language model based on the second feature and the prompt content.

[0084] Example 3. The method according to Example 1 or 2, further comprising:

[0085] Generating a first text feature of a training text by the text encoder;

[0086] The first text feature is converted into a second text feature by the conversion model, wherein the first text feature and the second text feature correspond to different feature spaces;

[0087] generating, by the language model, a training text description based on the second text feature; and

[0088] The conversion model and the language model are trained based on the training text and the loss between the training text descriptions.

[0089] Example 4. A method according to any one of Examples 1-3, wherein features of the training image in the training image-text pair generated by the visual encoder and features of the training text in the training image-text pair generated by the text encoder satisfy a similarity condition.

[0090] Example 5. The method of any one of Examples 1-4, wherein training the conversion model and the language model comprises:

[0091] Adjusting parameters in the conversion model and the language model based on the loss between the training text and the training text description, wherein during the adjustment process, the parameters in the visual encoder and the text encoder remain unchanged; and

[0092] In response to the loss between the training text and the training text description satisfying a convergence condition, it is determined that the training of the conversion model and the language model is completed.

[0093] Example 6. A method according to any one of Examples 1-5, wherein the visual encoder is a visual encoding sub-model in a text-image alignment model, and the text encoder is a text encoding sub-model in the text-image alignment model.

[0094] Example 7. The method according to any one of Examples 1-6, further comprising:

[0095] generating, by an image encoder semantically aligned with the text encoder, a first image feature of a training image in a training image-text pair;

[0096] The first image feature is converted into a second image feature by the conversion model, wherein the first image feature and the second image feature correspond to different feature spaces;

[0097] generating, by the language model, a fine-tuned text description of the training image based on the second image feature; and

[0098] The image encoder is fine-tuned based on the loss between the training text in the training image-text pair and the fine-tuning text description.

[0099] Example 8. The method of any one of Examples 1-7, wherein generating, by the language model, a fine-tuned text description of the training image based on the second image feature comprises:

[0100] Acquiring training prompt content for the training image-text pair, the training prompt content being used to indicate a type of the fine-tuning text description; and

[0101] The fine-tuned text description of the training image is generated by the language model based on the second image feature and the training prompt content.

[0102] Example 9. A method according to any one of Examples 1-8, wherein the feature space includes at least one of a feature size and a feature space distribution.

[0103] Example 10. The method according to any one of Examples 1-9, further comprising:

[0104] acquiring the image;

[0105] Wherein acquiring the image comprises:

[0106] Access the video; and

[0107] The image is extracted from a video frame in the video.

[0108] Example 11. An apparatus for generating a text description of an image, comprising:

[0109] A first feature generation module is configured to generate a first feature of an image by a visual encoder, and a text encoder semantically aligned with the visual encoder is used to train a conversion model and a language model;

[0110] A first feature conversion module is configured to convert the first feature into a second feature by using the conversion model, wherein the first feature and the second feature correspond to different feature spaces; and

[0111] A text description generation module is configured to generate a text description of the image based on the second feature by the language model.

[0112] Example 12. The apparatus according to Example 11, wherein the text description generation module is further configured to:

[0113] Acquire prompt content for the image, where the prompt content is used to indicate the type of the text description; and

[0114] The text description of the image is generated by the language model based on the second feature and the prompt content.

[0115] Example 13. The apparatus of Example 11 or 12, further comprising:

[0116] A first text feature generating module, configured to generate a first text feature of a training text by the text encoder;

[0117] A first text feature conversion module is configured to convert the first text feature into a second text feature by the conversion model, wherein the first text feature and the second text feature correspond to different feature spaces;

[0118] a training text description generating module, configured to generate a training text description based on the second text feature by the language model; and

[0119] The training module is configured to train the conversion model and the language model based on the loss between the training text and the description of the training text.

[0120] Example 14. An apparatus according to any one of Examples 11-13, wherein features of a training image in a training image-text pair generated by the visual encoder and features of a training text in the training image-text pair generated by the text encoder satisfy a similarity condition.

[0121] Example 15. The apparatus of any of Examples 11-14, wherein the training module is further configured to include:

[0122] Adjusting parameters in the conversion model and the language model based on the loss between the training text and the training text description, wherein during the adjustment process, the parameters in the visual encoder and the text encoder remain unchanged; and

[0123] In response to the loss between the training text and the training text description satisfying a convergence condition, it is determined that the training of the conversion model and the language model is completed.

[0124] Example 16. An apparatus according to any one of Examples 11-15, wherein the visual encoder is a visual encoding sub-model in a text-image alignment model, and the text encoder is a text encoding sub-model in the text-image alignment model.

[0125] Example 17. The apparatus of any one of Examples 11-16, further comprising:

[0126] A first image feature generation module is configured to generate a first image feature of a training image in a training image-text pair by an image encoder semantically aligned with the text encoder;

[0127] A first image feature conversion module is configured to convert the first image feature into a second image feature by the conversion model, wherein the first image feature and the second image feature correspond to different feature spaces;

[0128] a fine-tuning text description generation module, configured to generate a fine-tuning text description of the training image based on the second image feature by the language model; and

[0129] A fine-tuning module is configured to fine-tune the image encoder based on the loss between the training text in the training image-text pair and the fine-tuning text description.

[0130] Example 18. The apparatus according to any one of Examples 11-17, wherein the fine-tuning text description generation module is further configured to:

[0131] Acquiring training prompt content for the training image-text pair, the training prompt content being used to indicate a type of the fine-tuning text description; and

[0132] The fine-tuned text description of the training image is generated by the language model based on the second image features and the training prompt content.

[0133] Example 19. An apparatus according to any of Examples 11-18, wherein the feature space includes at least one of a feature size and a feature space distribution.

[0134] Example 20. The apparatus of any one of Examples 11-19, further comprising:

[0135] An image acquisition module, configured to acquire the image;

[0136] The image acquisition module is also configured as follows:

[0137] Access the video; and

[0138] The image is extracted from a video frame in the video.

[0139] Example 21. An electronic device comprising:

[0140] Processor; and

[0141] A memory coupled to the processor, the memory having instructions stored therein, wherein when the instructions are executed by the processor, the electronic device performs actions, the actions comprising:

[0142] A first feature of an image is generated by a visual encoder, and a text encoder semantically aligned with the visual encoder is used to train a conversion model and a language model;

[0143] The first feature is converted into a second feature by the conversion model, the first feature and the second feature correspond to different feature spaces; and

[0144] A text description of the image is generated by the language model based on the second feature.

[0145] Example 22. The electronic device of Example 21, wherein generating, by the language model, a text description of the image based on the second feature comprises:

[0146] Acquire prompt content for the image, where the prompt content is used to indicate the type of the text description; and

[0147] The text description of the image is generated by the language model based on the second feature and the prompt content.

[0148] Example 23. The electronic device according to Example 21 or 22, wherein the actions further include:

[0149] Generating a first text feature of a training text by the text encoder;

[0150] The first text feature is converted into a second text feature by the conversion model, wherein the first text feature and the second text feature correspond to different feature spaces;

[0151] generating, by the language model, a training text description based on the second text feature; and

[0152] The conversion model and the language model are trained based on the training text and the loss between the training text descriptions.

[0153] Example 24. An electronic device according to any one of Examples 21-23, wherein features of a training image in a training image-text pair generated by the visual encoder and features of a training text in the training image-text pair generated by the text encoder satisfy a similarity condition.

[0154] Example 25. The electronic device of any of Examples 21-24, wherein training the conversion model and the language model comprises:

[0155] Adjusting parameters in the conversion model and the language model based on the loss between the training text and the training text description, wherein during the adjustment process, the parameters in the visual encoder and the text encoder remain unchanged; and

[0156] In response to the loss between the training text and the training text description satisfying a convergence condition, it is determined that the training of the conversion model and the language model is completed.

[0157] Example 26. An electronic device according to any one of Examples 21-25, wherein the visual encoder is a visual encoding sub-model in a text-image alignment model, and the text encoder is a text encoding sub-model in the text-image alignment model.

[0158] Example 27. The electronic device of any one of Examples 21-26, wherein the actions further include:

[0159] generating, by an image encoder semantically aligned with the text encoder, a first image feature of a training image in a training image-text pair;

[0160] The first image feature is converted into a second image feature by the conversion model, wherein the first image feature and the second image feature correspond to different feature spaces;

[0161] generating, by the language model, a fine-tuned text description of the training image based on the second image feature; and

[0162] The image encoder is fine-tuned based on the loss between the training text in the training image-text pair and the fine-tuning text description.

[0163] Example 28. An electronic device according to any of Examples 21-27, wherein generating, by the language model, a fine-tuned text description of the training image based on the second image feature comprises:

[0164] Acquiring training prompt content for the training image-text pair, the training prompt content being used to indicate a type of the fine-tuning text description; and

[0165] The fine-tuned text description of the training image is generated by the language model based on the second image features and the training prompt content.

[0166] Example 29. An electronic device according to any of Examples 21-28, wherein the feature space includes at least one of a feature size and a feature space distribution.

[0167] Example 30. The electronic device of any one of Examples 21-29, wherein the actions further include:

[0168] acquiring the image;

[0169] Wherein acquiring the image comprises:

[0170] Access the video; and

[0171] The image is extracted from a video frame in the video.

[0172] Example 31. A computer-readable storage medium having computer-executable instructions stored thereon, wherein the computer-executable instructions are executed by a processor to implement a method according to any one of Examples 1 to 10.

[0173] Example 32 A computer program product, which is tangibly stored on a computer-readable medium and includes computer-executable instructions, which when executed by a device cause the device to perform a method according to any one of Examples 1 to 10.

[0174] Although the disclosure has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Instead, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A method for generating a text description of an image, comprising: A first feature of an image is generated by a visual encoder, and a text encoder semantically aligned with the visual encoder is used to train a conversion model and a language model; The first feature is converted into a second feature by the conversion model, and the first feature and the second feature correspond to different feature spaces; as well as A text description of the image is generated by the language model based on the second feature.

2. The method according to claim 1, wherein generating, by the language model, a text description of the image based on the second feature comprises: Acquire prompt content for the image, where the prompt content is used to indicate the type of the text description; as well as The text description of the image is generated by the language model based on the second feature and the prompt content.

3. The method according to claim 1, further comprising: Generating a first text feature of a training text by the text encoder; The first text feature is converted into a second text feature by the conversion model, wherein the first text feature and the second text feature correspond to different feature spaces; generating, by the language model, a training text description based on the second text feature; and The conversion model and the language model are trained based on the training text and the loss between the training text descriptions.

4. The method according to claim 3, wherein features of the training image in the training image-text pair generated by the visual encoder and features of the training text in the training image-text pair generated by the text encoder satisfy similarity conditions.

5. The method according to claim 3, wherein training the conversion model and the language model comprises: Adjusting parameters in the conversion model and the language model based on the loss between the training text and the training text description, wherein during the adjustment process, the parameters in the visual encoder and the text encoder remain unchanged; as well as In response to the loss between the training text and the training text description satisfying a convergence condition, it is determined that the training of the conversion model and the language model is completed.

6. The method according to claim 4, wherein the visual encoder is a visual encoding sub-model in a text-image alignment model, and the text encoder is a text encoding sub-model in the text-image alignment model.

7. The method according to claim 3, further comprising: generating, by an image encoder semantically aligned with the text encoder, a first image feature of a training image in a training image-text pair; The first image feature is converted into a second image feature by the conversion model, wherein the first image feature and the second image feature correspond to different feature spaces; generating, by the language model, a fine-tuned text description of the training image based on the second image feature; and The image encoder is fine-tuned based on the loss between the training text in the training image-text pair and the fine-tuning text description.

8. The method of claim 7, wherein generating, by the language model, a fine-tuned text description of the training image based on the second image feature comprises: Acquire training prompt content for the training image-text pair, wherein the training prompt content is used to indicate the type of the fine-tuning text description; as well as The fine-tuned text description of the training image is generated by the language model based on the second image feature and the training prompt content.

9. The method according to claim 1, wherein the feature space comprises at least one of a feature size and a feature space distribution.

10. The method according to any one of claims 1 to 9, further comprising: acquiring the image; Wherein acquiring the image comprises: Get the video; as well as The image is extracted from a video frame in the video.

11. An apparatus for generating a text description of an image, comprising: A first feature generation module is configured to generate a first feature of an image by a visual encoder, and a text encoder semantically aligned with the visual encoder is used to train a conversion model and a language model; A first feature conversion module is configured to convert the first feature into a second feature by using the conversion model, wherein the first feature and the second feature correspond to different feature spaces; and A text description generation module is configured to generate a text description of the image based on the second feature by the language model.

12. An electronic device comprising: processor; as well as A memory coupled to the processor, the memory having instructions stored therein, wherein when the instructions are executed by the processor, the electronic device executes the method according to any one of claims 1 to 10.

13. A computer-readable storage medium having computer-executable instructions stored thereon, wherein the computer-executable instructions are executed by a processor to implement the method according to any one of claims 1 to 10.

Images