Graph-to-text model training method and device and text generation method and device

Abstract

The invention provides a graph-to-text model training method and device, relates to the technical field of artificial intelligence, in particular to the technical fields of computer vision, deep learning, large models and the like, and can be applied to scenes such as content generation of artificial intelligence. According to the specific implementation scheme, an image sample set is obtained, wherein the image sample set comprises at least one image sample; obtaining an initial graph-to-text model, wherein the graph-to-text model comprises an image coding sub-network, a connection sub-network and a multi-modal large language model; training an image coding sub-network and a connection sub-network based on the image sample set and the image generator to obtain a trained image coding sub-network and a trained connection sub-network; and based on the image sample set, training the multi-modal large language model, the trained image coding sub-network and the trained connection sub-network to obtain a trained graph-to-text model.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, specifically to technical fields such as computer vision, deep learning, large models, and image recognition, and can be applied to scenarios such as artificial intelligence content generation, and in particular to a method and device for training a graph-to-text model, a method and device for generating text, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] In recent years, multimodal large language models have made significant progress and development, and have been widely used in various application scenarios. However, the text they generate is often inconsistent with the visual content, that is, multimodal large models have a common hallucination problem. This problem greatly affects the effect in practical applications. Therefore, it is necessary to optimize this problem and alleviate the hallucination problem. Summary of the invention

[0003] The present disclosure provides a method and device for training a graph-to-text model, a method and device for generating text, an electronic device, a computer-readable storage medium, and a computer program product.

[0004] According to a first aspect, a method for training an image-generated text model is provided, the method comprising: obtaining an image sample set, the image sample set comprising at least one image sample; obtaining an initial image-generated text model, the image-generated text model comprising: an image coding subnetwork, a connection subnetwork, and a multimodal large language model; training the image coding subnetwork and the connection subnetwork based on the image sample set and an image generator to obtain trained image coding subnetwork and connection subnetwork; training the multimodal large language model, the trained image coding subnetwork, and the trained connection subnetwork based on the image sample set to obtain a trained image-generated text model.

[0005] According to a second aspect, a text generation method is provided, the method comprising: obtaining an image to be processed; inputting the image to be processed into an image-to-text model generated by the method described in any implementation manner of the first aspect, to obtain an image description text of the image to be processed.

[0006] According to a third aspect, a device for training a graph-based text model is provided, the device comprising: a sample acquisition unit, configured to acquire an image sample set, the image sample set comprising at least one image sample; a model acquisition unit, configured to acquire an initial graph-based text model, the graph-based text model comprising: an image coding subnetwork, a connection subnetwork and a multimodal large language model; a network training unit, configured to train the image coding subnetwork and the connection subnetwork based on the image sample set and an image generator to obtain the trained image coding subnetwork and the connection subnetwork; a model training unit, configured to train the multimodal large language model, the trained image coding subnetwork and the trained connection subnetwork based on the image sample set to obtain the trained graph-based text model.

[0007] According to the fourth aspect, a text generation device is provided, which includes: an image acquisition unit, configured to acquire an image to be processed; an image input unit, configured to input the image to be processed and a prompt word into an image-to-text model generated by the device described in any implementation method of the third aspect, to obtain an image description text of the image to be processed.

[0008] According to the fifth aspect, an electronic device is provided, which includes: at least one processor; and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in any implementation of the first aspect or the second aspect.

[0009] According to a sixth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute the method described in any implementation of the first aspect or the second aspect.

[0010] According to a seventh aspect, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the method described in any implementation of the first aspect or the second aspect.

[0011] The method and device for training an image-generated text model provided by the embodiments of the present disclosure first obtain an image sample set, the image sample set includes at least one image sample; secondly, obtain an initial image-generated text model, the image-generated text model includes: an image coding subnetwork, a connection subnetwork and a multimodal large language model; thirdly, train the image coding subnetwork and the connection subnetwork based on the image sample set and the image generator to obtain the trained image coding subnetwork and the connection subnetwork; finally, train the multimodal large language model, the trained image coding subnetwork and the trained connection subnetwork based on the image sample set to obtain the trained image-generated text model. Therefore, the present disclosure verifies and optimizes the input of the image coding subnetwork and the output of the connection subnetwork through an image generator with fixed parameters, alleviates the illusion of text description generation by the image-generated text model, and improves the image description text generation effect of the image-generated text model.

[0012] The text generation method and device provided by the embodiments of the present disclosure obtain an image to be processed, input the image to be processed into a graph-to-text model generated by a graph-to-text model training method, and obtain an image description text of the image to be processed. Thus, the image-to-text model is used to generate a liveness detection result, thereby improving the reliability and accuracy of the image description text application.

[0013] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0015] Figure 1 is a flowchart of an embodiment of the method for training a graph-to-text model according to the present disclosure;

[0016] Figure 2 It is a structural schematic diagram of the training process of the disclosed graph-to-text model;

[0017] Figure 3 is a flow chart of an embodiment of a method for generating text according to the present disclosure;

[0018] Figure 4 is a structural schematic diagram of an embodiment of a device for training a graph-to-text model according to the present disclosure;

[0019] Figure 5 is a structural schematic diagram of an embodiment of a text generating device according to the present disclosure;

[0020] Figure 6 It is a block diagram of an electronic device used to implement the graph-to-text model training method or text generation method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0021] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0022] In response to the hallucination problem that often occurs in graph-generated text models, traditional technologies generally use two solutions to eliminate hallucinations. Solution 1: Introduce negative data and counterfactual data to increase the diversity of the data set of the graph-generated text model; reduce noise and errors in the data set to optimize sample quality. Solution 2: Use a higher-resolution visual encoder to enhance the visual perception ability of the graph-generated text model. Both Solution 1 and Solution 2 can alleviate the hallucination problem of the graph-generated text model, but are still limited by the output effects of the visual encoder and language encoder.

[0023] In view of the defects in traditional technologies, this paper proposes a method for training a graph-generated text model, which can effectively alleviate the hallucination problem of the graph-generated text model and improve the generation accuracy of the graph-generated text model. Figure 1 A process 100 according to an embodiment of a method for training a graph-to-text model in accordance with the present disclosure is shown. The method for training a graph-to-text model includes the following steps:

[0024] Step 101: Obtain an image sample set.

[0025] In this embodiment, the execution subject on which the image-to-text model training method runs can obtain the image sample set in a variety of ways. For example, the execution subject can obtain the image sample set stored in the database server through a wired connection or a wireless connection. For another example, the user can obtain the image sample set collected by the terminal by communicating with the terminal.

[0026] Here, the image sample set may include at least one image sample, the image sample includes an annotated image related to the image description text to be generated, the annotated image may include an image and the annotated text of the image, the annotated text is a text describing various objects, scenes and styles in the image, and when training the image-to-text model, the image in the image sample can be input into the image encoding subnetwork to detect whether the image description text output by the multimodal large language model is consistent with the annotated text.

[0027] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of the image sample sets involved are carried out after authorization and comply with relevant laws and regulations.

[0028] Step 102, obtaining an initial image-to-text model.

[0029] The image-to-text model includes: an image encoding subnetwork, a connection subnetwork, and a multimodal large language model. The image encoding subnetwork performs feature analysis on the input image to obtain visual features; the connection subnetwork is connected between the image encoding subnetwork and the multimodal large language model to align the visual features output by the image encoding subnetwork with the text, and sends the visual features after the visual-text alignment to the multimodal large language model; the multimodal large language model obtains the image description text of the image based on the aligned visual features output by the connection subnetwork. That is, the image encoding subnetwork is used to extract visual features and input the visual features into the connection subnetwork; the connection subnetwork is used to align the visual features with the text, and send the visual features after the visual-text alignment to the multimodal large language model.

[0030] like Figure 2 As shown in FIG. 1 , the selected image TX is input into the image encoding subnetwork, and the encoding subnetwork and the connection subnetwork can both be pre-trained networks. The image encoding subnetwork can adopt the VIT (Vision Transformer) model framework, and the connector can adopt the MLP (Multilayer Perceptron) model framework, the multimodal large language model (such as Figure 2 In the MLLM, the prompt word and the output of the connection sub-network are received to obtain the image description text To, wherein the multimodal large language model can adopt qwen8B.

[0031] In this embodiment, the multimodal large language model is a large model that converts aligned visual features into image description text. When analyzing the aligned visual features output by the connection subnetwork, it is necessary to input a fixed prompt word into the multimodal large language model. The prompt word is used to prompt the multimodal large language model to convert the input content into text to obtain the image description text. Among them, the image description text is the text that describes the image in the image sample. The content of the prompt word can be: Generate as detailed a description as possible based on the image. Please respect the facts of the picture and do not make excessive associations. It should be noted that the prompt word can be text or a token generated by the text.

[0032] In this embodiment, the image-to-text model includes a multimodal large language model. The multimodal large language model is a natural language processing model based on deep learning. It can automatically generate sentences, paragraphs or articles that conform to language rules mainly by learning a large amount of text and image data.

[0033] Optionally, the image-to-text model may further include: a mapping layer; the mapping layer is used to convert the dimension of the visual features after the connection sub-network output alignment into a dimension consistent with the multimodal large language model.

[0034] Step 103: train the image encoding subnetwork and the connection subnetwork based on the image sample set and the image generator to obtain trained image encoding subnetwork and connection subnetwork.

[0035] In this embodiment, the image generator is a model that converts visual features into images. Visual features are input to the image generator to obtain images output by the image generator. When training the image encoding subnetwork and the connection subnetwork, the input end of the image generator can be connected to the output end of the connection subnetwork, so that the image generator receives the visual features after visual text alignment output by the connection subnetwork, and generates an image based on the visual features after visual text alignment.

[0036] like Figure 2 As shown, the image generator SM receives the token output by the connection subnetwork MLP and generates a new image TX'. On this basis, the MLP output token is optimized by training the image encoding subnetwork and the connection subnetwork.

[0037] In this embodiment, the execution subject can select image samples from the image sample set obtained in step 101, and execute the training step of step 103 to complete an iterative training of the image coding subnetwork and the connection subnetwork. Among them, the selection method and the number of image samples selected from the image sample set are not limited in this application, and the number of iterative training of the image coding subnetwork and the connection subnetwork is not limited. For example, in one iterative training, multiple continuous image sets can be randomly selected, and the network loss values ​​of the image coding subnetwork and the connection subnetwork are calculated through the selected image samples, and the parameters of the image coding subnetwork and the connection subnetwork are adjusted to obtain the trained image coding subnetwork and the connection subnetwork.

[0038] In this embodiment, the loss function of the feature recognition subnetwork and the classification subnetwork in the graph-generated text model can adopt the cross entropy loss function, which can measure the degree of difference between two different probability distributions in the same random variable, and is expressed as the difference between the true probability distribution and the predicted probability distribution in machine learning. The smaller the value of the cross entropy loss function, the better the prediction effect of the feature recognition subnetwork and the classification subnetwork in the graph-generated text model.

[0039] Step 104: Based on the image sample set, train the multimodal large language model, the trained image encoding subnetwork, and the trained connection subnetwork to obtain a trained image-to-text model.

[0040] In this embodiment, the execution subject can select image samples from the image sample set obtained in step 101, and execute the training step of step 104 to complete an iterative training of the image-generated text model. Among them, the selection method and the number of image samples selected from the image sample set are not limited in this application, and the number of iterative training of the image-generated text model is not limited. It should be noted that prompt words can be input into the multimodal large language model when training the multimodal large language model, the trained image encoding subnetwork, and the trained connection subnetwork.

[0041] In this embodiment, during each iterative training of the image-to-text model, image samples are selected from the image sample set, and the selected image samples are input into the trained image encoding subnetwork, and the prompt words are input into the multimodal large language model. Based on the image description text output by the multimodal large language model, the loss values ​​of the multimodal large language model, the trained image encoding subnetwork, and the trained connection subnetwork are calculated.

[0042] In this embodiment, the trained image-to-text model is obtained by iteratively training the selected image samples into a multimodal large language model, a trained image encoding subnetwork, and a trained connection subnetwork, and adjusting the parameters of the multimodal large language model, the trained image encoding subnetwork, and the trained connection subnetwork.

[0043] In this embodiment, the loss values ​​of the multimodal large language model, the trained image coding subnetwork, and the trained connection subnetwork can be used to detect whether the multimodal large language model, the trained image coding subnetwork, and the trained connection subnetwork meet the training completion conditions. After the multimodal large language model, the trained image coding subnetwork, and the trained connection subnetwork meet the training completion conditions, a trained graph-to-text model is obtained.

[0044] Optionally, in this embodiment, in response to the multimodal large language model, the trained image coding subnetwork and the trained connection subnetwork not satisfying the training completion condition, relevant parameters in the multimodal large language model, the trained image coding subnetwork and the trained connection subnetwork are adjusted so that the loss values ​​of the multimodal large language model, the trained image coding subnetwork and the trained connection subnetwork converge, and based on the adjusted multimodal large language model, the trained image coding subnetwork and the trained connection subnetwork, the above-mentioned training step 104 is continued to be executed.

[0045] In this optional implementation, when the training completion conditions are not met, adjusting the relevant parameters of the multimodal large language model, the trained image encoding subnetwork, and the trained connection subnetwork can help converge the loss value.

[0046] The method for training a graph-generated text model provided by an embodiment of the present disclosure first obtains an image sample set, the image sample set includes at least one image sample; secondly, obtains an initial graph-generated text model, the graph-generated text model includes: an image coding subnetwork, a connection subnetwork, and a multimodal large language model; thirdly, trains the image coding subnetwork and the connection subnetwork based on the image sample set and the image generator to obtain the trained image coding subnetwork and the connection subnetwork; finally, based on the image sample set, trains the multimodal large language model, the trained image coding subnetwork, and the trained connection subnetwork to obtain the trained graph-generated text model. Therefore, the present disclosure verifies and optimizes the input of the image coding subnetwork and the output of the connection subnetwork through an image generator with fixed parameters, alleviates the illusion of text description generation by the graph-generated text model, and improves the image description text generation effect of the graph-generated text model.

[0047] In some optional implementations of the present disclosure, the above-mentioned training of the image encoding subnetwork and the connection subnetwork based on the image sample set and the image generator to obtain the trained image encoding subnetwork and the connection subnetwork includes: fixing the parameters of the image generator; selecting image samples from the image sample set; inputting the selected image samples into the image encoding subnetwork to obtain the visual features after the visual text alignment output by the connection subnetwork; inputting the visual features into the image generator to obtain the generated image generated by the image generator; based on the generated image and the selected image samples, calculating the branch loss of the image encoding subnetwork and the connection subnetwork; based on the branch loss, obtaining the trained image encoding subnetwork and the connection subnetwork.

[0048] In this optional implementation, the image generator is a model for generating images. Compared with the image-to-text model, the image generator is additionally added to facilitate the training of the image encoding subnetwork and the connection subnetwork. Specifically, the image generator can adopt the SD (system dynamics) model, which can effectively simulate the correction process.

[0049] like Figure 2As shown in the figure, in order to solve the problem that the current image encoding subnetwork tends to focus on global features and lacks fine-grained visual perception, a pre-trained image generator SM is used for supervised optimization, the weights of the image encoding subnetwork and the connection subnetwork are updated, and the weights of the image generator SM are frozen. The specific implementation is as follows: for the input image TX, the image encoding subnetwork (for example, a pre-trained VIT model based on CLIP contrastive learning) is used for feature extraction, and then the visual feature output after feature alignment in the visual-text space is performed based on the connection subnetwork MLP. Based on the output visual vector, it is input into the image generator as a visual control condition to guide the image generator to generate images. The effect of the feature description directly affects the details of the image generation of the image generator. The reconstruction loss calculation (similar to ip-adapter) is performed on the generated image TX' of the image generator and the original image TX, guiding the image encoding subnetwork and the connection subnetwork to update parameters and pay more attention to the fine-grained features of the image.

[0050] In this optional implementation, an intermediate loss function is set for the image encoding subnetwork and the connection subnetwork. The above-mentioned calculation of the branch losses of the image encoding subnetwork and the connection subnetwork based on the generated image and the selected image samples includes: inputting the selected image samples into the image encoding subnetwork to obtain the visual features of the visual text aligned with the output of the connection subnetwork; based on the intermediate loss function, calculating the branch losses of the image encoding subnetwork and the connection subnetwork.

[0051] This optional implementation provides a method for obtaining a trained image encoding subnetwork and a connection subnetwork, fixes the parameters of the image generator, selects images from an image sample set, trains the image encoding subnetwork and the connection subnetwork, and improves the accuracy and reliability of the image encoding subnetwork and the connection subnetwork in the image-to-text model.

[0052] In some optional implementations of the present disclosure, the above-mentioned branch loss-based, trained image coding subnetwork and connection subnetwork include: determining whether the branch loss is less than or equal to a branch loss threshold; in response to the determination result that the branch loss is less than or equal to the branch loss threshold, obtaining the trained image coding subnetwork and connection subnetwork.

[0053] In this optional implementation, the branch loss threshold is a loss limit value set for the image coding subnetwork and the connection subnetwork. When the branch loss is less than or equal to the branch loss threshold, it is determined that the image coding subnetwork and the connection subnetwork have been trained. The specific value of the branch loss threshold can be adjusted based on development requirements. For example, the branch loss threshold is 0.001.

[0054] The method for obtaining the trained image coding subnetwork and connection subnetwork provided by this optional implementation determines the timing of stopping the image coding subnetwork and the connection subnetwork by comparing the branch loss with the branch loss threshold, thereby providing suitable termination conditions for the completion of the training of the image coding subnetwork and the connection subnetwork.

[0055] Optionally, the above-mentioned branch loss-based trained image coding subnetwork and connection subnetwork also includes: in response to a judgment result that the branch loss is greater than a branch loss threshold, adjusting parameters of the image coding subnetwork and the connection subnetwork, inputting image samples selected from the image sample set into the image coding subnetwork after the parameters are adjusted, calculating the branch losses of the image coding subnetwork and the connection subnetwork after the parameters are adjusted, and returning to continue to judge whether the branch loss is less than the branch loss threshold.

[0056] In some optional implementations of the present disclosure, the above-mentioned training of a multimodal large language model, a trained image encoding subnetwork, and a trained connection subnetwork based on an image sample set to obtain a trained graph-to-text model includes: removing an image generator; selecting image samples from an image sample set; inputting the selected image samples into the trained image encoding subnetwork to obtain an image description text output by the multimodal large language model; calculating the total loss of the multimodal large language model, the trained image encoding subnetwork, and the trained connection subnetwork based on the image description text and the selected image samples; and obtaining the trained graph-to-text model based on the total loss.

[0057] In this optional implementation, the image generator is only applicable when training the image encoding subnetwork and the connection subnetwork. When jointly training a multimodal large language model, a trained image encoding subnetwork, and a trained connection subnetwork, the image generator no longer needs to participate.

[0058] In this optional implementation, the trained image encoding subnetwork and the trained connection subnetwork are used as one network branch, and a branch loss function is set; the multimodal large language model is used as another network branch, and a trunk loss function is set; when training the graph-to-text model, the loss values ​​of the loss functions corresponding to the two network branches are calculated respectively.

[0059] In this optional implementation, inputting the selected image sample into the trained image encoding subnetwork to obtain the image description text output by the multimodal large language model means: inputting the selected image sample into the trained image encoding subnetwork, sequentially processing through the trained connection subnetwork and the multimodal large language model, to obtain the image description text output by the multimodal large language model corresponding to the selected image sample.

[0060] In this optional implementation, the above-mentioned calculation of the total loss of the multimodal large language model, the trained image encoding subnetwork and the trained connection subnetwork based on the image description text and the selected image samples includes: inputting the selected image samples into the trained image encoding subnetwork to obtain the image description text output by the multimodal large language model; based on the branch loss function, calculating the first loss value of the trained image encoding subnetwork and the trained connection subnetwork; based on the trunk loss function, calculating the second loss value of the multimodal large language model, adding the first loss value and the second loss value to obtain the total loss of the multimodal large language model, the trained image encoding subnetwork and the trained connection subnetwork.

[0061] Optionally, the above-mentioned calculation of the total loss of the multimodal large language model, the trained image coding subnetwork and the trained connection subnetwork based on the image description text and the selected image samples can also include: inputting the selected image samples into the trained image coding subnetwork to obtain the image description text output by the multimodal large language model; based on the branch loss function, calculating the first loss value of the trained image coding subnetwork and the trained connection subnetwork; based on the trunk loss function, calculating the second loss value of the multimodal large language model, assigning weight values ​​to the first loss value and the second loss value respectively, and multiplying the first loss value and the second loss value by their corresponding weight values ​​to obtain the total loss of the multimodal large language model, the trained image coding subnetwork and the trained connection subnetwork.

[0062] The trained graph-to-text model provided by this optional implementation method, after removing the image generator, selects images from the image sample set, trains the graph-to-text model with the multimodal large language model, the trained image encoding subnetwork and the trained connection subnetwork as a whole, and obtains the trained graph-to-text model based on the optimized image encoding subnetwork and the connection subnetwork, thereby improving the accuracy and reliability of the graph-to-text model.

[0063] In some optional implementations of the present disclosure, the above-mentioned method of obtaining a trained graph-based text model based on the total loss includes: determining whether the total loss is less than or equal to a total loss threshold; in response to the determination result that the total loss is less than or equal to the total loss threshold, obtaining a trained graph-based text model.

[0064] In this optional implementation, the total loss threshold is the loss limit value set for the graph-generated text model. When the total loss is less than or equal to the total loss threshold, it is determined that the graph-generated text model has been trained. The specific value of the total loss threshold can be adjusted based on development requirements. For example, the total loss threshold is 0.01.

[0065] The method for obtaining a trained graph-to-text model provided by this optional implementation determines the timing to stop training the graph-to-text model by comparing the total loss with the total loss threshold, thereby providing a suitable termination condition for completing the training of the graph-to-text model.

[0066] Optionally, the above-mentioned trained graph-to-text model based on the total loss also includes: in response to a judgment result that the total loss is greater than a total loss threshold, adjusting the parameters of any one or more of the multimodal large language model, the trained image encoding subnetwork, and the trained connection subnetwork, inputting image samples selected from the image sample set into the trained image encoding subnetwork, calculating the total loss of the multimodal large language model, the image encoding subnetwork, and the connection subnetwork after adjusting the parameters, and returning to continue to determine whether the total loss is less than or equal to the total loss threshold.

[0067] Furthermore, based on the graph-to-text model training method provided in the above embodiment, the present disclosure also provides an embodiment of a text generation method. The text generation method of the present disclosure combines artificial intelligence fields such as computer vision and deep learning.

[0068] See also Figure 3 , shows a process 300 according to an embodiment of a text generation method of the present disclosure. The text generation method provided in this embodiment includes the following steps:

[0069] Step 301: Obtain an image to be processed.

[0070] In this embodiment, the image to be processed may include information such as people, objects, and scenery. The image to be processed is processed by the image-to-text model to obtain an image description text. The execution subject of the text generation method can obtain the image to be processed in a variety of ways. For example, the execution subject can obtain the image to be processed stored in the database server through a wired connection or a wireless connection. For another example, the execution subject can also receive the image to be processed collected in real time by a terminal or other device.

[0071] The text generation method disclosed in the present invention can be used in scenarios such as video understanding / description, and with the help of this technology, the accuracy of picture description can be improved.

[0072] Step 302: input the image to be processed and the prompt words into the image-to-text model to obtain the image description text of the image to be processed.

[0073] In this embodiment, the prompt word is a language that prompts the image-to-text model to generate text for the image to be processed. After obtaining the prompt word, the image-to-text model generates text for the image to be processed under the prompt of the prompt word to obtain the image description text. Specifically, the image-to-text model includes: an image encoding subnetwork, a connection subnetwork, and a multimodal large language model. The above step 302 includes: inputting the image to be processed into the image encoding subnetwork, and inputting the prompt word into the multimodal large language model.

[0074] In this embodiment, the execution subject may input the image to be processed obtained in step 301 into the image-to-text model, thereby obtaining the image description text of the image to be processed. The image description text is a text used to describe the content of the image to be processed.

[0075] In this embodiment, the graph-to-text model can be adopted as described above. Figure 1 The trained graph-to-text model is obtained by training the method described in the embodiment. For the specific training process, please refer to Figure 1 The relevant description of the embodiments will not be repeated here.

[0076] The text generation method provided by the embodiment of the present disclosure obtains an image to be processed, inputs the image to be processed into a graph-to-text model generated by a graph-to-text model training method, and obtains an image description text of the image to be processed. Thus, the image description text is generated by using the graph-to-text model, which improves the reliability and accuracy of liveness detection.

[0077] In some optional implementations of the present disclosure, the above-mentioned text generation method also includes: based on the image description text, using the target detection model to perform image detection on the image to be processed to obtain a detection result; based on the detection result, obtaining a new prompt word; inputting the image to be processed and the new prompt word into the image-to-text model to obtain a new image description text output by the image-to-text model.

[0078] In this optional implementation, the target detection model is a pre-trained model used to detect targets in an image, and the target may be any one or more subjects in the image.

[0079] In this optional implementation, the image to be processed is input into the target detection model to obtain the detection result output by the target detection model, wherein the detection result is the result of detecting the information related to the image description text in the image to be processed. Based on the image description text, the target detection model is used to perform image detection on the image to be processed to obtain the detection result, which includes: inputting the image to be processed into the target detection model to obtain the initial result output by the target detection model; calculating the similarity between the initial result and the image description text; in response to the similarity between the initial result and the image description text being greater than the similarity threshold, determining that the detection result is that the image description text is consistent with the description of the image to be processed; otherwise, determining that the detection result is that the image description text is inconsistent with the description of the image to be processed. When the detection result is that the image description text is inconsistent with the description of the image to be processed, the detection result is used as a new prompt word to enable the image-to-text model to obtain a new image description text.

[0080] The text generation method provided by this optional implementation method detects the image to be processed based on the target detection model to obtain the detection result; obtains a new prompt word based on the detection result, providing a reliable implementation method for updating the prompt word of the image generation model.

[0081] In some optional implementations of the present disclosure, the above-mentioned image detection is performed on the image to be processed based on the image description text using a target detection model, and the detection result obtained includes: extracting entity information in the image description text; inputting the entity information and the image to be processed into the target detection model, and obtaining the detection result of the entity information output by the target detection model.

[0082] In this optional implementation, the detection result may be a result of whether the image to be processed includes entity information. For example, if the entity information is an object, the detection result is whether the image to be processed includes the object.

[0083] In this optional implementation, entity information is information related to entities in the image description text, where entities include: people, objects, scenes, etc., and entity information includes: entity type, entity attributes, etc. Inputting entity information and the image to be processed into the target detection model together can enable the target detection model to more effectively identify information related to the entity information in the image to be processed and obtain detection results.

[0084] The method for obtaining the detection result provided by this optional implementation method extracts the entity information in the image description text, uses the target detection model to detect the image to be processed and the entity information, and obtains the detection result, which can make the detection result closer to the image to be processed and the entity information, and improve the accuracy of the detection result.

[0085] In some optional implementations of the present disclosure, the above-mentioned obtaining of new prompt words based on the detection results includes: detecting whether the detection results contain entity information; in response to detecting that the detection results do not contain entity information, generating prompt words to be used based on the entity information; inputting the prompt words to be used and the image to be processed into an image-to-text model to obtain new image description text for the image to be processed; based on the new image description text, using a target detection model to perform image detection on the image to be processed to obtain a new detection result; in response to the presence of entity information in the new detection result, using the new image description text as a new prompt word.

[0086] In this optional implementation, the entity information is information related to the subject in the image to be processed, and the prompt word to be used is a prompt word used to prompt the image-generated text model to generate text. The above-mentioned generating the prompt word to be used based on the entity information includes: adding the entity information to at least one preset command sentence to obtain the prompt word to be used. The command sentence is a sentence that instructs the image-generated text model to detect entity information.

[0087] The method for obtaining new prompt words provided by this optional implementation method generates prompt words to be used based on entity information when there is no entity information in the detection result; inputs the prompt words to be used and the image to be processed into the image-to-text model to obtain new image description text of the image to be processed; based on the new image description text, uses the target detection model to perform image detection on the image to be processed to obtain a new detection result; in response to the presence of entity information in the new detection result, uses the new image description text as a new prompt word, thereby providing a reliable implementation method for obtaining new prompt words.

[0088] Optionally, the above-mentioned obtaining of new prompt words based on the detection results includes: detecting whether the detection results contain entity information; in response to detecting that the detection results do not contain entity information, generating prompt words to be used based on the entity information; inputting the prompt words to be used and the image to be processed into a graph-to-text model to obtain a new image description text for the image to be processed; based on the new image description text, using a target detection model to perform image detection on the image to be processed to obtain a new detection result; in response to the new detection result not containing entity information, continuing to generate prompt words to be used based on the entity information; inputting the prompt words to be used and the image to be processed into a graph-to-text model to obtain a new image description text for the image to be processed; based on the new image description text, using a target detection model to perform image detection on the image to be processed to obtain a new detection result, until the new detection result has entity information, and the new image description text is used as a new prompt word.

[0089] In some optional implementations of the present disclosure, the obtaining of a new prompt word based on the detection result further includes: in response to detecting that the detection result contains entity information, using the prompt word as a new prompt word.

[0090] In this optional implementation, when there is entity information in the detection result, it is determined that the prompt function of the prompt word prompting the image-to-text model is more accurate, and the prompt word can continue to be used.

[0091] The method for obtaining a new prompt word provided by this optional implementation uses the prompt word as a new prompt word when there is entity information in the detection result, thereby providing a reliable implementation method for obtaining a new prompt word.

[0092] Optionally, the obtaining of a new prompt word based on the detection result further includes: in response to detecting that the detection result contains entity information, concatenating the prompt word with the entity information as a new prompt word.

[0093] In a specific example of the present disclosure, a multimodal large language model in the image-to-text model receives output tokens of the connection sub-network and text tokens of prompt words, and generates image description text for the image to be processed.

[0094] For the generated image description text, the execution subject on which the text generation method runs extracts entities in the image to be processed, such as banana, apple, etc., to obtain entity information.

[0095] The prompt words are generated by the entity body in the entity information to guide the target detection model (not shown in the figure) to perform target detection and determine whether bananas and apples exist. Based on the target detection results of the target detection model, if bananas exist but apples do not, multiple rounds of dialogues are conducted with the multimodal large language model to prompt that banana entities exist but apple entities do not exist. The multiple rounds of dialogues, i.e. the corrected model results, are output.

[0096] Further references Figure 4 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a device for training a graph-to-text model. Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0097] like Figure 4As shown, the image-generated text model training device 400 provided in this embodiment includes: a sample acquisition unit 401, a model acquisition unit 402, a network training unit 403, and a model training unit 404. Among them, the above-mentioned sample acquisition unit 401 can be configured to acquire an image sample set, and the image sample set includes at least one image sample. The above-mentioned model acquisition unit 402 can be configured to acquire an initial image-generated text model, and the image-generated text model includes: an image coding subnetwork, a connection subnetwork, and a multimodal large language model. The above-mentioned network training unit 403 can be configured to train the image coding subnetwork and the connection subnetwork based on the image sample set and the image generator to obtain the trained image coding subnetwork and the connection subnetwork. The above-mentioned model training unit 404 can be configured to train the multimodal large language model, the trained image coding subnetwork, and the trained connection subnetwork based on the image sample set to obtain a trained image-generated text model.

[0098] In this embodiment, in the image-generated text model training device 400, the specific processing of the sample acquisition unit 401, the model acquisition unit 402, the network training unit 403, and the model training unit 404 and the technical effects thereof can be referred to respectively. Figure 1 The relevant descriptions of step 101, step 102, step 103, and step 104 in the corresponding embodiment are not repeated here.

[0099] In some optional implementations of the present embodiment, the network training unit 403 is further configured to: fix the parameters of the image generator; select image samples from the image sample set; input the selected image samples into the image encoding subnetwork to obtain the visual features after visual text alignment output by the connection subnetwork; input the visual features into the image generator to obtain the generated image generated by the image generator; based on the generated image and the selected image samples, calculate the branch losses of the image encoding subnetwork and the connection subnetwork; based on the branch losses, obtain the trained image encoding subnetwork and the connection subnetwork.

[0100] In some optional implementations of this embodiment, the above-mentioned network training unit 403 is further configured to: determine whether the branch loss is less than or equal to the branch loss threshold; in response to the judgment result that the branch loss is less than or equal to the branch loss threshold, obtain the trained image encoding subnetwork and connection subnetwork.

[0101] In some optional implementations of the present embodiment, the model training unit 404 is further configured to: remove the image generator; select image samples from the image sample set; input the selected image samples into the trained image encoding subnetwork to obtain the image description text output by the multimodal large language model; based on the image description text and the selected image samples, calculate the total loss of the multimodal large language model, the trained image encoding subnetwork and the trained connection subnetwork; based on the total loss, obtain the trained image-to-text model.

[0102] In some optional implementations of this embodiment, the above-mentioned model training unit 404 is further configured to: determine whether the total loss is less than or equal to the total loss threshold; in response to the judgment result that the total loss is less than or equal to the total loss threshold, obtain a trained graph-to-text model.

[0103] The image-generated text model training device provided by the embodiment of the present disclosure, first, the sample acquisition unit 401 acquires an image sample set, the image sample set includes at least one image sample; secondly, the initial image-generated text model is acquired, the image-generated text model includes: an image coding subnetwork, a connection subnetwork and a multimodal large language model; thirdly, the network training unit 403 trains the image coding subnetwork and the connection subnetwork based on the image sample set and the image generator to obtain the trained image coding subnetwork and the connection subnetwork; finally, the model training unit 404 trains the multimodal large language model, the trained image coding subnetwork and the trained connection subnetwork based on the image sample set to obtain the trained image-generated text model. Therefore, the present disclosure verifies and optimizes the input of the image coding subnetwork and the output of the connection subnetwork through an image generator with fixed parameters, alleviates the illusion of text description generation by the image-generated text model, and improves the image description text generation effect of the image-generated text model.

[0104] Further references Figure 5 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a text generation device, which is similar to Figure 5 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0105] like Figure 5 As shown, the text generation device 500 provided in this embodiment includes: an image acquisition unit 501 and an image input unit 502. The image acquisition unit 501 can be configured to acquire an image to be processed. The image input unit 502 can be configured to input the image to be processed and the prompt word into the image to be processed. Figure 4 In the image-to-text model generated by the apparatus described in the embodiment, an image description text of the image to be processed is obtained.

[0106] In this embodiment, in the text generation device 500, the specific processing of the image acquisition unit 501 and the image input unit 502 and the technical effects thereof can be referred to in Figure 3 The relevant descriptions of step 301 and step 302 in the corresponding embodiment are not repeated here.

[0107] In some optional implementations of the present disclosure, the above-mentioned device 300 also includes: an updating unit (not shown in the figure), and the updating unit is configured to: based on the image description text, use the target detection model to perform image detection on the image to be processed to obtain a detection result; based on the detection result, obtain a new prompt word; input the image to be processed and the new prompt word into the image-to-text model to obtain a new image description text output by the image-to-text model.

[0108] In some optional implementations of the present disclosure, the above-mentioned updating unit is further configured to: extract entity information in the image description text; input the entity information and the image to be processed into the target detection model, and obtain the detection result of the entity information output by the target detection model.

[0109] In some optional implementations of the present disclosure, the above-mentioned update unit is further configured to: detect whether there is entity information in the detection result; in response to detecting that there is no entity information in the detection result, generate a standby prompt word based on the entity information; input the standby prompt word and the image to be processed into the image-to-text model to obtain a new image description text for the image to be processed; based on the new image description text, use the target detection model to perform image detection on the image to be processed to obtain a new detection result; in response to the image description text containing entity information in the new detection result, use the new image description text as a new prompt word.

[0110] In some optional implementations of the present disclosure, the updating unit is further configured to: in response to detecting that the detection result contains entity information, use the prompt word as a new prompt word.

[0111] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0112] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0113] Figure 6A schematic block diagram of an example electronic device 600 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0114] like Figure 6 As shown, the device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0115] A number of components in the device 600 are connected to the I / O interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0116] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above, such as a graph-generated text model training method or a text generation method. For example, in some embodiments, the graph-generated text model training method or the text generation method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the graph-generated text model training method or the text generation method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to execute the image-to-text model training method or the text generation method in any other appropriate manner (for example, by means of firmware).

[0117] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0118] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable graph-text model training device, or a text generation device, so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0119] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0120] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0121] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0122] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0123] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0124] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for training a graph-to-text model, the method comprising: Acquire an image sample set, wherein the image sample set includes at least one image sample; Acquire an initial graph-to-text model, wherein the graph-to-text model includes: an image encoding subnetwork, a connection subnetwork, and a multimodal large language model; Training the image encoding subnetwork and the connection subnetwork based on the image sample set and the image generator to obtain trained image encoding subnetwork and connection subnetwork; Based on the image sample set, the multimodal large language model, the trained image encoding subnetwork and the trained connection subnetwork are trained to obtain a trained image-to-text model.

2. The method according to claim 1, wherein: The training of the image encoding subnetwork and the connection subnetwork based on the image sample set and the image generator to obtain the trained image encoding subnetwork and the connection subnetwork comprises: Fix the parameters of the image generator; Selecting an image sample from the image sample set; Inputting the selected image sample into the image encoding subnetwork to obtain the visual features after visual text alignment output by the connection subnetwork; Inputting the visual feature into the image generator to obtain a generated image generated by the image generator; Based on the generated image and the selected image sample, calculating the branch losses of the image encoding subnetwork and the connection subnetwork; Based on the branch loss, the trained image encoding subnetwork and connection subnetwork are obtained.

3. The method according to claim 2, wherein: The image encoding subnetwork and the connection subnetwork that are trained based on the branch loss include: Determining whether the branch loss is less than or equal to a branch loss threshold; In response to the judgment result that the branch loss is less than or equal to the branch loss threshold, a trained image encoding subnetwork and a connection subnetwork are obtained.

4. The method according to claim 1, wherein: The step of training the multimodal large language model, the trained image encoding subnetwork, and the trained connection subnetwork based on the image sample set to obtain the trained image-to-text model includes: removing the image generator; Selecting an image sample from the image sample set; Inputting the selected image sample into the trained image encoding subnetwork to obtain the image description text output by the multimodal large language model; Based on the image description text and the selected image sample, calculating the total loss of the multimodal large language model, the trained image encoding subnetwork, and the trained connection subnetwork; Based on the total loss, a trained graph-to-text model is obtained.

5. The method according to claim 4, wherein: The trained graph-to-text model obtained based on the total loss includes: Determining whether the total loss is less than or equal to a total loss threshold; In response to the judgment result that the total loss is less than or equal to the total loss threshold, a trained graph-to-text model is obtained.

6. A text generation method, the method comprising: Get the image to be processed; The image to be processed and the prompt word are input into the image-to-text model generated by the method according to any one of claims 1 to 5 to obtain the image description text of the image to be processed.

7. The method according to claim 6, further comprising: Based on the image description text, using a target detection model to perform image detection on the image to be processed to obtain a detection result; Based on the detection result, a new prompt word is obtained; The image to be processed and the new prompt word are input into the image-to-text model to obtain a new image description text output by the image-to-text model.

8. The method according to claim 7, wherein: Based on the image description text, the target detection model is used to perform image detection on the image to be processed, and the detection result obtained includes: Extracting entity information from the image description text; The entity information and the image to be processed are input into the target detection model to obtain a detection result of the entity information output by the target detection model.

9. The method according to claim 8, wherein: The new prompt word obtained based on the detection result includes: Detecting whether the detection result contains the entity information; In response to detecting that the detection result does not contain the entity information, generating a prompt word to be used based on the entity information; Inputting the prompt words to be used and the image to be processed into the image-to-text model to obtain a new image description text of the image to be processed; Based on the new image description text, using the target detection model to perform image detection on the image to be processed to obtain a new detection result; In response to the entity information being included in the new detection result, the new image description text is used as a new prompt word.

10. The method according to claim 8, wherein: The obtaining of a new prompt word based on the detection result further includes: In response to detecting that the detection result contains the entity information, the prompt word is used as a new prompt word.

11. A device for training a graph-based text model, comprising: A sample acquisition unit, configured to acquire an image sample set, wherein the image sample set includes at least one image sample; A model acquisition unit is configured to acquire an initial image-generated text model, wherein the image-generated text model includes: an image encoding subnetwork, a connection subnetwork, and a multimodal large language model; A network training unit is configured to train the image encoding subnetwork and the connection subnetwork based on the image sample set and the image generator to obtain a trained image encoding subnetwork and connection subnetwork; The model training unit is configured to train the multimodal large language model, the trained image encoding subnetwork and the trained connection subnetwork based on the image sample set to obtain a trained image-to-text model.

12. The device according to claim 11, wherein The network training unit is further configured to: fix the parameters of the image generator; select image samples from the image sample set; input the selected image samples into the image encoding subnetwork to obtain the visual features after visual text alignment output by the connection subnetwork; Inputting the visual feature into the image generator to obtain a generated image generated by the image generator; Based on the generated image and the selected image sample, calculating the branch losses of the image encoding subnetwork and the connection subnetwork; Based on the branch loss, the trained image encoding subnetwork and connection subnetwork are obtained.

13. The device according to claim 12, wherein: The network training unit is further configured to: determine whether the branch loss is less than or equal to a branch loss threshold; in response to the determination result that the branch loss is less than or equal to the branch loss threshold, obtain a trained image encoding subnetwork and a connection subnetwork.

14. The device according to claim 11, wherein: The model training unit is further configured to: remove the image generator; select image samples from the image sample set; input the selected image samples into the trained image encoding subnetwork to obtain the image description text output by the multimodal large language model; Based on the image description text and the selected image sample, the total loss of the multimodal large language model, the trained image encoding subnetwork and the trained connection subnetwork is calculated; based on the total loss, a trained image-to-text model is obtained.

15. The device according to claim 14, wherein: The model training unit is further configured to: determine whether the total loss is less than or equal to a total loss threshold; in response to the determination result that the total loss is less than or equal to the total loss threshold, obtain a trained graph-to-text model.

16. A text generation device, the device comprising: An image acquisition unit, configured to acquire an image to be processed; The image input unit is configured to input the image to be processed and the prompt word into the image-to-text model generated by the device according to any one of claims 11 to 15 to obtain the image description text of the image to be processed.

17. The apparatus according to claim 16, further comprising: An updating unit is configured to: perform image detection on the image to be processed by using a target detection model based on the image description text to obtain a detection result; Based on the detection result, a new prompt word is obtained; The image to be processed and the new prompt word are input into the image-to-text model to obtain a new image description text output by the image-to-text model.

18. According to the device of claim 17, the updating unit is further configured to: extract entity information from the image description text; input the entity information and the image to be processed into the target detection model to obtain a detection result of the entity information output by the target detection model.

19. The apparatus according to claim 18, wherein the updating unit is further configured to: detect whether the detection result contains the entity information; in response to detecting that the detection result does not contain the entity information, generate a prompt word to be used based on the entity information; Inputting the prompt words to be used and the image to be processed into the image-to-text model to obtain a new image description text of the image to be processed; Based on the new image description text, the target detection model is used to perform image detection on the image to be processed to obtain a new detection result; in response to the new detection result of the image description text having the entity information, the new image description text is used as a new prompt word.

20. The apparatus according to claim 18, wherein the updating unit is further configured to: in response to detecting that the detection result contains the entity information, use the prompt word as a new prompt word.

21. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 10.

22. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 10.

23. A computer program product, comprising a computer program, which, when executed by a processor, implements the method of any one of claims 1 to 10.

Images