Text information generation method, electronic equipment and storage medium

Abstract

The invention discloses a text information generation method, electronic equipment and a storage medium, and relates to the field of computer vision and natural language processing. The method comprises the steps that a to-be-processed image is acquired, the to-be-processed image comprises a foreground image and a background image, the image content of the foreground image comprises at least one display object, and the image content of the background image comprises the background where the display object is located; inputting the to-be-processed image into a graph-to-text model, and analyzing the to-be-processed image by using the graph-to-text model to obtain text description information of a background image in the to-be-processed image, the graph-to-text model being obtained by training by using an image sample and a text description information sample, the image content of a background image sample in the image samples comprises a background where at least one display object sample is located, and the text description information is used for describing the image content of the background image; and outputting the text description information. The technical problem that the efficiency of generating the text description information of the background image is low is solved.

Description

Technical Field

[0001] The present application relates to the fields of computer vision and natural language processing, and in particular to a method for generating text information, an electronic device and a storage medium. Background Art

[0002] At present, text description information (Caption) can provide annotation data for the training of the image-generated text model, so that the image-generated text model can learn to match visual information with language information. In the model reasoning stage, using Caption as the input of the image-generated text model can guide the image-generated text model to generate content related to Caption, thereby enhancing the relevance and consistency of the generated results.

[0003] In the related art, the caption obtained by the image-to-text model usually describes the entire image to be processed. Since the display object is usually located in a prominent area, a large amount of space is used to describe the display object in the obtained caption. However, the caption required in the image-to-text process needs to only describe the image content of the background image in the image to be processed, so there is a technical problem of low efficiency in generating text description information of the background image.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present application provide a method for generating text information, an electronic device, and a storage medium to at least solve the technical problem of low efficiency in generating text description information of a background image.

[0006] According to one aspect of an embodiment of the present application, a method for generating text information is provided. The method may include: obtaining an image to be processed, wherein the image to be processed includes a foreground image and a background image, the image content of the foreground image includes at least one display object, and the image content of the background image includes the background where the display object is located; inputting the image to be processed into a graph-to-text model, and using the graph-to-text model to analyze the image to be processed to obtain text description information of the background image in the image to be processed, wherein the graph-to-text model is trained using image samples and text description information samples, the image content of the background image sample in the image sample includes the background where at least one display object sample is located, the display object sample is in an occluded state in the image sample, the text description information sample is used to describe the image content of the background image sample, and the text description information is used to describe the image content of the background image; and outputting the text description information.

[0007] According to another aspect of the embodiment of the present application, a method for determining a model is provided. The method may include: obtaining an image sample, wherein the image content of a background image sample in the image sample includes the background where at least one display object sample is located, and the display object sample is in an occluded state in the image sample; obtaining a text description information sample of the image sample, wherein the text description information sample is used to describe the image content of the background image sample; using the image sample and the text description information sample, generating a graph-to-text model, wherein the graph-to-text model is used to analyze the image to be processed, and obtain text description information of the background image in the image to be processed, wherein the image content of the foreground image in the image to be processed includes at least one display object, the image content of the background image includes the background where the display object is located, and the text description information is used to describe the image content of the background image.

[0008] According to another aspect of the embodiment of the present application, a method for generating text information is provided. The method may include: identifying an image to be processed from an e-commerce platform, wherein the image to be processed includes a foreground image and a background image, the image content of the foreground image includes at least one product to be traded on the e-commerce platform, and the image content of the background image includes the background where the product is located; inputting the image to be processed into a graph-to-text model, and using the graph-to-text model to analyze the image to be processed to obtain text description information of the background image in the image to be processed, wherein the graph-to-text model is trained using image samples and text description information samples, the image content of the background image sample in the image sample includes the background where at least one display object sample is located, the display object sample is in an occluded state in the image sample, the text description information sample is used to describe the image content of the background image sample, and the text description information is used to describe the image content of the background image; outputting the text description information to the e-commerce platform.

[0009] According to another aspect of the embodiment of the present application, a method for generating text information is provided, which is applied to a model training end. The method may include: obtaining text description information of a background image in an image to be processed, wherein the image to be processed includes a foreground image and a background image, the image content of the foreground image includes at least one display object, the image content of the background image includes the background where the display object is located, the text description information is obtained by analyzing the image to be processed using a graph-to-text model, and is used to describe the image content of the background image, the graph-to-text model is trained using image samples and text description information samples, the image content of the background image sample in the image sample includes the background where at least one display object sample is located, the display object sample is in an occluded state in the image sample, and the text description information sample is used to describe the image content of the background image sample; at least the text description information is used as a training sample to train an output model of the model training end.

[0010] According to another aspect of the embodiment of the present application, a method for generating text information is provided, which is applied to a model reasoning end. The method may include: obtaining text description information of a background image in an image to be processed, wherein the image to be processed includes a foreground image and a background image, the image content of the foreground image includes at least one display object, the image content of the background image includes the background where the display object is located, the text description information is obtained by analyzing the image to be processed using a graph-to-text model, and is used to describe the image content of the background image, the graph-to-text model is trained using image samples and text description information samples, the image content of the background image sample in the image sample includes the background where at least one display object sample is located, the display object sample is in an occluded state in the image sample, and the text description information sample is used to describe the image content of the background image sample; using the text description information and the image to be processed, an output image of the model reasoning end is generated.

[0011] According to another aspect of an embodiment of the present application, a method for generating text information is provided, which may include: responding to an input instruction on an operation interface, displaying an image to be processed on the operation interface, wherein the image to be processed includes a foreground image and a background image, the image content of the foreground image includes at least one display object, and the image content of the background image includes the background where the display object is located; responding to a text generation instruction on the operation interface, displaying text description information of the background image in the image to be processed on the operation interface, wherein the text description information is obtained by analyzing the image to be processed using a graph-to-text model and is used to describe the image content of the background image, the graph-to-text model is trained using image samples and text description information samples, the image content of the background image samples in the image samples includes the background where at least one display object sample is located, the display object sample is in an occluded state in the image sample, and the text description information sample is used to describe the image content of the background image sample.

[0012] According to another aspect of an embodiment of the present application, an electronic device is also provided. The electronic device may include a memory and a processor: the memory is used to store computer-executable instructions, and the processor is used to execute computer-executable instructions. When the above-mentioned computer-executable instructions are executed by the processor, any one of the above-mentioned methods is implemented.

[0013] According to another aspect of an embodiment of the present application, a processor is further provided, and the processor is used to run a program, wherein any one of the above methods is executed when the program is running.

[0014] According to another aspect of an embodiment of the present application, a computer-readable storage medium is further provided, the computer-readable storage medium including a stored program, wherein when the program is running, the device where the storage medium is located is controlled to execute any of the above methods.

[0015] According to another aspect of an embodiment of the present application, a computer program product is also provided, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, any one of the above methods is implemented.

[0016] In an embodiment of the present application, an image to be processed is obtained, wherein the image to be processed includes a foreground image and a background image, the image content of the foreground image includes at least one display object, and the image content of the background image includes the background where the display object is located; the image to be processed is input into a graph-generated text model, and the graph-generated text model is used to analyze the image to be processed to obtain text description information of the background image in the image to be processed, wherein the graph-generated text model is trained using image samples and text description information samples, the image content of the background image sample in the image sample includes the background where at least one display object sample is located, the display object sample is in an occluded state in the image sample, the text description information sample is used to describe the image content of the background image sample, and the text description information is used to describe the image content of the background image; and the text description information is output. That is, in the embodiment of the present application, the display object contained in the foreground image in the image to be processed is occluded to obtain the background image, and the background image is analyzed by the graph-generated text model to obtain the text description information of the background image, thereby achieving the purpose of generating text description information that only describes the image content of the background image, thereby achieving the technical effect of improving the efficiency of generating text description information of the background image, and solving the technical problem of low efficiency of generating text description information of the background image.

[0017] It is easy to notice that the above general description and the following detailed description are only for the purpose of exemplifying and explaining the present application, and do not constitute a limitation of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0019] Figure 1 is a schematic diagram of an application scenario of a method for generating text information according to an embodiment of the present application;

[0020] Figure 2 is a flow chart of a method for generating text information according to an embodiment of the present application;

[0021] Figure 3 is a flow chart of a method for determining a model according to an embodiment of the present application;

[0022] Figure 4 is a flowchart of another method for generating text information according to an embodiment of the present application;

[0023] Figure 5 is a flowchart of another method for generating text information according to an embodiment of the present application;

[0024] Figure 6 is a flowchart of another method for generating text information according to an embodiment of the present application;

[0025] Figure 7 is a flowchart of another method for generating text information according to an embodiment of the present application;

[0026] Figure 8 is a schematic diagram of a scene image according to an embodiment of the present application;

[0027] Fig. 9 is a schematic diagram of a mask image according to an embodiment of the present application;

[0028] Fig.10 is a schematic diagram of another scene image according to an embodiment of the present application;

[0029] Fig.11 is a schematic diagram of another mask image according to an embodiment of the present application;

[0030] Fig.12 is a schematic diagram of a commodity image according to an embodiment of the present application;

[0031] Fig.13 is a schematic diagram of a reference image according to an embodiment of the present application;

[0032] Fig.14 is a schematic diagram of generating an effect image according to an embodiment of the present application;

[0033] Fig.15 is a schematic diagram of producing and filtering seed data according to an embodiment of the present application;

[0034] Fig.16 is a schematic diagram of the structure of a U2Net network according to an embodiment of the present application;

[0035] Fig.17 is a schematic diagram of a black and white mask image according to an embodiment of the present application;

[0036] Fig.18 is a schematic diagram of a background image according to an embodiment of the present application;

[0037] Fig.19 is a schematic diagram of a region image of a scene image according to an embodiment of the present application;

[0038] Fig. 20is a structural block diagram of a computing environment of a method for generating text information according to an embodiment of the present application;

[0039] Fig.21 is a schematic diagram of a device for generating text information according to an embodiment of the present application;

[0040] Fig. 22 is a schematic diagram of a device for determining a model according to an embodiment of the present application;

[0041] Fig.23 is a schematic diagram of another device for generating text information according to an embodiment of the present application;

[0042] Fig.24 is a schematic diagram of another device for generating text information according to an embodiment of the present application;

[0043] Fig.25 is a schematic diagram of another device for generating text information according to an embodiment of the present application;

[0044] Fig.26 is a schematic diagram of another device for generating text information according to an embodiment of the present application;

[0045] Fig. 27 is a structural block diagram of a computer terminal according to an embodiment of the present application;

[0046] Fig.28 is a block diagram of an electronic device for a method for generating text information according to an embodiment of the present application;

[0047] Fig.29 It is a hardware structure block diagram of a computer terminal (or mobile device) for implementing a method for generating text information according to an embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0049] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0050] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following explanations:

[0051] Model distillation is an effective knowledge transfer technology that can transfer the knowledge of a large model to a small model, so that the small model has similar performance to the large model.

[0052] Caption: descriptive text of an image, video or other media content;

[0053] Seed data refers to an image pair, which includes a background image showing an occluded object and text description information of the background image.

[0054] The above-mentioned text information generation method provided in the embodiment of the present application can be applied to Figure 1 The application scenarios shown are not limited to these. Figure 1 is a schematic diagram of an application scenario of a method for generating text information according to an embodiment of the present application. Figure 1 In the application scenario shown, the server 10 can be a cloud. The server 10 can be connected to one or more client devices 20 via a local area network connection, a wide area network connection, an Internet connection, or other types of data networks. The client devices 20 here may include but are not limited to: smart phones, tablet computers, laptops, PDAs, personal computers, smart home devices, vehicle-mounted devices, etc. The client devices together constitute the client relative to the server. An operating interface for obtaining the image to be processed can be deployed on the graphical user interface on the client device. The client device 20 can interact with the user through the graphical user interface to implement the method for generating text information provided in the embodiment of the present application.

[0055] In the embodiment of the present application, the system composed of the client device 20 and the server 10 can perform the following steps: if the user needs to obtain the image to be processed, the user can perform the corresponding operation in the operation interface on the client device 20 to obtain the image to be processed. The client device can obtain the image to be processed and send it to the server through the network. After receiving the image to be processed, the server can perform the following steps: step S102, obtaining the image to be processed, wherein the image to be processed includes a foreground image and a background image, the image content of the foreground image includes at least one display object, and the image content of the background image includes the background where the display object is located; step S104, inputting the image to be processed into the image-generated text model, and using the image-generated text model to analyze the image to be processed to obtain text description information of the background image in the image to be processed, wherein the image-generated text model is trained using image samples and text description information samples, the image content of the background image sample in the image sample includes the background where at least one display object sample is located, the display object sample is in an occluded state in the image sample, the text description information sample is used to describe the image content of the background image sample, and the text description information is used to describe the image content of the background image; step S106, outputting the text description information.

[0056] Under the above operating environment, this application provides Figure 2 The method for generating the text information shown. Figure 2 is a flowchart of a method for generating text information according to an embodiment of the present application. Figure 2 As shown, the method may include the following steps:

[0057] Step S202, obtaining an image to be processed.

[0058] In the technical solution provided in the above step S202 of the present application, an image to be processed may be obtained, wherein the image to be processed may be a scene image (Image), and the image to be processed may at least include a foreground image and a background image.

[0059] It should be noted that the image content of the foreground image may include at least one display object, which may be a commodity, referred to as a main object, such as a sofa, bed, etc. in a furniture application scenario. This is only an example and does not impose specific restrictions on the type of display object.

[0060] Optionally, the image content of the background image may include the background where the display object is located. When the display object is a commodity, the image content of the background image may include the background where the commodity is located. For example, it may include the background above, below, left and right of a sofa or bed. This is only for example. The background at any position where the display object is located is included in the image content of the background image.

[0061] Step S204: input the image to be processed into the image-to-text model, and use the image-to-text model to analyze the image to be processed to obtain text description information of the background image in the image to be processed.

[0062] In the technical solution provided in the above step S204 of the present application, after obtaining the image to be processed, the obtained image to be processed can be input into the image-to-text model, and the image to be processed is analyzed by the image-to-text model to obtain text description information of the background image in the image to be processed. The text description information (Caption) can be used to describe the image content of the background image.

[0063] It should be noted that the image-to-text model can be trained using image samples and text description information samples. The image content of the background image sample in the image sample may include the background where at least one display object sample is located, and the display object sample is in an occluded state in the image sample. The text description information sample can be used to describe the image content of the background image sample.

[0064] In this embodiment, after acquiring the image to be processed, the foreground image and the background image can be identified from the image to be processed. Since the foreground image may contain a main object, and the main object has a large area or occupies a large area, the main object can be identified from the image to be processed first. Further, the main object can be occluded to obtain the background image. After the background image is identified from the image to be processed, the image-to-text model can be used to analyze the background image to obtain text description information of the background image.

[0065] For example, the image to be processed may be an image to be processed in a furniture application scenario. After acquiring the image to be processed, a foreground image and a background image may be identified from the image to be processed. Since the foreground image contains a main object such as a sofa or a bed, and the main object has a large area or occupies a large area, the main object such as the sofa or the bed may be identified from the image to be processed first. Furthermore, the main object such as the sofa or the bed may be occluded to obtain a background image. After the background image is identified from the image to be processed, the image-to-text model may be used to analyze the background image to obtain text description information of the background image, and the text description information only describes the background where the main object such as the sofa or the bed is located.

[0066] Step S206: output text description information.

[0067] In the technical solution provided in the above step S206 of the present application, after the image to be processed is analyzed using the image-to-text model to obtain the text description information of the background image in the image to be processed, the text description information can be output.

[0068] In this embodiment, the text description information can be directly output, or the text description information can be converted into audio for output. In addition, the text description information can also be transmitted to the client, and the text description information is displayed by the operation interface on the client to enable the user to view the text description information more intuitively.

[0069] Optionally, the embodiment may output text description information to the model training end and the model reasoning end respectively. After the text description information is output to the model training end, the text description information may be used as training data to train a large model of the raw image of the home decoration picture. After the text description information is output to the model reasoning end, the text description information is used to increase the ability to understand the scene image, thereby supporting the reference image capability in the reasoning process of the large model of the raw image of the home decoration picture.

[0070] Through the above steps S202 to S206 of the present application, an image to be processed is obtained, wherein the image to be processed includes a foreground image and a background image, the image content of the foreground image includes at least one display object, and the image content of the background image includes the background where the display object is located; the image to be processed is input into a graph-to-text model, and the image to be processed is analyzed using the graph-to-text model to obtain text description information of the background image in the image to be processed, wherein the graph-to-text model is trained using image samples and text description information samples, the image content of the background image sample in the image sample includes the background where at least one display object sample is located, the display object sample is in an occluded state in the image sample, the text description information sample is used to describe the image content of the background image sample, and the text description information is used to describe the image content of the background image; and the text description information is output. That is to say, the embodiment of the present application performs occlusion processing on the display object contained in the foreground image in the processed image to obtain a background image, and uses the image-to-text model to analyze the background image to obtain text description information of the background image, thereby achieving the purpose of generating text description information that only describes the image content of the background image, thereby achieving the technical effect of improving the efficiency of generating text description information of the background image, and solving the technical problem of low efficiency in generating text description information of the background image.

[0071] The above method of this embodiment is further introduced below.

[0072] As an optional implementation, step S204, using the image-to-text model to analyze the image to be processed to obtain text description information of the background image in the image to be processed, including: using the image-to-text model to identify the background image from the image to be processed; using the image-to-text model to perform image-to-text processing on the background image to obtain text description information.

[0073] In this embodiment, after obtaining the image to be processed, the image-to-text model can be used to identify the background image from the image to be processed. After identifying the background image from the image to be processed using the image-to-text model, the image-to-text model can be used to perform image-to-text processing on the background image to obtain text description information.

[0074] Optionally, after obtaining the image to be processed, since the display object (main object) contained in the foreground image of the image to be processed usually occupies a larger area, the display object in the image to be processed can be first identified and the display object can be occluded to obtain the background image. The background image can further be processed using the image-to-text model to obtain text description information that only describes the image content of the background image.

[0075] As an optional implementation, a background image is identified from an image to be processed using a graph-based model, including: using a graph-based model to determine a mask image corresponding to the image to be processed, wherein the mask image includes an area where a display object is displayed in the image to be processed; and using the graph-based model to perform masking processing on an area in the image to be processed corresponding to the mask image to obtain a background image.

[0076] In this embodiment, after obtaining the image to be processed, the image-generated text model can be used to determine the mask image corresponding to the image to be processed. After determining the mask image corresponding to the image to be processed, the image-generated text model can be used to perform masking processing on the area corresponding to the mask image in the image to be processed to obtain a background image. The mask image may include the area where the display object is displayed in the image to be processed, which can be called a black and white mask image.

[0077] It should be noted that Going Deeper with Nested U-Structure for Salient Object Detection (U2Net) is proposed for the Salient Object Detection (SOD) task. The Salient Object Detection task is similar to the semantic segmentation task. The Salient Object Detection task is a binary classification task, which is to segment the most attractive object or area in the image, so it includes two categories: foreground and background.

[0078] Optionally, the image to be processed is segmented using a U2Net network, and the display object area (product area, main product area) and the background area where the display object is located in the image to be processed can be segmented to obtain a black and white mask image. After the image to be processed is segmented to obtain a black and white mask image, the product area in the black and white mask image can be blackened, that is, the area corresponding to the black and white mask image in the image to be processed is blocked to obtain a background image.

[0079] Optionally, after performing image segmentation on the image to be processed to obtain a black and white mask image, image processing can be performed on the black and white mask image to set the pixel values ​​corresponding to areas in the black and white mask image whose pixel values ​​are greater than a specified threshold to 0, that is, convert the product area to black to obtain a background image.

[0080] This embodiment performs image segmentation on the image to be processed to obtain a black-and-white mask image, and further performs image processing on the black-and-white mask image to obtain a background image in which the main object is blocked.

[0081] As an optional implementation, the image-to-text model is used to perform image-to-text processing on the background image to obtain text description information, including: obtaining prompt information, wherein the prompt information is used to at least indicate that the text description information to be generated is prohibited from including information about the display object; and using the prompt information to guide the image-to-text model to perform image-to-text processing on the background image to obtain text description information.

[0082] In this embodiment, prompt information (Prompt) can be obtained, and the prompt information can also be called the guiding information of the image-to-text model. After obtaining the prompt information and using the image-to-text model to perform masking processing on the area corresponding to the mask image in the image to be processed to obtain the background image, the obtained prompt information can be used to guide the image-to-text model to perform image-to-text processing on the background image to obtain text description information. Among them, the prompt information can be used to at least indicate that the text description information to be generated is prohibited from including information about the display object, which can be called a prompt word and can be designed according to actual needs.

[0083] It should be noted that before using the prompt information to guide the image-to-text model to process the background image and obtain the text description information, the main product area needs to be ignored because the obtained text description information only describes the image content of the background image. Therefore, the above prompt words need to contain the complete text description information of the image to be processed to obtain more accurate text description information. In addition, the above prompt words also need to limit the output format of the text description information to ensure that the output Capt ion contains relevant information such as location layout.

[0084] It should be noted that the complete text description information of the image to be processed can be obtained by calling the natural language processing (Generative Pre-trained Transformer 4OpenAI, abbreviated as GPT-4o) model. First, a detailed and accurate prompt word is designed (the prompt word is different from the prompt word containing the complete text description information of the image to be processed), and the prompt word and the image to be processed are used as the input of the GPT-4o model. By calling the GPT-4o model, the complete text description information of the image to be processed output by the GPT-4o model is obtained.

[0085] Optionally, a prompt word is designed according to actual needs, and the prompt word needs to include complete text description information of the image to be processed, and the output format of the text description information is limited. Further based on the designed prompt word, the image-to-text model is guided to process the background image to obtain a Caption, and the Caption only describes the image content of the background image.

[0086] Optionally, the complete text description information of the image to be processed, the prompt words containing the complete text description information and limiting the output format of the text description information, and the background image in which the main object is occluded are taken as inputs of the GPT-4o model, and by calling the GPT-4o model, a Capt ion output by the GPT-4o model that only describes the image content of the background image is obtained.

[0087] As an optional implementation, step S206, outputting text description information, includes: outputting text description information to a model training end, wherein the text description information, the image to be processed, and a mask image corresponding to the image to be processed are used as training samples to train an output model of the model training end, wherein the mask image includes the area where the display object is displayed in the image to be processed.

[0088] In this embodiment, after analyzing the image to be processed using the image-to-text model and obtaining the text description information of the background image in the image to be processed, the text description information can be output to the model training end. Among them, the text description information, the image to be processed, and the mask image corresponding to the image to be processed can be used as training samples to train the output model of the model training end. The above-mentioned text description information, the image to be processed, and the mask image corresponding to the image to be processed can be called triple data. The output model can be a large model of the home decoration picture. The mask image can include the area where the display object is displayed in the image to be processed.

[0089] Optionally, after obtaining the image to be processed, the image to be processed is segmented to obtain a black-and-white mask image, and the black-and-white mask image is image processed to blacken the product area in the black-and-white mask image to obtain a background image. Further, the image-to-text model is guided to perform image-to-text processing on the background image using the complete text description information of the image to be processed and the prompt words that limit the output format of the text description information to obtain text description information.

[0090] Optionally, after obtaining the above-mentioned image to be processed, black-and-white mask image and text description information, the image to be processed, the black-and-white mask image and text description information can be used as triplet data to train a large model of the raw home decoration picture at the model training end.

[0091] This embodiment trains a large model of raw home decoration pictures by using the image to be processed, the black-and-white mask image and the text description information, thereby solving the problem of triplet data construction in the training process of the large model of raw home decoration pictures.

[0092] As an optional implementation, step S206, outputting text description information, includes: outputting the text description information to the model reasoning end, wherein the text description information and the image to be processed are used to generate an output image of the model reasoning end.

[0093] In this embodiment, after the image to be processed is analyzed using the image-to-text model and the text description information of the background image in the image to be processed is obtained, the text description information can be output to the model reasoning end. The text description information and the image to be processed can be used to generate an output image of the model reasoning end. The image to be processed can be a reference image. The text description information can be a Caption that only describes the background content of the reference image. The output image can be a generated effect image output based on the reference image and the Caption that only describes the background content of the reference image.

[0094] Optionally, the product image, the reference image and the Caption describing only the background of the reference image are used as inputs of the large model of the raw image of the home decoration picture. By calling the large model of the raw image of the home decoration picture, the generated effect image output by the large model of the raw image of the home decoration picture can be obtained.

[0095] This embodiment supports the reference graph capability during the model reasoning process, increases the ability to understand scene images, and solves problems such as unclear reference content of the reference graph during the reasoning process.

[0096] In an embodiment of the present application, an image to be processed is obtained, wherein the image to be processed includes a foreground image and a background image, the image content of the foreground image includes at least one display object, and the image content of the background image includes the background where the display object is located; the image to be processed is input into a graph-generated text model, and the graph-generated text model is used to analyze the image to be processed to obtain text description information of the background image in the image to be processed, wherein the graph-generated text model is trained using image samples and text description information samples, the image content of the background image sample in the image sample includes the background where at least one display object sample is located, the display object sample is in an occluded state in the image sample, the text description information sample is used to describe the image content of the background image sample, and the text description information is used to describe the image content of the background image; and the text description information is output. That is, in the embodiment of the present application, the display object contained in the foreground image in the image to be processed is occluded to obtain the background image, and the background image is analyzed by the graph-generated text model to obtain the text description information of the background image, thereby achieving the purpose of generating text description information that only describes the image content of the background image, thereby achieving the technical effect of improving the efficiency of generating text description information of the background image, and solving the technical problem of low efficiency of generating text description information of the background image.

[0097] The present application also provides a method for determining a model. Figure 3 is a flow chart of a method for determining a model according to an embodiment of the present application, such as Figure 3 As shown, the method may include the following steps:

[0098] Step S302, obtaining image samples.

[0099] In the technical solution provided in the above step S302 of the present application, an image sample can be obtained. Among them, the image content of the background image sample in the image sample includes the background where at least one display object sample is located, and the display object sample is in an obscured state in the image sample. The display object sample can be a commodity sample, which can be called a main object sample, that is, the image sample is an image sample whose main object sample is obscured.

[0100] In this embodiment, an initial image sample can be obtained and image segmentation can be performed on the initial image sample to segment the main object sample area (product sample area) and the background area in the initial image sample, and the main object sample area is set to black, thereby obtaining an image sample in which the main object sample is obscured.

[0101] Step S304: Obtain a text description information sample of the image sample.

[0102] In the technical solution provided in the above step S304 of the present application, after obtaining the image sample, a text description information sample of the image sample can be obtained, wherein the text description information sample can be used to describe the image content of the background image sample.

[0103] In this embodiment, after the image sample is obtained, since the display object sample in the image sample is in an occluded state, the image sample only contains the background image sample, thereby obtaining the background image sample. Further, a text description information sample of the background image sample can be obtained, and the text description information sample only describes the image content of the background image sample.

[0104] It should be noted that the image sample and the text description information sample of the image sample can be called seed data. Since the object sample displayed in the image sample is in an occluded state, the image sample is a background image sample, that is, the background image sample and the text description information sample of the background image sample can be called seed data.

[0105] Step S306: Generate an image-to-text model using the image samples and the text description information samples.

[0106] In the technical solution provided in the above step S306 of the present application, after obtaining the text description information sample of the image sample, the image sample and the text description information sample obtained can be used to generate a picture-to-text model. The picture-to-text model can be used to analyze the image to be processed to obtain the text description information of the background image in the image to be processed. The text description information Capt ion can be used to describe the image content of the background image.

[0107] It should be noted that the image to be processed may be a scene image Image, which may include at least a foreground image and a background image. The image content of the foreground image in the image to be processed includes at least one display object. The image content of the background image includes the background where the display object is located.

[0108] In this embodiment, after obtaining the image samples and the text description information samples, the image samples and the text description information samples may be used to train and obtain an image-to-text model.

[0109] Through the above steps S302 to S306 of the present application, an image sample is obtained, wherein the image content of the background image sample in the image sample includes the background where at least one display object sample is located, and the display object sample is in an occluded state in the image sample; a text description information sample of the image sample is obtained, wherein the text description information sample is used to describe the image content of the background image sample; and a graph-to-text model is generated using the image sample and the text description information sample, wherein the graph-to-text model is used to analyze the image to be processed to obtain text description information of the background image in the image to be processed, the image content of the foreground image in the image to be processed includes at least one display object, the image content of the background image includes the background where the display object is located, and the text description information is used to describe the image content of the background image, thereby achieving a technical effect and solving the technical problem of low efficiency in generating text description information of the background image.

[0110] The above method of this embodiment is further introduced below.

[0111] As an optional implementation, step S302, obtaining an image sample, includes: obtaining a mask image sample corresponding to an initial image sample, wherein the initial image sample includes a foreground image sample and a background image sample, the image content of the foreground image sample includes a display object sample, and the mask image sample includes an area displayed by the display object sample in the initial image sample; performing masking processing on an area corresponding to the mask image sample in the initial image sample to obtain an image sample.

[0112] In this embodiment, a mask image sample corresponding to the initial image sample may be obtained. After obtaining the mask image sample corresponding to the initial image sample, a masking process may be performed on the area corresponding to the mask image sample in the initial image sample to obtain an image sample. The initial image sample may include a foreground image sample and a background image sample. The image content of the foreground image sample may include a display object sample. The mask image sample may include an area displayed by the display object sample in the initial image sample, which may be referred to as a black and white mask image sample.

[0113] Optionally, after obtaining the initial image sample, since the initial image sample includes a foreground image sample and a background image sample, the image content of the foreground image sample includes a display object sample, and the display object sample usually occupies a larger area, therefore, the main object sample in the initial image sample can be first identified and the main object sample can be occluded to obtain the image sample.

[0114] As an optional implementation, obtaining a mask image sample corresponding to the initial image sample includes: performing image segmentation on the initial image sample to obtain the mask image sample; performing masking processing on the area corresponding to the mask image sample in the initial image sample to obtain the image sample, including: setting the area corresponding to the mask image sample in the initial image sample to black to obtain the image sample.

[0115] In this embodiment, after obtaining the initial image sample, the initial image sample can be segmented to obtain the mask image sample. After obtaining the mask image sample, the area corresponding to the mask image sample can be masked, that is, the area corresponding to the mask image sample is blackened to obtain the image sample.

[0116] It should be noted that the U2Net network is proposed for the salient object detection task SOD. The salient object detection task is similar to the semantic segmentation task. The salient object detection task is a binary classification task, and its task is to segment the most attractive object or area in the image, so it includes two categories: foreground and background.

[0117] Optionally, the U2Net network is used to perform image segmentation on the initial image sample, and the display object sample area (commodity sample area, main commodity sample area, main object sample area) and the background area where the display object sample is located in the initial image sample can be segmented to obtain a black and white mask image sample. After the initial image sample is segmented to obtain a black and white mask image sample, the commodity sample area in the black and white mask image sample can be set to black, that is, the area corresponding to the black and white mask image sample in the initial image sample is blocked to obtain an image sample.

[0118] Optionally, after performing image segmentation on the initial image sample to obtain a black and white mask image sample, image processing can be performed on the black and white mask image sample to set the pixel value corresponding to the area in the black and white mask image sample where the pixel value is greater than a specified threshold to 0, that is, convert the product sample area to black to obtain an image sample.

[0119] This embodiment performs image segmentation on the initial image samples to obtain black-and-white mask image samples, and further performs image processing on the black-and-white mask image samples to obtain image samples where the main object samples are blocked.

[0120] As an optional implementation, step S304, obtaining a text description information sample of the image sample, includes: obtaining a complete text description information sample of the image sample, wherein the complete text description information sample is used to describe the complete image content of the image sample; and generating a text description information sample based on the complete text description information sample and the image sample.

[0121] In this embodiment, after acquiring the image sample, a complete text description information sample of the image sample may be acquired. Further based on the acquired complete text description information sample and the image sample, a text description information sample may be generated. The complete text description information sample may be used to describe the complete image content of the image sample.

[0122] Optionally, based on the acquired image samples and the manually designed prompt word samples, a complete text description information sample of the image sample can be obtained. Further, the obtained complete text description information sample and the image sample of the main object sample being occluded are input into the natural language processing model to obtain a text description information sample.

[0123] As an optional implementation, obtaining a complete text description information sample of an image sample includes: obtaining a first prompt information sample, wherein the first prompt information sample is used to represent the rules that need to be satisfied by the complete text description information to be generated; using the first prompt information sample to guide a natural language processing model to analyze the image sample to obtain a complete text description information sample.

[0124] In this embodiment, after obtaining the image sample, a first prompt information sample can be obtained. Further using the obtained first prompt information sample, a natural language processing model can be guided to analyze the image sample to obtain a complete text description information sample. Among them, the first prompt information sample can be used to represent the rules that the complete text description information to be generated needs to meet, and can be called a first prompt word sample. The natural language processing model can be a GPT-4o model.

[0125] Optionally, a detailed and accurate first prompt word sample is designed according to actual needs. After obtaining the image sample and designing the first prompt word sample, the first prompt word sample and the image sample can be used as inputs of the GPT-4o model, and the complete text description information sample of the image sample output by the GPT-4o model can be obtained by calling the GPT-4o model.

[0126] As an optional implementation, a text description information sample is generated based on a complete text description information sample and an image sample, including: obtaining a second prompt information sample, wherein the second prompt information sample is used to represent a rule that needs to be satisfied by the text description information sample to be generated; using the second prompt information sample to guide a natural language processing model to analyze the complete text description information sample and the image sample to obtain a text description information sample.

[0127] In this embodiment, after obtaining the complete text description information sample, a second prompt information sample can be obtained. Further using the obtained second prompt information sample, the natural language processing model can be guided to analyze the obtained complete text description information sample and image sample to obtain the text description information sample. The second prompt information sample can be used to represent the rules that the text description information sample to be generated needs to meet, and can be called a second prompt word sample.

[0128] Optionally, before using the second prompt information sample to guide the natural language processing model to analyze the complete text description information sample and the image sample to obtain the text description information sample, the main product sample area needs to be ignored because the obtained text description information sample only describes the image content of the background image sample. Therefore, the above second prompt word sample needs to include the complete text description information sample of the image sample to obtain a more accurate text description information sample. In addition, the above second prompt word sample also needs to limit the output format of the text description information sample to ensure that the output Capt ion sample contains relevant information such as location layout.

[0129] It should be noted that the first prompt word sample is different from the second prompt word sample. The first prompt word sample needs to be detailed and accurate, while the second prompt word sample needs to include a complete text description information sample of the image sample and the output format of the text description information sample needs to be limited.

[0130] Optionally, a second prompt word sample is designed according to actual needs, and the second prompt word sample needs to include a complete text description information sample of the image sample, and the output format of the text description information sample is limited. Further based on the designed second prompt word sample, the natural language processing model is guided to analyze the complete text description information sample and the image sample to obtain a Caption sample, and the Caption sample only describes the image content of the background image sample.

[0131] Optionally, a complete text description information sample, a second prompt word sample containing the complete text description information sample and limiting the output format of the text description information sample, and an image sample in which the main object sample is occluded are taken as inputs of the GPT-4o model, and by calling the GPT-4o model, a Capt ion sample output by the GPT-4o model that only describes the image content of the background image sample is obtained.

[0132] As an optional implementation, the image sample includes image contents of multiple sub-regions associated with the display object sample, and the method further includes: obtaining multiple sub-text description information samples corresponding one-to-one to the multiple sub-regions from the text description information sample, wherein the sub-text description information samples are used to describe the image contents of the corresponding sub-regions; determining the correlation between the sub-text description information samples and the image contents of the sub-regions, wherein the correlation is used to express the degree of correlation between the sub-text description information samples and the image contents of the sub-regions; in response to the correlation being lower than a correlation threshold, prohibiting the use of the image sample and the text description information sample to generate an image-to-text model.

[0133] In this embodiment, the image sample includes the image contents of multiple sub-regions associated with the display object sample. After obtaining the text description information sample, multiple sub-text description information samples corresponding to the multiple sub-regions can be obtained from the obtained text description information sample. After determining the multiple sub-text description information samples, the correlation between the sub-text description information samples and the image contents of the sub-regions can be determined. When the correlation between the determined sub-text description information sample and the image contents of the sub-region is lower than the correlation threshold, in response to the correlation being lower than the correlation threshold, the use of the image sample and the text description information sample to generate the image-to-text model can be prohibited.

[0134] It should be noted that the sub-text description information sample can be used to describe the image content of the corresponding sub-region. The correlation can be used to express the correlation between the sub-text description information sample and the image content of the sub-region. The correlation threshold can be a threshold set according to actual conditions. For example, the correlation threshold can be 25. This is only an example and does not impose a specific limit on the value of the correlation threshold.

[0135] Optionally, after obtaining seed data including image samples and text description information samples, in order to obtain high-quality seed data, it is necessary to further filter out illusions or erroneous objects contained in the seed data. After obtaining the text description information samples, the text description information samples are formatted, and the image samples are divided into several regions according to the formatted text description information samples to obtain multiple sub-regions and multiple sub-text description information samples corresponding to the multiple sub-regions. The image-text correlation calculation is further performed on each sub-region and the corresponding sub-text description information sample to obtain the correlation between the sub-text description information sample and the image content of the sub-region.

[0136] It should be noted that the Contrastive Language-Image Pre-training (CLIP) can be used to calculate the image-text correlation of each sub-region of the image sample, significantly reducing the hallucination problem of the seed data. CLIP is a deep neural network that is trained through unsupervised learning of a large number of images and texts. It can embed images and texts into the same vector space, so that the semantic similarity across modalities can be calculated.

[0137] For example, the correlation threshold is set to 25. When the correlation between the sub-text description information sample and the image content of the sub-region is 28 calculated by CLIP, the image sample and the text description information sample can be used to generate the image-to-text model. When the correlation between the sub-text description information sample and the image content of the sub-region is 24 calculated by CLIP, the seed data is filtered to prohibit the use of the image sample and the text description information sample to generate the image-to-text model.

[0138] This embodiment uses CLIP to calculate the image-text correlation of each region of the image and filters the seed data, thereby reducing the illusion problem of the seed data to a certain extent.

[0139] As an optional implementation, an image-to-text model is generated using image samples and text description information samples, including: determining the image samples and text description information samples as seed data; using the seed data to perform model distillation on a student model to obtain an image-to-text model, wherein the parameter amount of the student model is less than a parameter amount threshold.

[0140] In this embodiment, after obtaining the image sample and the text description information sample, the image sample and the text description information sample can be determined as seed data. The seed data can be further used to perform model distillation on the student model to obtain the image-to-text model. Among them, the parameter amount of the student model is less than the parameter amount threshold. The student model can be an image annotation base model (Bl ip Image Capt ioning Base). The parameter amount threshold can be a threshold set according to actual conditions, for example, it can be 0.2B (B means billion, billion), which is only for example illustration, and no specific limitation is made to the value of the parameter amount threshold.

[0141] Optionally, after obtaining the image sample and the text description information sample, the image sample and the text description information sample can be determined as seed data. The seed data can be further filtered to filter the seed data whose correlation between the sub-text description information sample and the image content of the sub-region is lower than the correlation threshold, so as to reduce the hallucination problem of the seed data. After filtering the seed data, the filtered seed data can be used to perform model distillation on the student model to obtain an image-to-text model.

[0142] It should be noted that model distillation is a knowledge transfer technology that transfers the knowledge of a large and complex teacher model to a small and fast student model. This is very useful in the field of deep learning and can create smaller and faster models with performance close to that of large models, so that small models have similar performance to large models.

[0143] It should be noted that the Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation (BLIP) model can be used. For example, the image annotation basic model has outstanding performance in tasks such as image description and visual question answering, and the number of parameters is not high, which ensures the efficiency of model reasoning.

[0144] Optionally, the knowledge of the large model is mainly reflected through seed data. The Bl ip Image Capt ioning Base is used as the student model, and the student model is distilled using the filtered seed data to obtain an image-to-text model.

[0145] It should be noted that after completing the model distillation step, the distilled model can be evaluated to evaluate the training results of the distilled model under different hyperparameters, and the model that meets the performance requirements can be selected as the final result.

[0146] This embodiment uses the GPT-4o model to generate the seed data required for the BLIP model distillation, so that the distilled model can output more detailed Captions. During the distillation process, the student model used is the Bl ipImage Captioning Base, which has only 0.2B parameters, ensuring the effect of model reasoning.

[0147] In an embodiment of the present application, an image sample is obtained, wherein the image content of a background image sample in the image sample includes the background where at least one display object sample is located, and the display object sample is in an occluded state in the image sample; a text description information sample of the image sample is obtained, wherein the text description information sample is used to describe the image content of the background image sample; and a graph-to-text model is generated using the image sample and the text description information sample, wherein the graph-to-text model is used to analyze the image to be processed to obtain text description information of the background image in the image to be processed, wherein the image content of the foreground image in the image to be processed includes at least one display object, and the image content of the background image includes the background where the display object is located, and the text description information is used to describe the image content of the background image, thereby achieving a technical effect and solving the technical problem of low efficiency in generating text description information of the background image.

[0148] The present application also provides a method for generating text information. Figure 4 is a flowchart of a method for generating text information according to an embodiment of the present application. Figure 4 As shown, the method may include the following steps:

[0149] Step S402: identifying an image to be processed from the e-commerce platform.

[0150] In the technical solution provided in the above step S402 of the present application, an image to be processed can be identified from the e-commerce platform, wherein the image to be processed can be a scene image Image, which can at least include a foreground image and a background image.

[0151] It should be noted that the image content of the foreground image may include at least one product to be traded on the e-commerce platform. The product may be a commodity, which may be called a main object, such as a sofa, a bed, etc. This is only an example and does not impose specific restrictions on the type of product.

[0152] Optionally, the image content of the background image includes the background where the product is located. When the product is a commodity, the image content of the background image may include the background where the commodity is located. For example, it may include the background above, below, left and right of the sofa or bed. This is only for example. The background at any position of the product is included in the image content of the background image.

[0153] Step S404: input the image to be processed into the image-to-text model, and use the image-to-text model to analyze the image to be processed to obtain text description information of the background image in the image to be processed.

[0154] In the technical solution provided in the above step S404 of the present application, after the image to be processed is identified from the e-commerce platform, the identified image to be processed can be input into the image-to-text model, and the image to be processed is analyzed using the image-to-text model to obtain text description information of the background image in the image to be processed. The text description information Capt ion is used to describe the image content of the background image.

[0155] It should be noted that the image-to-text model can be trained using image samples and text description information samples. The image content of the background image sample in the image sample includes the background where at least one product sample is located, and the display object sample is in an occluded state in the image sample. The text description information sample is used to describe the image content of the background image sample.

[0156] In this embodiment, after the image to be processed is identified from the e-commerce platform, the foreground image and the background image can be identified from the image to be processed. Since the foreground image contains the main object, and the main object has a large area or occupies a large area, the main object can be identified from the image to be processed first. Further, the main object can be occluded to obtain the background image. After the background image is identified from the image to be processed, the image-to-text model can be used to analyze the background image to obtain text description information of the background image.

[0157] Step S406, outputting text description information to the e-commerce platform.

[0158] In the technical solution provided in the above step S406 of the present application, after the image to be processed is analyzed using the image-to-text model to obtain the text description information of the background image in the image to be processed, the text description information can be output to the e-commerce platform.

[0159] In this embodiment, the text description information can be output to the model training end and the model reasoning end respectively. After the text description information is output to the model training end, the text description information can be used as training data to train a large model of the raw image of the home decoration picture. After the text description information is output to the model reasoning end, the text description information is used to increase the ability to understand the scene image, thereby supporting the reference image capability in the reasoning process of the large model of the raw image of the home decoration picture.

[0160] Through the above steps S402 to S406 of the present application, an image to be processed is identified from an e-commerce platform, wherein the image to be processed includes a foreground image and a background image, the image content of the foreground image includes at least one product to be traded on the e-commerce platform, and the image content of the background image includes the background where the product is located; the image to be processed is input into a graph-to-text model, and the image to be processed is analyzed using the graph-to-text model to obtain text description information of the background image in the image to be processed, wherein the graph-to-text model is trained using image samples and text description information samples, the image content of the background image sample in the image sample includes the background where at least one display object sample is located, the display object sample is in an occluded state in the image sample, the text description information sample is used to describe the image content of the background image sample, and the text description information is used to describe the image content of the background image; the text description information is output to the e-commerce platform, thereby achieving the technical effect of improving the efficiency of generating text description information of the background image, and solving the technical problem of low efficiency in generating text description information of the background image.

[0161] The present application also provides a method for generating text information, which can be applied to a model training end. Figure 5 is a flowchart of a method for generating text information according to an embodiment of the present application. Figure 5 As shown, the method may include the following steps:

[0162] Step S502: obtaining text description information of the background image in the image to be processed.

[0163] In the technical solution provided in the above step S502 of the present application, text description information of the background image in the image to be processed can be obtained. The image to be processed can be a scene image Image, which can at least include a foreground image and a background image.

[0164] It should be noted that the image content of the foreground image includes at least one display object, and the image content of the background image includes the background where the display object is located.

[0165] Optionally, the text description information Capt ion can be obtained by analyzing the image to be processed using a graph-to-text model, and is used to describe the image content of the background image. The graph-to-text model is trained using image samples and text description information samples, and the image content of the background image sample in the image sample includes the background where at least one display object sample is located, and the display object sample is in an occluded state in the image sample. The text description information sample is used to describe the image content of the background image sample.

[0166] In this embodiment, an image to be processed can be obtained. After the image to be processed is obtained, the foreground image and the background image can be identified from the image to be processed. Since the foreground image contains a main object, and the main object has a large area or occupies a large area, the main object can be first identified from the image to be processed. Further, the main object can be occluded to obtain a background image. After the background image is identified from the image to be processed, the image-to-text model can be used to analyze the background image to obtain text description information of the background image.

[0167] Step S504, using at least the text description information as a training sample, training to obtain an output model of the model training end.

[0168] In the technical solution provided in the above step S504 of the present application, after obtaining the text description information of the background image in the image to be processed, at least the text description information is used as a training sample to train an output model of the model training end. The output model can be a large model of the raw image of the home decoration picture.

[0169] In this embodiment, after obtaining the image to be processed, the U2Net network can be used to perform image segmentation on the image to be processed, and the display object area (commodity area, main commodity area) and the background area where the display object is located in the image to be processed can be segmented to obtain a black and white mask image. After the image to be processed is segmented to obtain a black and white mask image, the commodity area in the black and white mask image can be set to black, that is, the area corresponding to the black and white mask image in the image to be processed is blocked to obtain the background image. The image-to-text model is further used to perform image-to-text processing on the background image to obtain text description information. The image to be processed, the black and white mask image and the text description information can be used as training samples to train a large model of the home decoration image.

[0170] Through the above steps S502 to S504 of the present application, text description information of the background image in the image to be processed is obtained, wherein the image to be processed includes a foreground image and a background image, the image content of the foreground image includes at least one display object, the image content of the background image includes the background where the display object is located, the text description information is obtained by analyzing the image to be processed using a graph-to-text model, and is used to describe the image content of the background image, the graph-to-text model is trained using image samples and text description information samples, the image content of the background image samples in the image samples includes the background where at least one display object sample is located, the display object sample is in an occluded state in the image sample, and the text description information sample is used to describe the image content of the background image samples; at least the text description information is used as a training sample to train to obtain an output model of the model training end, thereby achieving the technical effect of improving the efficiency of generating text description information of the background image, and solving the technical problem of low efficiency in generating text description information of the background image.

[0171] The above method of this embodiment is further introduced below.

[0172] As an optional implementation, step S504, at least using text description information as a training sample to train an output model of the model training end, includes: obtaining a mask image corresponding to the image to be processed, wherein the mask image includes the area displayed by the display object in the image to be processed; using the mask image, text description information and the image to be processed as training samples to train the output model.

[0173] In this embodiment, a mask image corresponding to the image to be processed can be obtained. After obtaining the image to be processed, the text description information and the mask image, the mask image, the text description information and the image to be processed are used as training samples to train an output model. Among them, the mask image includes the area displayed by the display object in the image to be processed, which can be called a black and white mask image. The above-mentioned text description information, the image to be processed, and the mask image corresponding to the image to be processed can be called triple data.

[0174] Optionally, after obtaining the image to be processed, the image to be processed is segmented to obtain a black-and-white mask image, and the black-and-white mask image is image processed to blacken the product area in the black-and-white mask image to obtain a background image. Further, using a prompt word containing a complete and detailed description of the image to be processed and limiting the output format of the text description information, the image-to-text model is guided to perform image-to-text processing on the background image to obtain text description information.

[0175] Optionally, after obtaining the above-mentioned image to be processed, black-and-white mask image and text description information, the image to be processed, the black-and-white mask image and text description information can be used as triplet data to train a large model of the raw home decoration picture at the model training end.

[0176] This embodiment trains a large model of raw home decoration pictures by using the image to be processed Image, the black-and-white mask image and the text description information Capt ion, thereby solving the problem of triple data construction in the training process of the large model of raw home decoration pictures.

[0177] The present application also provides a method for generating text information, which can be applied to the model reasoning end. Figure 6 is a flowchart of a method for generating text information according to an embodiment of the present application. Figure 6 As shown, the method may include the following steps:

[0178] Step S602: Obtain text description information of the background image in the image to be processed.

[0179] In the technical solution provided in the above step S602 of the present application, text description information of the background image in the image to be processed can be obtained. The image to be processed can be a scene image Image, which can at least include a foreground image and a background image.

[0180] It should be noted that the image content of the foreground image includes at least one display object, and the image content of the background image includes the background where the display object is located.

[0181] Optionally, the text description information Capt ion can be obtained by analyzing the image to be processed using a graph-to-text model, and is used to describe the image content of the background image. The graph-to-text model is trained using image samples and text description information samples, and the image content of the background image sample in the image sample includes the background where at least one display object sample is located, and the display object sample is in an occluded state in the image sample. The text description information sample is used to describe the image content of the background image sample.

[0182] In this embodiment, an image to be processed can be obtained. After the image to be processed is obtained, the foreground image and the background image can be identified from the image to be processed. Since the foreground image contains a main object, and the main object has a large area or occupies a large area, the main object can be first identified from the image to be processed. Further, the main object can be occluded to obtain a background image. After the background image is identified from the image to be processed, the image-to-text model can be used to analyze the background image to obtain text description information of the background image.

[0183] Step S604: Generate an output image of the model inference end by using the text description information and the image to be processed.

[0184] In the technical solution provided in the above step S604 of the present application, after obtaining the text description information of the background image in the image to be processed, the output image of the model inference end can be generated using the text description information and the image to be processed. Among them, the image to be processed can be a reference image. The text description information can be a Caption that only describes the background content of the reference image. The output image can be a generated effect image output based on the reference image and the Caption that only describes the background content of the reference image.

[0185] In this embodiment, after obtaining the image to be processed, the image to be processed can be analyzed using the image-to-text model to obtain text description information of the background image in the image to be processed. The text description information and the image to be processed can be further used to generate an output image of the model inference end.

[0186] Optionally, the product image, the reference image and the Caption describing only the background of the reference image are used as inputs of the large model of the raw image of the home decoration picture. By calling the large model of the raw image of the home decoration picture, the generated effect image output by the large model of the raw image of the home decoration picture can be obtained.

[0187] This embodiment supports the reference graph capability during the model reasoning process, increases the ability to understand scene images, and solves problems such as unclear reference content of the reference graph during the reasoning process.

[0188] Through the above-mentioned steps S602 and S604 of the present application, the text description information of the background image in the image to be processed is obtained, wherein the image to be processed includes a foreground image and a background image, the image content of the foreground image includes at least one display object, the image content of the background image includes the background where the display object is located, the text description information is obtained by analyzing the image to be processed using a graph-to-text model, and is used to describe the image content of the background image, the graph-to-text model is trained using image samples and text description information samples, the image content of the background image samples in the image samples includes the background where at least one display object sample is located, the display object sample is in an occluded state in the image sample, and the text description information sample is used to describe the image content of the background image sample; the text description information and the image to be processed are used to generate an output image of the model inference end, thereby achieving the technical effect of improving the efficiency of generating text description information of the background image, and solving the technical problem of low efficiency in generating text description information of the background image.

[0189] The above method of this embodiment is further introduced below.

[0190] As an optional implementation, step S604, using text description information and the image to be processed to generate an output image of the model reasoning end, includes: using the text description information and the image to be processed to generate a background image of the input image of the model reasoning end, wherein the input image is used to represent the image of the product to be traded, and the image content of the background image of the input image includes the background where the product to be traded is located; combining the background image of the input image and the input image to generate the output image of the model reasoning end.

[0191] In this embodiment, after obtaining the image to be processed, the image to be processed can be analyzed using the graph-to-text model to obtain text description information of the background image in the image to be processed. The text description information and the image to be processed can be further used to generate a background image of the input image of the model reasoning end. After the background image of the input image of the model reasoning end is generated, the background image of the input image and the input image can be combined to generate an output image of the model reasoning end. The input image can be used to represent an image of a product to be traded, and the image content of the background image of the input image can include the background where the product to be traded is located.

[0192] The present application also provides a method for generating text information. Figure 7 is a flowchart of a method for generating text information according to an embodiment of the present application. Figure 7 As shown, the method may include the following steps:

[0193] Step S702, in response to an input instruction on the operation interface, displaying the image to be processed on the operation interface.

[0194] In the technical solution provided in the above step S702 of the present application, in response to the input instruction acting on the operation interface, the image to be processed can be displayed on the operation interface. The operation interface can be an operation interface displayed on the above client device, for example, it can be an interface for interaction between a user and a computer or software system, it can be a graphical user interface, a command line interface, etc. This is only for example description, and no specific limitation is imposed on the form of the operation interface.

[0195] It should be noted that the input instruction can be an instruction for controlling the operation interface to display the image to be processed. For example, it can be an instruction entered by the user on the operation interface. The user can generate the input instruction by entering text through the keyboard, clicking buttons or options on the operation interface with the mouse, or dragging the image to be processed to a designated area.

[0196] Optionally, the image to be processed may be a scene image Image, which may include at least a foreground image and a background image. The image content of the foreground image includes at least one display object. The image content of the background image includes the background where the display object is located.

[0197] In an optional embodiment, the user can open the operation interface deployed on the client device and find a button or a designated area for uploading the image to be processed on the operation interface. An input instruction can be generated by clicking the button for uploading the image to be processed or dragging the image to be processed to the designated area. According to the generated input instruction, the corresponding image to be processed can be loaded and displayed on the operation interface.

[0198] Step S704, in response to the text generation instruction on the operation interface, displaying text description information of the background image in the image to be processed on the operation interface.

[0199] In the technical solution provided in the above step S704 of the present application, after the image to be processed is displayed on the operation interface, in response to the text generation instruction acting on the operation interface, the text description information of the background image in the image to be processed can be displayed on the operation interface. The text generation instruction can be an instruction to generate text for the image to be processed displayed on the operation interface, and the user can generate the text generation instruction by clicking on the image to be processed displayed on the operation interface. This is only an example and does not specifically limit the text generation instruction.

[0200] It should be noted that the text description information is obtained by analyzing the image to be processed using the image-to-text model, and is used to describe the image content of the background image. The image-to-text model is trained using image samples and text description information samples. The image content of the background image sample in the image sample includes the background where at least one display object sample is located, and the display object sample is in an occluded state in the image sample. The text description information sample is used to describe the image content of the background image sample.

[0201] In this embodiment, after the image to be processed is displayed on the operation interface, in response to the text generation instruction acting on the operation interface, text generation can be performed on the image to be processed displayed on the operation interface to obtain text description information of the background image in the image to be processed, and the text description information is displayed on the operation interface.

[0202] Optionally, after the image to be processed is displayed on the operation interface, the image to be processed can be sent to the server. After receiving the image to be processed, the server can identify the foreground image and the background image from the image to be processed. Since the foreground image contains the main object, and the main object has a large area or occupies a large area, the main object can be first identified from the image to be processed. The main object can be further occluded to obtain the background image. After the background image is identified from the image to be processed, the background image can be analyzed using the image-to-text model to obtain text description information of the background image. The user can click on the image to be processed displayed on the operation interface to generate a text generation instruction. According to the text generation instruction, the text description information generated by the server can be sent to the client device, and the text description information can be displayed by the operation interface deployed on the client device.

[0203] Through the above-mentioned steps S702 and S704 of the present application, in response to the input instruction acting on the operation interface, the image to be processed is displayed on the operation interface, wherein the image to be processed includes a foreground image and a background image, the image content of the foreground image includes at least one display object, and the image content of the background image includes the background where the display object is located; in response to the text generation instruction acting on the operation interface, the text description information of the background image in the image to be processed is displayed on the operation interface, wherein the text description information is obtained by analyzing the image to be processed using a graph-to-text model and is used to describe the image content of the background image, the graph-to-text model is trained using image samples and text description information samples, the image content of the background image samples in the image samples includes the background where at least one display object sample is located, the display object sample is in an occluded state in the image sample, and the text description information sample is used to describe the image content of the background image sample, thereby achieving the technical effect of improving the efficiency of generating text description information of the background image, and solving the technical problem of low efficiency in generating text description information of the background image.

[0204] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application, for example, the data for verification, are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0205] At present, based on the widespread application of diffusion models in generative models and the development of multimodal technology, the application of artificial intelligence generated content (AIGC) has also made progress, such as artificial intelligence (AI) painting (stable diffusion, dal le2, imagen) and three-dimensional (3D) model / voice / video generation under various conditions.

[0206] In the generative model, text description Caption plays a vital role in the training and reasoning process. Caption provides annotated data for training, enabling the model to learn how to match visual information (image) with language information (text description). In the reasoning stage, using Caption as the input of the model can guide the generative model to generate content related to Caption, thereby enhancing the relevance and consistency of the generated results.

[0207] In related technologies, the captions obtained by the image-generated text model usually have problems such as hallucinations, lack of details and unprofessionalism, and the content described by the obtained caption is the entire scene map, and the main product is usually located in a prominent area. Therefore, a large amount of space will be used to describe the main product in the entire caption. For example, Mianbi Intelligence has open-sourced the image-generated text model (MiniCPM-V2.6) and the new generation of multimodal large model (CogVLM2) launched by Zhipu AI, which has achieved significant performance improvements in visual and language understanding, supports text lengths of up to 8 kilobytes (K) and image inputs with a resolution of 1344*1344, and has powerful document image understanding capabilities. Although the MiniCPM-V2.6 model and the CogVLM2 model are open source, the captions obtained by these two models have problems such as hallucinations and lack of details.

[0208] In addition, in the process of generating text from images, we need to fill in the background content of the main product. The captions required in the training and reasoning process are descriptions of non-product areas in the scene graph, that is, a multimodal technology that only describes the background content of the scene graph is needed.

[0209] In order to solve the above problems, the GPT-4o model developed by the Artificial Intelligence Research Laboratory (OpenAI) is detailed and accurate, but the model is closed source and charged, and the reasoning time is slow. Therefore, this embodiment combines the multimodal capabilities of the GPT-4o model developed by OpenAI to produce seed data, and through model distillation, correlation calculation, etc., realizes an efficient image-to-text technology that only describes the background content of the scene graph, providing technical support for the iteration of the studio image-to-image model and the reference graph capability.

[0210] In an embodiment of the present application, a multimodal technology that only describes the background content of a scene graph is provided, which involves the ability to generate text from images and can be applied to the merchant tool "Homework". By inputting a scene graph containing commodities into the image-generated text model, the image-generated text model can be controlled to output a Capt ion that only describes the background content of the scene graph, thereby solving the problem of triple data construction in the training process of the large model of the home decoration image, and supporting the reference image capability in the model reasoning process. This technology increases the ability to understand scene graphs, making the triple data required in the training process of the large model of the home decoration image more independent and more effective. At the same time, it supports the function of studio reference images. In the reasoning process, it solves the problems of difficulty in user input of prompt words and unclear reference content of reference images.

[0211] This embodiment adopts a phased approach in the seed data production process to ensure the detail and accuracy of the seed data; uses CLIP to calculate the image-text correlation of each area of ​​the image, significantly reducing the hallucination problem of the seed data; uses the BLIP model, for example, the image annotation basic model performs outstandingly in tasks such as image description and visual question answering, and has a low number of parameters, ensuring the efficiency of model reasoning.

[0212] Figure 8 is a schematic diagram of a scene image according to an embodiment of the present application, Fig. 9 is a schematic diagram of a mask image according to an embodiment of the present application, such as Figure 8 and Fig. 9 As shown, Figure 8 is the scene image Image, Fig. 9The mask image is the mask image of the main object (product) in the scene image. The scene image Image and the mask image are used as the input of the large model of the home decoration picture raw image. The Capt ion output by the large model of the home decoration picture raw image that only describes the background content can be obtained. ion can be "To the right of the bed, there is a blue-grey nightstand with gold handles and support frame, which has a single drawer and an open shelf below the drawer. Above the head of the bed, there is a modern table lamp with a black lampshade and a gold base. To the left of the bed, there is a matching blue-grey nightstand with gold handles and support frame, which also has a single drawer and an open shelf below the drawer. On top of the nightstand is a modern table lamp with a black lampshade and a gold base. On the wall above the nightstand, a table lamp with a white cylindrical lampshade and a gold mounting frame is mounted. On the wall behind the bed, which is lined with vertical grooves and painted a soft grey, an abstract artwork hangs above the bed, showing geometric shapes in blue, white and gold, which ties the color palette of the room together. On the floor in front of the bed, there is a light-colored wood or laminate. Under the bed, a large area of ​​blue and gray area rug is placed to add texture and warmth to the space," the Capt. Capt ion can be expressed in Chinese, English, etc. This is just an example and does not impose any specific restrictions on the expression form of Capt ion.

[0213] Fig.10 is a schematic diagram of another scene image according to an embodiment of the present application, Fig.11 is a schematic diagram of another mask image according to an embodiment of the present application, such as Fig.10 and Fig.11 As shown, Fig.10 is the scene image Image, Fig.11 is the mask image of the main object in the scene image, and the scene image Image and the mask image are used as the input of the large model of the raw image of the home decoration picture, so as to obtain the Capt ion output by the large model of the raw image of the home decoration picture, which only describes the background content. The Capt ion can be "On the left side of the sofa, there is a huge green leaf plant and a white table with a beige lampshade on it. The table has a flip leg and an open shelf. On the right side of the sofa, there is a white cabinet with a glass door. There are books in the cabinet, and the books are rectangular in shape. Above the sofa, there are several framed pictures and paintings hanging on the wall, including a black frame, a picture of a tree and a certificate, and two other framed pictures with classic and simple designs. Under the sofa, there is a beige carpet with a black diamond pattern on it. The carpet has a rectangular shape and a soft texture. There is a stack of books on the floor in front of the sofa. Behind the sofa, there is a wall with vertical paneling and a white cabinet with books."

[0214] Fig.12 is a schematic diagram of a commodity image according to an embodiment of the present application, Fig.13 is a schematic diagram of a reference image according to an embodiment of the present application, and the caption that only describes the background content of the reference image may be "Above the sofa, there is a beige wall. Below the sofa, there is a light wooden floor and a white carpet with a black diamond pattern. Because the black has a subtle diamond pattern, it adds a layer of warmth and texture to the room. To the left of the sofa, there is a wall-mounted black lamp with a curved arm, a framed artwork with a floral pattern, and another framed artwork with a geometric pattern, wherein the wall-mounted black lamp has a modern design, adding visual interest. To the right of the sofa, the window has white curtains and beige curtains, which are flowing and complement the overall color of the room. In front of the sofa, there is a round wooden coffee table with a glass top, and the coffee table has a dark finish that matches the bookshelf, adding a coherent look to the room. Behind the sofa, there is a wooden bookshelf filled with various books and decorations. The bookshelf is open and made of dark wood, providing storage and stylish display." Fig.14 It is a schematic diagram of a generated effect image according to an embodiment of the present application. The product image, the reference image and the Capt ion that only describes the background of the reference image are used as the input of the large model of the raw image of the home decoration picture, and the generated effect image output by the large model of the raw image of the home decoration picture can be obtained.

[0215] This embodiment uses the multimodal capabilities of the GPT-4o model, and provides a multimodal technical capability of only describing the background content of the scene graph for the iteration of the large model of the home decoration picture raw picture through seed data production, seed data filtering, and model distillation. That is, the ability to only describe the background content of the scene graph is achieved through seed data production, seed data filtering, and model distillation. Fig.15 It is a schematic diagram of producing and filtering seed data according to an embodiment of the present application. In the process of seed data production, for indoor scene images, the first manually designed first prompt word is used and the GPT-4o model is called to obtain a detailed description of the scene graph. For unconditional Caption tasks, the results of the GPT-4o model are more credible. In order to further obtain the background content that only describes the scene graph, the image of the main object is occluded by technical means such as image segmentation and image processing. The complete details of the scene graph, the second prompt word and the image of the main object are used as the input of the GPT-4o model to obtain the background description of the scene graph output by the GPT-4o model. The background description is a Caption that only describes the background content of the scene graph.

[0216] Although the scene graph background description in the above steps is input by the scene graph detailed description, due to the increase in the conditional restrictions of the prompt words, the obtained seed data (image pair: the image of the main object occluded and the scene graph background description) has a low proportion of hallucinations or errors. Therefore, the seed data needs to be further filtered. The scene graph background description is formatted as a description of different areas around the main object, and the descriptions of different areas can be correlated with each area of ​​the image. When the correlation calculation result of a certain area is low, it is considered that the quality of the seed image data pair is unqualified, thereby completing the seed data filtering. After obtaining high-quality seed data, the open source BLIP model is distilled to enable the BLIP model to achieve better results in the scene graph background description task. After completing the model distillation, the distilled model is evaluated, and the model whose performance meets the conditions is selected, and its multimodal capabilities are used to achieve the goal of only describing the scene graph background content.

[0217] The scene graph background description produced by this embodiment meets the background generation requirements of the picture-generated text, is detailed and highly reliable, and the controllable data production provides technical support for the iteration of the studio picture generation model and the reference picture capability.

[0218] Regarding the step of obtaining a detailed description of the scene graph, for the Caption task in the graph text, there are many open source / closed source models that can be well implemented. This embodiment uses the highly recognized GPT-4o model to implement this task to obtain seed data. For indoor scene images, first use manually designed prompt words and call the GPT-4o model to obtain a complete and detailed description of the scene graph. The manually designed prompt words can be "You are an interior designer, please describe this scene in detail and make sure that the information you provide is accurate and valid." The prompt word design of this step requires detailed and accurate content. The above prompt words and Fig.13The image shown is used as the input of the GPT-4o model, and the GPT-4o model is called to obtain a complete description of the image output by the GPT-4o model, which can be described as "The image shows a modern and stylish living room with a sophisticated and comfortable environment. Here is a detailed description: Sofa: The main object in the room is a smooth black leather sectional sofa, which has a modern design and is decorated with several decorative pillows in different patterns and colors (including green and floral prints), adding a touch of personality and comfort. Carpet: A small rug is laid in front of the sofa, and the small rug has a black diamond pattern, adding a layer of warmth and texture to the room. Floor: The floor is light wood with a beautiful herringbone pattern, adding an elegant and timeless touch to the space. Coffee table: The round wooden coffee table is located in front of the sofa. The coffee table has a dark finish that matches the bookshelf and adds a coherent look to the room. There are some decorations on the coffee table, including a vase and some small accessories. Bookshelf Shelves: There is a tall open bookcase mounted on the wall, made of dark wood, neatly placed with books and decorative items, providing storage and stylish display. Wall Art: The wall behind the sofa has two picture frames. One is a larger picture frame with an abstract pattern, while the other is a smaller picture frame with more details, these pieces add visual interest. Lighting: A modern floor lamp with black trim and a curved design is placed next to the sofa, providing additional lighting and enhancing the modern feel of the room. Curtains: The windows are decorated with flowing curtains in a beige color, which are paired with pure white curtains, which allow natural light to filter through the room while maintaining privacy. Additional Seating: In the foreground, additional seating can be seen, which may be an armchair or legrest, upholstered in a green fabric that matches one of the pillows on the sofa. Overall, this living room is a mix of modern design and comfort, with a cohesive color palette of black, green and neutrals, and a mix of textures and patterns that create a welcoming and stylish environment. As can be seen from the detailed description obtained, the description is detailed enough, but lacks information such as location layout, while not meeting the functional needs of background filling.

[0219] Regarding the step of obtaining the background image, since the actual image-to-text function requires a Caption that only describes the background content of the scene graph, and also has requirements for information such as location layout, it is necessary to further obtain the background image and the scene graph background description. First, a black and white mask image is obtained through image segmentation, and then image processing is performed to obtain the background image where the main object is blocked.

[0220] In this embodiment, the image segmentation adopts the U2Net network (universal ability of image cutting). Fig.16This is a schematic diagram of the structure of a U2Net network according to an embodiment of the present application. U2Net is proposed for the salient object detection task SOD. The salient object detection task is similar to the semantic segmentation task. The salient object detection task is a binary classification task, and its task is to segment the most attractive object or area in the picture. Therefore, there are only two categories: foreground and background, that is, the main product area and background area required by this embodiment. Fig.17 is a schematic diagram of a black and white mask image according to an embodiment of the present application, such as Fig.17 As shown, through the above image segmentation, we can get Fig.13 The black and white mask image corresponding to the image shown.

[0221] Further based on Fig.13 The image shown in the figure and the black and white mask image are used to obtain the background image, and the product area in the image is set to black. The pixel values ​​corresponding to the part with pixel values ​​greater than the specified threshold can be set to [0,0,0], that is, the product area is converted to black, and then the processed red green blue (RGB) array is converted to an image object and saved to obtain the background image. In other words, the product area in the scene image is turned black to obtain the background image. Fig.18 is a schematic diagram of a background image according to an embodiment of the present application, such as Fig.18 As shown, the background image can be obtained by performing image processing on the black and white mask image.

[0222] This embodiment uses the U2Net network to perform image segmentation on the scene graph to obtain a black-and-white mask image, and further sets the main object area in the black-and-white mask image to black to obtain a background image in which the main object is occluded.

[0223] Regarding the step of obtaining the background description of the scene graph, since the input image in this stage is a background image and the main product (black area) is required to be ignored in the image-to-text process, the prompt words in this stage need to contain a detailed description of the complete scene graph so that the model can obtain complete information of the scene graph to ensure accurate model output. At the same time, in order to ensure that the output Capt ion contains relevant information such as location layout, it is necessary to further require the output format in the prompt words. The manually designed prompt words in this stage can be "You are an interior designer. The black object in the picture is a sofa. The description of this image is "{detailed description of the scene graph}". Please answer the layout, object, shape, color and other information of the image based on the image and text description. You need to ignore the description of the black object, and the answer has this format: above, below, left, right or front and back of the sofa, {{details about the surrounding environment and background on the right side of the sofa}}. Please provide detailed descriptions of each direction. If the direction has no content, you can skip it. Describe the shape, material, color, style, decoration, etc. as detailed as possible."

[0224] The complete description of the scene graph, the prompt words designed at this stage and Fig.18 The background image in which the main object is occluded is used as the input of the GPT-4o model to obtain the description of the background content of the scene graph output by the GPT-4o model. The description of the background content of the scene graph can be "Above the sofa, there is a beige wall. Below the sofa, there is a light wooden floor and a white carpet with a black diamond pattern. The black has a subtle diamond pattern, which adds a layer of warmth and texture to the room. To the left of the sofa, there is a wall-mounted black lamp with a curved arm, a framed artwork with a floral pattern, and another framed artwork with a geometric pattern. The wall-mounted black lamp has a modern design, which adds visual interest. To the right of the sofa, the window has white curtains and beige curtains that are flowing and complement the overall color of the room. In front of the sofa, there is a round wooden coffee table with a glass top and a dark finish that matches the bookshelf, adding a coherent look to the room. Behind the sofa, there is a wooden bookshelf filled with various books and decorations. The bookshelf is open and made of dark wood, providing storage and stylish display." So far, the seed data required for model distillation (image pair: background image of the main object being occluded and scene graph background description) has been obtained.

[0225] Regarding the seed data filtering step, in order to obtain high-quality seed data, it is necessary to further filter out hallucinations or erroneous objects contained in the seed data. This embodiment divides the image into several regions according to the formatted scene graph background description, and performs image-text correlation calculation on the image and Caption of each region. This embodiment uses the CLIP model for correlation calculation. The CLIP model is a version of the CLIP series model from OpenAI. CLIP is a deep neural network that is trained for unsupervised learning through a large number of image and text pairs. It can embed images and text into the same vector space so that cross-modal semantic similarity can be calculated.

[0226] Fig.19 is a schematic diagram of a region image of a scene image according to an embodiment of the present application. The Capt ion of the region image may be "to the right of the sofa, the window has white curtains and beige curtains, these curtains are flowing and complement the overall color of the room", and the CLIP score of the region can be calculated to be 28. Similarly, the CLIP score of each region can be calculated, and a threshold condition can be set to filter the seed data. For example, the threshold can be set to 25, and when the calculated CLIP score is less than the threshold 25, the corresponding seed data is filtered.

[0227] Regarding the step of model distillation, model distillation is a knowledge transfer technique that transfers the knowledge of a large and complex teacher model to a small and fast student model, which is very useful in the field of deep learning and can create smaller, faster models with performance close to that of large models. Model distillation is an effective knowledge transfer technique that can transfer the knowledge of a large model to a small model, so that the small model has similar performance to the large model.

[0228] In this embodiment, the knowledge of the large model is mainly reflected through seed data. The Bl ip Image Captioning Base is used as a student model and trained using the images obtained in the above steps. During the training process, the student model is trained on actual text descriptions and image pair data to learn the knowledge of generating scene graph background descriptions.

[0229] Regarding the model evaluation step, after completing the model distillation step, the distilled model can be evaluated to evaluate its training results under different hyperparameters, and the model that meets the performance conditions can be selected as the final result. After the multimodal technology of only describing the background content of the scene graph in this embodiment is used, the ability to understand the scene graph is further increased, and the triple data required in the training process of the large model of the home decoration picture is more independent and the effect is better. At the same time, because this technology can support the studio reference picture function, it solves the problems of difficulty in user input prompt words and unclear reference content of the reference picture during the reasoning process.

[0230] This embodiment uses the GPT-4o model to generate the seed data required for the distillation of the BLIP model, so that the distilled model can output more detailed Captions; CLIP is used to calculate the image-text correlation of each area of ​​the image, and the seed data is filtered to reduce the hallucination problem of the seed data to a certain extent; in the distillation process, the student model used is the Blip Image Captioning Base, which has a parameter volume of only 0.2B, ensuring the effect of model reasoning.

[0231] This embodiment adopts a phased approach in the seed data production process to ensure the detail and accuracy of the seed data; CLIP is used to calculate the image-text correlation of each area of ​​the image, significantly reducing the hallucination problem of the seed data; the BLIP model is used to perform outstanding performance in tasks such as image description and visual question answering with a low number of parameters, ensuring the efficiency of model reasoning.

[0232] In an embodiment of the present application, an image to be processed is obtained, wherein the image to be processed includes a foreground image and a background image, the image content of the foreground image includes at least one display object, and the image content of the background image includes the background where the display object is located; the image to be processed is input into a graph-generated text model, and the graph-generated text model is used to analyze the image to be processed to obtain text description information of the background image in the image to be processed, wherein the graph-generated text model is trained using image samples and text description information samples, the image content of the background image sample in the image sample includes the background where at least one display object sample is located, the display object sample is in an occluded state in the image sample, the text description information sample is used to describe the image content of the background image sample, and the text description information is used to describe the image content of the background image; and the text description information is output. That is, in the embodiment of the present application, the display object contained in the foreground image in the image to be processed is occluded to obtain the background image, and the background image is analyzed by the graph-generated text model to obtain the text description information of the background image, thereby achieving the purpose of generating text description information that only describes the image content of the background image, thereby achieving the technical effect of improving the efficiency of generating text description information of the background image, and solving the technical problem of low efficiency of generating text description information of the background image.

[0233] Fig. 20 is a structural block diagram of a computing environment of a method for generating text information according to an embodiment of the present application, such as Fig. 20 As shown, computing environment 2001 includes multiple computing nodes (such as servers) running on a distributed network (shown in the figure as 2010-1, 2010-2, ...). The computing nodes all contain local processing and memory resources, and end users 2002 can remotely run applications or store data in computing environment 2001. Applications can be provided as multiple services 2020-1, 2020-2, 2020-3 and 2020-4 in computing environment 2001, representing services "A", "D", "E" and "H" respectively.

[0234] The end user 2002 can provide and access services through a web browser or other software application on the client, and in some embodiments, the end user 2002's provision and / or request can be provided to the entry gateway 2030. The entry gateway 2030 can include a corresponding agent to handle the provision and / or request for the service (one or more services provided in the computing environment 2001).

[0235] Services are provided or deployed based on various virtualization technologies supported by the computing environment 2001. In some embodiments, services can be provided based on virtual machine (VM)-based virtualization, container-based virtualization, and / or similar methods. Virtual machine-based virtualization can be to simulate a real computer by initializing a virtual machine, and execute programs and applications without directly contacting any actual hardware resources. While the virtual machine virtualizes the machine, according to container-based virtualization, a container can be started to virtualize the entire operating system (Operat ingSystem, referred to as OS) so that multiple workloads can run on a single operating system instance.

[0236] In an embodiment based on container virtualization, several containers of a service can be assembled into a Pod (e.g., a Kubernetes Pod). Fig. 20 As shown, service 2020-2 can be equipped with one or more Pods 2040-1, 2040-2, ..., 2040-N (collectively referred to as Pods). Pods can include a proxy 2045 and one or more containers 2042-1, 2042-2, ..., 2042-M (collectively referred to as containers). One or more containers in a Pod process requests related to one or more corresponding functions of the service, and the proxy 2045 generally controls network functions related to the service, such as routing, load balancing, etc. Other services can also be equipped with Pods similar to Pods.

[0237] During operation, executing a user request from end user 2002 may require invoking one or more services in computing environment 2001, and executing one or more functions of a service may require invoking one or more functions of another service. Fig. 20 As shown, service "A" 2020-1 receives a user request from terminal user 2002 from the inlet gateway 2030, service "A" 2020-1 can call service "D" 2020-2, and service "D" 2020-2 can request service "E" 2020-3 to perform one or more functions.

[0238] The computing environment described above can be a cloud computing environment, where the allocation of resources is managed by the cloud service provider, allowing the development of functions without considering the implementation, adjustment or expansion of servers. The computing environment allows developers to execute code in response to events without building or maintaining complex infrastructure. Services can be divided into a set of functions that can be automatically and independently scaled, rather than expanding a single hardware device to handle potential loads.

[0239] According to an embodiment of the present application, there is also provided a method for implementing the above Figure 2 A text information generating device and a text information generating method shown.

[0240] Fig.21 is a schematic diagram of a device for generating text information according to an embodiment of the present application, such as Fig.21 As shown, the text information generating device 2100 may include: a first acquiring unit 2102 , a first analyzing unit 2104 and a first output unit 2106 .

[0241] The first acquisition unit 2102 is used to acquire an image to be processed, wherein the image to be processed includes a foreground image and a background image, the image content of the foreground image includes at least one display object, and the image content of the background image includes the background where the display object is located.

[0242] The first analysis unit 2104 is used to input the image to be processed into the image-to-text model, and use the image-to-text model to analyze the image to be processed to obtain text description information of the background image in the image to be processed, wherein the image-to-text model is trained using image samples and text description information samples, the image content of the background image sample in the image sample includes the background where at least one display object sample is located, and the display object sample is in an occluded state in the image sample, the text description information sample is used to describe the image content of the background image sample, and the text description information is used to describe the image content of the background image sample.

[0243] The first output unit 2106 is used to output text description information.

[0244] It should be noted that the first acquisition unit 2102, the first analysis unit 2104 and the first output unit 2106 correspond to steps S202 to S206, and the three units and the corresponding steps implement the same examples and application scenarios, but are not limited to the above disclosed content. It should be noted that the above units can be hardware components or software components stored in a memory (e.g., memory 2704) and processed by one or more processors (e.g., processors 2702a, 2702b..., 2702n), and the above units can also be run in a computer terminal A as part of the device.

[0245] In the device for generating text information, the image to be processed is acquired by the first acquisition unit 2102, wherein the image to be processed includes a foreground image and a background image, the image content of the foreground image includes at least one display object, and the image content of the background image includes the background where the display object is located. The image to be processed is input into the image-generated text model by the first analysis unit 2104, and the image to be processed is analyzed by the image-generated text model to obtain the text description information of the background image in the image to be processed, wherein the image-generated text model is trained by using image samples and text description information samples, the image content of the background image sample in the image sample includes the background where at least one display object sample is located, the display object sample is in an occluded state in the image sample, the text description information sample is used to describe the image content of the background image sample, and the text description information is used to describe the image content of the background image. The text description information is output by the first output unit 2106, thereby achieving the technical effect of improving the efficiency of generating text description information of the background image, and solving the technical problem of low efficiency of generating text description information of the background image.

[0246] According to an embodiment of the present application, there is also provided a method for implementing the above Figure 3 The model determination method shown is a model determination device.

[0247] Fig. 22 is a schematic diagram of a model determination device according to an embodiment of the present application, such as Fig. 22 As shown, the model determination device 2200 may include: a second acquisition unit 2202 , a third acquisition unit 2204 and a first generation unit 2206 .

[0248] The second acquisition unit 2202 is used to acquire an image sample, wherein the image content of the background image sample in the image sample includes the background where at least one display object sample is located, and the display object sample is in an obstructed state in the image sample.

[0249] The third acquisition unit 2204 is used to acquire a text description information sample of the image sample, wherein the text description information sample is used to describe the image content of the background image sample.

[0250] The first generating unit 2206 is used to generate an image-to-text model using image samples and text description information samples, wherein the image-to-text model is used to analyze the image to be processed to obtain text description information of the background image in the image to be processed, the image content of the foreground image in the image to be processed includes at least one display object, the image content of the background image includes the background where the display object is located, and the text description information is used to describe the image content of the background image.

[0251] Here, the second acquisition unit 2202, the third acquisition unit 2204 and the first generation unit 2206 correspond to steps S302 to S306, and the three units and the corresponding steps implement the same examples and application scenarios, but are not limited to the above disclosed content. It should be noted that the above units can be hardware components or software components stored in a memory (e.g., memory 2704) and processed by one or more processors (e.g., processors 2702a, 2702b..., 2702n), and the above units can also be run in computer terminal A as part of the device.

[0252] In the determination device of the model, an image sample is obtained by the second acquisition unit 2202, wherein the image content of the background image sample in the image sample includes the background where at least one display object sample is located, and the display object sample is in an occluded state in the image sample. A text description information sample of the image sample is obtained by the third acquisition unit 2204, wherein the text description information sample is used to describe the image content of the background image sample. The image sample and the text description information sample are used to generate a picture-to-text model by the first generation unit 2206, wherein the picture-to-text model is used to analyze the image to be processed to obtain the text description information of the background image in the image to be processed, the image content of the foreground image in the image to be processed includes at least one display object, the image content of the background image includes the background where the display object is located, and the text description information is used to describe the image content of the background image, thereby achieving the technical effect of improving the efficiency of generating text description information of the background image and solving the technical problem of low efficiency of generating text description information of the background image.

[0253] According to an embodiment of the present application, there is also provided a method for implementing the above Figure 4 A text information generating device and a text information generating method shown.

[0254] Fig.23 is a schematic diagram of a device for generating text information according to an embodiment of the present application, such as Fig.23 As shown, the text information generating device 2300 may include: a recognition unit 2302 , a second analysis unit 2304 and a second output unit 2306 .

[0255] The identification unit 2302 is used to identify the image to be processed from the e-commerce platform, wherein the image to be processed includes a foreground image and a background image, the image content of the foreground image includes at least one product to be traded on the e-commerce platform, and the image content of the background image includes the background where the product is located.

[0256] The second analysis unit 2304 is used to input the image to be processed into the image-to-text model, and use the image-to-text model to analyze the image to be processed to obtain text description information of the background image in the image to be processed, wherein the image-to-text model is trained using image samples and text description information samples, the image content of the background image sample in the image sample includes the background where at least one display object sample is located, and the display object sample is in an occluded state in the image sample, the text description information sample is used to describe the image content of the background image sample, and the text description information is used to describe the image content of the background image sample.

[0257] The second output unit 2306 is used to output text description information to the e-commerce platform.

[0258] It should be noted that the identification unit 2302, the second analysis unit 2304 and the second output unit 2306 correspond to steps S402 to S406, and the three units and the corresponding steps implement the same examples and application scenarios, but are not limited to the above disclosed content. It should be noted that the above units can be hardware components or software components stored in a memory (e.g., memory 2704) and processed by one or more processors (e.g., processors 2702a, 2702b..., 2702n), and the above units can also be run in a computer terminal A as part of the device.

[0259] In the device for generating text information, an image to be processed is identified from an e-commerce platform by an identification unit 2302, wherein the image to be processed includes a foreground image and a background image, the image content of the foreground image includes at least one product to be traded on the e-commerce platform, and the image content of the background image includes the background where the product is located. The image to be processed is input into a graph-to-text model by a second analysis unit 2304, and the image to be processed is analyzed by the graph-to-text model to obtain text description information of the background image in the image to be processed, wherein the graph-to-text model is trained using image samples and text description information samples, the image content of the background image sample in the image sample includes the background where at least one display object sample is located, the display object sample is in an occluded state in the image sample, the text description information sample is used to describe the image content of the background image sample, and the text description information is used to describe the image content of the background image. The text description information is output to the e-commerce platform by a second output unit 2306, thereby achieving the technical effect of improving the efficiency of generating text description information of the background image, and solving the technical problem of low efficiency of generating text description information of the background image.

[0260] According to an embodiment of the present application, there is also provided a method for implementing the above Figure 5 The text information generation device of the text information generation method shown is applied to the model training end.

[0261] Fig.24 is a schematic diagram of a device for generating text information according to an embodiment of the present application, such as Fig.24 As shown, the text information generating device 2400 may include: a fourth acquiring unit 2402 and a training unit 2404 .

[0262] The fourth acquisition unit 2402 is used to acquire text description information of the background image in the image to be processed, wherein the image to be processed includes a foreground image and a background image, the image content of the foreground image includes at least one display object, and the image content of the background image includes the background where the display object is located. The text description information is obtained by analyzing the image to be processed using a graph-to-text model and is used to describe the image content of the background image. The graph-to-text model is trained using image samples and text description information samples. The image content of the background image sample in the image sample includes the background where at least one display object sample is located, and the display object sample is in an occluded state in the image sample. The text description information sample is used to describe the image content of the background image sample.

[0263] The training unit 2404 is used to use at least the text description information as a training sample to train and obtain an output model of the model training end.

[0264] It should be noted that the fourth acquisition unit 2402 and the training unit 2404 correspond to steps S502 to S504, and the two units and the corresponding steps implement the same examples and application scenarios, but are not limited to the above disclosed content. It should be noted that the above units can be hardware components or software components stored in a memory (e.g., memory 2704) and processed by one or more processors (e.g., processors 2702a, 2702b..., 2702n), and the above units can also be run in computer terminal A as part of the device.

[0265] In the device for generating text information, the text description information of the background image in the image to be processed is obtained by the fourth acquisition unit 2402, wherein the image to be processed includes a foreground image and a background image, the image content of the foreground image includes at least one display object, the image content of the background image includes the background where the display object is located, the text description information is obtained by analyzing the image to be processed using a graph-generated text model, and is used to describe the image content of the background image, the graph-generated text model is obtained by training using image samples and text description information samples, the image content of the background image sample in the image sample includes the background where at least one display object sample is located, the display object sample is in an occluded state in the image sample, and the text description information sample is used to describe the image content of the background image sample. The training unit 2404 uses at least the text description information as a training sample to train and obtain an output model of the model training end, thereby achieving the technical effect of improving the efficiency of generating text description information of the background image, and solving the technical problem of low efficiency in generating text description information of the background image.

[0266] According to an embodiment of the present application, there is also provided a method for implementing the above Figure 6 The text information generation device of the text information generation method shown is applied to the model reasoning end.

[0267] Fig.25 is a schematic diagram of a device for generating text information according to an embodiment of the present application, such as Fig.25 As shown, the text information generating device 2500 may include: a fifth acquiring unit 2502 and a second generating unit 2504 .

[0268] The fifth acquisition unit 2502 is used to acquire text description information of a background image in an image to be processed, wherein the image to be processed includes a foreground image and a background image, the image content of the foreground image includes at least one display object, and the image content of the background image includes the background where the display object is located. The text description information is obtained by analyzing the image to be processed using a graph-to-text model and is used to describe the image content of the background image. The graph-to-text model is trained using image samples and text description information samples. The image content of the background image sample in the image sample includes the background where at least one display object sample is located, and the display object sample is in an occluded state in the image sample. The text description information sample is used to describe the image content of the background image sample.

[0269] The second generating unit 2504 is used to generate an output image of the model inference end by using the text description information and the image to be processed.

[0270] It should be noted that the fifth acquisition unit 2502 and the second generation unit 2504 correspond to steps S602 to S604, and the two units and the corresponding steps implement the same examples and application scenarios, but are not limited to the above disclosed content. It should be noted that the above units can be hardware components or software components stored in a memory (e.g., memory 2704) and processed by one or more processors (e.g., processors 2702a, 2702b..., 2702n), and the above units can also be run in computer terminal A as part of the device.

[0271] In the device for generating text information, the fifth acquisition unit 2502 acquires text description information of the background image in the image to be processed, wherein the image to be processed includes a foreground image and a background image, the image content of the foreground image includes at least one display object, the image content of the background image includes the background where the display object is located, the text description information is obtained by analyzing the image to be processed using a graph-to-text model, and is used to describe the image content of the background image, the graph-to-text model is trained using image samples and text description information samples, the image content of the background image sample in the image sample includes the background where at least one display object sample is located, the display object sample is in an occluded state in the image sample, and the text description information sample is used to describe the image content of the background image sample. The second generation unit 2504 uses the text description information and the image to be processed to generate an output image of the model inference end, thereby achieving the technical effect of improving the efficiency of generating text description information of the background image, and solving the technical problem of low efficiency in generating text description information of the background image.

[0272] According to an embodiment of the present application, there is also provided a method for implementing the above Figure 7 A text information generating device and a text information generating method shown.

[0273] Fig.26 is a schematic diagram of a device for generating text information according to an embodiment of the present application, such as Fig.26 As shown, the text information generating device 2600 may include: a first display unit 2602 and a second display unit 2604 .

[0274] The first display unit 2602 is used to respond to input instructions on the operation interface and display the image to be processed on the operation interface, wherein the image to be processed includes a foreground image and a background image, the image content of the foreground image includes at least one display object, and the image content of the background image includes the background where the display object is located.

[0275] The second display unit 2604 is used to respond to the text generation instruction acting on the operation interface, and display the text description information of the background image in the image to be processed on the operation interface, wherein the text description information is obtained by analyzing the image to be processed using a graph-to-text model and is used to describe the image content of the background image, the graph-to-text model is trained using image samples and text description information samples, the image content of the background image sample in the image sample includes the background where at least one display object sample is located, the display object sample is in an occluded state in the image sample, and the text description information sample is used to describe the image content of the background image sample.

[0276] It should be noted that the first display unit 2602 and the second display unit 2604 correspond to steps S702 to S704, and the two units and the corresponding steps implement the same examples and application scenarios, but are not limited to the above disclosed content. It should be noted that the above units can be hardware components or software components stored in a memory (e.g., memory 2704) and processed by one or more processors (e.g., processors 2702a, 2702b..., 2702n), and the above units can also be run in computer terminal A as part of the device.

[0277] In the text information generation device, the first display unit 2602 responds to the input instruction on the operation interface, and displays the image to be processed on the operation interface, wherein the image to be processed includes a foreground image and a background image, the image content of the foreground image includes at least one display object, and the image content of the background image includes the background where the display object is located. The second display unit 2604 responds to the text generation instruction on the operation interface, and displays the text description information of the background image in the image to be processed on the operation interface, wherein the text description information is obtained by analyzing the image to be processed using a graph-to-text model and is used to describe the image content of the background image, the graph-to-text model is trained using image samples and text description information samples, the image content of the background image sample in the image sample includes the background where at least one display object sample is located, the display object sample is in an occluded state in the image sample, and the text description information sample is used to describe the image content of the background image sample, thereby achieving the technical effect of improving the efficiency of generating text description information of the background image, and solving the technical problem of low efficiency of generating text description information of the background image.

[0278] The embodiment of the present application may provide a computer terminal, which may be any computer terminal device in a computer terminal group. Optionally, in this embodiment, the computer terminal may also be replaced by a terminal device such as a mobile terminal.

[0279] Optionally, in this embodiment, the computer terminal may be located in at least one network device among a plurality of network devices of the computer network.

[0280] In this embodiment, the above-mentioned computer terminal can execute the program code of the following steps in the method for generating text information: obtaining an image to be processed, wherein the image to be processed includes a foreground image and a background image, the image content of the foreground image includes at least one display object, and the image content of the background image includes the background where the display object is located; inputting the image to be processed into a picture-to-text model, and using the picture-to-text model to analyze the image to be processed to obtain text description information of the background image in the image to be processed, wherein the picture-to-text model is trained using image samples and text description information samples, the image content of the background image sample in the image sample includes the background where at least one display object sample is located, the display object sample is in an occluded state in the image sample, the text description information sample is used to describe the image content of the background image sample, and the text description information is used to describe the image content of the background image; outputting the text description information.

[0281] Optionally, Fig. 27 is a structural block diagram of a computer terminal according to an embodiment of the present application, such as Fig. 27 As shown, the computer terminal A may include: one or more (only one is shown in the figure) processors 2702 , a memory 2704 and a transmission device 2706 .

[0282] Among them, the memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the method and device for generating text information in the embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned method for generating text information. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the terminal A via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0283] Optionally, the processor may also execute program codes of the following steps: using an image-to-text model to identify a background image from the image to be processed; and using the image-to-text model to perform image-to-text processing on the background image to obtain text description information.

[0284] Optionally, the processor may also execute the program code of the following steps: using the graph-generated text model to determine a mask image corresponding to the image to be processed, wherein the mask image includes an area where the display object is displayed in the image to be processed; using the graph-generated text model to perform masking processing on the area corresponding to the mask image in the image to be processed to obtain a background image.

[0285] Optionally, the processor may also execute program code of the following steps: obtaining prompt information, wherein the prompt information is used to at least indicate that the text description information to be generated is prohibited from including information about the display object; using the prompt information, guiding the image-to-text model to perform image-to-text processing on the background image to obtain text description information.

[0286] Optionally, the processor may also execute the program code of the following steps: outputting text description information to a model training end, wherein the text description information, the image to be processed, and the mask image corresponding to the image to be processed are used as training samples to train an output model of the model training end, wherein the mask image includes the area where the display object is displayed in the image to be processed.

[0287] Optionally, the processor may also execute the program code of the following steps: outputting text description information to the model reasoning end, wherein the text description information and the image to be processed are used to generate an output image of the model reasoning end.

[0288] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: obtain an image sample, wherein the image content of the background image sample in the image sample includes the background where at least one display object sample is located, and the display object sample is in an occluded state in the image sample; obtain a text description information sample of the image sample, wherein the text description information sample is used to describe the image content of the background image sample; use the image sample and the text description information sample to generate a picture-to-text model, wherein the picture-to-text model is used to analyze the image to be processed to obtain the text description information of the background image in the image to be processed, the image content of the foreground image in the image to be processed includes at least one display object, the image content of the background image includes the background where the display object is located, and the text description information is used to describe the image content of the background image.

[0289] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: identify the image to be processed from the e-commerce platform, wherein the image to be processed includes a foreground image and a background image, the image content of the foreground image includes at least one product to be traded on the e-commerce platform, and the image content of the background image includes the background where the product is located; input the image to be processed into the image-to-text model, and use the image-to-text model to analyze the image to be processed to obtain text description information of the background image in the image to be processed, wherein the image-to-text model is trained using image samples and text description information samples, the image content of the background image sample in the image sample includes the background where at least one display object sample is located, the display object sample is in an occluded state in the image sample, the text description information sample is used to describe the image content of the background image sample, and the text description information is used to describe the image content of the background image; output the text description information to the e-commerce platform.

[0290] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: obtain text description information of the background image in the image to be processed, wherein the image to be processed includes a foreground image and a background image, the image content of the foreground image includes at least one display object, and the image content of the background image includes the background where the display object is located. The text description information is obtained by analyzing the image to be processed using a graph-to-text model and is used to describe the image content of the background image. The graph-to-text model is trained using image samples and text description information samples. The image content of the background image sample in the image sample includes the background where at least one display object sample is located, and the display object sample is in an occluded state in the image sample. The text description information sample is used to describe the image content of the background image sample; at least the text description information is used as a training sample to train and obtain an output model of the model training end.

[0291] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: obtain text description information of the background image in the image to be processed, wherein the image to be processed includes a foreground image and a background image, the image content of the foreground image includes at least one display object, and the image content of the background image includes the background where the display object is located. The text description information is obtained by analyzing the image to be processed using a graph-to-text model and is used to describe the image content of the background image. The graph-to-text model is trained using image samples and text description information samples. The image content of the background image sample in the image sample includes the background where at least one display object sample is located, and the display object sample is in an occluded state in the image sample. The text description information sample is used to describe the image content of the background image sample; and using the text description information and the image to be processed, an output image of the model inference end is generated.

[0292] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: responding to the input instruction acting on the operation interface, displaying the image to be processed on the operation interface, wherein the image to be processed includes a foreground image and a background image, the image content of the foreground image includes at least one display object, and the image content of the background image includes the background where the display object is located; responding to the text generation instruction acting on the operation interface, displaying the text description information of the background image in the image to be processed on the operation interface, wherein the text description information is obtained by analyzing the image to be processed using a graph-to-text model and is used to describe the image content of the background image, the graph-to-text model is trained using image samples and text description information samples, the image content of the background image sample in the image sample includes the background where at least one display object sample is located, the display object sample is in an occluded state in the image sample, and the text description information sample is used to describe the image content of the background image sample.

[0293] It can be understood by those skilled in the art that Fig. 27The structure shown is for illustration only, and the computer terminal A may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (Mobile Internet Devices, abbreviated as MID), a personal access display (Personal Access Display, abbreviated as PAD), and other terminal devices. Fig. 27 It does not limit the structure of the above-mentioned computer terminal A. For example, the computer terminal A may also include Fig. 27 More or fewer components (such as network interfaces, display devices, etc.) shown in, or having Fig. 27 Different configurations are shown.

[0294] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0295] The embodiment of the present application further provides a computer-readable storage medium. Optionally, in this embodiment, the computer-readable storage medium can be used to store the program code executed by the method for generating text information provided in the first embodiment.

[0296] Optionally, in this embodiment, the computer-readable storage medium may be located in any one of the computer terminals in a computer terminal group in a computer network, or in any one of the mobile terminals in a mobile terminal group.

[0297] Optionally, in this embodiment, a computer-readable storage medium is configured to store program codes for executing the following steps: obtaining an image to be processed, wherein the image to be processed includes a foreground image and a background image, the image content of the foreground image includes at least one display object, and the image content of the background image includes the background where the display object is located; inputting the image to be processed into a graph-to-text model, and using the graph-to-text model to analyze the image to be processed to obtain text description information of the background image in the image to be processed, wherein the graph-to-text model is trained using image samples and text description information samples, the image content of the background image samples in the image samples includes the background where at least one display object sample is located, the display object sample is in an occluded state in the image sample, the text description information sample is used to describe the image content of the background image sample, and the text description information is used to describe the image content of the background image; and outputting the text description information.

[0298] Optionally, the computer-readable storage medium may also execute program code for the following steps: using an image-to-text model to identify a background image from the image to be processed; and using the image-to-text model to perform image-to-text processing on the background image to obtain text description information.

[0299] Optionally, the computer-readable storage medium may also execute program code for the following steps: using the graph-generated text model to determine a mask image corresponding to the image to be processed, wherein the mask image includes an area where the display object is displayed in the image to be processed; using the graph-generated text model to perform masking processing on the area corresponding to the mask image in the image to be processed to obtain a background image.

[0300] Optionally, the computer-readable storage medium may also execute program code for the following steps: obtaining prompt information, wherein the prompt information is used to at least indicate that the text description information to be generated is prohibited from including information about the display object; using the prompt information, guiding the image-to-text model to perform image-to-text processing on the background image to obtain text description information.

[0301] Optionally, the computer-readable storage medium can also execute the program code of the following steps: output text description information to the model training end, wherein the text description information, the image to be processed, and the mask image corresponding to the image to be processed are used as training samples to train the output model of the model training end, wherein the mask image includes the area where the display object is displayed in the image to be processed.

[0302] Optionally, the computer-readable storage medium may also execute program code of the following steps: outputting text description information to a model reasoning end, wherein the text description information and the image to be processed are used to generate an output image of the model reasoning end.

[0303] As an optional example, a computer-readable storage medium is configured to store program code for performing the following steps: obtaining an image sample, wherein the image content of a background image sample in the image sample includes a background where at least one display object sample is located, and the display object sample is in an occluded state in the image sample; obtaining a text description information sample of the image sample, wherein the text description information sample is used to describe the image content of the background image sample; generating a graph-to-text model using the image sample and the text description information sample, wherein the graph-to-text model is used to analyze the image to be processed to obtain text description information of the background image in the image to be processed, wherein the image content of the foreground image in the image to be processed includes at least one display object, the image content of the background image includes the background where the display object is located, and the text description information is used to describe the image content of the background image.

[0304] As an optional example, a computer-readable storage medium is configured to store program code for performing the following steps: identifying an image to be processed from an e-commerce platform, wherein the image to be processed includes a foreground image and a background image, the image content of the foreground image includes at least one product to be traded on the e-commerce platform, and the image content of the background image includes the background where the product is located; inputting the image to be processed into a graph-to-text model, and using the graph-to-text model to analyze the image to be processed to obtain text description information of the background image in the image to be processed, wherein the graph-to-text model is trained using image samples and text description information samples, the image content of the background image sample in the image sample includes the background where at least one display object sample is located, the display object sample is in an occluded state in the image sample, the text description information sample is used to describe the image content of the background image sample, and the text description information is used to describe the image content of the background image; outputting the text description information to the e-commerce platform.

[0305] As an optional example, a computer-readable storage medium is configured to store program code for executing the following steps: obtaining text description information of a background image in an image to be processed, wherein the image to be processed includes a foreground image and a background image, the image content of the foreground image includes at least one display object, and the image content of the background image includes the background where the display object is located, the text description information is obtained by analyzing the image to be processed using a graph-to-text model and is used to describe the image content of the background image, the graph-to-text model is trained using image samples and text description information samples, the image content of the background image samples in the image samples includes the background where at least one display object sample is located, the display object sample is in an occluded state in the image sample, and the text description information sample is used to describe the image content of the background image sample; at least the text description information is used as a training sample to train an output model of the model training end.

[0306] As an optional example, a computer-readable storage medium is configured to store program code for performing the following steps: obtaining text description information of a background image in an image to be processed, wherein the image to be processed includes a foreground image and a background image, the image content of the foreground image includes at least one display object, and the image content of the background image includes the background where the display object is located, the text description information is obtained by analyzing the image to be processed using a graph-to-text model and is used to describe the image content of the background image, the graph-to-text model is trained using image samples and text description information samples, the image content of the background image samples in the image samples includes the background where at least one display object sample is located, the display object sample is in an occluded state in the image sample, and the text description information sample is used to describe the image content of the background image sample; and using the text description information and the image to be processed, generating an output image of the model inference end.

[0307] As an optional example, a computer-readable storage medium is configured to store program code for executing the following steps: in response to an input instruction acting on an operation interface, displaying an image to be processed on the operation interface, wherein the image to be processed includes a foreground image and a background image, the image content of the foreground image includes at least one display object, and the image content of the background image includes the background where the display object is located; in response to a text generation instruction acting on the operation interface, displaying text description information of the background image in the image to be processed on the operation interface, wherein the text description information is obtained by analyzing the image to be processed using a graph-to-text model and is used to describe the image content of the background image, the graph-to-text model is trained using image samples and text description information samples, the image content of the background image samples in the image samples includes the background where at least one display object sample is located, the display object sample is in an occluded state in the image sample, and the text description information sample is used to describe the image content of the background image sample.

[0308] The embodiment of the present application further provides a computer program product. Optionally, in this embodiment, the computer program product may include a computer program, and the computer program implements the method provided in the embodiment when executed by a processor.

[0309] Optionally, the computer program product may include a non-volatile computer-readable storage medium, and the non-volatile computer-readable storage medium may be used to store a computer program. When the computer program is executed by a processor, the method provided in the above embodiment is implemented.

[0310] An embodiment of the present application may provide an electronic device, which may include a memory and a processor.

[0311] Fig.28 It is a block diagram of an electronic device according to a method for generating text information in an embodiment of the present application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.

[0312] like Fig.28As shown, the device 2800 includes a computing unit 2801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 2802 or a computer program loaded from a storage unit 2808 into a random access memory (RAM) 2803. In the RAM 2803, various programs and data required for the operation of the device 2800 can also be stored. The computing unit 2801, the ROM 2802, and the RAM 2803 are connected to each other via a bus 2804. An input / output (I / O) interface 2805 is also connected to the bus 2804.

[0313] A number of components in the device 2800 are connected to the I / O interface 2805, including: an input unit 2806, such as a keyboard, a mouse, etc.; an output unit 2807, such as various types of displays, speakers, etc.; a storage unit 2808, such as a disk, an optical disk, etc.; and a communication unit 2809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 2809 allows the device 2800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0314] The computing unit 2801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 2801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSP), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 2801 performs the various methods and processes described above, such as a method for generating text information. For example, in some embodiments, the method for generating text information may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 2808. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 2800 via the ROM 2802 and / or the communication unit 2809. When the computer program is loaded into RAM 2803 and executed by computing unit 2801, one or more steps of the method for generating text information described above may be performed. Alternatively, in other embodiments, computing unit 2801 may be configured to perform the method for generating text information in any other appropriate manner (e.g., by means of firmware).

[0315] Various embodiments of the systems and techniques described above herein may be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard parts (ASSPs), system-on-a-chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs that are executable and / or interpreted on a programmable system including at least one programmable processor that may be a special purpose or general purpose programmable processor that may receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0316] The program code for implementing the method of the present application can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0317] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Fig.29 is a hardware structure block diagram of a computer terminal (or mobile device) for implementing a method for generating text information according to an embodiment of the present application, such as Fig.29As shown, the computer terminal 290 (or mobile device) may include one or more (2902a, 2902b, ..., 2902n are used to illustrate) processors 2902 (the processor 2902 may include but is not limited to a processing device such as a microprocessor (Microcontroller Unit, referred to as MCU) or a programmable logic device (Field Programmable Gate Array, referred to as FPGA)), a memory 2904 for storing data, and a transmission device 2906 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a Universal Serial Bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Fig.29 The structure shown is only for illustration and does not limit the structure of the above electronic device. Fig.29 More or fewer components as shown, or with Fig.29 Different configurations are shown.

[0318] Fig.29 The hardware structure block diagram shown can be used not only as an exemplary block diagram of the above-mentioned computer terminal 290 (or mobile device), but also as an exemplary block diagram of the above-mentioned server. In an optional embodiment, Fig.29 The block diagram shows the use of the above Fig.29 The computer terminal 290 (or mobile device) is shown as an embodiment of a computing node in the computing environment 701.

[0319] In the context of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by an instruction execution system, device or equipment or for use in combination with an instruction execution system, device or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory for short), an optical fiber, a portable compact disc read-only memory (CD-ROM for short), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0320] To provide interaction with a user, the systems and techniques described herein may be implemented on a computer having: a display device (e.g., a cathode ray tube (CRT) or a liquid crystal display (LCD), a monitor for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a path ball), through which the user may provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0321] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a Local Area Network (LAN), a Wide Area Network (WAN), and the Internet.

[0322] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0323] It should be noted that the serial numbers of the above-mentioned embodiments of the present application are only for description and do not represent the advantages or disadvantages of the embodiments.

[0324] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0325] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0326] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0327] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0328] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, read-only memories, random access memories, mobile hard disks, magnetic disks or optical disks.

[0329] The above are only preferred implementations of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for generating text information, characterized in that: include: Acquire an image to be processed, wherein the image to be processed includes a foreground image and a background image, the image content of the foreground image includes at least one display object, and the image content of the background image includes the background where the display object is located; Input the image to be processed into a graph-to-text model, and use the graph-to-text model to analyze the image to be processed to obtain text description information of the background image in the image to be processed, wherein the graph-to-text model is trained using image samples and text description information samples, the image content of the background image sample in the image sample includes the background where at least one display object sample is located, the display object sample is in an occluded state in the image sample, the text description information sample is used to describe the image content of the background image sample, and the text description information is used to describe the image content of the background image; Output the text description information.

2. The method according to claim 1, characterized in that: The image to be processed is analyzed by using the image-to-text model to obtain text description information of the background image in the image to be processed, including: Using the image-to-text model to identify the background image from the image to be processed; The image-to-text model is used to perform image-to-text processing on the background image to obtain the text description information.

3. The method according to claim 2, characterized in that Using the image-to-text model to identify the background image from the image to be processed includes: Determining a mask image corresponding to the image to be processed by using the image-to-text model, wherein the mask image includes an area where the display object is displayed in the image to be processed; The image-to-text model is used to perform masking processing on the area corresponding to the mask image in the image to be processed to obtain the background image.

4. The method according to claim 2, characterized in that: The image-to-text model is used to process the background image to obtain the text description information, including: Acquire prompt information, wherein the prompt information is used to at least indicate that the text description information to be generated is prohibited from including information about the display object; The prompt information is used to guide the image-to-text model to perform image-to-text processing on the background image to obtain the text description information.

5. The method according to any one of claims 1 to 4, characterized in that: Output the text description information, including: Output the text description information to the model training end, wherein the text description information, the image to be processed, and the mask image corresponding to the image to be processed are used as training samples to train the output model of the model training end, wherein the mask image includes the area where the display object is displayed in the image to be processed.

6. The method according to any one of claims 1 to 4, characterized in that Output the text description information, including: Output the text description information to the model inference end, wherein the text description information and the image to be processed are used to generate an output image of the model inference end.

7. A method for determining a model, characterized in that: include: Acquire an image sample, wherein the image content of a background image sample in the image sample includes a background where at least one display object sample is located, and the display object sample is in an obstructed state in the image sample; Acquire a text description information sample of the image sample, wherein the text description information sample is used to describe the image content of the background image sample; An image-to-text model is generated using the image samples and the text description information samples, wherein the image-to-text model is used to analyze the image to be processed to obtain text description information of the background image in the image to be processed, the image content of the foreground image in the image to be processed includes at least one display object, the image content of the background image includes the background where the display object is located, and the text description information is used to describe the image content of the background image.

8. The method according to claim 7, characterized in that Get image samples, including: Acquire a mask image sample corresponding to an initial image sample, wherein the initial image sample includes a foreground image sample and the background image sample, the image content of the foreground image sample includes the display object sample, and the mask image sample includes an area displayed by the display object sample in the initial image sample; Performing masking processing on the area corresponding to the mask image sample in the initial image sample to obtain the image sample.

9. The method according to claim 8, characterized in that Get the mask image sample corresponding to the initial image sample, including: Performing image segmentation on the initial image sample to obtain the mask image sample; Performing masking processing on the area corresponding to the mask image sample in the initial image sample to obtain the image sample includes: setting the area corresponding to the mask image sample in the initial image sample to black to obtain the image sample.

10. The method according to claim 7, characterized in that Obtaining a text description information sample of the image sample includes: Acquire a complete text description information sample of the image sample, wherein the complete text description information sample is used to describe the complete image content of the image sample; The text description information sample is generated based on the complete text description information sample and the image sample.

11. The method according to claim 10, characterized in that Obtain a complete text description information sample of the image sample, including: Acquire a first prompt information sample, wherein the first prompt information sample is used to represent a rule that the complete text description information to be generated needs to satisfy; The first prompt information sample is used to guide the natural language processing model to analyze the image sample to obtain the complete text description information sample.

12. The method according to claim 10, characterized in that Generating the text description information sample based on the complete text description information sample and the image sample includes: Acquire a second prompt information sample, wherein the second prompt information sample is used to represent a rule that the text description information sample to be generated needs to satisfy; The second prompt information sample is used to guide the natural language processing model to analyze the complete text description information sample and the image sample to obtain the text description information sample.

13. The method according to claim 7, characterized in that The image sample includes image contents of multiple sub-areas associated with the display object sample, and the method further includes: Acquire, from the text description information samples, a plurality of sub-text description information samples corresponding to the plurality of sub-regions one by one, wherein the sub-text description information samples are used to describe the image content of the corresponding sub-regions; Determining the correlation between the sub-text description information sample and the image content of the sub-region, wherein the correlation is used to express the degree of correlation between the sub-text description information sample and the image content of the sub-region; In response to the relevance being lower than a relevance threshold, it is prohibited to use the image sample and the text description information sample to generate an image-to-text model.

14. The method according to any one of claims 7 to 13, characterized in that Using the image sample and the text description information sample, generating an image-to-text model includes: Determine the image sample and the text description information sample as seed data; The student model is distilled using the seed data to obtain the graph-to-text model, wherein the parameter amount of the student model is less than a parameter amount threshold.

15. A method for generating text information, characterized in that: include: Identify an image to be processed from an e-commerce platform, wherein the image to be processed includes a foreground image and a background image, the image content of the foreground image includes at least one product to be traded on the e-commerce platform, and the image content of the background image includes the background where the product is located; Input the image to be processed into a graph-to-text model, and use the graph-to-text model to analyze the image to be processed to obtain text description information of the background image in the image to be processed, wherein the graph-to-text model is trained using image samples and text description information samples, the image content of the background image sample in the image sample includes the background where at least one product sample is located, and the product sample is in an occluded state in the image sample, the text description information sample is used to describe the image content of the background image sample, and the text description information is used to describe the image content of the background image; Output the text description information to the e-commerce platform.

16. A method for generating text information, characterized in that: Applied to model training, including: Obtaining text description information of a background image in an image to be processed, wherein the image to be processed includes a foreground image and a background image, the image content of the foreground image includes at least one display object, the image content of the background image includes the background where the display object is located, the text description information is obtained by analyzing the image to be processed using a graph-to-text model and is used to describe the image content of the background image, the graph-to-text model is trained using image samples and text description information samples, the image content of the background image sample in the image sample includes the background where at least one display object sample is located, the display object sample is in an occluded state in the image sample, and the text description information sample is used to describe the image content of the background image sample; At least the text description information is used as a training sample to train and obtain an output model of the model training end.

17. The method according to claim 16, characterized in that At least using the text description information as a training sample to train an output model of the model training end includes: Acquire a mask image corresponding to the image to be processed, wherein the mask image includes an area where the display object is displayed in the image to be processed; The mask image, the text description information and the image to be processed are used as training samples to train and obtain the output model.

18. A method for generating text information, characterized in that: Applied to the model inference end, including: Obtaining text description information of a background image in an image to be processed, wherein the image to be processed includes a foreground image and a background image, the image content of the foreground image includes at least one display object, the image content of the background image includes the background where the display object is located, the text description information is obtained by analyzing the image to be processed using a graph-to-text model and is used to describe the image content of the background image, the graph-to-text model is trained using image samples and text description information samples, the image content of the background image sample in the image sample includes the background where at least one display object sample is located, the display object sample is in an occluded state in the image sample, and the text description information sample is used to describe the image content of the background image sample; The output image of the model inference end is generated by using the text description information and the image to be processed.

19. The method according to claim 18, characterized in that Using the text description information and the image to be processed, generating an output image of the model inference end, including: Generate a background image of the input image of the model inference end by using the text description information and the image to be processed, wherein the input image is used to represent an image of the product to be traded, and the image content of the background image of the input image includes the background where the product to be traded is located; The background image of the input image and the input image are combined to generate an output image of the model inference end.

20. A method for generating text information, characterized in that: include: In response to an input instruction on the operation interface, displaying an image to be processed on the operation interface, wherein the image to be processed includes a foreground image and a background image, the image content of the foreground image includes at least one display object, and the image content of the background image includes a background where the display object is located; In response to a text generation instruction on the operation interface, text description information of the background image in the image to be processed is displayed on the operation interface, wherein the text description information is obtained by analyzing the image to be processed using a graph-to-text model and is used to describe the image content of the background image, the graph-to-text model is trained using image samples and text description information samples, the image content of the background image sample in the image sample includes the background where at least one display object sample is located, the display object sample is in an occluded state in the image sample, and the text description information sample is used to describe the image content of the background image sample.

21. An electronic device, characterized in that: include: A memory storing an executable program; A processor, configured to run the program, wherein the program executes the method according to any one of claims 1 to 19 when running.

22. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored executable program, wherein when the executable program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 19.

23. A computer program product, characterized in that The invention comprises a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 19.

Images