Point of interest image generation method and apparatus, electronic device, and storage medium

Abstract

This disclosure provides a method, apparatus, electronic device, and storage medium for generating point-of-interest (POI) images. The method includes: acquiring image description information of a target POI, the image description information including image description text; inputting the image description information into a pre-trained static image generation joint model; and executing the pre-trained static image generation joint model to generate a static image of the target POI. The static image generation joint model includes a first large-scale language model and a text-to-image model. The first large-scale language model generates text vectors based on the image description information, and the text-to-image model generates the static image of the target POI based on the text vectors. This technical solution can quickly generate high-quality POI images.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, specifically to a method, apparatus, electronic device, and storage medium for generating point-of-interest images. Background Technology

[0002] With the development of science and technology and the continuous improvement of people's quality of life, electronic map applications have emerged to facilitate people's travel. In existing electronic maps, Points of Interest (POI) data is provided to facilitate users' browsing and querying of useful information. The richness and attractiveness of POI data are strongly correlated with user experience. Higher quality content and more dynamic presentation can encourage users to stay in the map client, browse in-depth content, and thus promote conversion.

[0003] Points of interest (POI) images are a crucial part of conveying information to users. However, existing electronic maps suffer from low POI image coverage. Popular POIs typically have abundant images, while less popular POIs often lack sufficient images. Furthermore, some POI images are user-generated, and their shooting angles and resolutions may not meet display standards, resulting in low image quality. Therefore, generating high-quality POI images has become a pressing technical challenge. Summary of the Invention

[0004] To address the problems in the related technologies, this disclosure provides a method, apparatus, electronic device, and storage medium for generating point-of-interest images.

[0005] In a first aspect, this disclosure provides a method for generating point-of-interest (POI) images.

[0006] Specifically, the interest point image generation method includes:

[0007] Obtain image description information of the target point of interest, wherein the image description information includes image description text;

[0008] The image description information is input into a pre-trained static image generation joint model, and the pre-trained static image generation joint model is executed to generate a static image of the target interest point. The static image generation joint model includes a first large-scale language model and a text-to-image model. The first large-scale language model is used to generate text vectors based on the image description information, and the text-to-image model is used to generate a static image of the target interest point based on the text vectors.

[0009] Secondly, this disclosure provides a training method for a joint model for static image generation, comprising:

[0010] Obtain a first training dataset, which includes multiple positive samples and / or multiple negative samples, wherein both the positive samples and the negative samples include sample images of sample interest points and sample image text.

[0011] The initial static image generation joint model is trained using the first training dataset to obtain a trained static image generation joint model, which includes a first large-scale language model and a text-to-image model.

[0012] Wherein, the loss function of the static image generation joint model is L = αL1 + (1-α)L2, where L1 is the difference between the noise predicted by the text-to-image model at each time step during the denoising process and the Gaussian noise added during the diffusion process, L2 is the difference between the predicted image text output by the first large-scale language model and the sample image text, α is a predetermined parameter value, and the text vector generated by the first large-scale language model is the image generation condition of the text-to-image model.

[0013] Thirdly, this disclosure provides a method for training a joint model for dynamic image generation, comprising:

[0014] Obtain sample images and basic information of sample points of interest;

[0015] Using a predetermined image segmentation model, mask information for each mask corresponding to the sample image is generated based on the sample image;

[0016] Obtain the operation information of each mask corresponding to the sample image;

[0017] Based on the second training dataset, the second large-scale language model is fine-tuned to obtain the trained second large-scale language model. The second training dataset includes basic information of multiple sample interest points, sample images and their corresponding mask information and operation information.

[0018] The trained dynamic image generation joint model includes a trained second large-scale language model, a predetermined image segmentation model, and a predetermined dynamic image editing model. The trained second large-scale language model is used to generate image editing instructions based on the mask information and basic information of each mask, and the predetermined dynamic image editing model is used to edit the image according to the image editing instructions to generate dynamic images.

[0019] Fourthly, this disclosure provides an interest point image generation apparatus, comprising:

[0020] The information acquisition module is configured to acquire image description information of the target point of interest, wherein the image description information includes image description text;

[0021] A static image generation module is configured to input the image description information into a pre-trained static image generation joint model, and execute the pre-trained static image generation joint model to generate a static image of the target interest point. The static image generation joint model includes a first large-scale language model and a text-to-image model. The first large-scale language model is used to generate text vectors based on the image description information, and the text-to-image model is used to generate a static image of the target interest point based on the text vectors.

[0022] Fifthly, embodiments of this disclosure provide an electronic device including a memory and a processor, wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the method as described in any one of the first aspects.

[0023] In a sixth aspect, embodiments of this disclosure provide a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the method as described in any one of the first aspects.

[0024] In a seventh aspect, this disclosure provides a computer program product including computer instructions that, when executed by a processor, implement the method steps as described in any one of the first aspects.

[0025] According to the technical solution provided in this disclosure, a pre-trained static image generation joint model can be used to generate static images of the target points of interest based on the image description information. This image description information includes image description text. The powerful natural language understanding capability of the first large-scale language model in the static image generation joint model can be utilized to understand the image description information and generate text vectors. These text vectors can then be used to restrict image generation conditions in the Chinese-language image generation model of the static image generation joint model. By utilizing the image generation capability of this text-based image generation model, static images conforming to the description information can be generated based on the text vectors. Thus, by using text vectors understood by the first large-scale language model to restrict generation conditions in the text-based image generation model, high-quality point-of-interest images can be generated, filling in missing or replacing low-quality point-of-interest images, making the point-of-interest data richer and more attractive, and providing users with better map services.

[0026] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0027] Other features, objects, and advantages of this disclosure will become more apparent from the following detailed description of non-limiting embodiments, taken in conjunction with the accompanying drawings. In the drawings:

[0028] Figure 1 A flowchart illustrating a method for generating point-of-interest images according to embodiments of the present disclosure is shown;

[0029] Figure 2 A flowchart illustrating a training method for a joint model for generating static images according to an embodiment of the present disclosure is provided.

[0030] Figure 3 A flowchart illustrating a training method for a joint model for dynamic image generation according to an embodiment of the present disclosure is provided.

[0031] Figure 4 A structural block diagram of an interest point image generation apparatus according to an embodiment of the present disclosure is shown;

[0032] Figure 5 A structural block diagram of a training apparatus for a static image generation joint model according to an embodiment of the present disclosure is shown.

[0033] Figure 6 A structural block diagram of a training apparatus for a joint model of dynamic image generation according to an embodiment of the present disclosure is shown.

[0034] Figure 7 A structural block diagram of an electronic device according to an embodiment of the present disclosure is shown;

[0035] Figure 8 A schematic diagram of the structure of a computer system suitable for implementing the method according to embodiments of the present disclosure is shown. Detailed Implementation

[0036] In the following, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings to enable those skilled in the art to readily implement them. Furthermore, for clarity, portions unrelated to the description of exemplary embodiments have been omitted from the drawings.

[0037] In this disclosure, it should be understood that terms such as “comprising” or “having” are intended to indicate the presence of features, figures, steps, behaviors, components, parts or combinations thereof disclosed in this specification, and are not intended to exclude the possibility of the presence or addition of one or more other features, figures, steps, behaviors, components, parts or combinations thereof.

[0038] It should also be noted that, unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other. This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0039] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0040] As mentioned above, with the development of science and technology and the continuous improvement of people's quality of life, electronic maps have emerged to facilitate travel. Existing electronic maps provide Points of Interest (POI) data to facilitate users' browsing and querying of useful information. The richness and attractiveness of POI data are strongly correlated with user experience. Higher-quality content and more dynamic presentation formats can encourage users to stay on the map client, browse in-depth content, and thus drive conversions. POI images are a crucial part of conveying information to users. However, in existing electronic maps, the POI image fill rate is low. Popular POIs typically have a large number of images, while less popular POIs suffer from significant image loss. Furthermore, some POI images are taken by users, and the shooting angle / resolution may not meet display standards, resulting in low image quality. Therefore, how to generate high-quality POI images has become a pressing technical problem to be solved.

[0041] This disclosure provides a method for generating point-of-interest (POI) images. This method uses a trained joint static image generation model to autonomously generate static images of the target POI based on its image description information. It leverages the powerful natural language understanding capabilities of the first large-scale language model within the joint static image generation model to understand the image description information and generate text vectors. These text vectors are then used to constrain the image generation conditions of the Chinese-language image generation model within the joint static image generation model. Utilizing the image generation capabilities of this text-based image generation model, static images conforming to the image description information are generated based on the text vectors. Thus, by using text vectors understood by the first large-scale language model to constrain generation conditions within the text-based image generation model, high-quality POI images can be generated, filling in missing POI images or replacing low-quality POI images, making the POI data richer and more attractive, and providing users with better map services.

[0042] Figure 1 A flowchart illustrating a method for generating point-of-interest images according to embodiments of the present disclosure is shown. Figure 1 As shown, the point of interest image generation method includes the following steps S101-S102:

[0043] In step S101, image description information of the target point of interest is obtained, wherein the image description information includes image description text;

[0044] In step S102, the image description information is input into a pre-trained static image generation joint model, and the pre-trained static image generation joint model is executed to generate a static image of the target interest point based on the image description information. The static image generation joint model includes a first large-scale language model and a text-to-image model. The large-scale language model is used to generate text vectors based on the image description information, and the text-to-image model is used to generate a static image of the target interest point based on the text vectors.

[0045] In one possible implementation, the point of interest image generation method is applicable to devices such as computers, computing devices, servers, and server clusters capable of generating point of interest images.

[0046] In one possible implementation, the target point of interest can be a point of interest in a missing point of interest image, or a point of interest in an image with low image quality that needs to be replaced.

[0047] In one possible implementation, the image description text refers to any text that can indirectly infer the image to be generated or intuitively describe the image to be generated. It can be at least one of various texts describing a point of interest, such as attribute information text, product description text, or comment text. The attribute information text includes text with inherent attributes such as the name, brand, and category of the point of interest. The product description text includes SKU (Stock Keeping Unit) description text and / or SPU (Standard Product Unit) description text. For example, when the product is a mobile phone, the SPU description text might be "Model XX mobile phone," and the SKU description text might be "Model XX mobile phone with 16GB of memory and black color."

[0048] In one possible implementation, the static image generation joint model is used to generate a static image of the target interest point based on the image description information. Its input is the image description information, and its output is a static image. This static image generation joint model is a joint model of a Large Language Model (LLM) and a text-to-image model. The text-to-image model can be an SDM (Stable Diffusion Model), a text-to-image generation model based on Latent Diffusion Models (LDMs). Specifically, Stable Diffusion uses a text-to-image dataset to train a Latent Diffusion Model, which is specifically designed for text-to-image generation. The SDM model, after training, can progressively denoise random Gaussian noise to obtain an image matching the text description. It should be noted that, besides SDM, this text-to-image model can also be other types of intelligent drawing models.

[0049] In one possible implementation, the first large-scale language model is used to understand image description information, generate text vectors that understand natural language meaning, and then the output layer of the first large-scale language model outputs the understood natural language meaning based on the text vectors. In this implementation, the first large-scale language model and the text-to-image model are a joint model. The text vectors generated by the first large-scale language model serve as the input to the text-to-image model, constraining the text-to-image model to generate static images that conform to the image description information, thus generating static images for the target point of interest. For example, the text-to-image model can be an SDM (Search Engine Denoising Model), which includes a denoising network and a diffusion network. The input to the diffusion network is the encoding of a predetermined noisy image. The diffusion network can gradually add Gaussian noise to the noisy image to generate a random noise image. The text vector is input to the denoising network, which estimates the noise in the random noise image based on the text vector. Based on the estimated noise, it generates an image with less noise using a posterior probability formula. Through iteration, the denoising network gradually generates the content constrained by the text vector and injects it into the decoder to generate a realistic static image.

[0050] This implementation can use a pre-trained static image generation joint model to generate static images of the target points of interest based on the image description information, which includes image description text. The powerful natural language understanding capabilities of the first large-scale language model within the static image generation joint model can be utilized to understand the image description information and generate text vectors. These text vectors can then be used to restrict image generation conditions within the Chinese-language image generation model of the static image generation joint model. By leveraging the image generation capabilities of this text-based image generation model, static images conforming to the description information can be generated based on the text vectors. In this way, by using text vectors understood by the first large-scale language model to restrict generation conditions within the text-based image generation model, high-quality point-of-interest images can be generated, filling in missing or replacing low-quality point-of-interest images, making the point-of-interest data richer and more attractive, and providing users with better map services.

[0051] In one possible implementation, the image description information further includes a low-quality image for describing the target interest point, and the step of inputting the image description information into a pre-trained static image generation joint model and executing the pre-trained static image generation joint model to generate a static image of the target interest point includes:

[0052] The image description text and the low-quality image are input into the first large-scale language model, the first large-scale language model is executed, and the text vector output by the first large-scale language model is obtained.

[0053] The text vector and the low-quality image are input into the text-to-image model, and the text-to-image model is executed to obtain a static image of the target interest point output by the text-to-image model.

[0054] In this embodiment, the target point of interest already has an image describing the target point of interest, but the image quality of the image is low and needs to be replaced. In this case, the image description information may include the low-quality image in addition to the image description text. The static image generation joint model is used to generate a static image based on the image description text and the low-quality image.

[0055] In this embodiment, the first large-scale language model is a multimodal large-scale language model. In addition to understanding the image description text, it can also understand the natural language meaning of the low-quality image. After the image description text and the low-quality image are input into the first large-scale language model, the first large-scale language model can embed the low-quality image and concatenate it with the image description text, and generate a text vector accordingly. In this way, the text vector output by the first large-scale language model can be obtained.

[0056] In this embodiment, the text-to-image model can be an SDM (Search Engine Design Model), which includes a denoising network and a diffusion network. When the low-quality image is included in the image description information, the low-quality image can be input into the encoder of the SDM to obtain the encoding of the low-quality image. At this time, the input of the diffusion network in the SDM is not a predetermined noise image but the encoding of the low-quality image. The diffusion network can gradually add Gaussian noise to the low-quality image to generate a random noise image. The text vector and the random noise image can be input into the denoising network in the SDM. The denoising network will estimate the noise in the random noise image based on the text vector. Based on the estimated noise, it will generate an image with less noise through a posterior probability formula. Through iteration, the denoising network will gradually generate the content defined by the text vector and inject it into the decoder to generate a realistic static image.

[0057] This embodiment can also use low-quality images and image description text to generate static images. By referencing existing low-quality images, more realistic and lifelike static images can be generated.

[0058] In one possible implementation, the first large-scale language model corresponds to multiple prompt templates.

[0059] The step of inputting the image description information into a pre-trained joint static image generation model and executing the pre-trained joint static image generation model to generate a static image of the target interest point includes:

[0060] The image description information is input into a pre-trained static image generation joint model. The pre-trained static image generation joint model is executed multiple times to generate static images of the target interest points, resulting in multiple different static images.

[0061] The method further includes:

[0062] For each static image, a corresponding prompt is determined. The prompt is generated by the first large-scale language model based on the image description information and the prompt template.

[0063] The static image is input into the first large-scale language model to obtain the static image text output by the first large-scale language model;

[0064] Calculate the quality score of the static image based on the static image text and the corresponding prompt.

[0065] Based on the quality scores of the multiple different still images, at least one target still image is selected from the multiple different still images.

[0066] In this implementation, the prompt template helps the large-scale language model "recall" the knowledge learned during the pre-training phase, guiding the large-scale language model to perform tasks. The first large-scale language model corresponds to multiple prompt templates. For example, one prompt template is: "Shop name 'XXXXX', products 'XXX' and 'XXX', user reviews 'XXXXXX'. Based on the above information, generate a realistic signboard image and produce various product images." After obtaining the image description text, the content of the image description text can be used to fill in "XXXXXX" in the prompt template. Assume the image description text for the target point of interest is shown in Table 1 below:

[0067] Image description text field name Image description text field values Target Point of Interest Name Xiao Dong Department Store - Hardware, Electrical Supplies, Plumbing Fittings Target Point of Interest Product Name Tin scissors, size 8 nut Target Interest Points User Reviews High quality, low price, truly affordable, and trustworthy.

[0068] Table 1

[0069] By adding the image description text shown in Table 1 above into the prompt template, you can obtain the following prompt: "Shop name 'Xiao Dong Department Store Hardware, Electrical Supplies and Plumbing Fittings', products include 'Tin Scissors' and 'No. 8 Nuts', user reviews 'High quality, low price, affordable, and trustworthy'. Based on the above information, generate a realistic signboard image and produce various product images."

[0070] In this embodiment, after inputting the image description information into a pre-trained static image generation joint model, the model can be executed multiple times to generate static images of the target interest point. Each execution generates one static image. Different prompt templates or the same prompt template can be used for any two executions. Using different prompt templates will generate different static images. Due to the randomness of static image generation, even using the same prompt template, the generated static images will be different each time. Therefore, multiple static images can be generated by using the static image generation joint model multiple times.

[0071] In this embodiment, in order to obtain higher quality still images, one or more target still images of the highest quality can be selected from the generated still images. The selection method can be manual selection, but manual selection requires a lot of cost. Therefore, in order to reduce costs, quality assessment can be performed automatically to select at least one target still image of the highest quality.

[0072] In this embodiment, the automatic selection method can be to use an image quality scoring model to automatically score each static image to obtain the quality score of the static image, and then select one or more target static images with the highest quality scores.

[0073] In one possible implementation, selecting at least one target still image from a plurality of different still images includes:

[0074] For each static image, a corresponding prompt is determined. The prompt is generated by the first large-scale language model based on the image description information and the prompt template.

[0075] The static image is input into the first large-scale language model to obtain the static image text output by the first large-scale language model;

[0076] Calculate the first similarity between the static image text and the corresponding prompt of the static image;

[0077] The quality score of the static image is determined based on the first similarity.

[0078] Based on the quality scores of the multiple different still images, at least one target still image is selected from the multiple different still images.

[0079] In this embodiment, the image quality scoring model may include a first large-scale language model and a computational layer. For each static image, a corresponding prompt can be determined. The static image is input into the first large-scale language model, which utilizes its powerful understanding capabilities to obtain static image text describing the static image. The computational layer is used to calculate the first similarity between the static image text and the prompt. For example, the inner product of the static image text and the prompt can be used as the first similarity. This first similarity can be used as the quality score of the static image. The higher the first similarity, the more the semantics of the generated static image matches the prompt, and the more it meets the image generation requirements described by the prompt. The higher the quality of the generated static image, the higher the quality score of the target static image can be obtained.

[0080] This embodiment can use the static image generation joint model to generate multiple different static images. Then, based on the prompts corresponding to the static images and the first similarity between the static image text generated after understanding the static image by the first large-scale language model, the quality score of the static image is calculated. Based on the quality scores of the multiple different static images, one or more target static images with the highest quality are selected from the multiple static images, thus obtaining a higher quality static image.

[0081] In one possible implementation, if the image description information further includes a low-quality image, the method further includes:

[0082] The low-quality image is input into the first large-scale language model to obtain the low-quality image text output by the first large-scale language model.

[0083] Calculate the second similarity between the static image text and the low-quality image text of the static image;

[0084] Determining the quality score of the static image based on the first similarity includes:

[0085] The quality score of the static image is determined based on the first similarity and the second similarity.

[0086] Based on the quality scores of the multiple different still images, at least one target still image is selected from the multiple different still images.

[0087] In this embodiment, if the image description information also includes a low-quality image, the quality score of the static image can be calculated by referring to the second similarity between the static image and the low-quality image. The low-quality image can be input into the first large-scale language model to obtain the low-quality image text output by the first large-scale language model. The second similarity between the static image text and the low-quality image text can be calculated. For example, the inner product of the static image text and the low-quality image text can be used as the second similarity. The higher the second similarity, the closer the generated static image is to the meaning expressed by the low-quality image.

[0088] In this embodiment, the quality score of the static image can be determined based on the first similarity and the second similarity. For example, the first similarity and the second similarity can be weighted and averaged to obtain the quality score of the static image, and one or more target static images with the highest quality scores can be obtained.

[0089] This implementation method can comprehensively consider the first similarity between the static image text and the prompt and the second similarity between the static image and the low-quality image text to determine the quality score of the static image. This can avoid the semantic difference between the acquired target static image and the low-quality image being too large, and avoid large semantic changes before and after image generation.

[0090] In one possible implementation, the method may further include the following steps:

[0091] Obtain basic information about the target point of interest;

[0092] For any static image, a pre-trained dynamic image generation joint model is used to generate a dynamic image of the target interest point based on the basic information of the target interest point and the target static image.

[0093] In this implementation, the basic information of the target interest point can be various descriptive information, such as the target interest point's name, category (e.g., food, shopping mall, school), and other attribute information, as well as comment information (including comment text and comment images), etc. Different target interest points require different dynamic effects. A pre-trained dynamic image generation joint model can be used to configure the corresponding dynamic effects for the target static image based on the basic information of the target interest point, thereby generating the corresponding dynamic image. The dynamic image generation joint model includes a second large-scale language model, an image segmentation model, and a dynamic image editing model. The input of the dynamic image generation joint model is the basic information of the target interest point and the static image, and the output of the dynamic image generation joint model is the dynamic image of the target interest point.

[0094] In this embodiment, any static image can be a single static image of the target interest point generated by a pre-trained static image generation joint model, or it can be a single target static image selected from multiple different static images generated by the pre-trained static image generation joint model.

[0095] In this embodiment, the input of the image segmentation model is a static image of the target interest point, and the output of the image segmentation model is the mask information of each mask image of the static image. The image segmentation model is used to segment the static image of the target interest point into multiple masks and generate mask information for each mask. The mask information includes information such as mask size, mask type, and mask position in the static image. For example, the specific content of the mask information is shown in Table 2 below:

[0096]

[0097] Table 2

[0098] In this embodiment, the input of the second large-scale language model is the mask information of each mask and the basic information of the target interest point. The output of the second large-scale language model is an image editing instruction. This second large-scale language model is used to generate image editing instructions based on the mask information of the multiple mask images and the basic information of the target interest point. It should be noted that the output of this large-scale language model is actually natural language text, which needs to be converted into image operation instructions. This can be achieved by adding a conversion layer to the second large-scale language model. This conversion layer can be implemented through code. The conversion layer can convert the corresponding mask number and operation information for that mask in the natural language text into image editing instructions that the dynamic image editing model can accept. The operation information includes the operation type and operation range. For example, the specific content of the operation information is shown in Table 3 below:

[0099]

[0100] Table 3

[0101] In this embodiment, a filtering layer can be added before the conversion layer in the second large-scale language model. This filtering layer can control the number of masks to be operated on. If there are many masks to be operated on, many areas of the generated dynamic image will have dynamic effects, which is not visually appealing to users. If there are few masks to be operated on, only a few areas of the generated dynamic image will have dynamic effects, which may go unnoticed by users and fail to achieve the desired appeal of the dynamic image. Therefore, when there are too many masks to be operated on, the filtering layer can select some masks to operate on, and when there are few masks to be operated on, it can discard the creation of dynamic effects for the static image. It should be noted that when selecting some masks to operate on, the selection can be based on the mask position and / or the operation information of the mask according to predetermined rules. For example, the predetermined rules could be to retain one mask for the same type of operation, one mask for the same area, or to retain masks with higher operation priority according to the predetermined operation priority of each operation information, etc.

[0102] In this embodiment, the input of the dynamic image editing model is an image editing instruction and the static image, and the output of the dynamic image editing model is a dynamic image edited according to the image editing instruction. The dynamic image editing model is used to edit the static image according to the image editing instruction to obtain the dynamic image of the target point of interest.

[0103] It should be noted that when the image editing command is a drag command, the dynamic image editing model will generate multiple intermediate results when executing the drag command. We can denote the image before the drag command is executed as A0, and the image at the final position as A... n Then there will be intermediate results A1, A2, ..., A n-1 At this point, the dynamic image editing model can construct an image sequence of A0, A1, A2, ..., A n-1 A n A n-1 By using A1, ..., A2, A1, a dynamic effect of image gradation can be achieved.

[0104] This implementation can use a pre-trained joint model for generating dynamic images to configure corresponding dynamic effects for the static image of the target based on the basic information of the target interest point, thereby generating the corresponding dynamic image efficiently and at low cost. This can increase the data types of the target interest point, making the data of the target interest point richer and more attractive.

[0105] This disclosure also provides a training method for a joint model for static image generation. Figure 2 A flowchart illustrating a training method for a joint model for generating static images according to embodiments of the present disclosure is shown. Figure 2 As shown, the training method may include the following steps S201-S202:

[0106] In step S201, a first training dataset is obtained, which includes multiple positive samples and / or multiple negative samples. The positive samples and the negative samples each include sample images of sample interest points, sample image text, and sample weights.

[0107] In step S202, the initial static image generation joint model is trained using the first training dataset to obtain a trained static image generation joint model, which includes a first large-scale language model and a text-to-image model.

[0108] In one possible implementation, the training method for the static image generation joint model is applicable to devices such as computers, computing devices, servers, and server clusters that can perform the training of the static image generation joint model.

[0109] In one possible implementation, existing LLM models trained on general corpora and text-based image models trained on general images cannot be directly applied to point-of-interest (POI) image production. Firstly, there's a gap between the corpora; LBS (Location Based Service) scenarios contain a large amount of geographically relevant descriptions and terminology, and models trained on general corpora lack understanding of this knowledge. Secondly, the prompts used in LLM generation are often complex and difficult to pronounce, requiring high proficiency in prompt design from technical personnel. Therefore, LLM needs fine-tuning to allow downstream users to access the platform in a more natural language-like manner. Another crucial aspect is the gap between images; general image training sets contain various painting styles and expressions, most of which are unsuitable for LBS scenarios. Therefore, fine-tuning training using PPO images is necessary. Furthermore, the training processes for LLM and text-based image models are currently separate. Here, the two models are trained jointly, their loss calculated together, and both optimized together to achieve better results. The two jointly trained models can be referred to as the static image generation joint model, which includes a first large-scale language model and a text-to-image model.

[0110] In one possible implementation, the first training dataset includes multiple positive samples and / or multiple negative samples, each of which includes sample images of interest points, sample image text, and sample weights. For example, the sample images, their image sources, and sample categories in the first training dataset can be shown in Table 4 below:

[0111]

[0112] Table 4

[0113] As shown in Table 4 above, sample images can be obtained by filtering and selecting from existing interest header images, product data, and comment data. However, different sample categories (positive / negative samples) are assigned to different image sources.

[0114] In one possible implementation, for sample image text, when the sample image is an interest header image of a sample interest point, the interest point name, interest point brand, interest point category, etc. in the interest header image can be used as sample image text; when the sample image is an image in product data, the description of SKU / SPU can be used as sample image text; when the sample image is an image in comment data, processed (e.g., summarized & extracted) user comments can be used as sample image text.

[0115] In one possible implementation, a first large-scale language model and a text-to-image model are jointly combined into a static image generation joint model. This combination can be achieved by using the text vector output by the first large-scale language model as the image generation condition for the text-to-image model, and the text vector output by the first large-scale language model as the input to its output layer. The output layer can then output the predicted image text based on this text vector. Training this static image generation joint model using the aforementioned first training dataset includes the following two sub-tasks:

[0116] The first subtask is to minimize the difference between the noise predicted at each time step during the denoising process and the Gaussian noise added during the diffusion process. During denoising, a Cross-Attention mechanism can be used, where the prompts from natural language are used as conditional constraints. These are encoded using a first large-scale language to generate text vectors, which serve as the Q element in the attention mechanism. Therefore, during the gradual denoising process, static images conforming to the prompts can be generated; the difference between the noise predicted at each time step during denoising and the Gaussian noise added during the diffusion process can be denoted as L1.

[0117] The second subtask is multimodal training in the first large-scale language. Sample images are embedded and then concatenated with the input sample image text. An autoregressive approach is used for training, aiming to enable the first large-scale language to learn the natural language meaning represented by the sample images, thus deepening its understanding of images. The image encoding part is not considered in the loss function; only the difference between the predicted image text and the sample image text (e.g., calculating the cross-entropy when predicting text from an image) is calculated. This loss is denoted as L2.

[0118] Finally, the loss function of the joint static image generation model is L = αL1 + (1-α)L2, where α is a predetermined parameter value. The model parameters of the two models in the joint static image generation model can be continuously adjusted until the loss function reaches its minimum, thus obtaining the trained joint static image generation model.

[0119] In other possible implementations, a warm-up process can be set up during model training. In the initial stage of training, the generated images output by the text-based image model are not yet stable, and sample images are used to calculate L2. When L1 gradually stabilizes, the generated images output by the text-based image model are used to calculate L2 (i.e., the difference between the L2 predicted image text and the generated image text), so as to ensure the stability of the sample input distribution in subsequent use.

[0120] In other possible implementations, the static image generation joint model may also use a predetermined text-to-image model, in which the parameters of the text-to-image model are not adjusted during training, but the first large-scale language model in the static image generation joint model is fine-tuned, and so on.

[0121] In one possible implementation, the text-to-image model can be an SDM, which is a text-to-image generation model based on Latent Diffusion Models (LDMs). Of course, in addition to SDM, the text-to-image model can also be other types of intelligent drawing models.

[0122] In one possible implementation, both the positive and negative samples also include sample weights, as shown in Table 4. Different sample weights (confidence levels) can be assigned to different image sources. The sample weights are implemented through a loss function during model training, which can be achieved by multiplying the training loss of each sample by its weight.

[0123] When calculating the loss function, the sample weights of each sample can be used to calculate the weights of the samples or categories. The loss function is ultimately implemented in the training model. When calculating the loss of the model, the loss corresponding to each sample is multiplied by its weight and then weighted to obtain the loss.

[0124] In one possible implementation, the training method described above includes the first training dataset comprising original positive samples, original negative samples, expanded positive samples, and / or expanded negative samples, and obtaining the first training dataset may include:

[0125] Obtain original positive samples and / or original negative samples, wherein both the original positive samples and the original negative samples include the original sample image of the sample interest point and the original sample image text;

[0126] The original positive samples are augmented using at least one of the following steps to obtain expanded positive samples and / or expanded negative samples:

[0127] In response to the absence of the original sample image, the interest point images of other interest points under the same brand as the sample interest point are used as the extended sample images of the extended positive samples, and the original sample image text of the sample interest point is used as the extended sample image text of the extended positive samples.

[0128] Since the source of the original positive samples is comment data, the comment data is broken down to obtain multiple extended positive samples;

[0129] For any two original positive samples, construct an extended negative sample by combining the original sample image in one original positive sample with the original sample image text in the other original positive sample.

[0130] In response to the fact that the source of the original positive sample is product data, an extended negative sample is constructed by combining the product image of one product with the product description text of another product.

[0131] In this embodiment, the original positive samples and / or original negative samples can be obtained as shown in Table 4 above. However, considering the limited number of samples, at least one of the following methods can be used to obtain more samples:

[0132] Option 1: For sample points of interest that lack images but are part of a brand chain, images of other points of interest under the same brand can be used as extended sample images for extended positive samples, while the original sample image text of the sample point of interest is still used as the extended sample image text for extended positive samples; in this case, the sample weight of the extended positive samples is low.

[0133] Option 2: Since the source of the original positive sample is comment data, the comment text in the comment data can be broken down to obtain multiple texts as extended sample image texts of the extended positive sample, and multiple comment images in the comment data can be used as extended sample images of the extended positive sample. At this time, the sample weight of the extended positive sample is low weight.

[0134] Option 3: For two original positive samples, such as the original sample image a and original sample image text a for interest point a, and the original sample image b and original sample image text b for interest point b, extended negative samples can be constructed as <original sample image a, original sample image text b> and <original sample image b, original sample image text a>. It should be noted that these two original positive samples are original positive samples of two different types of interest points, and the sample weights of the resulting extended negative samples are medium weights.

[0135] Option 4: In response to the fact that the source of the original positive sample is product data, which includes product image A and product description text A of product A, and product image B and product description text B of product B, extended negative samples can be constructed as <product image A, product description text B> and <product image B, product description text A>. The sample weight of the extended negative samples obtained at this time is medium weight.

[0136] In this embodiment, the original positive sample, the original negative sample, the expanded positive sample, and the expanded negative sample can all be samples from the first training dataset.

[0137] This disclosure also provides a training method for a joint model for dynamic image generation. Figure 3 A flowchart illustrating a training method for a joint model for dynamic image generation according to embodiments of the present disclosure is shown. Figure 3 As shown, the training method may include the following steps S301-S304:

[0138] In step S301, sample images and basic information of sample interest points are obtained;

[0139] In step S302, a predetermined image segmentation model is used to generate mask information for each mask corresponding to the sample image based on the sample image;

[0140] In step S303, the operation information of each mask corresponding to the sample image is obtained;

[0141] In step S304, the second large-scale language model is fine-tuned based on the second training dataset to obtain the trained second large-scale language model. The second training dataset includes basic information of multiple sample interest points, sample images and their corresponding mask information and operation information.

[0142] In one possible implementation, the training method for the joint dynamic image generation model is applicable to devices such as computers, computing devices, servers, and server clusters capable of training joint dynamic image generation models.

[0143] In one possible implementation, the image segmentation model and the dynamic image editing model in the joint dynamic image generation model do not need to be fine-tuned using interest point data. Instead, they can directly use existing image segmentation models and dynamic image editing models. For example, the image segmentation model can be SAM (Segment Anything Model), and the dynamic image editing model can be DragGAN (Generative Adversarial Network) model, which is a GAN-based algorithm for intuitive point-based image editing. Of course, in addition to SAM, the image segmentation model can also be other models capable of image segmentation, and in addition to DragGAN, it can also be other models capable of editing dynamic images.

[0144] In one possible implementation, a new large-scale language model, namely a second large-scale language model, can be selected and fine-tuned to learn how to manipulate different parts of an image, i.e., to learn to use the image segmentation model and the dynamic image editing model. During training, only the parameters of the second large-scale language model are fine-tuned, while the parameters of the image segmentation model and the dynamic image editing model are fixed.

[0145] In one possible implementation, the second training dataset for training the second large-scale language model includes multiple samples. Each sample includes a sample image of a sample interest point, basic sample information, mask information of each mask corresponding to the sample image, and operation information of each mask. The mask information of each mask corresponding to the sample image is generated by a predetermined image segmentation model, and this mask information can be as shown in Table 2 above. The operation information of each mask is manually annotated. For example, the image segmentation model can be used to segment and obtain masks of each part of the sample image. The trained annotator determines which operation to use to achieve the dynamic effect in the sample image based on the mask information of each mask, such as mask type, mask size, and mask position in the sample image. Here, the operation needs to be structured into operation information that can be processed by a computer, and this operation information can be as shown in Table 3 above.

[0146] It's important to note that the samples in this second training dataset also include image types. These image types can be real-world scenes (buildings), real-world scenes (people), real-world scenes (shops), design-icons, design-text, or combinations of these types. This image type is the output of the second large-scale language model. This output is not applied to subsequent animation generation, but this auxiliary task helps the second large-scale language model understand that different image types require different operations.

[0147] It should be noted that the second large-scale language model needs to learn from the above samples how to select appropriate elements for specific elements in the image to achieve different dynamic effects.

[0148] In one possible implementation, in order to ensure that each type of image can be adequately trained, sample images can be selected according to the image type, for example, one thousand sample images can be selected for each image type.

[0149] In one possible implementation, during the training process, the second large-scale language model can generate the following information from the aforementioned samples using a preset set of prompt templates.

[0150] "You are a dynamic image generation module that needs to generate dynamic effects for sample images of sample interest points. You can obtain the mask information of each mask in the sample image and generate the dynamic operation suitable for each mask."

[0151] Basic information about the sample interest points: ${sample interest point name} is a ${sample interest point type}. Based on the image segmentation results, this image contains ${mask number} masks, with specific information including [{"serial number":${serial number},"mask size":${mask size},"mask type":${mask type},"mask location":${mask location}},...]

[0152] Output image type:

[0153] ${Image Type}

[0154] Output operation information:

[0155] [{"Serial Number":${Serial Number},"Operation Type':${Operation Type}},...]”

[0156] The difference between the operational information output by the second large-scale language model and the operational information corresponding to the sample image is calculated. The model parameters of the second large-scale language model are continuously adjusted until the difference is minimized. For example, the training objective can adopt an autoregressive approach for large-scale language models, calculating the distribution difference between each output operational information and the operational information corresponding to the sample image, using a negative log-maximum likelihood loss function to calculate the loss, and continuously adjusting the model parameters of the second large-scale language model until the loss is minimized.

[0157] It should be noted that the second large-scale language model can also be jointly trained with the image segmentation model and the dynamic image editing model. While fine-tuning the second large-scale language model, the image segmentation model and the dynamic image editing model are also trained. In this case, the training dataset can include basic information about multiple sample interest points, static images of samples, and their corresponding dynamic images. Of course, besides the training methods mentioned above, other training methods can be used to obtain the joint dynamic image generation model, which will not be elaborated upon here.

[0158] Figure 4 A structural block diagram of a point-of-interest image generation apparatus according to an embodiment of the present disclosure is shown. This apparatus can be implemented as part or all of an electronic device through software, hardware, or a combination of both. Figure 4 As shown, the point of interest image generation device includes:

[0159] Information acquisition module 401 is configured to acquire image description information of target points of interest, wherein the image description information includes image description text;

[0160] The static image generation module 402 is configured to input the image description information into a pre-trained static image generation joint model, and execute the pre-trained static image generation joint model to generate a static image of the target interest point. The static image generation joint model includes a first large-scale language model and a text-to-image model. The first large-scale language model is used to generate text vectors based on the image description information, and the text-to-image model is used to generate a static image of the target interest point based on the text vectors.

[0161] In one possible implementation, the image description information further includes a low-quality image for describing the target point of interest; the still image generation module 402 is configured to:

[0162] The image description text and the low-quality image are input into the first large-scale language model, the first large-scale language model is executed, and the text vector output by the first large-scale language model is obtained.

[0163] The text vector and the low-quality image are input into the text-to-image model, and the text-to-image model is executed to obtain a static image of the target interest point output by the text-to-image model.

[0164] In one possible implementation, the first large-scale language model corresponds to multiple prompt templates, and the static image generation module 402 is configured to:

[0165] The image description information is input into a pre-trained static image generation joint model. The pre-trained static image generation joint model is executed multiple times to generate static images of the target interest points, resulting in multiple different static images.

[0166] The device further includes:

[0167] The filtering module is configured to filter at least one target still image from multiple different still images.

[0168] In one possible implementation, the filtering module is configured as follows:

[0169] For each static image, a corresponding prompt is determined. The prompt is generated by the first large-scale language model based on the image description information and the prompt template.

[0170] The static image is input into the first large-scale language model to obtain the static image text output by the first large-scale language model;

[0171] Calculate the first similarity between the static image text and the corresponding prompt of the static image;

[0172] The quality score of the static image is determined based on the first similarity.

[0173] Based on the quality scores of the multiple different still images, at least one target still image is selected from the multiple different still images.

[0174] In one possible implementation, if the image description information further includes a low-quality image, the apparatus further includes:

[0175] The calculation module is configured to input the low-quality image into the first large-scale language model to obtain low-quality image text output by the first large-scale language model; and to calculate a second similarity between the static image text of the static image and the low-quality image text.

[0176] The part of the filtering module that determines the quality score of the static image based on the first similarity is configured as follows:

[0177] The quality score of the static image is determined based on the first similarity and the second similarity.

[0178] Based on the quality scores of the multiple different still images, at least one target still image is selected from the multiple different still images.

[0179] In one possible implementation, the device further includes:

[0180] The basic information acquisition module is configured to acquire basic information about the target point of interest;

[0181] A dynamic image generation module is configured to, for any static image, use a pre-trained joint dynamic image generation model to generate a dynamic image of the target interest point based on the basic information of the target interest point and the static image. The joint dynamic image generation model includes a second large-scale language model, an image segmentation model, and a dynamic image editing model. The image segmentation model is used to segment the static image of the target interest point into multiple masks and generate mask information for each mask. The second large-scale language model is used to generate image editing instructions based on the mask information of each mask and the basic information of the target interest point. The dynamic image editing model is used to edit the static image according to the image editing instructions to obtain the dynamic image of the target interest point.

[0182] Figure 5 A structural block diagram of a training apparatus for a joint model of static image generation according to an embodiment of the present disclosure is shown. This apparatus can be implemented as part or all of an electronic device through software, hardware, or a combination of both.

[0183] like Figure 5 As shown, the training device for the static image generation joint model includes:

[0184] The training data acquisition module 501 is configured to acquire a first training dataset, which includes multiple positive samples and / or multiple negative samples, wherein both the positive samples and the negative samples include sample images of sample interest points and sample image text.

[0185] The static model training module 502 is configured to train the initial static image generation joint model using the first training dataset to obtain a trained static image generation joint model, wherein the static image generation joint model includes a first large-scale language model and a text-to-image model.

[0186] Wherein, the loss function of the static image generation joint model is L = αL1 + (1-α)L2, where L1 is the difference between the noise predicted by the text-to-image model at each time step during the denoising process and the Gaussian noise added during the diffusion process, L2 is the difference between the predicted image text output by the first large-scale language model and the sample image text, α is a predetermined parameter value, and the text vector generated by the first large-scale language model is the image generation condition of the text-to-image model.

[0187] In one possible implementation, the first training dataset includes original positive samples, original negative samples, expanded positive samples, and / or expanded negative samples, and the training data acquisition module 501 is configured to:

[0188] Obtain original positive samples and / or original negative samples, wherein both the original positive samples and the original negative samples include the original sample image of the sample interest point and the original sample image text;

[0189] The original positive samples are augmented using at least one of the following steps to obtain expanded positive samples and / or expanded negative samples:

[0190] In response to the absence of the original sample image, the interest point images of other interest points under the same brand as the sample interest point are used as the extended sample images of the extended positive samples, and the original sample image text of the sample interest point is used as the extended sample image text of the extended positive samples.

[0191] Since the source of the original positive samples is comment data, the comment data is broken down to obtain multiple extended positive samples;

[0192] For two original positive samples, an expanded negative sample is constructed by combining the original sample image from one original positive sample with the original sample image text from the other original positive sample.

[0193] In response to the fact that the source of the original positive sample is product data, an extended negative sample is constructed by combining the product image of one product with the product description text of another product.

[0194] Figure 6 A structural block diagram of a training apparatus for a joint model of dynamic image generation according to an embodiment of the present disclosure is shown. This apparatus can be implemented as part or all of an electronic device through software, hardware, or a combination of both.

[0195] like Figure 6 As shown, the training device for the joint dynamic image generation model includes:

[0196] The sample information acquisition module 601 is configured to acquire sample images and basic sample information of sample interest points;

[0197] The segmentation module 602 is configured to use a predetermined image segmentation model to generate mask information for each mask corresponding to the sample image based on the sample image;

[0198] The operation information acquisition module 603 is configured to acquire the operation information of each mask corresponding to the sample image;

[0199] The fine-tuning module 604 is configured to fine-tune the second large-scale language model based on the second training dataset to obtain the trained second large-scale language model. The second training dataset includes sample basic information of multiple sample interest points, sample images and their corresponding mask information and operation information.

[0200] The trained dynamic image generation joint model includes a trained second large-scale language model, a predetermined image segmentation model, and a predetermined dynamic image editing model. The trained second large-scale language model is used to generate image editing instructions based on the mask information and basic information of each mask, and the predetermined dynamic image editing model is used to edit the image according to the image editing instructions to generate dynamic images.

[0201] The technical terms and features mentioned in this device implementation are the same or similar. For the explanation and description of the technical terms and features involved in this device, please refer to the explanation of the above method implementation, which will not be repeated here.

[0202] This disclosure also discloses an electronic device. Figure 7 A structural block diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0203] like Figure 7 As shown, the electronic device 700 includes a memory 701 and a processor 702, wherein the memory 701 is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor 702 to implement the method according to an embodiment of the present disclosure.

[0204] Figure 8 A schematic diagram of the structure of a computer system suitable for implementing the method according to embodiments of the present disclosure is shown.

[0205] like Figure 8 As shown, the computer system 800 includes a processing unit 801, which can execute various processes described in the above embodiments according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage portion 808 into a random access memory (RAM) 803. The RAM 803 also stores various programs and data required for the operation of the computer system 800. The processing unit 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0206] The following components are connected to I / O interface 805: an input section 806 including a keyboard, mouse, etc.; an output section 807 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to I / O interface 805 as needed. A removable medium 811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 810 as needed so that computer programs read from it can be installed into storage section 808 as needed. The processing unit 801 can be implemented as a CPU, GPU, TPU, FPGA, NPU, etc.

[0207] In particular, according to embodiments of this disclosure, the methods described above can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising computer instructions that, when executed by a processor, implement the steps of the methods described above. In such embodiments, the computer program product can be downloaded and installed from a network via communication section 809, and / or installed from removable media 811.

[0208] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0209] The units or modules described in the embodiments of this disclosure can be implemented in software or programmable hardware. The described units or modules can also be located in a processor, and the names of these units or modules do not necessarily constitute a limitation on the unit or module itself.

[0210] In another aspect, this disclosure also provides a computer-readable storage medium, which may be a computer-readable storage medium included in the electronic device or computer system described above; or it may be a standalone computer-readable storage medium not assembled into a device. The computer-readable storage medium stores one or more programs, which are used by one or more processors to perform the methods described in this disclosure.

[0211] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

Claims

1. A method for generating point of interest (POI) images, comprising: Obtain image description information of the target point of interest, the image description information including image description text and a low-quality image used to describe the target point of interest; The image description information is input into a pre-trained static image generation joint model, and the pre-trained static image generation joint model is executed to generate a static image of the target interest point. The static image generation joint model includes a first large-scale language model and a text-to-image model. The first large-scale language model is used to generate text vectors based on the image description information, and the text-to-image model is used to generate a static image of the target interest point based on the text vectors. The first large-scale language model corresponds to multiple prompt templates. Specifically, generating a static image of the target interest point by executing the pre-trained joint static image generation model includes: Based on the multiple prompt templates, the pre-trained static image generation joint model is executed multiple times to generate multiple different static images of the target interest point; For each static image, a corresponding prompt is determined, wherein the prompt is generated by the first large-scale language model based on the image description information and the prompt template; The static image is input into the first large-scale language model to obtain the static image text output by the first large-scale language model; Calculate the first similarity between the static image text and the corresponding prompt of the static image; The low-quality image is input into the first large-scale language model to obtain the low-quality image text output by the first large-scale language model. Calculate the second similarity between the static image text and the low-quality image text of the static image; The quality score of the static image is determined based on the first similarity and the second similarity. Based on the quality scores of the multiple different still images, at least one target still image is selected from the multiple different still images.

2. The method according to claim 1, wherein, The step of inputting the image description information into a pre-trained joint static image generation model and executing the pre-trained joint static image generation model to generate a static image of the target interest point includes: The image description text and the low-quality image are input into the first large-scale language model, the first large-scale language model is executed, and the text vector output by the first large-scale language model is obtained. The text vector and the low-quality image are input into the text-to-image model, and the text-to-image model is executed to obtain a static image of the target interest point output by the text-to-image model.

3. The method according to claim 1 or 2, wherein, The method further includes: Obtain basic information about the target point of interest; For any static image, a pre-trained joint dynamic image generation model is used to generate a dynamic image of the target interest point based on the basic information of the target interest point and the static image. The joint dynamic image generation model includes a second large-scale language model, an image segmentation model, and a dynamic image editing model. The image segmentation model is used to segment the static image of the target interest point into multiple masks and generate mask information for each mask. The second large-scale language model is used to generate image editing instructions based on the mask information of each mask and the basic information of the target interest point. The dynamic image editing model is used to edit the static image according to the image editing instructions to obtain the dynamic image of the target interest point.

4. A training method for a joint model for static image generation, comprising: Obtain a first training dataset, which includes multiple positive samples and / or multiple negative samples, wherein both the positive samples and the negative samples include sample images of sample interest points and sample image text. The initial static image generation joint model is trained using the first training dataset to obtain a trained static image generation joint model, which includes a first large-scale language model and a text-to-image model. Wherein, the loss function of the static image generation joint model is L = αL1 + (1-α)L2, where L1 is the difference between the noise predicted by the text-to-image model at each time step during the denoising process and the Gaussian noise added during the diffusion process, L2 is the difference between the predicted image text output by the first large-scale language model and the sample image text, α is a predetermined parameter value, and the text vector generated by the first large-scale language model is the image generation condition of the text-to-image model.

5. The method according to claim 4, wherein, The first training dataset includes original positive samples, original negative samples, expanded positive samples, and / or expanded negative samples. Obtaining the first training dataset includes: Obtain original positive samples and / or original negative samples, wherein both the original positive samples and the original negative samples include the original sample image of the sample interest point and the original sample image text; The original positive samples are augmented using at least one of the following steps to obtain expanded positive samples and / or expanded negative samples: In response to the absence of the original sample image, the interest point images of other interest points under the same brand as the sample interest point are used as the extended sample images of the extended positive samples, and the original sample image text of the sample interest point is used as the extended sample image text of the extended positive samples. Since the source of the original positive samples is comment data, the comment data is broken down to obtain multiple extended positive samples; For two original positive samples, an expanded negative sample is constructed by combining the original sample image from one original positive sample with the original sample image text from the other original positive sample. In response to the fact that the source of the original positive sample is product data, an extended negative sample is constructed by combining the product image of one product with the product description text of another product.

6. An interest point image generation apparatus, comprising: The information acquisition module is configured to acquire image description information of a target point of interest, the image description information including image description text and a low-quality image used to describe the target point of interest; The static image generation module is configured to use a pre-trained static image generation joint model to generate a static image of the target interest point based on the image description information. The static image generation joint model includes a first large-scale language model and a text-to-image model. The large-scale language model is used to generate text vectors based on the image description information, and the text-to-image model is used to generate a static image of the target interest point based on the text vectors. The first large-scale language model corresponds to multiple prompt templates. Specifically, generating a static image of the target interest point by executing the pre-trained joint static image generation model includes: Based on the multiple prompt templates, the pre-trained static image generation joint model is executed multiple times to generate multiple different static images of the target interest point; For each static image, a corresponding prompt is determined, wherein the prompt is generated by the first large-scale language model based on the image description information and the prompt template; The static image is input into the first large-scale language model to obtain the static image text output by the first large-scale language model; Calculate the first similarity between the static image text and the corresponding prompt of the static image; The low-quality image is input into the first large-scale language model to obtain the low-quality image text output by the first large-scale language model. Calculate the second similarity between the static image text and the low-quality image text of the static image; The quality score of the static image is determined based on the first similarity and the second similarity. Based on the quality scores of the multiple different still images, at least one target still image is selected from the multiple different still images.

7. An electronic device comprising a memory and a processor; wherein, The memory is used to store one or more computer instructions, which are executed by the processor to implement the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having computer instructions stored thereon, wherein, When executed by a processor, the computer instructions implement the method described in any one of claims 1 to 5.

Images