Commodity introduction information generation method, system and device and storage medium

Abstract

The invention discloses a commodity introduction information generation method, system and device and a storage medium, and is applied to the technical field of data processing, and the method comprises the steps: receiving a commodity image, and determining a prompt text used for guiding a multi-modal model to describe the commodity image; after the commodity image and the prompt text are input into the multi-modal model, text information which is output by the multi-modal model and used for describing the commodity image is obtained; and inputting the text information and the commodity image into an image generation model to obtain a plurality of expanded images which are output by the image generation model and are related to the commodity image, and taking each expanded image and the text information as generated commodity introduction information corresponding to the commodity image. By applying the scheme of the invention, the generation efficiency and quality of the commodity introduction information are effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method, system, device and storage medium for generating product introduction information. Background Art

[0002] At present, more and more people will buy goods online. Whether it is the product details page of the e-commerce platform, or the promotional page of social software or public account, its production is completed by artists and editors. The artists will put the pictures they take into the corresponding templates, and the editors are responsible for writing the corresponding texts. The work content is simple and repetitive, but it takes a long time and has high labor costs. In addition, the pace of new products is very fast now, especially for companies such as fast fashion clothing, which have to update dozens or hundreds of products every week. The content design of the product details page is an important factor in the conversion rate of the product. If it is not well made or updated slowly, the expected publicity and display effect will not be achieved, and the merchant's stocking cycle will be extended. If you want to meet a faster update speed, the labor cost will be higher.

[0003] At present, there are also solutions that use artificial intelligence algorithms to improve the quality and efficiency of generating product introduction information. Usually, the product data of the product object is entered in the input page on the operation interface. When the product data is an image, an artificial intelligence recognition algorithm is used to identify the image content and generate the product introduction information of the product object. For example, through the artificial intelligence recognition algorithm such as ANN (Artificial Neural Network, artificial neural network) for image recognition, the image content is recognized as pants, and the generated product introduction information is "pants". When the product data is text, the artificial intelligence recognition algorithm can also be used to directly extract relevant information to generate the product introduction information of the product object.

[0004] Such a solution processes text input and image input separately, which easily leads to incomplete output information and is highly dependent on the quality of input data. In addition, there is also a problem of low correlation between the description text and the image in the product introduction information. Moreover, such a solution only extracts and generates recognized text information, and cannot supplement more description information, which is not conducive to enriching the user's browsing experience.

[0005] In summary, how to improve the efficiency and quality of generating product introduction information is a technical problem that technical personnel in this field urgently need to solve. Summary of the invention

[0006] The purpose of the present invention is to provide a method, system, device and storage medium for generating product introduction information, so as to effectively improve the generation efficiency and quality of product introduction information.

[0007] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0008] In a first aspect, the present invention provides a method for generating product introduction information, comprising:

[0009] Receiving a product image, and determining a prompt text for guiding the multimodal model to describe the product image;

[0010] After the product image and the prompt text are input into the multimodal model, text information output by the multimodal model for describing the product image is obtained;

[0011] The text information and the product image are input into an image generation model to obtain a plurality of extended images related to the product image output by the image generation model, and each of the extended images and the text information is used as the generated product introduction information corresponding to the product image.

[0012] On the other hand, it also includes:

[0013] Receiving auxiliary description text for the product image;

[0014] The auxiliary explanation text is added to the prompt text.

[0015] On the other hand, the multimodal model is a pre-trained multimodal model based on a large-scale language and visual assistant architecture, and the multimodal model includes a visual encoder, a large-scale language model, and a multi-layer perceptron for connecting the visual encoder and the large-scale language model.

[0016] On the other hand, determining a prompt text for guiding the multimodal model to describe the product image includes:

[0017] According to the commodity type in the commodity image, a prompt text corresponding to the commodity type is selected from a prompt text library as the determined prompt text for guiding the multimodal model to describe the commodity image.

[0018] On the other hand, it also includes:

[0019] For each commodity type, K candidate prompt texts corresponding to the commodity type are pre-constructed;

[0020] Wherein, K is a preset positive integer not less than 2;

[0021] After inputting a test product image of the product type and an i-th candidate prompt text of the product type into the multimodal model, obtaining i-th text information output by the multimodal model for describing the test product image;

[0022] Wherein, i is a positive integer and i ranges from 1 to K in sequence;

[0023] After obtaining the first to Kth text information for describing the test product image, sequentially determining the degree of similarity between each of the first to Kth text information and the reference text information;

[0024] Selecting the maximum similarity degree from the K similarities, and using the corresponding candidate prompt text as the optimal prompt text corresponding to the product type;

[0025] The optimal prompt text is placed in the prompt text library, and a corresponding relationship between the optimal prompt text and the commodity type is established, so that the optimal prompt text is used as the prompt text corresponding to the commodity type in the prompt text library.

[0026] On the other hand, sequentially determining the similarity between each of the first text information to the Kth text information and the reference text information includes:

[0027] The first to the Kth text information and the reference text information are all converted into vectors, and the similarity between the first to the Kth text information and the reference text information is determined in turn by calculating the cosine similarity between the vectors.

[0028] On the other hand, it also includes:

[0029] When a prompt text adjustment instruction is received, a prompt text corresponding to the product type is selected from the backup text library as the prompt text determined to guide the multimodal model to describe the product image, and the operation of inputting the product image and the prompt text into the multimodal model is returned to regenerate the product introduction information corresponding to the product image.

[0030] In a second aspect, the present invention provides a system for generating product introduction information, comprising:

[0031] A prompt text determination module, used for receiving a product image and determining a prompt text for guiding the multimodal model to describe the product image;

[0032] A text information generating module, configured to input the product image and the prompt text into the multimodal model, and obtain text information output by the multimodal model for describing the product image;

[0033] A product introduction information determination module is used to input the text information and the product image into an image generation model, obtain multiple extended images related to the product image output by the image generation model, and use each of the extended images and the text information as the generated product introduction information corresponding to the product image.

[0034] In a third aspect, the present invention provides a device for generating product introduction information, comprising:

[0035] Memory for storing computer programs;

[0036] A processor is used to execute the computer program to implement the steps of the method for generating product introduction information as described above.

[0037] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method for generating product introduction information as described above are implemented.

[0038] By applying the technical solution provided by the embodiment of the present invention, considering that the product introduction information is generated based on the multimodal model and the image generation model, the generation efficiency and quality of the product introduction information can be effectively improved. Specifically, the multimodal model supports the input of text and images. Therefore, after receiving the product image, it is also necessary to determine the prompt text used to guide the multimodal model to describe the product image. After that, the product image and the prompt text can be input into the multimodal model, and the multimodal model will describe the product image according to the requirements of the prompt text, so that the text information output by the multimodal model for describing the product image can be obtained. The text information output by the multimodal model will provide an extended supplementary description of the product image, which is also conducive to enriching the user's browsing experience. After that, the text information and the product image are input into the image generation model, and the image generation model can expand the image. Moreover, since the text information describes the product image, the image generation model can understand the product image more accurately, which is conducive to ensuring the quality of the obtained extended image, that is, the obtained multiple extended images can have a greater correlation with the text information. And it can be seen that the present application scheme does not require manual generation of product introduction information, so efficiency can be guaranteed.

[0039] In summary, the present application scheme effectively improves the generation efficiency and quality of product introduction information. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0041] Figure 1 This is a flowchart of a method for generating product introduction information provided by a specific implementation method of the present invention;

[0042] Figure 2 A schematic diagram of the structure of a system for generating product introduction information provided by a specific implementation method of the present invention.

[0043] Figure 3 A schematic diagram of the structure of a device for generating product introduction information provided by a specific implementation method of the present invention;

[0044] Figure 4 The present invention is a schematic diagram of the structure of a computer-readable storage medium. DETAILED DESCRIPTION

[0045] The core of the present invention is to provide a method for generating commodity introduction information, which effectively improves the generation efficiency and quality of commodity introduction information.

[0046] In order to enable those skilled in the art to better understand the scheme of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0047] Please refer to Figure 1 , Figure 1 The present invention provides a flowchart of a method for generating product introduction information in accordance with a specific embodiment of the present invention. The method for generating product introduction information may include the following steps:

[0048] Step S101: receiving a product image, and determining a prompt text for guiding a multimodal model to describe the product image.

[0049] Product images can be input by users, for example, the user inputs a product image of a shirt. Prompt text can also be called prompt, which can be composed of one or several words, or one or more sentences. In the era of artificial intelligence, especially AGI (artificial general intelligence), prompt text plays a vital role. It is a bridge for users to interact with AI (Artificial Intelligence) models, and can guide the model to produce specific outputs.

[0050] The specific content of the prompt text can be set and adjusted according to actual needs. For example, considering that the present application scheme can introduce products, the prompt text can be, for example, "Please generate a description of the product information in the picture."

[0051] Step S102: After the product image and the prompt text are input into the multimodal model, the text information output by the multimodal model for describing the product image is obtained.

[0052] After obtaining the product image and the prompt text, the product image and the prompt text can be input into the multimodal model. The multimodal model can accept text and image input, and since the prompt text is used to guide the multimodal model to describe the product image, the multimodal model will output text information used to describe the product image according to the requirements of the prompt text.

[0053] The specific form of the multimodal model can be set and adjusted according to actual needs. For example, in a specific embodiment of the present invention, the multimodal model can be a pre-trained multimodal model based on a large language and vision assistant architecture. That is, the multimodal model is a multimodal model based on the LLaVA (Large Language and Vision Assistant) architecture. For example, in a specific scenario, LLaVA-v1.6 can be used as the multimodal model of the present application. The multimodal model of the LLaVA architecture includes a visual encoder, an LLM (LLM Large Language Model) and an MLP (Multilayer Perceptron) for connecting the visual encoder and the large-scale language model.

[0054] The MLP layer is responsible for aligning or converting visual and language features so that they can be operated in a unified space. Since the features extracted from the visual encoder are not in the same semantic expression space as the LLM model, it is necessary to align the image token embedding to the semantic expression space of the text word embedding through pre-training. Therefore, in practical applications, when training a multimodal model, the weight parameters of the visual encoder and the LLM model must first be frozen, and the MLP layer must be trained and updated, which is the feature alignment pre-training part of the multimodal model training process. Then the end-to-end training part of the multimodal model training process is carried out, which means that for tasks such as semantic question answering and multi-round dialogue, the weights of the MLP layer and the LLM model are updated while the weights of the visual encoder are still frozen.

[0055] In the above example, LLaVA-v1.6 is used as the multimodal model of this application because it has the advantage of low-cost training and can be adapted and applied in different languages ​​and cultural backgrounds, greatly expanding the practicality and universality of AI. Compared with traditional deep learning models that usually require a lot of computing resources and time for training, LLaVA-v1.6 can use efficient training methods to achieve rapid training and deployment at a lower cost. In terms of performance, LLaVA-v1.6 has also demonstrated strong strength, showing higher accuracy and robustness when handling complex tasks, which is conducive to the application scheme to obtain higher quality and more comprehensive product image and text information.

[0056] Step S103: input text information and product image into the image generation model, obtain multiple extended images related to the product image output by the image generation model, and use each extended image and text information as the generated product introduction information corresponding to the product image.

[0057] The text information output by the multimodal model is a description of the product image. The text information and the product image are input into the image generation model, so that the image generation model can understand the product image more accurately, which is conducive to ensuring the quality of the obtained extended image, that is, the obtained multiple extended images can have a greater correlation with the text information.

[0058] In addition, when inputting text information and product images into the image generation model, the number of required extended images can also be specified. For example, after inputting text information and product images into the image generation model, if "please generate 10 related images" is also input, the image generation model will output 10 extended images related to the product image. Of course, in actual applications, the number of specific extended images does not need to be specified, and the image generation model will usually output a number of extended images, or output a default number of extended images, which can be set according to actual needs.

[0059] There can be many specific forms of image generation models, as long as they can effectively expand images based on text information and product images. For example, the Stable diffusion model is specifically used in one occasion. The Stablediffusion model is a diffusion model based on stable distribution, which simulates the diffusion process by gradually converting input data into output data. Compared with traditional diffusion models, the Stable diffusion model has stronger stability and higher accuracy, and can be well applied to occasions such as image processing, natural language processing and sound recognition, and can be used to process large-scale data and complex data structures. In addition, for image processing, the Stable diffusion model can also realize image denoising, image enhancement, image restoration and other functions, thereby improving the visual effect and quality of the obtained expanded image. For the processing of language text, the Stable diffusion model can realize information extraction, thereby better mining and processing text data.

[0060] After obtaining each extended image and text information, these contents can be used as the generated product introduction information corresponding to the product image, and then fed back to the user.

[0061] In a specific embodiment of the present invention, it may also include:

[0062] Receive auxiliary description text for the product image;

[0063] Adds help text to the tooltip.

[0064] This implementation method takes into account that the role of the prompt text is to guide the multimodal model to describe the product image, and the product type is different in different situations, so the prompt text usually uses a more general expression to ensure that it has a certain degree of versatility. For example, in the above example, the prompt text can be "Please generate the product information description in the picture", rather than "Please generate the product information description of the men's linen shirt in the picture", because for the latter, only the men's linen shirt can use this prompt text, and other products cannot use this prompt text.

[0065] On the other hand, the more specific the prompt text is, the more conducive it is to the multimodal model to output more accurate and high-quality text information. Therefore, this implementation method takes into account that, for specific products, users can be allowed to provide auxiliary descriptions of product images, that is, receive auxiliary description text for product images, and then add the auxiliary description text to the prompt text.

[0066] Still taking a shirt as an example, for example, the user inputs a product image of a shirt and also inputs a sentence as the auxiliary description text of the product image: "The product in the picture is a men's shirt made of linen". After obtaining the auxiliary description text, the auxiliary description text can be added to the prompt text "Please generate information description of the product in the picture", so that after adding, the prompt text becomes "The product in the picture is a men's shirt made of linen; please generate information description of the product in the picture". That is, in this implementation method, the auxiliary description text for the specific product is combined with the universal default prompt text to obtain the prompt text required to be input into the multimodal model.

[0067] In a specific embodiment of the present invention, determining a prompt text for guiding the multimodal model to describe a product image includes:

[0068] According to the commodity type in the commodity image, a prompt text corresponding to the commodity type is selected from the prompt text library as the determined prompt text for guiding the multimodal model to describe the commodity image.

[0069] In actual applications, default prompt texts can be used for various product images. For example, in the above example, the default prompt text is set to "Please generate a description of the product information in the image", which is very versatile. The advantage of this is that there is no need to pre-set too many prompt texts.

[0070] This implementation further takes into account that, in actual applications, for a certain product, when different prompt texts are used, the quality of the text information output by the multimodal model is different, that is, the prompt texts also have advantages and disadvantages. Therefore, the most ideal way is to preset an optimal prompt text suitable for different specific products. However, this is a lot of work, especially in situations where products are updated quickly and there are many styles.

[0071] In this regard, this implementation method takes into account that a prompt text suitable for the product type can be set according to the different product types. Since the product types are limited, such an implementation method will not require too much workload and can effectively improve the quality of the text information output by the multimodal model.

[0072] Specifically, after obtaining the product image, the prompt text corresponding to the product type can be selected from the prompt text library according to the product type in the product image. It can be understood that the prompt text corresponding to the product type in the prompt text library is a pre-set prompt text that is very suitable for the product type.

[0073] In addition, it should be pointed out that there are many ways to determine the type of product in a product image. For example, the type of product in a product image can be identified through a simple recognition model. For another example, the user can independently select the type of product in the product image on the operation interface.

[0074] In a specific embodiment of the present invention, it may also include:

[0075] For each product type, K candidate prompt texts corresponding to the product type are pre-constructed;

[0076] Wherein, K is a preset positive integer not less than 2;

[0077] After inputting the test product image of the product type and the i-th candidate prompt text of the product type into the multimodal model, the i-th text information output by the multimodal model for describing the test product image is obtained;

[0078] Wherein, i is a positive integer and i ranges from 1 to K in sequence;

[0079] After obtaining the first to Kth text information for describing the test product image, the similarity between the first to Kth text information and the reference text information is determined in sequence;

[0080] Select the maximum similarity from the K similarities, and use the corresponding candidate prompt text as the optimal prompt text corresponding to the product type;

[0081] The optimal prompt text is placed in a prompt text library, and a corresponding relationship between the optimal prompt text and the commodity type is established, so that the optimal prompt text is used as the prompt text corresponding to the commodity type in the prompt text library.

[0082] As described above, the prompt text corresponding to the product type in the prompt text library is a pre-set prompt text that is very suitable for the product type. This implementation method discloses a specific implementation method for obtaining the prompt text in the prompt text library, which can effectively obtain the corresponding prompt text suitable for different product types.

[0083] Specifically, for each product type, K candidate prompt texts corresponding to the product type need to be pre-constructed. It can be understood that the contents of the K candidate prompt texts will guide the multimodal model to describe the test product image of the product type, but the K candidate prompt texts use different expressions.

[0084] For any product type, when setting K candidate prompt texts corresponding to the product type, there can be multiple specific settings. For example, some candidate prompt texts can use the same expression method, but use different separators. Some candidate prompt texts will require the multimodal model to perform structured output, while some candidate prompt texts do not require the multimodal model to perform structured output. Some candidate prompt texts can give examples, while some candidate prompt texts may not give examples. Setting K candidate prompt texts in this way is conducive to ensuring that the K candidate prompt texts have diverse forms and ultimately screening out the best candidate prompt texts.

[0085] For delimiters, choosing appropriate delimiters can clearly indicate the different parts of the input. For structured output, in some cases, users want the multimodal model to give some structured output, not just continuous text. For example, it is hoped that the multimodal model can return content organized in a certain format, such as JSON, HTML, etc. This structure makes it convenient for users to process in the code. For the provision of examples, it means providing one or two reference examples to the multimodal model in the candidate prompt text, so that the multimodal model can understand the user's requirements and expected output style, which can guide the multimodal model to produce better reasoning results. In addition, different expressions will also affect the output of the multimodal model.

[0086] For each product type, it is necessary to input a test product image of the product type and the i-th candidate prompt text of the product type into the multimodal model, so as to obtain the i-th text information output by the multimodal model for describing the test product image.

[0087] Taking the product type "fruit" as an example, since i is a positive integer and i takes values ​​from 1 to K in sequence, it is first necessary to input the test product image of the fruit and the first candidate prompt text of the product type into the multimodal model to obtain the first text information used to describe the test product image, and then input the test product image of the fruit and the second candidate prompt text of the product type into the multimodal model to obtain the second text information used to describe the test product image, and so on, the first to Kth text information used to describe the test product image of the fruit can be obtained.

[0088] The reference text information can be set by the user, and represents the description information that the user expects to obtain for the test product image. For the first to Kth text information obtained, it is necessary to determine the degree of similarity between the first to Kth text information and the reference text information in turn. It can be understood that the more similar the first to Kth text information is to the reference text information, the higher the quality of the text information is, and the more it meets the user's expectations.

[0089] In addition, there may be many specific implementations of determining the degree of similarity. For example, in a specific implementation of the present invention, sequentially determining the degree of similarity between each of the first text information to the Kth text information and the reference text information may specifically include:

[0090] The first to Kth text information and the reference text information are all converted into vectors, and the similarity between the first to Kth text information and the reference text information is determined in turn by calculating the cosine similarity between the vectors.

[0091] This implementation method takes into account that the similarity between text information is not convenient to compare directly, but the text can be converted into a vector first, and then the similarity between the vectors can be easily determined. For this purpose, the first text information to the Kth text information and the reference text information all need to be converted into vectors. For ease of description, after the first text information to the Kth text information are converted into vectors, they are referred to as the first vector to the Kth vector, and after the reference text information is converted into a vector, it is referred to as the reference vector. Taking the first vector as an example, the cosine similarity between the first vector and the reference vector can be calculated. The larger the value of the cosine similarity, the lower the similarity between the first vector and the reference vector. Conversely, the smaller the value of the cosine similarity, the higher the similarity between the first vector and the reference vector. In practical applications, after obtaining the cosine similarity between the first vector and the reference vector, it can be converted into a corresponding similarity score value, so that when the similarity between the first vector and the reference vector is higher, the similarity score value is higher, and the similarity score value is used as the obtained similarity between the first vector and the reference vector, that is, the similarity between the first text information and the reference text information is obtained. Furthermore, in other implementations, in addition to the cosine similarity, other measurement methods may be set to determine the similarity between vectors, and may be set according to actual needs.

[0092] After determining the similarity between the first to the Kth text information and the reference text information, the maximum similarity can be selected from the K similarities, that is, the text information most similar to the reference text information is selected from the first to the Kth text information, and the corresponding candidate prompt text is used as the optimal prompt text corresponding to the commodity type. For example, among the first to the Kth text information, the fifth text information has the highest similarity with the reference text information, and the candidate prompt text corresponding to the fifth text information, that is, the fifth candidate prompt text, can be used as the optimal prompt text corresponding to the commodity type of fruit.

[0093] Finally, the obtained optimal prompt text is placed in the prompt text library, and a corresponding relationship between the optimal prompt text and the commodity type is established, so that the optimal prompt text can be used as the prompt text corresponding to the commodity type in the prompt text library. That is to say, in the subsequent application process, for the commodity type of fruit, the prompt text corresponding to the commodity type selected from the prompt text library is the optimal prompt text corresponding to the commodity type of fruit pre-placed in the prompt text library through the above process.

[0094] In a specific embodiment of the present invention, it also includes:

[0095] When a prompt text adjustment instruction is received, a prompt text corresponding to the product type is selected from the backup text library as the prompt text determined to guide the multimodal model to describe the product image, and the operation of inputting the product image and the prompt text into the multimodal model is returned to regenerate the product introduction information corresponding to the product image.

[0096] This implementation takes into account that, although according to the above principles, for different product types, the optimal prompt text that is very suitable for the product type is pre-set, in actual applications, since the forms of product images of the same product type are very diverse, it is still possible that the text information output by the multimodal model may be of poor quality and not meet user expectations.

[0097] In this regard, in this implementation, the user is allowed to adjust the prompt text. For example, when the user clicks on the option to change the prompt text on the operation interface, the back end can receive the prompt text adjustment instruction, and then select the prompt text corresponding to the product type from the backup text library as the updated prompt text, and then return to execute the operation of step S102, so as to finally regenerate the product introduction information corresponding to the product image based on the updated prompt text.

[0098] It is understandable that this implementation requires setting up a backup text library, which contains prompt texts corresponding to different commodity types. There are many ways to construct the backup text library. For example, in the above implementation, for any commodity type, the text information most similar to the reference text information is selected from the first to the Kth text information, and the corresponding candidate prompt text is used as the optimal prompt text corresponding to the commodity type. Then, the text information with the second highest degree of similarity to the reference text information can be selected, and the corresponding candidate prompt text can be used as the prompt text corresponding to the commodity type stored in the backup text library, so that the construction of the backup text library can be easily completed.

[0099] By applying the technical solution provided by the embodiment of the present invention, considering that the product introduction information is generated based on the multimodal model and the image generation model, the generation efficiency and quality of the product introduction information can be effectively improved. Specifically, the multimodal model supports the input of text and images. Therefore, after receiving the product image, it is also necessary to determine the prompt text used to guide the multimodal model to describe the product image. After that, the product image and the prompt text can be input into the multimodal model, and the multimodal model will describe the product image according to the requirements of the prompt text, so that the text information output by the multimodal model for describing the product image can be obtained. The text information output by the multimodal model will provide an extended supplementary description of the product image, which is also conducive to enriching the user's browsing experience. After that, the text information and the product image are input into the image generation model, and the image generation model can expand the image. Moreover, since the text information describes the product image, the image generation model can understand the product image more accurately, which is conducive to ensuring the quality of the obtained extended image, that is, the obtained multiple extended images can have a greater correlation with the text information. And it can be seen that the present application scheme does not require manual generation of product introduction information, so efficiency can be guaranteed.

[0100] In summary, the present application scheme effectively improves the generation efficiency and quality of product introduction information.

[0101] Corresponding to the above method embodiment, the embodiment of the present invention further provides a system for generating product introduction information, which can be referred to in correspondence with the above.

[0102] See also Figure 2 , the system for generating the product introduction information may include:

[0103] A prompt text determination module 201 is used to receive a product image and determine a prompt text for guiding the multimodal model to describe the product image;

[0104] A text information generating module 202 is used to input the product image and the prompt text into the multimodal model, and obtain the text information output by the multimodal model for describing the product image;

[0105] The product introduction information determination module 203 is used to input text information and product images into the image generation model, obtain multiple extended images related to the product image output by the image generation model, and use each extended image and text information as the generated product introduction information corresponding to the product image.

[0106] In a specific embodiment of the present invention, a prompt text adding module is also included, which is used to:

[0107] Receive auxiliary description text for the product image; add the auxiliary description text to the prompt text.

[0108] In a specific embodiment of the present invention, the multimodal model is a pre-trained multimodal model based on a large language and visual assistant architecture, and the multimodal model includes a visual encoder, a large-scale language model, and a multi-layer perceptron for connecting the visual encoder and the large-scale language model.

[0109] In a specific implementation of the present invention, the prompt text determination module 201 is specifically used to:

[0110] A product image is received, and according to the product type in the product image, a prompt text corresponding to the product type is selected from a prompt text library as the determined prompt text for guiding the multimodal model to describe the product image.

[0111] In a specific embodiment of the present invention, it also includes:

[0112] A candidate prompt text construction module is used to pre-construct K candidate prompt texts corresponding to each commodity type;

[0113] Wherein, K is a preset positive integer not less than 2;

[0114] A candidate text information generation module is used to input a test product image of a product type and an i-th candidate prompt text of the product type into a multimodal model, and obtain an i-th text information output by the multimodal model for describing the test product image;

[0115] Wherein, i is a positive integer and i ranges from 1 to K in sequence;

[0116] A similarity determination module is used to determine the similarity between each of the first to Kth text information and the reference text information after obtaining the first to Kth text information for describing the test product image;

[0117] An optimal prompt text determination module is used to select the maximum similarity degree from the K similarities and use the corresponding candidate prompt text as the optimal prompt text corresponding to the product type;

[0118] The prompt text library establishment module is used to place the optimal prompt text into the prompt text library and establish a corresponding relationship between the optimal prompt text and the commodity type, so as to use the optimal prompt text as the prompt text corresponding to the commodity type in the prompt text library.

[0119] In a specific implementation of the present invention, sequentially determining the similarity between each of the first text information to the Kth text information and the reference text information includes:

[0120] The first to Kth text information and the reference text information are all converted into vectors, and the similarity between the first to Kth text information and the reference text information is determined in turn by calculating the cosine similarity between the vectors.

[0121] In a specific implementation of the present invention, a prompt text adjustment module is also included, which is used to:

[0122] When a prompt text adjustment instruction is received, a prompt text corresponding to the product type is selected from the backup text library as the prompt text determined to guide the multimodal model to describe the product image, and returned to the trigger text information generation module 202 to regenerate the product introduction information corresponding to the product image.

[0123] Corresponding to the above method and system embodiments, the embodiments of the present invention further provide a device for generating product introduction information, a computer-readable storage medium, and a computer program product, which can be referred to in correspondence with the above.

[0124] See also Figure 3 As shown, the device may include:

[0125] Memory 301, used for storing computer programs;

[0126] The processor 302 is used to execute a computer program to implement the steps of the method for generating product introduction information in any of the above embodiments.

[0127] The computer program product includes a computer program / instruction, and when the computer program / instruction is executed by a processor, the steps of the method for generating product introduction information in any of the above embodiments are implemented.

[0128] See also Figure 4The computer readable storage medium 40 stores a computer program 41, and when the computer program 41 is executed by the processor, the steps of the method for generating product introduction information in any of the above embodiments are implemented. The computer readable storage medium 40 mentioned here includes RAM (Random Access Memory), memory, ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read Only Memory), register, hard disk, removable disk, or any other form of storage medium known in the technical field.

[0129] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0130] Professionals may further realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to the function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention, and the description of the above embodiments is only used to help understand the technical solution and its core idea of ​​the present invention. It should be pointed out that for ordinary technicians in the technical field, without departing from the principle of the present invention, the present invention can also be improved and modified in a number of ways, and these improvements and modifications also fall within the scope of protection of the present invention.

Claims

1. A method for generating product introduction information, characterized in that: include: Receiving a product image, and determining a prompt text for guiding the multimodal model to describe the product image; After the product image and the prompt text are input into the multimodal model, text information output by the multimodal model for describing the product image is obtained; The text information and the product image are input into an image generation model to obtain a plurality of extended images related to the product image output by the image generation model, and each of the extended images and the text information is used as the generated product introduction information corresponding to the product image.

2. The method for generating product introduction information according to claim 1, characterized in that: Also includes: Receiving auxiliary description text for the product image; The auxiliary explanation text is added to the prompt text.

3. The method for generating product introduction information according to claim 1, characterized in that: The multimodal model is a pre-trained multimodal model based on a large-scale language and visual assistant architecture, and the multimodal model includes a visual encoder, a large-scale language model, and a multi-layer perceptron for connecting the visual encoder and the large-scale language model.

4. The method for generating product introduction information according to any one of claims 1 to 3, characterized in that: Determining a prompt text for guiding the multimodal model to describe the product image includes: According to the commodity type in the commodity image, a prompt text corresponding to the commodity type is selected from a prompt text library as the determined prompt text for guiding the multimodal model to describe the commodity image.

5. The method for generating product introduction information according to claim 4, characterized in that: Also includes: For each commodity type, K candidate prompt texts corresponding to the commodity type are pre-constructed; Wherein, K is a preset positive integer not less than 2; After inputting a test product image of the product type and an i-th candidate prompt text of the product type into the multimodal model, obtaining i-th text information output by the multimodal model for describing the test product image; Wherein, i is a positive integer and i ranges from 1 to K in sequence; After obtaining the first to Kth text information for describing the test product image, sequentially determining the degree of similarity between each of the first to Kth text information and the reference text information; Selecting the maximum similarity degree from the K similarities, and using the corresponding candidate prompt text as the optimal prompt text corresponding to the product type; The optimal prompt text is placed in the prompt text library, and a corresponding relationship between the optimal prompt text and the commodity type is established, so that the optimal prompt text is used as the prompt text corresponding to the commodity type in the prompt text library.

6. The method for generating product introduction information according to claim 5, characterized in that: Determining in sequence the similarity between each of the first text information to the Kth text information and the reference text information includes: The first to the Kth text information and the reference text information are all converted into vectors, and the similarity between the first to the Kth text information and the reference text information is determined in turn by calculating the cosine similarity between the vectors.

7. The method for generating product introduction information according to claim 4, characterized in that: Also includes: When a prompt text adjustment instruction is received, a prompt text corresponding to the product type is selected from the backup text library as the prompt text determined to guide the multimodal model to describe the product image, and the operation of inputting the product image and the prompt text into the multimodal model is returned to regenerate the product introduction information corresponding to the product image.

8. A system for generating product introduction information, characterized in that: include: A prompt text determination module, used for receiving a product image and determining a prompt text for guiding the multimodal model to describe the product image; A text information generating module, configured to input the product image and the prompt text into the multimodal model, and obtain text information output by the multimodal model for describing the product image; A product introduction information determination module is used to input the text information and the product image into an image generation model, obtain multiple extended images related to the product image output by the image generation model, and use each of the extended images and the text information as the generated product introduction information corresponding to the product image.

9. A device for generating product introduction information, characterized in that: include: Memory for storing computer programs; A processor, configured to execute the computer program to implement the steps of the method for generating product introduction information as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method for generating product introduction information according to any one of claims 1 to 7 are implemented.

Images