Intelligent automatic product publishing method

Abstract

The invention discloses an intelligent automatic product publishing method, which is characterized by comprising the following steps of: pre-training a product picture selection model according to simple information input by a user, selecting a product picture which is most matched with the product according to the characteristics of the product, inputting a text and a candidate picture set, and outputting a matching score of the text and each picture. Selecting a picture which is most matched with the input text according to the matching score of the image-text; in combination with algorithms such as recall, classification and generation, information such as keywords, categories and descriptions corresponding to products is generated in an auxiliary manner, a deep learning technology is used, a user is assisted to publish new products, and the user is assisted to acquire information required for publishing the products according to a small amount of simple data and historical product data of the user, so that the labor efficiency of the user is greatly reduced.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, specifically to the field of computer vision and natural language processing technology, and in particular to an intelligent product automatic publishing method. Background Art

[0002] With the development of mobile Internet, various product e-commerce platforms are booming. However, the release and management of product information still basically requires manual participation. Traditionally, the release of new products basically relies on manual filling of various product information, which is inefficient.

[0003] In recent years, with the rapid development of computer technology, in the field of computer vision, with the introduction of visual models such as CLIP and VIT, machines have made great progress in understanding image information; in the field of natural language processing, since the release of ChatGPT, the generation model has continued to be popular, and foreign models such as Bard and Claude have been released one after another, while domestic models such as Wenxin Yiyan, Baichuan and ChatGLM have made further progress.

[0004] Newly released products involve the combination of two technologies, using computer vision and natural language processing to express the same product content information, achieving a deeper understanding of the product. Some studies have begun to try to apply multimodal technology to product recommendations, such as product recommendations in a text-and-picture dialogue style and product image generation in a text-and-picture dialogue style. However, there are not many applications of using multimodal technology for automated extraction of product content information and product release. In product release scenarios, most of them still use different models to deal with different tasks for different tasks.

[0005] Therefore, there is a need for an intelligent automatic publishing method that can widely and accurately understand the content information of various e-commerce products. Summary of the invention

[0006] The technical problem solved by the present invention is mainly to overcome the lack of intelligence in the current product release, and provide an intelligent product automatic release method, including collecting and cleaning existing product data, organizing product names, pictures, keywords, directories, descriptions and other information; using the CLIP encoder to train an image selector, and the user selects the picture that best matches the product; establishing a multimodal feature fusion model to fuse text information and image information to obtain a richer product feature representation; finally, based on the multimodal fusion model, different modules are established for different tasks to complete the automatic release of product content information.

[0007] To solve the above problems, the present invention provides an intelligent product automatic release method, comprising the following steps:

[0008] Step 1: confirm the content information required for publishing the product, the content information includes text and pictures, the text includes name, keywords, category and description; collect and analyze the existing content information from existing products to make a product data set, the analysis of the existing content information includes filtering and cleaning the content information required by the product picture selection model;

[0009] Step 2, in the content information, the features other than the picture are converted into text information, the content information is manually annotated to obtain the consistency between the picture and the text, and the positive sample of the picture-text matching and the negative sample of the picture-text mismatch are constructed according to the correlation between the picture and the text; the positive and negative samples are used to train a product picture selection model based on the CLIP model, which is used to select the picture of the product that best matches it through the content information. The input is text and more than one candidate picture, and the matching score between the text and each picture is output, and the picture that best matches the input text is selected according to the matching score of the picture and text;

[0010] Step 3: Use the content information in the preset product data set to train a multimodal fusion model for image-text fusion. In the multimodal fusion model, the image encoder and the text encoder are used to extract content information of different modalities, and the multimodal fusion module composed of a multi-layer neural network is used to extract a multimodal fusion vector.

[0011] Step 4: Connect different multimodal task modules after the multimodal fusion model to complete the product release task, which includes recall, classification and generation.

[0012] 2. The intelligent product automatic publishing method according to claim 1, characterized in that in step 1, filtering and cleaning the content information required by the product image selection model specifically includes:

[0013] Removing irrelevant content from the text, including HTML tags and digital noise;

[0014] Remove redundant keywords from the text;

[0015] Correct grammatical and spelling errors in the text and standardize technical terms in the text;

[0016] Verify the consistency between product images and text information to ensure that they correspond to the same product;

[0017] Remove invalid images that do not match the text.

[0018] 3. An intelligent product automatic release method as described in claim 2 is characterized in that the step 2 includes using the name as text input, encoding the name vector through the trained CLIP model, encoding multiple candidate pictures belonging to the same product using the fine-tuned CLIP model respectively to obtain representations of multiple candidate picture vectors, calculating the similarity between the name vector and each picture vector, and selecting the picture with the highest similarity to the name vector as the positive sample picture; the CLIP model encodes the picture into a 512-dimensional picture vector and the text into a 512-dimensional text vector by combining the calculation of cosine similarity with the cross entropy loss function, and additionally mixing product pictures of other products into multiple pictures of the same product.

[0019] 4. The intelligent product automatic release method according to claim 3, characterized in that in step 3, training a multimodal fusion model specifically includes:

[0020] Step 3.1, use the trained CLIP model to extract the text vector and image vector of the product in the product data set, splice the text vector and image vector to obtain a 1024-dimensional multimodal fusion vector of the product, and input it into a fusion module composed of a 3-layer 1024-dimensional fully connected neural network to form a multimodal fusion model;

[0021] Step 3.2: Use the trained text encoder and image encoder, freeze the parameters of the trained text encoder and image encoder, obtain the corresponding text vector and image vector, and then use the multimodal fusion model to perform feature fusion. Only train the parameters of the fusion module and use the content information as the general information input to learn the semantic fusion knowledge of the product domain.

[0022] In step 3.3, the preset image-text comparison learning loss function is used to shorten the feature distance of positive samples and move away from the feature distance of negative samples; the preset image-text matching loss function (ITM) is used to evaluate the consistency between images and texts.

[0023] 5. An intelligent product automatic release method according to claim 4, characterized in that in step 4, the release task includes recall, classification and generation, specifically:

[0024] For recall in the release task, the product image and name are encoded into the first multimodal fusion vector through the multimodal fusion model, and each keyword is encoded using a text encoder based on the Bert model to obtain a keyword vector. The first multimodal fusion vector of the product and the keyword vector are recalled by similarity. The 10 keyword vectors with the highest similarity are recalled, and the product's own keywords are used as labels. The parameters of keyword encoding are optimized through the cross entropy loss function.

[0025] For the classification in the release task, the existing content information of the product is input into the multimodal fusion model to obtain the multimodal fusion vector, the existing classification directory is determined, and the classifier is connected to the multimodal fusion model. The prepared product data set is used, the fusion model parameters are fixed, and the classifier is optimized through the cross entropy loss function to obtain the classifier for this type of task;

[0026] For the catalog of product releases, after filling in keywords in the product catalog according to the order of releasing new products, the multimodal fusion model accesses the corresponding classifier to select the catalog with the highest current keyword consistency for user assistance or reference;

[0027] For the generation in the publishing task, the open source large model is used for long text generation training; the autoregressive log-likelihood loss is used to optimize the parameters, and the multimodal fusion vector in the multimodal fusion model is aligned with the input of the open source large model to generate the required text.

[0028] 6. An intelligent product automatic release method according to claim 5, characterized in that in the step 3.3, the formula of the image-text contrast learning loss is:

[0029]

[0030] in, represents the expectation of the image vector I and the text vector T in the dataset D, H represents the cross entropy, I represents the image vector, T represents the text vector, s(W,I) represents the similarity between the text vector and the image vector, s(I,W) represents the similarity between the image vector and the text vector, y i2t (I) and y t2i (T) is a 0, 1 label, used to indicate whether the real image and text match;

[0031] The image-text matching loss function is used to evaluate the consistency between the image and the generated text, and the formula is:

[0032]

[0033] in, represents the expectation of the image vector I and the text vector T in the dataset D, H represents the cross entropy, I represents the image vector, T represents the text vector, and y itm represents the real image and text classification label, p itm Represents the predicted probability of binary classification of image and text;

[0034] In the step 3.2, the content information is split into multiple training samples of different granularities according to the order of information filled in when the product is released, and the training samples include {product image, product name, product keyword}, ​​{product image, product name, product keyword, product catalog} and {product image, product name, product keyword, product catalog, product description}.

[0035] Beneficial effects achieved by the present invention: The present invention can combine existing product content information, and realize in-depth understanding of product details by combining graphic and text information, thereby intelligently extracting product content information, integrating product visual information with text information, and constructing a multimodal automatic product release method, which can ensure the original product release process, while using deep learning technology, and related modules share multimodal fusion features, and refine according to different product release subtasks, which can give full play to the advantages of multi-source heterogeneous information, enhance the semantic understanding and generation capabilities of the model, assist users to release new products more conveniently, greatly reduce labor costs, and improve the efficiency of product release. The designed system process is clear and reasonable, and the technical means are advanced and feasible. It can be applied to the new product release process of various platforms, providing intelligent and refined audit services for them, improving audit efficiency and quality, and helping to promote the further improvement and perfection of e-commerce intelligence. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 is a simplified flow chart of a method according to an embodiment of the present invention;

[0037] Figure 2 It is a framework diagram of a multimodal model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0038] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings. The embodiments described in the accompanying drawings are specific examples, which only show the application of the present invention under the example and do not represent the upper limit of the present invention.

[0039] Figure 1 This is a simplified flow chart of the method of this patent. The example of the present invention is an intelligent product automatic release method, which mainly includes:

[0040] Step 1: Confirm the relevant information required to release a new product, such as product name, product keywords, product category, product description, and upload product pictures. Collect various information from the released products, use different fields to store, analyze the collected data, filter and clean the text and image information required for subsequent models.

[0041] Specifically, remove irrelevant content in the text information, such as HTML tags and digital noise; delete redundant keywords, avoid duplication and invalid keywords; clean product description text, correct grammar and spelling errors, unify product attributes, etc.; verify the consistency of product images and text information, ensure that the images and text match the product, and delete unmatched product samples.

[0042] Through the above-mentioned multi-step data cleaning and enhancement, the quality of training data can be effectively improved, and the model's ability to learn product visual and semantic information can be enhanced, thereby improving the robustness of the model.

[0043] Step 2: In order to select the main image that best matches the product from multiple product images, manual annotation is performed on the collected product data and image data. The relevance of the product image group and the product is manually marked, and the most suitable image is selected and labeled as a positive sample "1". The remaining images and the product are used as negative samples and labeled as "0". A product name corresponds to multiple images. The labeled data set is used to train the product image selection model based on the CLIP model to select the image that best matches the product itself from multiple images when the product is displayed.

[0044] Specifically, the training set of product names and manually labeled product pictures is trained based on the CLIP model. CLIP consists of an image encoder and a text encoder. Each product picture is encoded into a 512-dimensional picture vector by the image encoder, and the product name is encoded into a 512-dimensional text vector by the text encoder. By calculating the cosine similarity between the product name and each picture, the cross entropy loss function is used to optimize the product picture selection model based on the pre-labeled labels. In order to improve the robustness of the model, three additional product pictures of other products will be selected during training to expand the negative samples and increase the difficulty of training.

[0045] Step 3: After the user enters the product name and several candidate images through the trained product image selection model, the pre-trained image selection model is used to automatically select the main image that best matches the product name. Collect and integrate product names, keywords, catalogs, descriptions, etc., build a multimodal product set containing text and images, and use it to train and establish a unified multimodal fusion model to obtain richer multimodal features for each product.

[0046] Step 3.1 Use the product image selection model to encode multiple candidate photos of each product to obtain image feature vectors, encode each product name into a text feature vector, and select the image with the highest similarity as the main image of the product based on the selection model. Combine the main image of the product with text data such as product name, keywords, catalog and description to form a multimodal product set.

[0047] Specifically, the multimodal product set contains multiple groups of instances, such as {product image, product name, product keywords}, {product image, product name, product keywords, product catalog}, {product image, product name, product keywords, product catalog, product description}. The data corresponding to the same product will be made into several corresponding groups of instance data combinations, and instances of different compositions will be used for model training in different time periods.

[0048] Step 3.2 The framework of training the multimodal fusion model is as follows Figure 2 As shown in the figure, the text information is passed into the text encoder to obtain the product text vector, the image information is passed into the image encoder to obtain the product image vector, the product text vector and the product image vector are passed into the multimodal fusion model to obtain the multimodal fusion vector, and the multimodal fusion vector is passed into the generation module, classification module and recall module as needed for subsequent processing. By fixing the model parameters in the trained text encoder and image encoder, the corresponding text vector and image vector are obtained after encoding, and the 6-layer Transformers Encoder module is used as the multimodal fusion module for vector feature fusion, and then the parameters of the multimodal fusion model are optimized according to different image and text product examples and loss function training.

[0049] Specifically, when constructing the multimodal fusion module, it includes 6 layers of Transformer Encoder layers. The first layer uses the Cross-Attention mechanism to achieve the fusion and learning of text vectors and image vectors, and obtain more comprehensive multimodal vector features of the product.

[0050] Specifically, we will combine the two loss functions to train the multimodal fusion model.

[0051] ITC loss takes the text and image of the same product as positive samples and the combination of different products as negative samples. The loss function brings the features of positive samples closer and moves the features of negative samples further away.

[0052]

[0053] in, represents the expectation of the image vector I and the text vector T in the dataset D, H represents the cross entropy, I represents the image vector, T represents the text vector, s(W,I) represents the similarity between the text vector and the image vector, s(I,W) represents the similarity between the image vector and the text vector, y i2t (I) and y t2i (T) is a 0, 1 label, used to indicate whether the real image and text match;

[0054] Image-Text Matching Loss Function (ITM),The text features output by the text encoder need to be highly matched with the image features. The loss function evaluates the consistency between the image and the generated text.

[0055]

[0056] in, represents the expectation of the image vector I and the text vector T in the dataset D, H represents the cross entropy, I represents the image vector, T represents the text vector, and y itm represents the real image and text classification label, p itm Represents the predicted probability of the image and text classification ；

[0057] Specifically, according to the order of filling in information when releasing products, each product content information is split into multiple training samples, {product image, product name, product keyword}, ​​{product image, product name, product keyword, product catalog}, {product image, product name, product keyword, product catalog, product description}, and all product information is combined into training samples of different granularities. These training samples of different granularities can be used to train multimodal models to implicitly learn the relationship between product attributes and improve understanding ability.

[0058] The above strategy can effectively enhance the correlation between the model's learning of product image and text information, making the multimodal fusion vector more expressive. In the multimodal framework, sharing feature information between text and pictures can also further enhance the model's ability to understand products.

[0059] Step 4: In actual development, you need to fill in the product name, keywords, catalog, description and other text, as well as input product pictures. Filling in this information can be divided into three types of tasks: recall, classification and generation. For future multimodal fusion models, such as Figure 2 As shown in the figure, we only need to train different application modules with future data sets according to different needs. Different modules are connected after the multimodal fusion model according to different tasks to assist users in all their needs when releasing new products, thereby building a complete multimodal product release method. This method can effectively use product images and a small amount of text information to automatically assist users in releasing new products and improve all aspects of product release.

[0060] Step 4.1, for the recall in the release task, the product image and name are encoded into the first multimodal fusion vector through the multimodal fusion model, and the multimodal feature vector of the product is obtained through the multimodal fusion model.

[0061] Specifically, for the recall in the release task, it is necessary to recall 10 keywords most relevant to the product from all candidate keywords for users to choose. Using the {product image, product name, product keyword} dataset, the product image and name are encoded into the first multimodal fusion vector through the multimodal fusion model. The keyword uses a text encoder based on the Bert model to encode each keyword. The product vector and the keyword vector are recalled by similarity. The 10 with the highest similarity are recalled, and the product's own keywords are used as labels. The parameters of the keyword encoder are optimized through the cross entropy loss function to obtain a more accurate keyword encoder.

[0062] Specifically, by combining the multimodal fusion model with the keyword encoder, when a user publishes a product, 10 keywords that best match the product name entered by the user and the uploaded product picture can be provided for the user to select and confirm.

[0063] Step 4.2, for the classification in the release task, obtain the product multimodal fusion vector by passing the product information of the text input through the multimodal fusion model. Confirm the candidate catalog, connect the classifier after the multimodal fusion model, use the prepared product data set, fix the multimodal fusion model parameters, and train and optimize the classifier parameters through the cross entropy loss function to obtain a classifier that meets the requirements of the keyword classification task.

[0064] Specifically, for the task of product catalog classification, according to the current order of new product releases, the product catalog is filled in after the keywords. Using the {product image, product name, product keyword, product catalog} dataset, the multimodal fusion model is used to encode the existing product content information to obtain a 512-dimensional product multimodal feature vector. We use a 6-layer TransformersEncoder module with randomly initialized parameters as a classifier, use the catalog classification dataset, fix the multimodal fusion model parameters, and only use the cross entropy loss function to optimize the classifier parameters.

[0065] Specifically, in actual use, the text and image information already entered by the user are combined, and by connecting the multimodal fusion model to the classifier, the product catalog that best suits the current product can be selected to assist the user in filling in or referring to it.

[0066] Step 4.3 is for the generation of publishing tasks, which is a more complex task. It inputs the existing product content information and obtains the multimodal fusion vector through the multimodal fusion model. In order to generate long text information with good results, the open source large model is used for long text generation training. Through fine-tuning training, the autoregressive log-likelihood loss is used for parameter optimization, and the multimodal feature vector is aligned with the input of the open source large model to generate the required text.

[0067] Specifically, for product description generation, you need to select a generative large model for the generation of long texts. The framework and training methods of the Baichuan open source large model are transparent, and it has strong Chinese and English comprehension capabilities. The 7B parameter Baichuan model can run on resource-constrained devices when deployed on mobile terminals. Different prompt words are set for different generation requirements to prompt the Baichuan model, such as "Based on the text information provided below, combined with the image information, generate a description of the corresponding product:". Since the output of the multimodal fusion model and the input of the Baichuan model are different in shape, we need to use a 3-layer fully connected layer for dimensionality increase. In this task, the parameters of the multimodal fusion model and the Baichuan model will be fixed, and some high-quality product description samples will be selected for better generation.

[0068] Specifically, since the multimodal feature vector output by the multimodal fusion model has a different input shape from the Baichuan generation model, we use the {product image, product name, product keyword, product catalog, product description} dataset and 3 fully connected layers as an alignment module to align the parameter dimensions of the multimodal fusion model with the Baichuan model. During the training process, we freeze the parameters of the multimodal fusion model and the Baichuan model at the same time, and only train the alignment layer parameters of these 3 layers to train an alignment module that can perfectly adapt the multimodal fusion model to the Baichuan model, thereby generating better quality text content.

[0069] The above-mentioned shared multimodal fusion vector can adapt different tasks according to different product release subtasks, which can give full play to the advantages of multi-source heterogeneous information, enhance the semantic understanding and generation capabilities of the model, thereby realizing automatic high-quality product release and saving a lot of manual time.

[0070] In order to solve the cumbersome process in the product release process, the present invention can be used to build a product image selection model to select highly relevant product images, and at the same time build a multimodal product coding model, which can fuse the features of product images and text information to capture information between different modalities. In different tasks for releasing products, the multimodal fusion model is combined to train different modules for different tasks, thereby more efficiently assisting users in releasing new products.

[0071] The beneficial effects achieved by the present invention are:

[0072] The present invention can combine existing product content information and achieve in-depth understanding of product details by combining graphic and text information, thereby intelligently extracting product content information, integrating product visual information with text information, and constructing a multimodal-based automatic product release method. While ensuring the original product release process, the deep learning technology is used, and related modules share multimodal fusion features, which are refined according to different product release subtasks. The advantages of multi-source heterogeneous information can be fully utilized, and the semantic understanding and generation capabilities of the model are enhanced, assisting users to release new products more conveniently, greatly reducing labor costs, and improving product release efficiency. The designed system process is clear and reasonable, and the technical means are advanced and feasible. It can be applied to the new product release process of various platforms, providing them with intelligent and refined review services, improving review efficiency and quality, and helping to promote the further improvement and perfection of e-commerce intelligence.

[0073] The present invention may also have many other implementation modes. The above embodiments do not limit the present invention in any way. Without departing from the spirit and essence of the present invention, technicians familiar with the field may make various corresponding changes and modifications based on the present invention. All other improvements and applications made to the above embodiments in an equivalent transformation manner should fall within the scope of protection of the claims attached to the present invention.

Claims

1. An intelligent product automatic release method, characterized by comprising the following steps: Step 1: confirm the content information required for publishing the product, the content information includes text and pictures, the text includes name, keywords, category and description; collect and analyze the existing content information from existing products to make a product data set, the analysis of the existing content information includes filtering and cleaning the content information required by the product picture selection model; Step 2, in the content information, the features other than the picture are converted into text information, the content information is manually annotated to obtain the consistency between the picture and the text, and the positive sample of the picture-text matching and the negative sample of the picture-text mismatching are constructed according to the correlation between the picture and the text; the positive and negative samples are used to train a product picture selection model based on the CLIP model, which is used to select the picture of the product that best matches it through the content information. The input is text and more than one candidate picture, and the matching score between the text and each picture is output, and the picture that best matches the input text is selected according to the matching score of the picture and text; Step 3: Use the content information in the preset product data set to train a multimodal fusion model for image-text fusion. In the multimodal fusion model, the image encoder and the text encoder are used to extract content information of different modalities, and the multimodal fusion module composed of a multi-layer neural network is used to extract a multimodal fusion vector. Step 4: Connect different multimodal task modules after the multimodal fusion model to complete the product release task, which includes recall, classification and generation.

2. An intelligent product automatic release method as claimed in claim 1, characterized in that: In step 1, filtering and cleaning the content information required by the product image selection model specifically includes: Removing irrelevant content from the text, including HTML tags and digital noise; Remove redundant keywords from the text; Correct grammatical and spelling errors in the text and standardize technical terms in the text; Verify the consistency between product images and text information to ensure that they correspond to the same product; Remove invalid images that do not match the text.

3. An intelligent product automatic release method as claimed in claim 2, characterized in that: The step 2 includes using the name as text input, encoding the name vector representation through the trained CLIP model, encoding multiple candidate images belonging to the same product using the fine-tuned CLIP model respectively to obtain representations of multiple candidate image vectors, calculating the similarity between the name vector and each image vector, and selecting the image with the highest similarity to the name vector as the positive sample image; the CLIP model encodes the image into a 512-dimensional image vector and the text into a 512-dimensional text vector by combining the calculation of cosine similarity with the cross entropy loss function, and additionally mixing product images of other products into multiple images of the same product.

4. An intelligent product automatic release method as claimed in claim 3, characterized in that: In step 3, training a multimodal fusion model specifically includes: Step 3.1, use the trained CLIP model to extract the text vector and image vector of the product in the product data set, splice the text vector and image vector to obtain a 1024-dimensional multimodal fusion vector of the product, and input it into a fusion module composed of a 3-layer 1024-dimensional fully connected neural network to form a multimodal fusion model; Step 3.2: Use the trained text encoder and image encoder, freeze the parameters of the trained text encoder and image encoder, obtain the corresponding text vector and image vector, and then use the multimodal fusion model to perform feature fusion. Only train the parameters of the fusion module and use the content information as the general information input to learn the semantic fusion knowledge of the product domain. In step 3.3, the preset image-text comparison learning loss function is used to shorten the feature distance of positive samples and move away from the feature distance of negative samples; the preset image-text matching loss function (ITM) is used to evaluate the consistency between images and texts.

5. An intelligent product automatic release method as claimed in claim 4, characterized in that: In step 4, the release task includes recall, classification and generation, specifically: For recall in the release task, the product image and name are encoded into the first multimodal fusion vector through the multimodal fusion model, and each keyword is encoded using the text encoder based on the Bert model to obtain the keyword vector. The first multimodal fusion vector of the product and the keyword vector are recalled by similarity. The 10 keyword vectors with the highest similarity are recalled, and the product's own keywords are used as labels. The cross entropy loss function is used to optimize the keyword encoding parameters. For the classification in the release task, the existing content information of the product is input into the multimodal fusion model to obtain the multimodal fusion vector, the existing classification directory is determined, and the classifier is connected to the multimodal fusion model. The prepared product data set is used, the fusion model parameters are fixed, and the classifier is optimized through the cross entropy loss function to obtain the classifier for this type of task; For the catalog of product releases, after filling in keywords in the product catalog according to the order of releasing new products, the multimodal fusion model accesses the corresponding classifier to select the catalog with the highest current keyword consistency for user assistance or reference; For the generation of publishing tasks, we use open source large models to train the generation of long texts; The autoregressive log-likelihood loss is used to optimize the parameters and align the multimodal fusion vector in the multimodal fusion model with the input of the open source large model to generate the required text.

6. An intelligent product automatic release method as claimed in claim 5, characterized in that: In step 3.3, the formula for the image-text contrast learning loss is: in, represents the expectation of the image vector I and the text vector T in the dataset D, H represents the cross entropy, I represents the image vector, T represents the text vector, s(W,I) represents the similarity between the text vector and the image vector, s(I,W) represents the similarity between the image vector and the text vector, y i2t (I) and y t2i (T) is a 0, 1 label, used to indicate whether the real image and text match; The image-text matching loss function is used to evaluate the consistency between the image and the generated text, and the formula is: in, represents the expectation of the image vector I and the text vector T in the dataset D, H represents the cross entropy, I represents the image vector, T represents the text vector, and y itm represents the real image and text classification label, p itm Represents the predicted probability of binary classification of image and text; In the step 3.2, the content information is split into multiple training samples of different granularities according to the order of information filled in when the product is released, and the training samples include {product image, product name, product keyword}, ​​{product image, product name, product keyword, product catalog} and {product image, product name, product keyword, product catalog, product description}.

Images