Automatic commodity publishing method based on artificial intelligence
Through artificial intelligence technology, computer vision and deep learning models are used to automatically process and integrate product information, and intelligently typed with multimodal large models, it solves the problems of inefficient release of traditional products and poor display effects, and achieves efficient and accurate product release and enhance user experience.
Patent Information
- Application Number
- CN202411614741.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-13
- Publication Date
- 2025-02-18
AI Technical Summary
The prior art is inefficient and prone to errors during product release, and cannot fully understand the multimodal information of text and pictures, resulting in poor product display effect.
The automated product release method based on artificial intelligence is adopted, and product information is automatically processed and analyzed through computer vision models, combined with deep learning models to integrate intelligent product information and layout, and multi-modal large models are used for intelligent board arrangement.
It realizes efficient extraction, multi-dimensional classification and intelligent typesetting of product information, improves the efficiency and accuracy of product release, and improves the user's shopping experience.
Smart Images
Figure CN119477474A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the fields of natural language processing, large language models and computer vision, and in particular to an automatic commodity publishing method based on artificial intelligence. Background Art
[0002] In recent years, the rapid development of e-commerce has driven the demand for digital management of product information. Traditional product release methods usually rely on a large number of manual operations, which are inefficient and prone to errors. In order to improve the efficiency and accuracy of product release, the use of artificial intelligence technology for automated processing has become a research hotspot. In particular, the advancement of computer vision and natural language processing technology has provided new solutions for automated product information processing.
[0003] By providing formatted product information in advance by the user and combining it with OCR technology, text blocks and text information can be quickly extracted from images. The information can then be broken down and filled into corresponding positions according to a fixed template, so that the product display page can be automatically generated.
[0004] However, OCR technology cannot deeply understand the layout of text. Understanding input information depends largely on detailed regulations and splitting of user input, requiring users to provide information in a fixed way. At the same time, the product display results given by templates lack diversity, and cannot automatically understand the information provided by users, flexibly display product information, and cannot fully understand the multimodal information of text and pictures when arranging them in a layout, resulting in poor product display effects.
[0005] Therefore, a more efficient and automated product release method is needed. Summary of the invention
[0006] The technical problem solved by the present invention is to overcome the deficiencies of the prior art and provide an automatic product release method based on artificial intelligence, which mainly includes the following steps:
[0007] Step 1: Automatic processing and analysis of product information based on computer vision models;
[0008] Step 2: Intelligent product information and layout integration based on deep learning models;
[0009] Step 3: Intelligent plate arrangement based on multi-modal large model;
[0010] Step 4: Export the module.
[0011] In step 1, the automatic processing and analysis of commodity information based on the computer vision model specifically includes:
[0012] The files, pictures, and web links containing product information input by the user are uniformly converted into picture format. For the product information in picture format, the trained target detection model is combined with OCR to perform layout recognition and extract product information. Different areas are segmented by bounding boxes and text, table, and picture information are extracted to obtain the layout of different types of product information and the parsed product information.
[0013] In step 1, the use of the trained object detection model combined with OCR to perform layout recognition and extract product information specifically includes:
[0014] Use the target detection model that has been annotated and trained with actual data to detect user input converted into image form, and output bounding boxes representing the positions of text, images, image-text pairs, and tables in the image. The corresponding labels are text, image, image-text pair, and table, respectively, to obtain the overall element layout; use the bounding boxes to capture the text, image, and table areas in the input image, and output the images of each area; use the OCR model to detect and extract the text in the text and table parts, respectively, to obtain text and table information in character representation, and associate the corresponding image with the extracted text according to the bounding box of the image-text pair, and regard them as product information of the same product for subsequent classification.
[0015] In step 2, the smart product information and layout integration based on the deep learning model specifically includes:
[0016] For the parsed product information, a large language model combined with an image classification model is used to split the multimodal information of the product, and the images and texts matched by image text pairs are marked during sorting. The images after dimensioning are attributed to the corresponding products according to the corresponding texts, and product introductions of different dimensions of each product are obtained. The parsed product information is divided into several products using a large language model, and different product names and corresponding company information are extracted. The large language model is then used to extract text and table information related to each product from the product information, and the extraction results are classified into different navigation bars. The sub-images in each area of the image captured after parsing are classified into different navigation bars using the trained image classification model. The information of each navigation bar is integrated to obtain product introductions of different dimensions of each product, which are output in json format.
[0017] In step 3, the intelligent plate arrangement based on the multimodal large model specifically includes:
[0018] For the text, pictures, and table elements contained in each navigation bar, add digital IDs for encoding, then use the multimodal large model to understand the semantics and layout, output a digital ID list containing layout information, check and correct the quality after typesetting, compare the returned digital ID list with the input digital ID, check the generation result of the large language model, complete the missing digital IDs, and remove duplicate or non-existent digital IDs; decode the digital ID of the typesetting result, and decode each digital ID into the corresponding text, table, and picture element according to the digital ID list generated by the large language model, and arrange each element according to the corresponding digital ID; arrange the sub-elements corresponding to the digital ID in each navigation bar in combination with the layout to obtain the typeset product introduction.
[0019] In step 3, the multimodal large model understands the semantics and formats the layout, which specifically includes:
[0020] For each navigation bar, the text, image, table elements, digital IDs and prompt words contained therein are provided to the large language model, and the large language model arranges the positions of each element and returns a list of digital IDs with position information and layout details.
[0021] In step 4, the output module specifically includes:
[0022] According to the text, table and picture in each navigation bar of each product after typeset, they are arranged according to the corresponding typesetting detail format to obtain product introduction pages containing different dimensions of different products. The typesetting detail format includes: single text, single picture, single table, left picture and right text, left text and right picture, two pictures in parallel and three pictures in parallel.
[0023] In step 1, the documents, pictures and links input by the user are converted into picture form through the python library, and the four types of elements, namely text, image, image text pair and table, are detected using the yolo-doclayout layout recognition model. The text content of text and table is extracted using the PaddleOCR algorithm model, and then arranged according to the coordinates of the center point of the bounding box;
[0024] In step 2, the image classification model is a ResNet model.
[0025] In step 1, the python libraries include Pillow, pdf2image and selenium.
[0026] The beneficial effects achieved by the present invention are:
[0027] This paper proposes an automated product publishing method based on artificial intelligence, which realizes the automated extraction, integration and layout output of product information through the combination of multiple algorithms. By training the target detection model and combining OCR technology, efficient layout analysis and information extraction of product information are realized, and product information can be processed without manual data splitting. Further combined with the large language model and the trained image classification model, the product information is refined and multi-dimensionally classified, and the product information is automatically integrated into different navigation bars. In addition, a multimodal large model is used for intelligent layout, and the layout is automatically adjusted to ensure the aesthetics of the visual display. Finally, the layout content is presented as a visual product introduction page through the output module. This method is highly automated and can greatly reduce manual intervention. It can not only improve the efficiency and accuracy of product release, but also enhance the user's shopping experience. It is an important technology for the construction of future intelligent product management systems and has broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] Figure 1 A simplified flow chart of an exemplary embodiment of the present invention.
[0029] Figure 2 A detailed flow chart of an exemplary embodiment of the present invention. DETAILED DESCRIPTION
[0030] The present invention is an automated product release method based on artificial intelligence, which mainly includes automated processing and analysis of product information based on computer vision models, intelligent product information and layout integration based on deep learning models, intelligent plate layout based on multimodal large models, and output modules.
[0031] To implement the above method, the following will be combined with the attached Figure 1 The method flow of an exemplary embodiment of the present invention is described as follows:
[0032] Step 1: Automatic processing and analysis of product information based on computer vision models, including:
[0033] Process the product information in different forms input by users, such as word documents, pictures, PDF documents, web links, etc., and use Python image conversion libraries such as Pillow, pdf2image, and selenium to process the product information in the form of pictures. Pillow image processing library provides a wide range of image file format support, pdf2image can quickly convert PDF files into various image formats, and Selenium supports obtaining images from multiple browsers and web links. It is efficient and compatible when processing rich forms of product information uploaded by users.
[0034] Then, the trained target detection model is used to perform layout recognition on the product information to obtain the layout of different types of information. The target detection base model uses the YOLO-doclayout layout detection model in the YOLO series model as the base model. The target detection dataset is annotated by itself. The annotated categories include text, image, image text pair, and table. YOLO-doclayout is fine-tuned and trained.
[0035] By detecting the image with the trained target detection model, we can obtain the bounding boxes of the four types of elements in the image: text, image, imagetext pair, and table. The image area corresponding to the coordinates of the text and image category bounding boxes is intercepted, and the OCR model is used in turn to detect and extract text and tables, and extract product information in different layouts. The OCR model uses the PaddleOCR algorithm model to extract text and table elements in the image.
[0036] Finally, the elements are arranged from left to right and from top to bottom according to the coordinates of the center point of the bounding box. Then, using the image-text pair bounding box, the pictures and text elements corresponding to the image and text bounding boxes contained in each image-text pair bounding box are arranged together as the same product information, and finally the parsed product information is obtained.
[0037] Step 2: Intelligent product information and layout integration based on deep learning models, including:
[0038] For the parsed product information, a large language model is used to classify the product text information. The large language model is used to extract different product names and corresponding company information from the parsed product information text. The large language model is then used to extract the text and table information related to each product from the product information and classify them into different navigation bars. The categories include Product Name, Product Description, Product Parameters, Certifications, Packaging & Shipping, Installation Instructions, Our Advantages, After Sales Service, and FAQ, obtaining different dimensional information for each product.
[0039] Then use the image classification model to classify product image information. The images in each area of the image extracted after parsing are classified using the ResNet model trained by fine-tuning the self-annotated data, and classified into different navigation bars, including Product main image, Product Name, Product Description, Product Parameters, Certifications, Packaging & Shipping, Installation Instructions, Our Advantages, After Sales Service, and FAQ, corresponding to the images in different dimensions of product information.
[0040] For the images and texts that match the image text pair during sorting, the images after dimension division are assigned to the corresponding products according to the corresponding texts. The product descriptions of different dimensions of each product are obtained and output in JSON format.
[0041] The json file contains a list list, each list contains several dict dictionaries, one dict dictionary for each product, and the specific format is {"Product main image":dict(),"Product Name":dict(),"Product Description":dict(),"Product Parameters":dict(),"Certifications":dict(),"Packaging&Shipping":dict(),"InstallationInstructions":dict(),"Our Advantages":dict(),"After Sales Service":dict(),"FAQ":dict(),"Product Description":dict()}, where each key value represents a dimension, and each dimension contains a dict dictionary. The dictionary corresponding to the Product main image dimension contains the image field and several corresponding product main pictures, in the format of {"image":[image1,image2...]}, and the dictionaries of the other dimensions contain text, table, image fields and corresponding text, tables, and pictures, in the format of {"text":"xxx","table":[table1,table2...],"image":
[0042] [image1,image2...]}.
[0043] Step 3: Intelligent plate arrangement based on multi-modal large model, including:
[0044] For each product after content classification, for each navigation bar, encode each sub-element. For each text, picture, and table element in each navigation bar of the product introduction of each commodity, add a digital ID field. The digital ID starts from 1 and increases incrementally. It is used to distinguish different elements in each navigation bar.
[0045] Then, the multimodal big model is used to understand the semantics and layout. For each navigation bar, the text, picture, table elements, digital IDs and prompt words contained therein are provided to the big language model. The big language model arranges the positions of each element and returns a list of digital IDs with position information and arrangement details. The digital ID list has a dimension of 2 and a format such as [[2], [1, 3], [4, 5, 6]...]. Each small list in the digital ID list represents the elements in the same row of the page. For example, the element with digital ID 2 is in one row, and the elements with digital IDs 1 and 3 are in one row, below the element with digital ID 2, and so on. The arrangement details represent the arrangement styles of the elements in the same row, including single text, single picture, single table, left picture and right text, left text and right picture, two pictures in parallel, three pictures in parallel, etc. For example, the arrangement styles of [2], [1, 3], and [4, 5, 6] here can be single picture, left text and right picture, and three pictures in parallel, respectively.
[0046] After that, the quality of the typesetting is checked and corrected, the returned digital ID list is compared with the input digital ID, the generation results of the large language model are checked, the missing digital IDs are completed, and the duplicate and non-existent digital IDs are removed.
[0047] Finally, the digital id of the typesetting result is decoded. According to the digital id list generated by the large language model, each digital id is decoded into the corresponding text, table, and picture elements, and each element is arranged according to the corresponding digital id. The sub-elements corresponding to the digital id in each navigation bar of each product are arranged in combination with the layout to obtain the typeset product introduction, which is output as a json file. The json file contains several product lists in the form of [product list 1, product list 2...], and each product list contains several navigation bar dimension lists in the form of [navigation bar dimension list 1, navigation bar dimension list 2...], and each navigation bar dimension list contains several element block dictionaries in the form of {"layout":"image-text","detail":[{"type":"image","content":
[0048] "xxx"},{"type":"text","content":"xxx"}]}, each element block dictionary represents several elements arranged in a row, where layout represents the arrangement details and detail contains the element type and specific content.
[0049] Step 4: Output the module output, including:
[0050] According to the text, tables, and pictures in each navigation bar of each product after typeset layout, they are arranged in formats such as left picture and right text, left text and right picture, two pictures side by side, and three pictures side by side. The corresponding elements are drawn into visual pictures through Python libraries such as Pillow, and finally merged to obtain product introduction pages containing different dimensions for different products.
[0051] like Figure 2 As shown, an exemplary embodiment of the present invention includes:
[0052] The automated processing and parsing steps of product information based on computer vision models include processing product information input by users, using target inspection models and OCR for layout recognition and extraction of product information, and content sorting; the intelligent product information and layout integration steps based on deep learning models include using a large language model combined with an image classification model to split product multimodal information; the intelligent layout steps based on a multimodal large model include digital id encoding of each sub-element, multimodal large model understanding of semantics and typeset, post-typesetting quality inspection, and digital id encoding of typesetting results.
[0053] In the process of automatic processing and parsing of product information based on computer vision models, the documents, pictures, links, etc. input by users are converted into images through Python, and then the fine-tuned yolo-doclayout layout recognition model is used to detect the four types of elements, namely text, image, image text pair, and table. The regional images of each element are captured by the bounding box, and the PaddleOCR algorithm model is used to extract the text and table elements. Then, the elements are arranged according to the coordinates of the center point of the bounding box to obtain the parsed product information after the content is sorted. The yolo-doclayout network is trained on the layout recognition dataset. Compared with the general pre-trained yolo model, it has a stronger ability in layout recognition scenarios. On this basis, using self-annotated data for training can achieve better results and require less training time. On the basis of efficient text recognition, the PaddleOCR algorithm can also extract and output table content at the same time, which can better adapt to the table information that often appears in product information.
[0054] In the process of integrating intelligent product information and layout based on deep learning models, a large language model and a fine-tuned ResNet model are used to classify product text and image information respectively, and the product information of different products is divided into different navigation bars for dimensional classification.
[0055] In the process of intelligent layout based on multimodal big model, for each product after content classification, for each navigation bar, each sub-element is encoded to obtain a digital id, and then the multimodal big model is used to understand the semantics and layout to obtain a list of digital ids with location information and layout details. After completing the missing digital ids after layout, after removing the repeated or non-existent digital ids, each digital id is decoded into the corresponding text, table, and picture elements, and each element is arranged according to the corresponding digital id. The sub-elements corresponding to the digital id in each navigation bar of each product are arranged according to the layout to obtain the product introduction after layout.
[0056] In the output module, according to the text, table, and picture in each navigation bar of each product after typeset, the corresponding elements are arranged in formats such as left picture and right text, left text and right picture, two pictures side by side, and three pictures side by side. The corresponding elements are drawn into visual pictures through Python libraries such as Pillow, and finally merged to obtain product introduction pages containing different dimensions for different products.
[0057] The beneficial effects achieved by the present invention are:
[0058] An automated product release method based on artificial intelligence is proposed. By training the target detection model and combining it with OCR technology, efficient layout analysis and information extraction of product information can be achieved, and product information can be processed without manual data splitting. Further combined with the large language model and the image classification model, the product information is refined and multi-dimensionally classified, and the product information is automatically integrated into different navigation bars to ensure that the generated product introduction is comprehensive and accurate. Further, intelligent layout is performed through the multimodal large model, and the layout is automatically adjusted to ensure the aesthetics of the visual display. Finally, the typeset content is presented as a visual product introduction page through the output module, thereby achieving a high degree of automation of the process, reducing manual intervention, and improving the efficiency of product release and user experience.
[0059] At present, traditional product publishing methods usually rely on manual information extraction, classification and layout, which is not only inefficient but also prone to errors. Existing technologies mostly use a single text or image processing model, which cannot fully combine multimodal information, resulting in poor product display effects. At the same time, the information layout lacks intelligent adjustment, it is difficult to ensure aesthetics, and the user experience needs to be improved. With the improvement of large language models in text understanding and generation capabilities, the classification and description of product information has become more sophisticated and intelligent.
[0060] The present invention proposes an artificial intelligence-based automated product release method, which realizes efficient layout analysis and information extraction of product information by training target detection models combined with OCR technology, and automatically performs refined classification using a large language model and a trained image classification model, and then performs intelligent typesetting through a multimodal large model, automatically adjusts the layout, and ensures the beauty and rationality of the display. This method is highly automated and can greatly reduce manual intervention. It can not only improve the efficiency and accuracy of product release, but also enhance the user's shopping experience. It is an important technology for the construction of future intelligent product management systems and has broad application prospects.
[0061] The above embodiments do not limit the present invention in any way. Through the above description, relevant personnel can make various changes and modifications without departing from the technical idea of the present invention. All other improvements and applications made to the above embodiments in an equivalent manner belong to the protection scope of the present invention. The technical scope of the present invention is not limited to the contents of the specification, and its technical scope must be determined according to the scope of the claims.
Claims
1. An automated product release method based on artificial intelligence, characterized in that: The following steps are involved: Step 1: Automatic processing and analysis of product information based on computer vision models; Step 2: Intelligent product information and layout integration based on deep learning models; Step 3: Intelligent plate arrangement based on multi-modal large model; Step 4: Export the module.
2. The method for automatic product release based on artificial intelligence according to claim 1, characterized in that: In step 1, the automatic processing and analysis of commodity information based on the computer vision model specifically includes: The files, pictures, and web links containing product information input by the user are uniformly converted into picture format. For the product information in picture format, the trained target detection model is combined with OCR to perform layout recognition and extract product information. Different areas are segmented by bounding boxes and text, table, and picture information are extracted to obtain the layout of different types of product information and the parsed product information.
3. The method for automatic product release based on artificial intelligence according to claim 2, characterized in that: In step 1, the use of the trained object detection model combined with OCR to perform layout recognition and extract product information specifically includes: Use the target detection model that has been annotated and trained with actual data to detect user input converted into image form, and output bounding boxes representing the positions of text, images, image-text pairs, and tables in the image. The corresponding labels are text, image, image-text pair, and table, respectively, to obtain the overall element layout; use the bounding boxes to capture the text, image, and table areas in the input image, and output the images of each area; use the OCR model to detect and extract the text in the text and table parts, respectively, to obtain text and table information in character representation, and associate the corresponding image with the extracted text according to the bounding box of the image-text pair, and regard them as product information of the same product for subsequent classification.
4. The method for automatic product release based on artificial intelligence according to claim 3, characterized in that: In step 2, the smart product information and layout integration based on the deep learning model specifically includes: For the parsed product information, a large language model combined with an image classification model is used to split the multimodal information of the product, and the images and texts matched by image text pairs are marked during sorting. The images after dimensioning are attributed to the corresponding products according to the corresponding texts, and product introductions of different dimensions of each product are obtained. The parsed product information is divided into several products using a large language model, and different product names and corresponding company information are extracted. The large language model is then used to extract text and table information related to each product from the product information, and the extraction results are classified into different navigation bars. The sub-images in each area of the image captured after parsing are classified into different navigation bars using the trained image classification model. The information of each navigation bar is integrated to obtain product introductions of different dimensions of each product, which are output in json format.
5. The method for automatic product release based on artificial intelligence according to claim 4, characterized in that: In step 3, the intelligent plate arrangement based on the multimodal large model specifically includes: For the text, pictures, and table elements contained in each navigation bar, add digital IDs for encoding, then use the multimodal large model to understand the semantics and layout, output a digital ID list containing layout information, check and correct the quality after typesetting, compare the returned digital ID list with the input digital ID, check the generation result of the large language model, complete the missing digital IDs, and remove duplicate or non-existent digital IDs; decode the digital ID of the typesetting result, and decode each digital ID into the corresponding text, table, and picture element according to the digital ID list generated by the large language model, and arrange each element according to the corresponding digital ID; arrange the sub-elements corresponding to the digital ID in each navigation bar in combination with the layout to obtain the typeset product introduction.
6. The method for automatic product release based on artificial intelligence according to claim 5, characterized in that: In step 3, the multimodal large model understands the semantics and formats the layout, which specifically includes: For each navigation bar, the text, image, table elements, digital IDs and prompt words contained therein are provided to the large language model, and the large language model arranges the positions of each element and returns a list of digital IDs with position information and layout details.
7. The method for automatic product release based on artificial intelligence according to claim 6, characterized in that: In step 4, the output module specifically includes: According to the text, table and picture in each navigation bar of each product after typeset, they are arranged according to the corresponding typesetting detail format to obtain product introduction pages containing different dimensions of different products. The typesetting detail format includes: single text, single picture, single table, left picture and right text, left text and right picture, two pictures in parallel and three pictures in parallel.
8. The method for automatic product release based on artificial intelligence according to claim 7, characterized in that: In step 1, the documents, pictures and links input by the user are converted into picture form through the python library, and the four types of elements, namely text, image, image text pair and table, are detected using the yolo-doclayout layout recognition model. The text content of text and table is extracted using the PaddleOCR algorithm model, and then arranged according to the coordinates of the center point of the bounding box; In step 2, the image classification model is a ResNet model.
9. The method for automatic product release based on artificial intelligence according to claim 8, characterized in that: In step 1, the python libraries include Pillow, pdf2image and selenium.