A multi-model based product label auditing method, system, device and medium

Abstract

The application relates to a product label auditing method, system, device and medium based on a multi-model, wherein the method comprises the following steps: obtaining a product label picture of a preset product, wherein the product label picture comprises an image label and a text label; performing text recognition on the text label to extract label text information of the preset product; based on the image label, performing content correction on the label text information by a multi-modal model and a first large language model to obtain corrected label text information; and based on the label text information, performing compliance auditing on the product label of the preset product by a second large language model. Through the application, the label information is identified and audited by combining the product image label and the text label under the multi-model architecture of the multi-modal model and the large language model, the identification and auditing accuracy of the label information is effectively improved, and the problem of how to improve the identification and auditing accuracy of the product label information is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image recognition, in particular to a product label auditing method, system and device based on multiple models and a medium. BACKGROUND

[0002] The product label contains a lot of key information, and the accuracy and compliance of these information are directly related to the safety and health of consumers and the brand image of enterprises, so accurate identification and auditing of product labels are particularly important.

[0003] At present, the content of the labels of agricultural chemical products, disinfection and sterilization products, cosmetics, pharmaceuticals and other products are usually audited by manual methods, which is inefficient, prone to errors and unable to meet the processing needs of massive data. Although some image recognition-based auditing systems have appeared on the market, there are still problems of poor recognition accuracy and auditing errors.

[0004] At present, there is no effective solution to how to improve the identification and auditing accuracy of product label information in related technologies. SUMMARY

[0005] The embodiments of the present application provide a product label auditing method, system, device and medium based on multiple models to at least solve the problem of how to improve the identification and auditing accuracy of product label information in related technologies.

[0006] In a first aspect, the embodiments of the present application provide a product label auditing method based on multiple models, which comprises:

[0007] Obtaining a product label picture of a preset product, wherein the product label picture contains an image label and a text label;

[0008] Performing text recognition on the text label to extract label text information of the preset product;

[0009] Based on the image label, performing content correction on the label text information by a multi-modal model and a first large language model to obtain corrected label text information;

[0010] Based on the label text information, performing compliance auditing on the product label of the preset product by a second large language model.

[0011] In some embodiments, based on the image label, performing content correction on the label text information by a multi-modal model and a first large language model to obtain corrected label text information comprises:

[0012] The first large language model is used to structure the label text information, and structured label text information is obtained, wherein the label text information includes several pieces of structured label text;

[0013] The image label and the structured label text are input into a multi-modal model, and the multi-modal model is used to determine whether the structured label text has a recognition error;

[0014] In the case where the structured label text has a recognition error, the first large language model is further used to correct the content of the structured label text, so as to obtain corrected label text information.

[0015] In some embodiments, inputting the image label and the label text information into a multi-modal model and determining whether the structured label text has a recognition error by using the multi-modal model include:

[0016] The image label is subjected to feature extraction by using an image encoder in the CLIP multi-modal model, so as to obtain an image feature vector of the image label;

[0017] The structured label text is subjected to feature extraction by using a text encoder in the CLIP multi-modal model, so as to obtain a text feature vector of the structured label text;

[0018] Based on the image feature vector and the text feature vector, it is determined whether the structured label text has a recognition error.

[0019] In some embodiments, based on the image feature vector and the text feature vector, it is determined whether the label text information has a recognition error, including:

[0020] The cosine similarity between the image feature vector and the text feature vector is calculated;

[0021] Based on the cosine similarity, the structured label text related to the image feature vector is determined from the structured label text;

[0022] Based on the cosine similarity, it is determined whether the related structured label text has a recognition error.

[0023] In some embodiments, the method includes:

[0024] The image encoder in the CLIP multi-modal model is a ViT image encoder constructed based on a Vision Transformer network;

[0025] The text encoder in the CLIP multi-modal model is a BERT image encoder constructed based on a Transformer network.

[0026] In some embodiments, based on the label text information, the compliance review of the product label of the preset product by the second large language model includes:

[0027] Each structured label text in the label text information is reviewed for compliance by the second large language model according to the product label supervision regulations of the preset product.

[0028] In some embodiments, the text recognition of the character label includes extracting the label text information of the preset product.

[0029] The text recognition of the character label by an optical character recognition tool extracts the label text information of the preset product, wherein the label text information includes ingredient composition, production date, quality assurance period, producer information, operator information, place of origin, product standard number, bar code, and license information.

[0030] In a second aspect, the embodiments of the present application provide a product label review system based on multiple models, which is used to execute the method of the first aspect described above, and the system includes a data acquisition module, an identification and extraction module, a content correction module, and a review and comparison module.

[0031] The data acquisition module is configured to acquire a product label image of a preset product, wherein the product label image includes an image label and a character label.

[0032] The identification and extraction module is configured to perform text recognition on the character label to extract label text information of the preset product.

[0033] The content correction module is configured to perform content correction on the label text information by a multi-modal model and a first large language model according to the image label, to obtain corrected label text information.

[0034] The review and comparison module is configured to perform compliance review of the product label of the preset product by a second large language model according to the label text information.

[0035] In a third aspect, the embodiments of the present application provide an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method of the first aspect described above.

[0036] In a fourth aspect, the embodiments of the present application provide a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the method of the first aspect.

[0037] Compared with the related art, the method, system, device and medium for product label auditing based on multiple models provided by the embodiments of the present application, wherein the method comprises: obtaining a product label picture of a preset product, wherein the product label picture comprises an image label and a text label; performing text recognition on the text label to extract label text information of the preset product; based on the image label, performing content correction on the label text information by a multi-modal model and a first large language model to obtain corrected label text information; and based on the label text information, performing compliance auditing on the product label of the preset product by a second large language model. The embodiments of the present application realize the identification and auditing of label information by combining the product image label and the text label under the multi-model architecture of the multi-modal model and the large language model, effectively improve the identification and auditing accuracy of the label information, and solve the problem of how to improve the identification and auditing accuracy of the product label information. BRIEF DESCRIPTION OF DRAWINGS

[0038] The accompanying drawings, which are included to provide a further understanding of the present application, constitute a part of the present application and illustrate the illustrative embodiments of the present application and their description serve to explain the present application, but do not constitute improper limitations on the present application. In the drawings:

[0039] Figure 1 is a step flow chart of the method for product label auditing based on multiple models according to the embodiments of the present application;

[0040] Figure 2 is a schematic diagram of the internal structure of the electronic device according to the embodiments of the present application. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solutions and advantages of the present application more clear, the present application is described and explained in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and do not limit the present application. Based on the embodiments provided by the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of the present application.

[0042] It is apparent that the drawings in the following description merely show some examples or embodiments of the present application, and for those skilled in the art, the present application can be applied to other similar situations without creative labor based on these drawings. In addition, it can be understood that although the efforts made in this development process can be complex and lengthy, some design, manufacture or production changes based on the technical content disclosed in the present application are only routine technical means for those skilled in the art related to the disclosure of the present application, and should not be understood as insufficient disclosure of the present application.

[0043] In the present application, "embodiment" means that the specific features, structures or properties described in conjunction with the embodiment can be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment to other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in the present application can be combined with other embodiments without conflict.

[0044] Unless otherwise defined, the technical terms or scientific terms involved in the present application should be understood as the usual meaning understood by those skilled in the art in the technical field to which the present application belongs. The "one", "a", "an", "the" and similar words involved in the present application do not represent quantity limitation, which can represent singular or plural. The terms "include", "contain", "have" and any variations thereof involved in the present application are intended to cover non-exclusive inclusion; for example, the process, method, system, product or device including a series of steps or modules (units) is not limited to the listed steps or units, but can also include steps or units not listed, or can also include other steps or units inherent to these processes, methods, products or devices. The terms "connected", "connected", "coupled" and similar words involved in the present application are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. The "multiple" involved in the present application means two or more. The "and / or" describes the association between the associated objects, which means that there can be three relationships, for example, "A and / or B" can represent three cases: A exists alone, A and B exist together, and B exists alone. The character " / " generally represents that the associated objects before and after are in an "or" relationship. The terms "first", "second", "third" and the like involved in the present application are only to distinguish similar objects, and do not represent a specific order for the objects.

[0045] The embodiment of the present application provides a product label auditing method based on multiple models, Figure 1 is a step flow chart of the product label auditing method based on multiple models according to the embodiment of the present application, as shown in Figure 1As shown, the method comprises the following steps:

[0046] In step S102, a product label picture of a preset product is acquired, wherein the product label picture contains image labels and text labels.

[0047] It should be noted that the application scenarios of the product label auditing method provided in this embodiment include but are not limited to: retail logistics scenarios (such as automatically identifying and auditing product labels on store shelves), medical material management (such as automatically identifying and auditing product label information of medicines in storage), food information analysis scenarios (such as consumers wanting to know food information). As can be seen, the way to acquire the product label picture of the preset product also includes but is not limited to: being acquired by a store inspection camera, being acquired from a medical database, and being acquired by a consumer uploading.

[0048] In step S104, text recognition is performed on the text labels to extract label text information of the preset product.

[0049] In step S104, text recognition is performed on the text labels by an optical character recognition (OCR) tool to extract label text information of the preset product, wherein the label text information includes ingredient composition, production date, quality guarantee period, producer information, operator information, place of origin, product standard number, barcode, license information, etc.

[0050] It should be noted that the image labels of the product mainly refer to the patterns and designs on the product packaging. These images can include product real object pictures, brand logos, product use scenarios, etc. The label text information in the text labels should include ingredient composition, production date, quality guarantee period, producer information, operator information, place of origin, product standard number, barcode, license information, etc.

[0051] In step S106, based on the image labels, content correction is performed on the label text information by a multi-modal model and a first large language model to obtain corrected label text information.

[0052] Step S106 specifically comprises the following steps:

[0053] In step S1061, the first large language model is used to perform structured processing on the label text information to obtain structured label text information, wherein the label text information contains a plurality of structured label texts.

[0054] It should be noted that the label text information is extracted by using the OCR technology, and the recognition accuracy of the OCR is easily affected by many factors such as image quality, font layout, and text background. Moreover, the OCR generally only focuses on converting the text in the image into text, and cannot always understand the actual meaning or context information of the text, which means that even if there is an error in the text recognition, it is difficult to spontaneously correct the error by connecting the context. Therefore, in step S1061, the label text information extracted by the OCR is structured by the first large language model, and the large language model has the ability to understand the context. In other words, during the process of structured processing, the large language model can connect the context information in the label text information and preliminarily correct the recognition errors therein. It should be noted that based on the preset Prompt prompt word (such as "You are a product label parser. Please extract the following fields from the following OCR text: product name, production date, shelf life, shelf life expiration date. Requirements: only output the JSON format result, and do not miss any field."), the LLM is guided to structure the label text information extracted by the OCR, and the output structured label text information can be:

[0055] "Product Name": ["XXX Health Products"],

[0056] "Production Date": "2023-03-22",

[0057] "Shelf Life": "6 months",

[0058] "Shelf Life Expiration Date": "2023-09-22",

[0059] In step S1062, the image label and the structured label text are input into the multi-modal model, and the multi-modal model is used to determine whether the structured label text has recognition errors.

[0060] In step S1062, the image label is feature-extracted by the image encoder in the CLIP multi-modal model to obtain an image feature vector of the image label; the structured label text is feature-extracted by the text encoder in the CLIP multi-modal model to obtain a text feature vector of the structured label text; wherein the image encoder is a ViT image encoder constructed based on a VisionTransformer network, and the text encoder is a BERT image encoder constructed based on a Transformer network.

[0061] The cosine similarity between the image feature vector and the text feature vector is calculated; the structured label text related to the image feature vector is determined from the structured label text based on the cosine similarity; and it is determined whether the related structured label text has a recognition error based on the cosine similarity.

[0062] It should be noted that the CLIP (Contrastive Language-Image Pre-training) multi-modal model adopts a dual-tower structure, i.e., an image encoder and a text encoder process image and text inputs respectively and map them into a unified semantic space. Preferably, a ViT (Vision Transformer) image encoder is used as the image encoder of CLIP, which introduces the Transformer architecture into computer vision, breaks through the limitation of the local receptive field of traditional CNN, and greatly improves the feature extraction accuracy of image labels, thereby effectively improving the recognition accuracy of structured label text.

[0063] Further, since the image label of the product may not be associated with all the contents in the text label, after obtaining a plurality of pieces of structured label text in step S1061, the cosine similarity between the text feature vector of each piece of structured label text and the image feature vector of the image label is calculated in step S1062 to determine the structured label text related to the image label (e.g., the structured label text with a cosine similarity greater than a first threshold value) and then determine the structured label text with recognition error (e.g., the structured label text with a cosine similarity less than a second threshold value) from the related structured label text. By combining the image label and the text label of the product to identify and review the label information, the accuracy of the identification and review of the label information is effectively improved.

[0064] In step S1062, in the case where the structured label text has recognition error, the first large language model is used to correct the content of the structured label text to obtain the corrected label text information.

[0065] For example, in the case where the structured label text has recognition error, assuming that the structured label text is: "product name": ["XXX cat food"], and the image label shows that the product is dog food, the first large language model is used to correct the content of the structured label text in combination with the image label to obtain the corrected label text information.

[0066] In step S108, based on the label text information, the second large language model is used to perform compliance review on the product label of the preset product.

[0067] In step S108, the second large language model is used to perform compliance review on each piece of structured label text in the label text information according to the product label supervision regulations of the preset product.

[0068] Through the above steps in the embodiments of the present application, under the multi-model architecture of the multi-modal model and the large language model, the identification and review of label information are performed by combining product image labels and text labels, the identification and review accuracy of label information is effectively improved, and the problem of how to improve the identification and review accuracy of product label information is solved.

[0069] It should be noted that the steps shown in the above process or the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from here.

[0070] The embodiments of the present application provide a product label review system based on a multi-model, which comprises a data acquisition module, an identification and extraction module, a content correction module, and a review and checking module.

[0071] The data acquisition module is configured to acquire product label pictures of a preset product, wherein the product label pictures comprise image labels and text labels.

[0072] The identification and extraction module is configured to perform text recognition on the text labels to extract label text information of the preset product.

[0073] The content correction module is configured to perform content correction on the label text information by a multi-modal model and a first large language model according to the image labels, to obtain corrected label text information.

[0074] The review and checking module is configured to perform compliance review on the product label of the preset product by a second large language model according to the label text information.

[0075] Through the data acquisition module, the identification and extraction module, the content correction module, and the review and checking module in the embodiments of the present application, under the multi-model architecture of the multi-modal model and the large language model, the identification and review of label information are performed by combining product image labels and text labels, the identification and review accuracy of label information is effectively improved, and the problem of how to improve the identification and review accuracy of product label information is solved.

[0076] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can be located in the same processor; or the above modules can also be located in different processors in any combination.

[0077] The embodiment provides an electronic device, comprising a memory and a processor, the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the method embodiments.

[0078] Optionally, the electronic device can further comprise a transmission device and an input / output device, wherein the transmission device is connected with the processor, and the input / output device is connected with the processor.

[0079] Optionally, the electronic device can further comprise a processor, a memory, a network interface, a display screen and an input device connected through a system bus. The processor of the electronic device is configured to provide computing and control capabilities. The memory of the electronic device comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The network interface of the electronic device is configured to communicate with an external terminal through a network connection. The computer program is executed by the processor to implement a product label auditing method based on multiple models. The display screen of the electronic device can be a liquid crystal display screen or an electronic ink display screen. The input device of the electronic device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the shell of the electronic device, or an external keyboard, touchpad or mouse, etc.

[0080] It should be noted that the specific examples in the embodiment can refer to the examples described in the above embodiments and optional implementation manners, and the embodiment will not be described here.

[0081] In addition, in combination with the product label auditing method based on multiple models in the above embodiments, the embodiment of the application can provide a storage medium to implement. The storage medium stores a computer program; the computer program is executed by the processor to implement any of the product label auditing methods based on multiple models in the above embodiments.

[0082] In one embodiment, Figure 2 is a schematic diagram of the internal structure of an electronic device according to the embodiment of the application, as Figure 2 shown, an electronic device is provided, which can be a server, and the internal structure diagram thereof can be as Figure 2 shown. The electronic device comprises a processor, a network interface, an internal memory and a non-volatile memory connected through an internal bus, wherein the non-volatile memory stores an operating system, a computer program and a database. The processor is configured to provide computing and control capabilities, the network interface is configured to communicate with an external terminal through a network connection, the internal memory is configured to provide an environment for running the operating system and the computer program, the computer program is executed by the processor to implement a product label auditing method based on multiple models, and the database is configured to store data.

[0083] Those skilled in the art can understand that, Figure 2 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0084] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, which can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments of each method. Any reference to memory, storage, database or other medium used in each embodiment provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM) and the like.

[0085] Those skilled in the art should understand that each technical feature of the above-mentioned embodiments can be combined arbitrarily, and in order to make the description simple, not all possible combinations of each technical feature in the above-mentioned embodiments are described, however, as long as the combination of these technical features does not exist contradictory, it should be considered as the scope of the present application.

[0086] The above-mentioned embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent. It should be noted that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of the present application. Therefore, the scope of the patent of the present application should be subject to the appended claims.

Claims

1. A multi-model based product label auditing method, characterized in that, The method comprises: acquiring a product label picture of a preset product, wherein the product label picture comprises an image label and a text label; performing text recognition on the text label to extract label text information of the preset product; performing structural processing on the label text information by a first large language model to obtain structured label text information, wherein the label text information comprises a plurality of pieces of structured label text; extracting features of the image label by an image encoder in a CLIP multi-modal model to obtain an image feature vector of the image label, extracting features of the structured label text by a text encoder in the CLIP multi-modal model to obtain a text feature vector of the structured label text, and determining whether the structured label text has a recognition error based on the image feature vector and the text feature vector; in the case where the structured label text has a recognition error, performing content correction on the structured label text by the first large language model to obtain corrected label text information; performing compliance auditing on the product label of the preset product by a second large language model based on the label text information.

2. The method of claim 1, wherein, The method comprises: calculating a cosine similarity between the image feature vector and the text feature vector; determining, based on the cosine similarity, a piece of structured label text related to the image feature vector from the structured label text; determining, based on the cosine similarity, whether the piece of structured label text related to the image feature vector has a recognition error.

3. The method of claim 1, wherein, The method comprises: the image encoder in the CLIP multi-modal model is a ViT image encoder constructed based on a Vision Transformer network; the text encoder in the CLIP multi-modal model is a BERT image encoder constructed based on a Transformer network.

4. The method of claim 1, wherein, The method comprises: performing compliance auditing on each piece of structured label text in the label text information by the second large language model according to product label supervision regulations of the preset product.

5. The method of claim 1, wherein, The method comprises: performing text recognition on the text label by an optical character recognition tool to extract the label text information of the preset product, wherein the label text information comprises ingredient composition, production date, quality assurance period, producer information, operator information, place of origin, product standard number, bar code, and license information.

6. A multi-model based product label auditing system, characterized in that, The system is used to perform the method of any one of claims 1 to 5, and the system comprises a data acquisition module, an identification and extraction module, a content correction module, and an auditing and checking module; the data acquisition module is configured to acquire a product label picture of a preset product, wherein the product label picture comprises an image label and a text label; The identification extraction module is configured to perform text identification on the text label to extract label text information of the preset product; The content correction module is configured to perform content correction on the label text information by a multi-modal model and a first large language model according to the image label to obtain corrected label text information. The review and check module is configured to perform compliance review on a product label of the preset product by a second large language model according to the label text information. 7.An electronic device comprising a memory and a processor, the electronic device characterized by, The memory stores a computer program, and the processor is configured to execute the computer program to perform the method in any one of claims 1 to 5.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the method in any one of claims 1 to 5. The program is executed by the processor to implement the method in any one of claims 1 to 5.

Images