Method, device and equipment for automatically identifying and analyzing commodity information in commodity inspection

Abstract

The invention discloses a method, device and equipment for automatically identifying and analyzing commodity information in commodity inspection, and the method comprises the steps: receiving a commodity identification request which carries a commodity picture and is transmitted by inspection table equipment, and identifying an identification code which uniquely identifies a commodity in the commodity picture; matching the identification code with an identification code in a database to obtain an identification result, and sending the identification result to the inspection table equipment; receiving an automatic input request sent by the inspection table equipment for the target commodity which is not successfully identified, and identifying the commodity picture by using an OCR model to obtain text information and a text position; inputting the commodity picture, the text information and the text position into a multi-modal model, and outputting to-be-input commodity information; and sending the commodity information to a commodity information input interface of the inspection table equipment for display, and storing the commodity information of the target commodity in a database according to input confirmation operation. Therefore, the problem of low commodity information input efficiency caused by long time consumption of manually inputting commodity related information in the prior art is solved.

Description

Background Art

[0002] In the field of customs entry commodity inspection, AI identification of commodities is performed using relevant equipment at the inspection site, and the identified unique identifier of the commodity is matched with the unique identifier of the commodity in the database. If the match is successful, the commodity is inspected according to the commodity information in the database. If the match is not successful, it is determined that the commodity has not been put into storage, and the relevant information of the commodity needs to be entered into the database.

[0003] In order to input commodity information, the current method is to use OCR to recognize the text of the commodity as the description information, and other information needs to be manually input by customs officers. However, manually inputting commodity-related information takes a long time, resulting in low efficiency in commodity information input. Summary of the invention

[0004] The purpose of this application is to provide a method, device and equipment for automatically identifying and analyzing commodity information for commodity inspection, which is used to solve the problem that the existing manual entry of commodity-related information is time-consuming and leads to low efficiency in commodity information entry.

[0005] In a first aspect, an embodiment of the present application provides a method for automatically identifying and analyzing commodity information during commodity inspection, the method comprising:

[0006] Receive a commodity identification request with a commodity picture sent by the inspection station device, and identify an identification code that uniquely identifies the commodity in the commodity picture;

[0007] Matching the identification code with the identification code in the database to obtain an identification result, and sending the identification result to the inspection station device;

[0008] Receiving an automatic entry request sent by the inspection station device for a target product that has not been successfully identified, and using an OCR model to identify the product image to obtain text information and text position;

[0009] Input the product image, text information and text position into the multimodal model, and output the product information to be input;

[0010] The commodity information is sent to the commodity information entry interface of the inspection station device for display, and the commodity information of the target commodity is saved in the database according to the confirmation entry operation.

[0011] In some possible embodiments, the method further includes:

[0012] Obtain sample images, and annotation information of text information and text positions in the sample images;

[0013] The sample image is input into an OCR model, and the OCR model is trained with the goal of outputting the annotation information.

[0014] In some possible embodiments, the method further includes:

[0015] Obtaining a sample image, and annotated text information, text position, and product information to be entered in the sample image;

[0016] The sample image, the annotated text information, and the text position are input into a multimodal model, and the multimodal model is trained with the goal of outputting the annotated commodity information to be entered.

[0017] In some possible embodiments, the method further includes at least one of the following steps:

[0018] When it is determined that the commodity information to be entered includes the name of the commodity, based on a classification set obtained by pre-classifying the names of the commodities in the database, the category to which the target commodity belongs is determined and sent to the commodity information entry interface of the inspection station device for display;

[0019] When it is determined that the product information to be entered includes a brand, the brand is compared with each brand in the brand set obtained based on the brand of the product in the database for similarity, and the target brand with the highest similarity exceeding the similarity threshold is determined as the revised brand. If the similarities are all lower than the similarity threshold, the brand is not revised.

[0020] In some possible embodiments, the method further includes at least one of the following steps:

[0021] Receive the bill picture sent by the inspection station device, identify the product name and price in the bill picture, and send the identified price to the product information input interface of the inspection station device for display;

[0022] After the product information of the target product is saved in the database, the tax request sent by the inspection station device is received, the tax amount is determined by matching the product information of the target product with the tax-paid price list, and a payment code for paying the tax amount is generated and sent to the tax interface of the inspection station device for display.

[0023] In a second aspect, the present application also provides a method for automatically identifying and analyzing commodity information by commodity inspection, including:

[0024] In response to the commodity identification instruction, take a picture of the commodity, and send a commodity identification request carrying the commodity picture to a server, so that the server identifies an identification code that uniquely identifies the commodity in the commodity picture;

[0025] Receiving the identification result returned by the server after matching the identification code with the identification code in the database, and displaying the successfully identified and / or unsuccessfully identified commodities on the intelligent inspection interface;

[0026] According to the new adding operation of the target commodity that has not been successfully identified, a commodity information entry interface is displayed, and an identification code that uniquely identifies the target commodity is displayed on the commodity information entry interface;

[0027] According to the operation on the product information input interface, an automatic input request is sent to the server, and the server uses the OCR model to identify the product image to obtain text information and text position, and inputs the product image, text information and text position into the multimodal model, and outputs the product information to be input;

[0028] Receive the product information to be entered returned by the server and display it on the product information entry interface, notify the server to save the product information of the target product according to the confirmation entry operation, and update the target product to a successfully identified product on the smart inspection interface.

[0029] In some possible embodiments, in response to the bill recognition instruction, taking a photo of the bill of the target commodity that has not been successfully recognized, and sending the bill photo to the server;

[0030] The price of the target product returned by the server is received, and the price of the target product is displayed on the product information entry interface.

[0031] In some possible embodiments, after the intelligent inspection interface updates the target commodity to a commodity that has been successfully identified, the method further includes:

[0032] Switching to the tax payment interface according to the tax payment operation on the intelligent inspection interface, and sending a tax payment request to the server;

[0033] Receive the payment code for paying the corresponding tax amount returned by the server and display it on the tax payment interface.

[0034] In a third aspect, an embodiment of the present application provides a device for automatically identifying and analyzing commodity information for commodity inspection, the device comprising:

[0035] An identification code recognition module is used to receive a commodity identification request with a commodity picture sent by the inspection station device, and identify an identification code that uniquely identifies the commodity in the commodity picture;

[0036] A commodity inspection module, used to match the identification code with the identification code in the database to obtain an identification result, and send the identification result to the inspection station device;

[0037] A text recognition module is used to receive an automatic entry request sent by the inspection station device for a target product that has not been successfully identified, and use an OCR model to identify the product image to obtain text information and text position;

[0038] A product information acquisition module, used to input the product image, text information and text position into the multimodal model, and output the product information to be entered;

[0039] The commodity information input module is used to send the commodity information to the commodity information input interface of the inspection station device for display, and save the commodity information of the target commodity in the database according to the confirmation input operation.

[0040] In a fourth aspect, an embodiment of the present application further provides a device for automatically identifying and analyzing commodity information for commodity inspection, the device comprising:

[0041] A commodity identification request module, used to take a picture of a commodity in response to a commodity identification instruction, and send a commodity identification request carrying the commodity picture to a server, so that the server identifies an identification code that uniquely identifies the commodity in the commodity picture;

[0042] An identification result display module, used to receive the identification result returned by the server after matching the identification code with the identification code in the database, and display the successfully identified and / or unsuccessfully identified commodities on the intelligent inspection interface;

[0043] A commodity input and display module, configured to display a commodity information input interface according to a new addition operation on the target commodity that has not been successfully identified, and to display an identification code that uniquely identifies the target commodity on the commodity information input interface;

[0044] A product entry request module, used to send an automatic entry request to a server according to an operation on the product information entry interface, wherein the server uses an OCR model to identify the product image to obtain text information and text position, and inputs the product image, text information and text position into a multimodal model, and outputs the product information to be entered;

[0045] The product entry confirmation module is used to receive the product information to be entered returned by the server and display it on the product information entry interface, notify the server to save the product information of the target product according to the confirmation entry operation, and update the target product to a successfully identified product on the intelligent inspection interface.

[0046] In the fifth aspect, another embodiment of the present application also provides a device for automatically identifying and analyzing commodity information for commodity inspection, comprising at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any of the methods for automatically identifying and analyzing commodity information for commodity inspection provided in the embodiments of the present application.

[0047] In a sixth aspect, another embodiment of the present application further provides a computer storage medium, wherein the computer storage medium stores a computer program, and the computer program is used to enable a computer to execute any method for automatically identifying and analyzing commodity information for commodity inspection provided in an embodiment of the present application.

[0048] The method, device and equipment for automatically identifying and analyzing commodity information for commodity inspection provided in the embodiment of the present application can automatically identify commodity information by using a multimodal model after using an OCR model to identify the text information and text position of a commodity image when commodity identification based on commodity information in a database fails, and save the commodity information on a server after confirming that the entered commodity information is correct through an inspection station. This solves the problem of low efficiency in commodity information entry due to the long time consumption caused by the existing manual entry of commodity-related information.

[0049] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. Obviously, the drawings introduced below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0051] Figure 1 A schematic diagram of an application scenario according to an embodiment of the present application;

[0052] Figure 2 A flowchart of a method for automatically identifying and analyzing commodity information for commodity inspection applied to a server according to an embodiment of the present application;

[0053] Figure 3 A schematic diagram of a process for training an OCR model according to an embodiment of the present application;

[0054] Figure 4 A schematic diagram of a process for training a multimodal model according to an embodiment of the present application;

[0055] Figure 5 A schematic diagram of modifying a brand according to an embodiment of the present application;

[0056] Figure 6 A schematic diagram of a process for determining the classification of commodity names according to an embodiment of the present application;

[0057] Figure 7 This is a schematic diagram of the overall process of a method for automatically identifying and analyzing commodity information for commodity inspection according to an embodiment of the present application;

[0058] Figure 8 This is an exemplary schematic diagram of modifying a brand according to an embodiment of the present application;

[0059] Fig. 9 This is an example schematic diagram of the name classification of commodities determined according to one embodiment of the present application;

[0060] Fig.10 A schematic diagram of a process for determining a commodity price according to an embodiment of the present application;

[0061] Fig.11 A schematic diagram of a process for paying taxes on goods according to an embodiment of the present application;

[0062] Fig.12 A flowchart of a method for automatically identifying and analyzing commodity information for commodity inspection applied to an inspection station device according to an embodiment of the present application;

[0063] Fig.13 A schematic diagram of placing commodities in the inspection area according to an embodiment of the present application;

[0064] Fig.14 A schematic diagram of displaying recognition results on a smart inspection interface according to an embodiment of the present application;

[0065] Fig.15 A schematic diagram of re-photographing a product according to an embodiment of the present application;

[0066] Fig.16 This is a schematic diagram of a product information input interface according to an embodiment of the present application;

[0067] Fig.17 A flowchart of displaying prices on a product information entry interface according to an embodiment of the present application;

[0068] Fig.18 A schematic diagram of switching to a tax payment interface according to an embodiment of the present application;

[0069] Fig.19 A schematic diagram of a tax payment interface according to an embodiment of the present application;

[0070] Fig. 20 A schematic diagram of a device for automatically identifying and analyzing commodity information according to an embodiment of the present application. DETAILED DESCRIPTION

[0071] To further illustrate the technical solution provided by the embodiment of the present application, this is described in detail below in conjunction with the accompanying drawings and specific implementation methods. Although the embodiment of the present application provides the method operation steps as shown in the following embodiments or drawings, more or fewer operation steps may be included in the method based on routine or no creative labor. In the steps where there is no necessary causal relationship logically, the execution order of these steps is not limited to the execution order provided by the embodiment of the present application. The method can be executed in the order of the method shown in the embodiment or drawings or in parallel during the actual processing process or when the control device is executed.

[0072] In view of the problem that the commodity-related information currently used in the related art is manually input, which takes a long time and leads to low efficiency in commodity information input, the present application proposes a method, device and equipment for automatically identifying and analyzing commodity information during commodity inspection, which can be applied to, but not limited to, customs commodity inspection scenarios.

[0073] See also Figure 1 , is a schematic diagram of an application environment according to an embodiment of the present application. The application scenario of the embodiment of the present application is an application scenario of commodity inspection by an inspection station device, which mainly includes a front-end inspection station device 10, a server 20, and a database 30. The front-end inspection station device 10 is used to take pictures of commodities in the inspection area on the inspection station device, send the commodity pictures to the server, and the server recognizes the commodity pictures to obtain barcodes, compares the barcodes with the barcodes in the database, and if the comparison is successful, sends the commodity information corresponding to the successfully matched barcode to the inspection station device for commodity inspection. If the comparison is unsuccessful, returns commodity identification failure information to the inspection station device. In the related technology, the relevant operator enters the commodity information on the inspection station device and sends it to the database for storage.

[0074] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings.

[0075] The following is a detailed description of the method for automatically identifying and analyzing commodity information for commodity inspection in an embodiment of the present application in conjunction with the accompanying drawings.

[0076] Figure 2 A schematic diagram of a method for automatically identifying and analyzing commodity information for commodity inspection provided by an embodiment of the present application is shown. The method is applied to a server communicating with an inspection station device, and includes:

[0077] Step 201: receiving a commodity identification request with a commodity picture sent by an inspection station device, and identifying an identification code that uniquely identifies the commodity in the commodity picture.

[0078] The inspection station equipment controls the camera to take pictures of the goods according to the operations of the relevant operators, and sends the product pictures obtained by taking pictures to the server through the product identification request.

[0079] Step 202, matching the identification code with the identification code in the database to obtain an identification result, and sending the identification result to the inspection station device;

[0080] In the embodiment of the present application, the server can use the AI ​​model to identify the identification code of the product. The above identification code is used to uniquely identify the product and can be but is not limited to a barcode.

[0081] The above-mentioned product image may include multiple products, so multiple barcodes will be identified and each barcode will be compared with the barcode in the database. If the comparison is successful, it is determined that the product has been successfully identified. Otherwise, it is determined that the product has not been successfully identified. The identification result is returned to the inspection station device for display. For the successfully identified products, the corresponding product information in the database is also returned.

[0082] Step 203, receiving the automatic input request sent by the inspection station device for the target product that has not been successfully identified, and using the OCR model to identify the product image to obtain text information and text position;

[0083] After the inspection station device displays the successfully identified goods and / or unidentified goods, the inspection station device sends an automatic entry request to the server for the unidentified target goods according to the operation of the relevant operator. In the embodiment of the present application, the server uses a pre-trained OCR model to identify the product image to obtain the text information and text position of the target product. The text position can be used as coordinate information or as annotation box information.

[0084] Step 204, inputting the product image, text information and text position into the multimodal model, and outputting the product information to be input;

[0085] The above-mentioned multimodal model can be but is not limited to a LayoutLM model. The above-mentioned product information to be entered is the key information of the product, which can be but is not limited to including at least one of the brand, name and specifications of the product. As a possible implementation method, the product image, text information and text position are input into the LayoutLM model, and the brand, name and specifications of the product are output.

[0086] Step 205: Send the commodity information to the commodity information input interface of the inspection station device for display, and save the commodity information of the target commodity in the database according to the confirmation input operation.

[0087] The server sends the identified product information to the inspection station device for display. The inspection station device notifies the server to save the product information of the target product in the database based on the confirmation and entry operation of the relevant operator, thereby realizing automatic analysis and entry of the product information.

[0088] The above-mentioned automatic entry request may also be actively triggered after choosing to take a picture. After obtaining the product information, the embodiment of the present application can further check for duplication in the database based on the barcode of the product. If no duplication is found, the above-mentioned product information will be entered into the database, otherwise it will not be entered.

[0089] In some possible embodiments, Figure 3 As shown, the method also includes:

[0090] Step 301, obtaining a sample image, and the annotation information of the text information and the text position in the sample image, specifically the text information and the text position related to the outer packaging of the product in the sample image of the annotated product;

[0091] Step 302: Input the sample image into an OCR model, and train the OCR model with the goal of outputting the annotation information.

[0092] The OCR model is tuned through the above process to improve the adaptability of OCR to product text recognition.

[0093] In some possible embodiments, Figure 4 As shown, the method also includes:

[0094] Step 401, obtaining a sample image, and the text information, text position, and product information to be entered in the sample image that are marked;

[0095] Step 402: Input the sample image, the annotated text information, and the text position into a multimodal model, and train the multimodal model with the goal of outputting the annotated commodity information to be entered.

[0096] Taking the use of the LayoutLM pre-trained model to label the brand, name, and specifications of the product as an example, the named entity recognition model training is implemented through the above process. After the training is completed, the performance of the LayoutLM model is tested, focusing on checking samples with poor recognition effects, finding the reasons, and annotating the corresponding samples based on the reasons. On this basis, fine-tuning training or retraining is carried out.

[0097] After the OCR model and LayoutLM model are built, the trained OCR model can be used to identify the text and text position on the product packaging. The text, text position, and product image can be input into the LayoutLM model to analyze and extract the key information of the product: brand, name, specifications, etc.

[0098] The above OCR model cannot recognize text with 100% accuracy. Therefore, the brand information identified by the OCR model and extracted by the LayoutLM model may contain incorrect characters. It is necessary to build a brand text correction method.

[0099] In some possible embodiments, Figure 5 As shown, the method also includes:

[0100] Step 501, when it is determined that the product information to be entered includes a brand, the brand is compared with each brand in a brand set obtained based on the brand of the product in the database for similarity;

[0101] In the embodiment of the present application, the server pre-sorts all the commodity brands in the database to form a brand set, compares the commodity brand text extracted by the LayoutLM model with the brands in the brand set one by one for similarity sequences, and sets a similarity threshold.

[0102] Step 502, determine the target brand with the highest similarity exceeding the similarity threshold as the revised brand. If the similarities are all lower than the similarity threshold, the brand will not be revised, that is, the brand extracted by the LayoutLM model will be directly returned to the inspection station device.

[0103] After the product names are extracted by the LayoutLM model, the embodiment of the present application can also determine the classification of the product based on the collection of classifications of the product names pre-built based on the product information database. In some possible embodiments, for example Figure 6 As shown, the method also includes:

[0104] Step 601, when it is determined that the commodity information to be entered includes the name of the commodity, the category to which the target commodity belongs is determined based on a classification set obtained by pre-classifying the names of the commodities in the database;

[0105] Each classification set includes multiple product names, and the product name of the target product is matched one by one with the product names in each classification set. If the match is successful, the category to which the target product belongs is determined according to the successfully matched classification set.

[0106] Step 602: Send the category to which the target product belongs to the product information entry interface of the inspection station device for display.

[0107] In the above, the product name of the target product is compared with the product names in each classification set one by one in similar sequence, and a similarity threshold is also set. Among all the options that exceed the similarity threshold, the category to which the highest one belongs is selected as the product category; if all comparison results are lower than the similarity threshold, the category is not output.

[0108] like Figure 7 The figure shows an overall flow chart of the method for automatically identifying and analyzing product information during product inspection according to an embodiment of the present application. The product image of the target product that has not been successfully identified is subjected to the OCR model to obtain text information and text position, and the product image and product text (including text information and text position) are input into the LayoutLM model to identify the brand, specification and name of the target product. The brand is corrected based on the brand set and the category to which the product belongs is determined based on the category set (the above-mentioned classification set) according to the method provided in the above embodiment.

[0109] like Figure 8 As shown, a specific example of brand correction in an embodiment of the present application mainly includes: obtaining a product image, using an OCR model to recognize text information and text position, and after inputting the product image, text information and text position into a LayoutLM model, the extracted brand information is Esteo lauder; performing a similarity comparison between Esteo lauder and each brand in a brand set such as HERA, Esteelauder, etc., and determining that the similarity with Estee lauder is the highest based on the comparison result, then correcting Esteolauder to Estee lauder and outputting it.

[0110] like Fig. 9 As shown, it is a specific example of determining the category to which the target product belongs in an embodiment of the present application, which mainly includes: obtaining a product image; using the OCR model to recognize text information and text position, and after inputting the product image, text information and text position into the LayoutLM model, the extracted name is face cream; matching the extracted face cream with each name in a classification set such as cosmetics, food, etc.; determining that the face cream in the cosmetics set is successfully matched; and outputting the classification as cosmetics.

[0111] In some possible embodiments, Fig.10 As shown, the method of the embodiment of the present application also includes:

[0112] Step 1001, receiving a bill image sent by an inspection station device, and identifying the commodity name and price in the bill image;

[0113] The inspection station equipment can use a camera to take a picture of the receipt of the target product, and the server uses the OCR model to identify the product name and price in the receipt image.

[0114] Step 1002, the identified price is sent to the commodity information entry interface of the inspection station device for display.

[0115] After the server identifies the invoice image and obtains the product name and price, it will match the name extracted by the LayoutLM model with the product name on the invoice. After a successful match, it will determine that the price in the corresponding invoice is the price of the target product, and send the identified price to the product information entry interface of the inspection station device for display.

[0116] In some possible embodiments, in order to implement tax collection for unidentified target goods, such as Fig.11 As shown, the method also includes:

[0117] Step 1101, after storing the commodity information of the target commodity in the database, receiving the tax payment request sent by the inspection station device;

[0118] The commodity information of the target commodity is entered into the database according to the confirmation entry operation of the inspection station equipment, and then the relevant operator can send a tax request by operating the inspection station equipment.

[0119] Step 1102, the tax amount is determined by matching the commodity information of the target commodity with the duty-paid price list, and a payment code for paying the tax amount is generated and sent to the tax interface of the inspection station device for display.

[0120] In the embodiment of the present application, the database also stores a post-duty-paid price list, which includes relevant product information and tax information, such as tax number, tax rate, etc. The product information is used to search for the corresponding tax number, tax rate, etc. in the duty-paid price list, and the tax amount is determined using the price of the product. A payment code for paying the tax amount is generated and sent to the tax interface of the inspection station device for display.

[0121] like Fig.12 As shown, the embodiment of the present application provides a method for automatically identifying and analyzing commodity information during commodity inspection, and the method is applied to an inspection station device, comprising:

[0122] Step 1201, taking a picture of the product in response to the product identification instruction, and sending a product identification request carrying the product picture to a server, and the server identifying an identification code that uniquely identifies the product in the product picture;

[0123] like Fig.13 As shown, when inspecting goods, multiple goods are placed in the goods inspection area. The relevant operator clicks the AI ​​button on the intelligent inspection interface of the inspection station equipment to trigger the taking of photos of the goods in the inspection area. The obtained product pictures include multiple pictures, and the product pictures are sent to the server.

[0124] Step 1202, receiving the recognition result returned by the server after matching the recognition code with the recognition code in the database, and displaying the successfully recognized and / or unrecognized commodities on the intelligent inspection interface;

[0125] The recognition results obtained by the server in the above manner, including the successfully recognized commodities and / or the successfully recognized commodities, are displayed in different columns on the intelligent inspection interface, and can also be further distinguished by different colors, such as Fig.14 A schematic diagram showing the recognition results on the intelligent inspection interface, which includes two columns: recognized products and unrecognized products. For successfully recognized products, the server also returns product information to the inspection station device, which then displays the product information.

[0126] For target products that have not been successfully identified, you can also place them separately in a designated area and take another photo of them, such as Fig.15 As shown, the target product that has not been successfully identified is placed in a designated area and re-photographed, and the product image obtained by the re-photographing is sent to the server.

[0127] Step 1203: displaying a product information entry interface according to the new adding operation for the target product that has not been successfully identified, and displaying an identification code that uniquely identifies the target product on the product information entry interface;

[0128] In the embodiment of the present application, a new button is displayed on the intelligent inspection interface. The relevant operator clicks the new button to trigger a pop-up product information entry interface. The product information entry interface is as follows: Fig.16 shown.

[0129] Step 1204, according to the operation of the product information input interface, an automatic input request is sent to the server, and the server uses the OCR model to identify the product image to obtain text information and text position, and inputs the product image, text information and text position into the multimodal model, and outputs the product information to be input;

[0130] The product information entry interface in the embodiment of the present application has an OCR button. The relevant operator clicks the OCR button to send an automatic entry request to the server to determine that the product image of the newly added product needs to be automatically analyzed to obtain the product information.

[0131] The above-mentioned commodity information to be entered includes at least one item, and the name of each item and the corresponding information filling box are displayed on the commodity information entry interface.

[0132] In the commodity identification stage of the embodiment of the present application, the identification result returned by the server includes the barcode of the target commodity, and the corresponding barcode is displayed in the fill box of the commodity barcode item in the commodity information entry interface.

[0133] Step 1205, receiving the product information to be entered returned by the server, and displaying it on the product information entry interface, notifying the server to save the product information of the target product based on the confirmation entry operation, and updating the target product to a successfully identified product on the smart inspection interface.

[0134] In the embodiment of the present application, the commodity information to be entered returned by the server includes commodity name, category, brand and specification, and is automatically filled in the corresponding fill-in box. Fig.16 As shown, the product information such as product name, product category, capacity, quantity, etc. is automatically filled in.

[0135] In some possible embodiments, Fig.17 As shown, the method further comprises:

[0136] Step 1701, in response to the bill recognition instruction, taking a photo of the bill of the target commodity that has not been successfully recognized, and sending the bill photo to the server;

[0137] The product receipt includes information such as the product barcode, name, price, specifications, etc. The inspection station device in the embodiment of the present application can further take a photo of the receipt according to the operation of relevant personnel and send the receipt image to the server.

[0138] Step 1702: Receive the price of the target product returned by the server, and display the price of the target product on the product information entry interface.

[0139] The server of the embodiment of the present application identifies the product name and price of the bill image in the manner of the above embodiment, obtains the price of the target product after relevant matching, and returns the price as the product information to be entered to the inspection station device. The inspection station device fills the returned price in the fill box of the duty-paid price item on the product information entry interface.

[0140] After obtaining the above-mentioned product information on the product information entry interface, the embodiment of the present application determines that when the relevant operator clicks the OK button, it is determined that a confirmation entry operation needs to be performed and the server is notified. The server will enter the various information displayed on the product information entry interface as the product information of the target product into the database according to the notification.

[0141] In the embodiment of the present application, after the relevant operator clicks the OK button, the user returns to the smart inspection interface and updates the target product to a successfully identified product in the smart inspection interface.

[0142] like Fig.18 As shown, after the target product is updated to a successfully identified product on the intelligent inspection interface, the following further includes:

[0143] Step 1801, switching to the tax payment interface according to the tax payment operation on the intelligent inspection interface, and sending a tax payment request to the server;

[0144] After returning to the smart verification interface, the embodiment of the present application triggers switching to the tax payment interface by clicking the next button and sends a tax payment request to the server.

[0145] Step 1802, receiving the payment code for paying the corresponding tax amount returned by the server and displaying it on the tax payment interface.

[0146] When receiving a tax payment request, the server of the embodiment of the present application calculates the tax amount that the passenger should pay for the commodity according to the commodity classification, price, and specifications, and can further summarize the total tax amount to form a payment QR code to be displayed on the tax payment interface. Fig.19 shown.

[0147] Based on the same inventive concept, the present application also provides a device for automatically identifying and analyzing commodity information for commodity inspection, including:

[0148] An identification code recognition module is used to receive a commodity identification request with a commodity picture sent by the inspection station device, and identify an identification code that uniquely identifies the commodity in the commodity picture;

[0149] A commodity inspection module, used to match the identification code with the identification code in the database to obtain an identification result, and send the identification result to the inspection station device;

[0150] A text recognition module is used to receive an automatic entry request sent by the inspection station device for a target product that has not been successfully identified, and use an OCR model to identify the product image to obtain text information and text position;

[0151] A product information acquisition module, used to input the product image, text information and text position into the multimodal model, and output the product information to be entered;

[0152] The commodity information input module is used to send the commodity information to the commodity information input interface of the inspection station device for display, and save the commodity information of the target commodity in the database according to the confirmation input operation.

[0153] Based on the same inventive concept, the present application also provides a device for automatically identifying and analyzing commodity information for commodity inspection, the device comprising:

[0154] A commodity identification request module, used to take a picture of a commodity in response to a commodity identification instruction, and send a commodity identification request carrying the commodity picture to a server, so that the server identifies an identification code that uniquely identifies the commodity in the commodity picture;

[0155] An identification result display module, used to receive the identification result returned by the server after matching the identification code with the identification code in the database, and display the successfully identified and / or unsuccessfully identified commodities on the intelligent inspection interface;

[0156] A commodity input and display module, configured to display a commodity information input interface according to a new addition operation on the target commodity that has not been successfully identified, and to display an identification code that uniquely identifies the target commodity on the commodity information input interface;

[0157] A product entry request module, used to send an automatic entry request to a server according to an operation on the product information entry interface, wherein the server uses an OCR model to identify the product image to obtain text information and text position, and inputs the product image, text information and text position into a multimodal model, and outputs the product information to be entered;

[0158] The product entry confirmation module is used to receive the product information to be entered returned by the server and display it on the product information entry interface, notify the server to save the product information of the target product according to the confirmation entry operation, and update the target product to a successfully identified product on the intelligent inspection interface.

[0159] After introducing the method and apparatus for automatically identifying and analyzing commodity information for commodity inspection according to an exemplary embodiment of the present application, next, an apparatus for automatically identifying and analyzing commodity information for commodity inspection according to another exemplary embodiment of the present application will be introduced.

[0160] Those skilled in the art will appreciate that various aspects of the present application may be implemented as a system, method or program product. Therefore, various aspects of the present application may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to as "circuit", "module" or "system" herein.

[0161] In some possible implementations, the device for automatically identifying and analyzing commodity information for commodity inspection according to the present application may include at least one processor and at least one memory. The memory stores program code, and when the program code is executed by the processor, the processor executes the steps of the method for automatically identifying and analyzing commodity information for commodity inspection according to various exemplary implementations of the present application described above in this specification.

[0162] Refer to the following Fig. 20 To describe the device 200 for automatically identifying and analyzing commodity information according to this embodiment of the present application. Fig. 20The device 200 for automatically identifying and analyzing commodity information for commodity inspection is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0163] like Fig. 20 As shown, the device 200 for automatically identifying and analyzing commodity information for commodity inspection is presented in the form of a general electronic device. The components of the device 200 for automatically identifying and analyzing commodity information for commodity inspection may include, but are not limited to: at least one processor 201, at least one memory 202, and a bus 203 connecting different system components (including the memory 202 and the processor 201).

[0164] Bus 203 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a processor, or a local bus using any of a variety of bus architectures.

[0165] The memory 202 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 2021 and / or a cache memory 2022 , and may further include a read-only memory (ROM) 2023 .

[0166] The memory 202 may also include a program / utility 2025 having a set (at least one) of program modules 2024, such program modules 2024 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0167] The device 200 for automatically identifying and analyzing commodity information for commodity inspection can also communicate with one or more external devices 204 (e.g., keyboards, pointing devices, etc.), and can also communicate with one or more devices that enable users to interact with the device 200 for automatically identifying and analyzing commodity information for commodity inspection, and / or communicate with any device (e.g., router, modem, etc.) that enables the device 200 for automatically identifying and analyzing commodity information for commodity inspection to communicate with one or more other electronic devices. Such communication can be performed through an input / output (I / O) interface 205. In addition, the device 200 for automatically identifying and analyzing commodity information for commodity inspection can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN) and / or public network, such as the Internet) through a network adapter 206. As shown in the figure, the network adapter 206 communicates with other modules of the device 200 for automatically identifying and analyzing commodity information for commodity inspection through a bus 203. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the device 200 for automatically identifying and analyzing commodity information during commodity inspection, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0168] In some possible implementations, various aspects of the method for automatically identifying and analyzing commodity information for commodity inspection provided by the present application may also be implemented in the form of a program product, which includes program code. When the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of the method for automatically identifying and analyzing commodity information for commodity inspection according to various exemplary implementations of the present application described above in this specification.

[0169] The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0170] The program product for automatically identifying and analyzing commodity information for commodity inspection of the embodiment of the present application can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on an electronic device. However, the program product of the present application is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, device or device.

[0171] The readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, wherein the readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0172] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0173] The program code for performing the operations of the present application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and also conventional procedural programming languages ​​such as "C" language or similar programming languages. The program code can be executed entirely on the user electronic device, partially on the user device, as an independent software package, partially on the user electronic device and partially on the remote electronic device, or entirely on the remote electronic device or server. In the case of a remote electronic device, the remote electronic device can be connected to the user electronic device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external electronic device (for example, using an Internet service provider to connect through the Internet).

[0174] It should be noted that, although several units or subunits of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into multiple units to be embodied.

[0175] In addition, although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0176] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0177] The present application is described with reference to the flowcharts and block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and block diagram, as well as the combination of the processes and boxes in the flowchart and block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and block diagram. Figure 1 Process or multiple processes and boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0178] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 Process or multiple processes and boxes Figure 1 A function specified in one or more boxes.

[0179] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 Process or multiple processes and boxes Figure 1 A step that specifies a function in one or more boxes.

[0180] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0181] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for automatically identifying and analyzing commodity information during commodity inspection, characterized in that: The method comprises: Receive a commodity identification request with a commodity picture sent by the inspection station device, and identify an identification code that uniquely identifies the commodity in the commodity picture; Matching the identification code with the identification code in the database to obtain an identification result, and sending the identification result to the inspection station device; Receiving an automatic entry request sent by the inspection station device for a target product that has not been successfully identified, and using an OCR model to identify the product image to obtain text information and text position; Input the product image, text information and text position into the multimodal model, and output the product information to be input; The commodity information is sent to the commodity information entry interface of the inspection station device for display, and the commodity information of the target commodity is saved in the database according to the confirmation entry operation.

2. The method according to claim 1, characterized in that Also includes: Obtain sample images, and annotation information of text information and text positions in the sample images; The sample image is input into an OCR model, and the OCR model is trained with the goal of outputting the annotation information.

3. The method according to claim 1, characterized in that Also includes: Obtaining a sample image, and the text information, text position, and product information to be entered in the sample image that are annotated; The sample image, the annotated text information, and the text position are input into a multimodal model, and the multimodal model is trained with the goal of outputting the annotated commodity information to be entered.

4. The method according to claim 1, characterized in that: It also includes at least one of the following steps: When it is determined that the commodity information to be entered includes the name of the commodity, based on a classification set obtained by pre-classifying the names of the commodities in the database, the category to which the target commodity belongs is determined and sent to the commodity information entry interface of the inspection station device for display; When it is determined that the product information to be entered includes a brand, the brand is compared with each brand in the brand set obtained based on the brand of the product in the database for similarity, and the target brand with the highest similarity exceeding the similarity threshold is determined as the revised brand. If the similarities are all lower than the similarity threshold, the brand is not revised.

5. The method according to any one of claims 1 to 4, characterized in that: It also includes at least one of the following steps: Receive the bill picture sent by the inspection station device, identify the product name and price in the bill picture, and send the identified price to the product information input interface of the inspection station device for display; After the product information of the target product is saved in the database, the tax request sent by the inspection station device is received, the tax amount is determined by matching the product information of the target product with the tax-paid price list, and a payment code for paying the tax amount is generated and sent to the tax interface of the inspection station device for display.

6. A method for automatically identifying and analyzing commodity information during commodity inspection, characterized in that: include: In response to the commodity identification instruction, take a picture of the commodity, and send a commodity identification request carrying the commodity picture to a server, so that the server identifies an identification code that uniquely identifies the commodity in the commodity picture; Receiving the identification result returned by the server after matching the identification code with the identification code in the database, and displaying the successfully identified and / or unsuccessfully identified commodities on the intelligent inspection interface; According to the new adding operation of the target commodity that has not been successfully identified, a commodity information entry interface is displayed, and an identification code that uniquely identifies the target commodity is displayed on the commodity information entry interface; According to the operation on the product information input interface, an automatic input request is sent to the server, and the server uses the OCR model to identify the product image to obtain text information and text position, and inputs the product image, text information and text position into the multimodal model, and outputs the product information to be input; Receive the product information to be entered returned by the server and display it on the product information entry interface, notify the server to save the product information of the target product according to the confirmation entry operation, and update the target product to a successfully identified product on the smart inspection interface.

7. The method according to claim 6, characterized in that In response to the bill recognition instruction, taking a photo of the bill of the target commodity that has not been successfully recognized, and sending the bill photo to the server; The price of the target product returned by the server is received, and the price of the target product is displayed on the product information entry interface.

8. The method according to claim 6, characterized in that After the target product is updated to a successfully identified product on the intelligent inspection interface, the following steps are further included: Switching to the tax payment interface according to the tax payment operation on the intelligent inspection interface, and sending a tax payment request to the server; Receive the payment code for paying the corresponding tax amount returned by the server and display it on the tax payment interface.

9. A device for automatically identifying and analyzing commodity information for commodity inspection, characterized in that: It comprises at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method as described in any one of claims 1-8.

10. A computer storage medium, characterized in that: The computer storage medium stores a computer program, and the computer program is used to enable a computer to execute the method according to any one of claims 1 to 8.

Images