Database updating method and device based on bill identification and commodity matching

Abstract

The invention provides a database updating method and device based on bill recognition and commodity matching, and the method comprises the steps: obtaining commodity information obtained through the recognition of a commodity, and obtaining the successfully matched target commodity information in a database according to the matching result of the commodity information and the commodity information in the database; acquiring bill information obtained by identifying the bill, and extracting bill key information from the bill information; the target commodity information is matched with the bill key information, and the target commodity information in the database is updated by utilizing the bill key information when it is determined that matching succeeds according to the matching result of the target commodity information and the bill key information. The commodity information in the database can be updated in real time, the commodity information in the database can be newly added, the commodity checking efficiency is improved, and the method has important significance for checking of customs personnel.

Description

Background Art

[0002] Customs data refers to the import and export statistics generated by customs in the performance of import and export trade statistics. The data is specific to the specific import and export situation of commodities of each enterprise and each customs port. At present, customs data is divided by product and code and updated monthly. The commodity inspection method is mainly to identify the commodity information through manual experience, and the price of commodities can not be updated in real time through online search or manual experience. Summary of the invention

[0003] The present application proposes a database updating method and device based on bill recognition and commodity matching, which can update the commodity information in the database in real time and add new commodity information in the database, thereby improving the efficiency of commodity inspection, which is of great significance to customs officers' inspection.

[0004] In a first aspect, the present application proposes a database updating method based on bill recognition and commodity matching, comprising:

[0005] Acquire commodity information obtained by identifying the commodity, and obtain target commodity information successfully matched in the database according to the matching result between the commodity information and the commodity information in the database;

[0006] After obtaining the bill information obtained by identifying the bill, extracting the bill key information from the bill information;

[0007] According to the matching result between the target commodity information and the key information of the bill, when it is determined that the match is successful, the key information of the bill is used to update the target commodity information in the database.

[0008] In some possible embodiments, it further includes:

[0009] When it is determined that no target product information successfully matched in the database is obtained according to the matching result between the product information and the product information in the database, the product information in the database is added using the key information of the ticket according to the user's instruction;

[0010] According to the matching result between the target commodity information and the key information of the bill, if it is determined that the match is not successful, the target commodity information in the database is updated using the key information of the bill according to the user's instruction.

[0011] In some possible embodiments, after extracting the key bill information from the bill information, the method further includes:

[0012] Determine the number of the first product from the identified product information;

[0013] Determine the number of the second commodity from the bill key information / bill information;

[0014] Compare the number of the first product with the number of the second product;

[0015] When the number of the first commodities is equal to the number of the second commodities, matching the target commodity information with the key information of the bill;

[0016] When the number of the first commodities is greater than the number of the second commodities, the commodity information is added to the database according to the user's instruction using the commodity information;

[0017] When the number of the first commodities is less than the number of the second commodities, commodity information is newly added to the database using the key information of the bill.

[0018] In some possible embodiments, the step of obtaining commodity information obtained by identifying the commodity includes:

[0019] Obtaining product photos obtained by taking photos of the products;

[0020] Using a rotating object detection model to perform product detection on the product photos, obtaining a single image of each product and cropping it;

[0021] Using a rotating target detection model to perform barcode detection on a single image of each commodity, respectively obtaining and cropping a barcode image of each commodity;

[0022] The width of each vertical line in the barcode image of each product is analyzed to obtain the product information.

[0023] In some possible embodiments, the step of obtaining the bill information obtained by identifying the bill includes:

[0024] Obtain a photo of the bill obtained by taking a photo of the bill;

[0025] Use optical character recognition (OCR) to perform text recognition on bill photos to obtain text bill information and corresponding location information;

[0026] Use the non-text recognition model to detect non-text information on the bill photo to obtain non-text bill information and corresponding location information;

[0027] According to the characteristics of different types of privacy information, the corresponding privacy information and the corresponding location information are detected from the obtained text / non-text ticket information and the corresponding location information;

[0028] The privacy information in the text / non-text ticket information is erased to obtain the ticket information.

[0029] In some possible embodiments, the key bill information extracted from the bill information includes:

[0030] Input the bill information and questions about the key bill information into the multimodal big model API to obtain the multimodal big model API answer;

[0031] The multimodal large model API answers are analyzed and screened to obtain the key information of the bill.

[0032] In some possible embodiments, matching the target commodity information with the bill key information includes:

[0033] According to different types of information respectively contained in the bill key information and the target product information, for the same type of information, corresponding sequences are respectively selected from the bill key information and the target product information;

[0034] According to the string similarity matching method, the similarity of two sequences corresponding to the same type of information is calculated;

[0035] According to the calculated similarity, it is determined whether the target product information and the bill key information are matched successfully.

[0036] In some possible embodiments, the similarity between two sequences corresponding to the same type of information is calculated in the following manner:

[0037] in:

[0038] Wherein, similarity(a, b) represents similarity, |a| and |b| represent the lengths of sequence a and sequence b corresponding to the same type of information, respectively, and |c| represents a matching sequence, which is the longest continuous subsequence existing in both sequences corresponding to the same type of information.

[0039] In the second aspect, the present application also proposes a database update device based on bill recognition and commodity intelligent matching, comprising at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a database update method based on bill recognition and commodity intelligent matching as described in the first aspect above.

[0040] In a third aspect, the present application also proposes a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are used to enable the computer to execute a database update method based on bill recognition and commodity intelligent matching as described in the first aspect above.

[0041] The present application provides a database update method and device based on bill identification and commodity matching, which obtains commodity information obtained by identifying commodities, and obtains target commodity information successfully matched in the database according to the matching result between the commodity information and the commodity information in the database; after obtaining the bill information obtained by identifying the bill, extracts the bill key information from the bill information; according to the matching result between the target commodity information and the bill key information, when it is determined that the match is successful, updates the target commodity information in the database using the bill key information. This solves the problem that the existing commodity price cannot be updated in real time by searching the Internet or judging the commodity price by manual experience, improves the efficiency of commodity inspection, and is of great significance to the inspection of customs officers.

[0042] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] To further illustrate the technical solution provided by the embodiments of the present application, a detailed description is given below in conjunction with the accompanying drawings and specific implementation methods.

[0044] Figure 1 A schematic diagram of an application environment in an embodiment of the present application;

[0045] Figure 2 This is a database update flow chart in an embodiment of the present application;

[0046] Figure 3 This is a schematic diagram of a newly added page in an embodiment of the present application;

[0047] Figure 4 This is a flowchart of obtaining product information in an embodiment of the present application;

[0048] Figure 5 A schematic diagram comparing two detection algorithms in the embodiments of the present application;

[0049] Figure 6 This is a schematic diagram of an interactive interface in which a successful match is made in an embodiment of the present application;

[0050] Figure 7 This is a schematic diagram of an interaction interface that fails to match in an embodiment of the present application;

[0051] Figure 8 This is a flowchart for obtaining bill information in an embodiment of the present application;

[0052] Fig. 9 This is a flowchart for obtaining text-type bill information in an embodiment of the present application;

[0053] Fig.10 This is a flowchart of obtaining non-text bill information in an embodiment of the present application;

[0054] Fig.11 This is a flow chart of erasing private information in an embodiment of the present application;

[0055] Fig.12 This is a schematic diagram of a bill including private information in an embodiment of the present application;

[0056] Fig.13 This is a flow chart of extracting key information of a bill in an embodiment of the present application;

[0057] Fig.14 This is a schematic diagram of a bill including commodity information in an embodiment of the present application;

[0058] Fig.15 This is a flow chart of matching target commodity information with key bill information in an embodiment of the present application;

[0059] Fig.16 This is a specific threshold matching flow chart in the embodiment of the present application;

[0060] Fig.17 A schematic diagram of a database updating device based on bill recognition and commodity matching in an embodiment of the present application;

[0061] Fig.18 This is a schematic diagram of a database updating device based on bill recognition and commodity matching in an embodiment of the present application. DETAILED DESCRIPTION

[0062] The technical solutions in the embodiments of the present application will be described clearly and in detail below in conjunction with the accompanying drawings. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present invention, and are not intended to limit the scope of the present invention in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0063] At the customs port, it is often necessary to inspect and value the items carried by passengers. The previous method of commodity inspection was mainly to identify commodity information through manual experience, and to determine the price of commodities through online search or manual experience. The relevant technology provides customs with a customs intelligent inspection station based on intelligent recognition algorithms to improve the efficiency of commodity inspection.

[0064] The effectiveness of on-site customs inspections using smart inspection stations depends on the types and quantity of data in the database. In particular, if the goods inspected on-site can be updated or added to the supervision station database in a timely manner, it will not only enrich the number of goods in the database in a targeted manner, but also update the latest prices of the goods, providing customs officers with more accurate commodity prices, thereby reducing commodity valuation time and improving commodity inspection efficiency.

[0065] The shopping receipt is the direct proof of the passenger's purchase of goods. Customs officers will take the receipt as the first priority for passengers to provide purchase price certification. At the same time, the price of the receipt also reflects the latest price of the product and has the highest price reference significance. Therefore, the embodiment of the present application identifies the product information of the receipt to complete the update and addition of the database, which is of great significance to the inspection of customs officers.

[0066] Since the current customs inspection method is mainly to identify commodity information through manual experience, and to determine commodity prices through network search or manual experience, it is impossible to achieve real-time update. This application proposes a database update method and device based on bill recognition and commodity matching.

[0067] See also Figure 1 , is a schematic diagram of an application environment according to an embodiment of the present application. The application scenario of the embodiment of the present application is an application scenario of a supervision station inspecting goods, which mainly includes an intelligent inspection station 101, an algorithm server 102 and a database 103. The above-mentioned intelligent inspection station is connected to an RGB camera. The intelligent inspection station 101 is used to take pictures of the goods to be inspected on the supervision station, identify the images obtained by the pictures to obtain the goods information, and match the identified goods information with the goods information in the database.

[0068] In this application, the intelligent inspection station uses an RGB camera to take pictures of the goods and bills respectively, uses the intelligent inspection station / algorithm server to extract the bill information and match it with the commodity information in the database, and updates and adds to the database according to the size relationship between the similarity and the threshold, providing a new database update method. It mainly obtains the commodity information obtained by identifying the goods, and obtains the target commodity information that successfully matches in the database according to the matching results of the commodity information and the commodity information in the database; after obtaining the bill information obtained by identifying the bill, extract the bill key information from the bill information; according to the matching results of the target commodity information and the bill key information, when it is determined that the match is successful, the target commodity information in the database is updated using the bill key information. To solve the problem that the current way of inspecting goods by customs is mainly to identify the information of the goods through manual experience, and to determine the price of the goods through network search or manual experience, and real-time update cannot be achieved. This application can update the commodity information in the database in real time and add the commodity information in the database, improve the efficiency of commodity inspection, and is of great significance to the inspection of customs officers.

[0069] A database updating method and device based on bill recognition and commodity matching in an embodiment of the present application are described in detail below with reference to the accompanying drawings.

[0070] The present application embodiment provides a database update method based on bill recognition and commodity matching, and its process is as follows: Figure 2 As shown, including:

[0071] Step 201, obtaining commodity information obtained by identifying the commodity, and obtaining target commodity information successfully matched in the database according to the matching result between the commodity information and the commodity information in the database;

[0072] In the embodiment of the present application, when the front-end interactive interface of the front-end supervision device detects that a user clicks the AI ​​photo button on the front-end interactive interface, the camera will be turned on to take a photo, and the taken photo of the product will be transmitted to the AI ​​algorithm for processing to identify the product information; when the front-end interactive interface detects that a user clicks the receipt photo button on the front-end interactive interface, the receipt will be photographed to identify the product information in the receipt.

[0073] In the embodiment of the present application, the product information obtained by identifying the product photo is matched with the product information of different products in the database. If the product information of a certain product is successfully matched, the product information is determined to be the target product information.

[0074] Step 202, after acquiring the bill information obtained by identifying the bill, extracting the bill key information from the bill information;

[0075] The above-mentioned key information of the bill is the characteristic information of the product, which may include one or more pieces of information. In the embodiment of the present application, the key information of the bill may include but is not limited to the name of the product.

[0076] Step 203, based on the matching result between the target product information and the key information of the bill, when it is determined that the match is successful, the target product information in the database is updated using the key information of the bill.

[0077] In the embodiment of the present application, the target product information is compared with the key information of the bill. Specifically, but not limited to, the text of the "product name" of the target product information is compared with the text of the "product name" of the key information of the bill. If the sequence matching degree of the two exceeds the threshold, it is considered that the target product information and the key information of the bill are successfully matched. The product in the key information of the bill corresponds to the product in the target product information. The two will be automatically associated, and the target product information in the database is updated using the key information of the bill. Specifically, the "product name" in the target product information is updated to the "product name" of the product in the key information of the bill, and the "price" in the target product information is updated to the "price" of the product in the key information of the bill. The real-time update of the product information in the database is achieved, and the efficiency of product inspection is improved.

[0078] In some possible embodiments, it further includes:

[0079] According to the matching result between the commodity information and the commodity information in the database, when it is determined that the target commodity information successfully matched in the database is not obtained, the commodity information in the database is added using the key information of the bill according to the user's instruction.

[0080] In the embodiment of the present application, at this time, after the commodity information is identified, if the target commodity information matching the identified commodity information is not found in the database, it can be considered that the target commodity information is empty, and the two need to be manually associated, and the "commodity name", "quantity" and "price" of this commodity in the bill key information are added to the corresponding "commodity name", "quantity" and "price" fields in the target commodity information in the database, and the commodity information in the database is added using the bill key information. The commodity information in the database can be added to provide more accurate commodity prices for customs officers, thereby reducing commodity valuation time and improving commodity inspection efficiency.

[0081] In some possible embodiments, the method further includes: if it is determined that no match is successful based on the matching result between the target product information and the ticket key information, the target product information in the database is updated using the ticket key information according to the user's instruction.

[0082] In the embodiment of the present application, the text of the "product name" of the target product information is sequenced with the text of the "product name" of the bill key information, and the sequence matching degree of the two is less than the threshold. At this time, if the bill key information is inconsistent with the target product information, it is possible that either one of them is incorrect, and manual inspection and manual association of the two are required to update the "product name" in the target product information to the "product name" of this product in the bill key information, update the "price" in the target product information to the "price" of this product in the bill key information, and update the "quantity" in the target product information to the "quantity" of this product in the bill key information, and complete the update of the target product information in the database using the bill key information.

[0083] In some possible embodiments, after extracting the key bill information from the bill information, the method further includes:

[0084] Determine the number of the first product from the identified product information;

[0085] Determine the number of the second commodity from the bill key information / bill information;

[0086] Compare the number of the first product with the number of the second product;

[0087] When the number of the first commodities is equal to the number of the second commodities, matching the target commodity information with the key information of the bill;

[0088] When the number of the first commodities is greater than the number of the second commodities, the commodity information is added to the database according to the user's instruction using the commodity information;

[0089] When the number of the first commodities is less than the number of the second commodities, commodity information is newly added to the database using the key information of the bill.

[0090] The embodiment of the present application mainly aims at the situation that the commodity information in the database does not correspond to the key information / invoice information one by one, and adds the target commodity information in the database. When the identified commodity information is consistent with the number of commodities in the key information of the bill, the target commodity information is matched with the key information of the bill, and the method of the embodiment of the present application is performed to update the database by the matching method. For the situation that the identified commodity information is inconsistent with the number of commodities in the key information of the bill:

[0091] When the number of the first commodity is greater than the number of the second commodity, there is redundant commodity information, that is, for commodity information that is not included in the bill key information / bill information and has not been successfully matched with the target commodity information in the database, a barcode scanner is used to obtain the barcode (if there is no barcode, click the default barcode button), click the "OCR" button, take a front picture, obtain the front picture of the commodity, commodity name, category and capacity and other information, and finally the user fills in the tax-paid price table according to the actual price checked, clicks the OK button to complete the addition of data, and the schematic diagram of the new page is as follows Figure 3 shown.

[0092] When the number of the first commodities is less than the number of the second commodities, there is redundant bill information / bill key information, and the bill key information is directly used to add new commodity information in the database, and the addition is performed through a new page in the interactive interface.

[0093] In a possible embodiment, the adding of product information in the database using the product information according to the user instruction includes:

[0094] Get the product image, barcode, product name and other information of the product, and add the product.

[0095] In a possible embodiment, using the bill key information to add commodity information to the database includes: automatically filling the bill key information / the redundant bill key information / the bill information in the bill information into the adding interface to complete the adding.

[0096] In some possible embodiments, the process of obtaining the commodity information obtained by identifying the commodity is as follows: Figure 4 As shown, including:

[0097] Step 401, obtaining a product photo obtained by taking a photo of the product;

[0098] Step 402, using the rotating target detection model yolov5-obb algorithm model to perform commodity detection on the commodity photos, and obtaining and cropping a single image of each commodity;

[0099] Step 403, using the rotating target detection model yolov5-obb algorithm model to perform barcode detection on the single image of each commodity, respectively obtaining the barcode image of each commodity and cropping it;

[0100] Step 404 , performing width analysis on each vertical line in the barcode image of each product to obtain information about each product.

[0101] Since goods may be placed at any angle, the traditional rectangular frame-based detection algorithm is not applicable. Therefore, to address the problem of arbitrary angle placement, a rotating target detection algorithm model in the field of aerial images is introduced. Rotating target detection is to mark targets placed at any angle in the image through a computational vision algorithm. General target detection algorithms use a rectangular frame represented by four values ​​[x, y, w, h] to mark a target; and in order to handle arbitrary angles, rotating target detection algorithms usually use a rotating rectangular frame represented by five values ​​[x, y, w, h, \theta] or eight values ​​[x1, y1, x2, y2, x3, y3, x4, y4] to mark a target. Figure 5 A comparison of two detection algorithms is shown.

[0102] The yolov5-obb model is used, that is, the yolov5 model is used as the backbone network to extract the target features, and the rotation target detection prediction head structure is used as the classifier and regressor. Based on the compromise between speed and performance, this project uses the yolov5-m model as the backbone structure. The rotation target detection prediction head structure adds angle regression on the basis of traditional position regression, and the angle value range is [0,179] with a total of 180 categories.

[0103] Since the barcode image of each product is unique, the AI ​​recognition function uses the method of detecting and identifying the barcode image of each product to identify the product.

[0104] In the embodiment of the present application, the barcode parsing algorithm is used to parse the width of each vertical line in the barcode image of each commodity. However, the barcode parsing algorithm cannot directly identify the number below the barcode, but according to the composition rules of the barcode, the width of each vertical line in the barcode is parsed into a corresponding number.

[0105] In some possible embodiments, the product information is matched with the product information in the database to obtain successfully matched target product information.

[0106] Place the product with the barcode facing upwards and use the AI ​​recognition function. The product photo taken by the camera will be transmitted to the algorithm server. After being processed by the algorithm, the barcode of the product will be identified and matched with the barcode in the product database. If the barcode can be found in the data, the recognition is successful; otherwise, the recognition fails.

[0107] The barcode number parsed by the AI ​​algorithm will be matched with the barcode field in the database. If the match is successful, the product information in the database will be displayed as the target product information in the interactive interface, such as Figure 6 As shown; if the match fails (there is no product data in the database), the corresponding field is empty, as shown below Figure 7 shown.

[0108] In some possible embodiments, the process of obtaining the bill information obtained by identifying the bill is as follows: Figure 8 As shown, including:

[0109] Step 801, obtaining a bill photo obtained by taking a photo of the bill;

[0110] Step 802, using optical character recognition (OCR) to perform text recognition on the bill photo to obtain text bill information and corresponding position information;

[0111] Step 803, using a non-text recognition model to detect non-text information on the bill photo, and obtaining non-text bill information and corresponding position information;

[0112] Step 804, detecting corresponding privacy information and corresponding location information from the acquired text / non-text ticket information and corresponding location information according to the characteristics of different types of privacy information;

[0113] Step 805, erasing the private information in the text / non-text ticket information to obtain the ticket information;

[0114] Specifically, Fig. 9 As shown, obtain text-based ticket information, including:

[0115] Step 901, obtaining a bill photo obtained by taking a photo of the bill;

[0116] Step 902, using optical character recognition (OCR) to perform text recognition on the bill photo;

[0117] Step 903, obtaining text-type ticket information and corresponding location information;

[0118] Specifically, Fig.10 As shown, obtain non-text ticket information, including:

[0119] Step 1001, obtaining a bill photo obtained by taking a photo of the bill;

[0120] Step 1002: Use the barcode / QR code model to detect non-text information on the bill photo:

[0121] Step 1003, obtaining non-text ticket information and corresponding location information;

[0122] Specifically, Fig.11 As shown, the private information is erased and the ticket information is obtained, including:

[0123] Step 1101, obtaining text and non-text ticket information and corresponding position information;

[0124] Step 1102, detecting corresponding private information and corresponding location information according to the characteristics of different types of private information;

[0125] Step 1103, erasing the privacy information in the text / non-text ticket information;

[0126] Step 1104, obtaining ticket information;

[0127] In the embodiment of the present application, the characteristics of the different types of privacy information include: passport number, name, delivery address, telephone number and other text information (the relevant text fields are gradually accumulated and enriched as receipt samples are collected) and image information such as QR codes and bar codes.

[0128] by Fig.12 For example, the receipt contains the consumer's passport number, name and barcode, which involves personal privacy. Therefore, before sending the receipt information to the big model API, detect and erase the private information. Compare the receipt OCR text one by one, detect the text containing fields such as passport no. or passport name, and use the text in the same or next line of the detected text as private text; use the non-text recognition model to detect the location of the barcode, erase the image in the private information location box, and generate the receipt information after privacy erasure.

[0129] In some possible embodiments, the process of extracting the key information of the bill from the bill information is as follows: Fig.13 As shown, including:

[0130] Step 1301, input the bill information and questions about the key bill information into the multimodal big model API;

[0131] Step 1302, obtaining a multimodal large model API answer;

[0132] Step 1303, analyzing and screening the multimodal large model API answers;

[0133] Step 1304, obtaining the key information of the bill.

[0134] In this embodiment, the questions about the key information of the bill include but are not limited to the following questions:

[0135] What is the name of the commodity in this note?

[0136] What is the quantity of merchandise in this bill?

[0137] What is the unit price of the goods in this bill?

[0138] What is the price of the goods in this note?

[0139] Output the results in table format.

[0140] By inputting the bill information and asking questions about the key information of the bill into the multimodal large model API, you can get a relatively accurate answer. However, the answers of the multimodal large model API often contain a lot of redundant information, so character analysis is required to eliminate the redundant information. For example, if you ask "What is the name of the product on this receipt, how much is it, how much is the unit price, and how much is the price, please output it in the form of a table." The answer output by the multimodal large model API uses the "|" symbol to distinguish the table, and you can filter out the key information of the product by looking for the "|" position. Fig.14 The answer output by the multimodal large model API of the ticket shown is:

[0141] |Product Name|Quantity|Unit Price|Price|

[0142] |YSL NAKE EDP LE CUSN 2020 10FG CN|1|670|629|

[0143] |NARS MAKE LR SETTING PD-2019PKG|1|370|347|

[0144] |TFORD B MAKE GLOSS LUXE-22SUNRISE PINK|1|510|479|

[0145] |GIVENCHY MAKE LE ROUGE SHEER VELVET N16|1|360|338|

[0146] |E.LAUDER MAKE DW SIP MU DUO BONE|1|790|740|

[0147] |E.LAUDER MAKE PC L / S BOISDE ROSE|2|315|595|

[0148] In this embodiment, a bill recognition method based on a combination of a small model and a large model is used to perform key extraction of bill information.

[0149] See also Fig.15 In some possible embodiments, matching the target commodity information with the bill key information includes:

[0150] Step 1501, according to different types of information respectively included in the bill key information and the target product information, for the same type of information, corresponding sequences are selected from the bill key information and the target product information respectively;

[0151] Step 1502, calculating the similarity of two sequences corresponding to the same type of information according to a string similarity matching method;

[0152] Step 1503: Determine whether the target product information and the bill key information are matched successfully based on the calculated similarity.

[0153] For the sequence alignment of the same type of "product name" text information, the "string similarity" matching method is used, which is to find the longest common subsequence in the two sequences, thereby breaking the problem into smaller sub-problems and storing intermediate results to avoid repeated calculations. During the comparison process, it identifies a series of matching character blocks that are the same in the two sequences and records their starting position and length.

[0154] In the embodiment of the present application, the specific matching process is as follows: Fig.16 As shown, including:

[0155] Step 1601, obtaining the key information of the bill and the target product information, and calculating the similarity between the two;

[0156] Step 1602, matching the key information of the bill with the target product information according to the threshold;

[0157] When the similarity between the two is greater than the set threshold, it means that the key information of the bill already exists in the database, and step 1603 is performed to fill the commodity price in the key information of the bill back into the price field of the target commodity information in the database, completing the price update of the commodity in the database;

[0158] When the similarity between the two is less than the set threshold, the target product information is empty, indicating that the product information does not exist in the database, and step 1604 is performed to fill the ticket key information back into the corresponding field in the target product information in the database to complete the addition of the product information in the database.

[0159] In some possible embodiments, the similarity between two sequences corresponding to the same type of information is calculated in the following manner:

[0160] in:

[0161] Wherein, similarity(a, b) represents similarity, |a| and |b| represent the lengths of sequence a and sequence b corresponding to the same type of information, respectively, and |c| represents a matching sequence, which is the longest continuous subsequence existing in both sequences corresponding to the same type of information.

[0162] In the embodiment of the present application, the similarity is calculated by dividing twice the length of the matching sequence by the total length of the two sequences. Specifically, it calculates the ratio of twice the length of the matching sequence (i.e., the number of identical elements in the two sequences) to the sum of the lengths of the two sequences. Wherein, the matching sequence refers to the longest continuous subsequence that exists in both sequences.

[0163] For example, the name of the target product information in the database is "YSL MAKE EDP LE CUSH 202010FG CN", but due to various factors when recognizing the bill, the product name in the result output by the multimodal large model API is "YSL NAKN EDP LE CUSN 2020 10FG CN".

[0164] The similarity of the two strings is: 2*(6+2+11+14) / (36+36)=0.91666666666666666

[0165] Matching block information:

[0166] Match(a=0,b=0,size=6)

[0167] Match(a=7,b=7,size=2)

[0168] Match(a=10,b=10,size=11)

[0169] Match(a=22,b=22,size=14)

[0170] Match(a=36,b=36,size=0)

[0171] In some possible embodiments, the value of similarity(a, b) is between 0 and 1, and a larger value indicates that the two sequences are more similar.

[0172] In some possible embodiments, the threshold of similarity is set to 0.8. When the similarity is greater than 0.8, the commodity in the key information of the bill corresponds to the commodity in the database, and the two will be automatically associated. At the same time, the "commodity name" of the target commodity information in the database is updated to the "commodity name" of the corresponding commodity in the key information of the bill, and the "price" of the target commodity information in the database is updated to the "price" of the corresponding commodity in the key information of the bill, thereby completing the update of the database. When the similarity is less than 0.8, at this time, the commodity information is not recognized, that is, there is no such commodity in the database, and the user needs to manually associate the two, fill the "commodity name", "quantity" and "price" of the corresponding commodity in the key information of the bill into the corresponding commodity information in the unrecognized list in the database, and complete the database entry of the commodity information.

[0173] A database updating method based on ticket recognition and commodity matching provided in an embodiment of the present application adopts a ticket recognition method based on a combination of a small model and a large model, which can effectively protect the privacy information of passengers in the ticket; by setting a threshold, when the commodity information recognized by the ticket matches the commodity information recognized by AI and is greater than the set threshold, the commodity information recognized by the ticket and the commodity information recognized by AI are intelligently associated; through the intelligent association of the commodity information recognized by the ticket and the commodity information recognized by AI, for commodities already existing in the database, the old prices in the database can be updated according to the prices of the tickets, so that the commodity prices in the database are updated in real time; for commodities not in the database, new data can be added through manual association of the commodity information recognized by the ticket, so as to continuously expand the commodity database.

[0174] Based on the same inventive concept, the embodiment of the present application also provides a database updating device based on bill recognition and commodity matching, such as Fig.17 As shown, including:

[0175] The commodity matching module 1701 is used to obtain commodity information obtained by identifying commodities, and obtain target commodity information successfully matched in the database according to the matching result between the commodity information and the commodity information in the database;

[0176] The bill processing module 1702 is used to obtain bill information obtained by identifying the bill and extract key bill information from the bill information;

[0177] The data updating module 1703 is used to update the target product information in the database using the key information of the bill when it is determined that the match is successful based on the matching result between the target product information and the key information of the bill.

[0178] In some possible embodiments, the data updating module is further used for:

[0179] When it is determined that no target product information successfully matched in the database is obtained according to the matching result between the product information and the product information in the database, the product information in the database is added using the key information of the ticket according to the user's instruction;

[0180] According to the matching result between the target commodity information and the key information of the bill, if it is determined that the match is not successful, the target commodity information in the database is updated using the key information of the bill according to the user's instruction.

[0181] In some possible embodiments, after extracting the key bill information from the bill information, the bill processing module is further used to:

[0182] Determine the number of the first product from the identified product information;

[0183] Determine the number of the second commodity from the bill key information / bill information;

[0184] Compare the number of the first product with the number of the second product;

[0185] When the number of the first commodities is equal to the number of the second commodities, matching the target commodity information with the key information of the bill;

[0186] When the number of the first commodities is greater than the number of the second commodities, the commodity information is added to the database according to the user's instruction using the commodity information;

[0187] When the number of the first commodities is less than the number of the second commodities, commodity information is newly added to the database using the key information of the bill.

[0188] In some possible embodiments, the commodity matching module obtains commodity information obtained by identifying the commodity, including:

[0189] Obtaining product photos obtained by taking photos of the products;

[0190] Using a rotating object detection model to perform product detection on the product photos, obtaining a single image of each product and cropping it;

[0191] Using a rotating target detection model to perform barcode detection on a single image of each commodity, respectively obtaining and cropping a barcode image of each commodity;

[0192] The width of each vertical line in the barcode image of each product is analyzed to obtain the product information.

[0193] In some possible embodiments, the bill processing module obtains bill information obtained by identifying the bill, including:

[0194] Obtain a photo of the bill obtained by taking a photo of the bill;

[0195] Use optical character recognition (OCR) to perform text recognition on bill photos to obtain text bill information and corresponding location information;

[0196] Use the non-text recognition model to detect non-text information on the bill photo to obtain non-text bill information and corresponding location information;

[0197] According to the characteristics of different types of privacy information, the corresponding privacy information and the corresponding location information are detected from the obtained text / non-text ticket information and the corresponding location information;

[0198] The privacy information in the text / non-text ticket information is erased to obtain the ticket information.

[0199] In some possible embodiments, the key bill information extracted from the bill information by the bill processing module includes:

[0200] Input the bill information and questions about the key bill information into the multimodal big model API to obtain the multimodal big model API answer;

[0201] The multimodal large model API answers are analyzed and screened to obtain the key information of the bill.

[0202] In some possible embodiments, the data updating module matches the target commodity information with the key ticket information, including:

[0203] According to different types of information respectively contained in the bill key information and the target product information, for the same type of information, corresponding sequences are respectively selected from the bill key information and the target product information;

[0204] According to the string similarity matching method, the similarity of two sequences corresponding to the same type of information is calculated;

[0205] According to the calculated similarity, it is determined whether the target product information and the bill key information are matched successfully.

[0206] In some possible embodiments, the data updating module calculates the similarity of two sequences corresponding to the same type of information in the following manner:

[0207] in:

[0208] Wherein, similarity(a, b) represents similarity, |a| and |b| represent the lengths of sequence a and sequence b corresponding to the same type of information, respectively, and |c| represents a matching sequence, which is the longest continuous subsequence existing in both sequences corresponding to the same type of information. After introducing the database updating method and apparatus based on bill recognition and commodity matching according to an exemplary embodiment of the present application, next, a database updating device based on bill recognition and commodity matching according to another exemplary embodiment of the present application is introduced.

[0209] Those skilled in the art will appreciate that various aspects of the present application may be implemented as a system, method or program product. Therefore, various aspects of the present application may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to as "circuit", "module" or "system" herein.

[0210] In some possible implementations, the database updating device based on bill identification and commodity matching according to the present application may include at least one processor and at least one memory. The memory stores program code, and when the program code is executed by the processor, the processor executes the steps of the database updating method based on bill identification and commodity matching according to various exemplary implementations of the present application described above in this specification.

[0211] Refer to the following Fig.18 The database updating device 180 based on bill recognition and commodity matching according to this embodiment of the present application is described. Fig.18 The commodity tax number determination device 180 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0212] like Fig.18 As shown, the database updating device 180 based on bill recognition and commodity matching is presented in the form of a general electronic device. The components of the database updating device 180 based on bill recognition and commodity matching may include but are not limited to: at least one processor 181, at least one memory 182, and a bus 183 connecting different system components (including the memory 182 and the processor 181).

[0213] Bus 183 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a processor, or a local bus using any of a variety of bus architectures.

[0214] The memory 182 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 1821 and / or a cache memory 1822 , and may further include a read-only memory (ROM) 1823 .

[0215] The memory 182 may also include a program / utility 1825 having a set (at least one) of program modules 1824, such program modules 1824 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0216] The database updating device 180 based on bill recognition and commodity matching can also communicate with one or more external devices 184 (e.g., keyboard, pointing device, etc.), and can also communicate with one or more devices that enable a user to interact with the database updating device 180 based on bill recognition and commodity matching, and / or communicate with any device that enables the database updating device 180 based on bill recognition and commodity matching to communicate with one or more other electronic devices (e.g., router, modem, etc.). Such communication can be performed through an input / output (I / O) interface 185. In addition, the database updating device 180 based on bill recognition and commodity matching can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN) and / or public network, such as the Internet) through a network adapter 186. As shown in the figure, the network adapter 186 communicates with other modules of the database updating device 180 based on bill recognition and commodity matching through a bus 183. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the database update device 180 based on bill recognition and commodity matching, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0217] In some possible implementations, various aspects of a database update method based on bill identification and commodity matching provided by the present application may also be implemented in the form of a program product, which includes a program code. When the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of a database update method based on bill identification and commodity matching according to various exemplary implementations of the present application described above in this specification.

[0218] The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0219] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0220] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A database updating method based on bill recognition and commodity matching, comprising: Acquire commodity information obtained by identifying the commodity, and obtain target commodity information successfully matched in the database according to the matching result between the commodity information and the commodity information in the database; After obtaining the bill information obtained by identifying the bill, extracting the bill key information from the bill information; According to the matching result between the target commodity information and the key information of the bill, when it is determined that the match is successful, the key information of the bill is used to update the target commodity information in the database.

2. The method according to claim 1, characterized in that Also includes: When it is determined that no target product information successfully matched in the database is obtained according to the matching result between the product information and the product information in the database, the product information in the database is added using the key information of the ticket according to the user's instruction; According to the matching result between the target commodity information and the key information of the bill, if it is determined that the match is not successful, the target commodity information in the database is updated using the key information of the bill according to the user's instruction.

3. The method according to claim 1, characterized in that After extracting the key information of the bill from the bill information, it also includes: Determine the number of the first product from the identified product information; Determine the number of the second commodity from the bill key information / bill information; Compare the number of the first product with the number of the second product; When the number of the first commodities is equal to the number of the second commodities, matching the target commodity information with the key information of the bill; When the number of the first commodities is greater than the number of the second commodities, the commodity information is added to the database according to the user's instruction using the commodity information; When the number of the first commodities is less than the number of the second commodities, commodity information is newly added to the database using the key information of the bill.

4. The method according to claim 1, characterized in that The obtaining of commodity information obtained by identifying the commodity includes: Obtaining product photos obtained by taking photos of the products; Using a rotating object detection model to perform product detection on the product photos, obtaining a single image of each product and cropping it; Using a rotating target detection model to perform barcode detection on a single image of each commodity, respectively obtaining and cropping a barcode image of each commodity; The width of each vertical line in the barcode image of each product is analyzed to obtain the product information.

5. The method according to claim 1, characterized in that The step of obtaining the bill information obtained by identifying the bill includes: Obtain a photo of the bill obtained by taking a photo of the bill; Use optical character recognition (OCR) to perform text recognition on bill photos to obtain text bill information and corresponding location information; Use the non-text recognition model to detect non-text information on the bill photo to obtain non-text bill information and corresponding location information; According to the characteristics of different types of privacy information, the corresponding privacy information and the corresponding location information are detected from the obtained text / non-text ticket information and the corresponding location information; The privacy information in the text / non-text ticket information is erased to obtain the ticket information.

6. The method according to claim 1, characterized in that The key bill information extracted from the bill information includes: Input the bill information and questions about the key bill information into the multimodal big model API to obtain the multimodal big model API answer; The multimodal large model API answers are analyzed and screened to obtain the key information of the bill.

7. The method according to claim 3, characterized in that Matching the target product information with the key bill information includes: According to different types of information respectively contained in the bill key information and the target product information, for the same type of information, corresponding sequences are respectively selected from the bill key information and the target product information; According to the string similarity matching method, the similarity of two sequences corresponding to the same type of information is calculated; According to the calculated similarity, it is determined whether the target product information and the bill key information are matched successfully.

8. The method according to claim 7, characterized in that The similarity of two sequences corresponding to the same type of information is calculated as follows: in: Wherein, similarity(a, b) represents similarity, |a| and |b| represent the lengths of sequence a and sequence b corresponding to the same type of information, respectively, and |c| represents a matching sequence, which is the longest continuous subsequence existing in both sequences corresponding to the same type of information.

9. A database updating device based on bill recognition and commodity intelligent matching, characterized in that: It comprises at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable the computer to execute the method according to any one of claims 1 to 8.

Images