Commodity order information intelligent identification method based on natural language processing

Abstract

The invention discloses a commodity order information intelligent identification method based on natural language processing, and belongs to the technical field of natural language processing. Aiming at the problem of low recognition accuracy of unstructured text order information, the invention provides an intelligent recognition method. The method comprises the following steps: receiving an unstructured text; preprocessing, including conversion from traditional Chinese characters to simplified Chinese characters, space removal and keyword filtering; commodity names are extracted by adopting accurate matching and pinyin fuzzy matching; extracting numbers and quantity identifiers; analyzing the number of each commodity, including processing the expression of each character, the expression of a direct number, the expression of a negative number and the expression of 0; and outputting the structured order information. In addition, the method can also process agency orders and serial number orders, identifies commodity name variants through a sliding window technology and Pinyin similarity calculation, improves order processing efficiency, enhances the capability of processing complex expressions, reduces labor cost, and improves user experience.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a method for intelligently identifying commodity order information based on natural language processing, which is used for automatically extracting and parsing commodity order information from unstructured text. Background Art

[0002] With the development of e-commerce and social media, more and more businesses are receiving customer orders through instant messaging apps like WeChat. However, these orders often appear in the form of unstructured natural language text, such as "5 apples, 2 bananas and 2 pears each," "Daifa Xiaowang = 3 boxes of strawberries, 1 orange," and "No mangoes, lychees cancelled." This type of order information has the following characteristics: 1. Diverse forms of expression and lack of a unified format; 2. The product name may have multiple expressions or contain typos; 3. Complex expressions of quantity, such as "2 each", "two servings", etc. 4. It may contain information about the person sending the order, and the real person who placed the order needs to be identified; 5. Serial numbers may be used instead of product names, such as "2 copies each of 1 and 3".

[0003] Traditional order processing relies primarily on manual identification and data entry, which is subject to low efficiency, error-proneness, and high costs. Existing automated solutions, primarily based on keyword matching and rule engines, suffer from poor accuracy and robustness when dealing with complex and varied natural language expressions.

[0004] Therefore, there is an urgent need for a method that can intelligently identify and parse product order information in unstructured text to improve order processing efficiency and reduce labor costs. Summary of the Invention

[0005] The purpose of the present invention is to provide a method for intelligently identifying commodity order information based on natural language processing, which can accurately extract commodity names and quantities from unstructured text, and solve the technical problem of low recognition accuracy in the prior art when faced with diverse expressions, commodity name variants, complex quantity expressions and other problems.

[0006] To achieve the above objectives, the present invention provides the following technical solutions: A method for intelligently identifying commodity order information based on natural language processing, comprising the following steps: (1) Receive unstructured text containing product order information; (2) preprocessing the text, including converting traditional Chinese characters to simplified Chinese characters, removing spaces and specific keywords; (3) Extract product names from the preprocessed text, including: ①Perform exact matching based on a preset list of product names; ②Replace the matched product names in the text with specific markers; ③Deduplicate the extracted product names and retain the first-occurring product; (4)For product names that cannot be exactly matched, use a pinyin fuzzy matching algorithm for identification, including: ①Segment the text and use the sliding window technique; ②Calculate the pinyin similarity between the text segment and the product name; ③When the similarity exceeds the preset threshold, confirm a successful match; (5)Extract numbers and quantity identifiers from the preprocessed text, including: ①Identify Arabic numerals and Chinese numerals; ②Identify the character "each" as a quantity allocation identifier; ③Identify the positions of product placeholders; (6)Based on the extracted numbers, quantity identifiers, and product placeholders, parse the quantity of each product, including: ①Segment the text by the character "each"; ②Associate the numbers with the products according to the segmentation and the positions of the numbers; ③Calculate the final quantity of each product; (7)Output structured order information containing the product names and their corresponding quantities. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] To more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required in the embodiments. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings.

[0008] Figure 1 It is a flowchart of the intelligent identification method for product order information based on natural language processing of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0009] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art without creative efforts based on the embodiments of the present invention belong to the scope of protection of the present invention.

[0010] Embodiment 1 This embodiment details the complete implementation process of an intelligent recognition method for commodity order information based on natural language processing, which is carried out strictly according to the Figure 1 process shown.

[0011] Step S1: Receive unstructured text input The system receives the unstructured order text input by the user, for example: "5 portions of apples, 2 portions each of bananas and pears"; this text may contain various complex expressions, including but not limited to: Chinese numerical expressions, English numbers, commodity aliases, dialect expressions, typos, etc.

[0012] Step S2: Text preprocessing Perform multi-level preprocessing on the input text: (1) Drop-ship order detection: Check if the text starts with "Drop-ship". If so, extract the drop-ship information; use the regular expression r"Drop-ship(.*?)=(.*)" to match the user name between "Drop-ship" and "=", and the actual order content after "="; (2) Traditional Chinese to simplified Chinese conversion: Call the convert function of the zhconv library to convert all traditional Chinese characters to simplified Chinese characters to ensure the unity of the text for subsequent processing; (3) Remove spaces and special characters: Use the regular expression re.sub(r'\s+', '', message) to remove all spaces and line breaks; (4) Keyword filtering: According to the preset interference word library, filter out interfering words with quantities such as "Are they the same?" and "In total" to avoid affecting order parsing.

[0013] Step S$3$: Commodity name extraction (accurate extraction) (1) Exact matching based on the preset commodity list: Convert the commodity list into a processing format, supporting aliases separated by "|", such as "Apple|Red Fuji|Fuji Apple"; Sort in descending order of commodity name length, and match long names first to avoid the problem of short names covering long names; Construct a regular expression for exact matching; (2) Replace with "#" marker: Use re.finditer(item_regex, input_string) to find all matching commodities and their positions; Record the name, position, and matching text of each matching commodity; Replace the matching commodity names with "#" markers for subsequent quantity parsing; ($3$) Duplicate removal processing: Remove duplicates from the extracted commodity list and keep the commodity that appears first; Record the index of the removed commodity for subsequent corresponding cleaning of "#" symbols.

[0014] Step S4: Pinyin fuzzy matching (optional) It should be noted that in the translation, the "步骤S3" in the original text is translated as "Step S$3$" to maintain the consistency of the original text's tags. You may need to correct it to "Step S3" according to the actual situation. Also, some terms might need to be adjusted according to the context and specific requirements of the patent text. (1) Sliding window technique: Split the text into segments using the regular expression re.split(r'[0-9\s,,、]+', input_string); Filter out segments with a length less than 2; Convert the product name to pinyin using the pypinyin library; Use a 2-6 character sliding window in the text segments; (2) Pinyin similarity calculation: Use the ratio function of the fuzzywuzzy library to calculate the similarity between the window content and the product pinyin; (3) Threshold judgment: Set the threshold to 95%; When the similarity reaches the threshold, record the matching result and replace it with the "#" placeholder.

[0015] Step S5: Judgment of whether there is a product name (1) If the number of product lists is 0, enter the no-product branch; (2) If the number of product lists is greater than 0, enter the product branch.

[0016] Step S5A: Processing of the no-product branch (1) Conversion to serial number order parsing: Convert the input text to the serial number order format; (2) Serial number order conversion algorithm: Convert Chinese numbers to English numbers; Use the regular expression re.sub(r'[^\w]', '', message) to remove non-alphanumeric characters; Judge whether the processed text is pure numbers or contains the character "各"; If it is pure numbers, replace them with product names in the order of product IDs, such as "123" → "1 apple, banana, and pear"; If it contains the character "各", process it by segmenting according to the character "各", such as "12各3" → "3 apples and bananas"; (3) Re-perform product identification: Re-execute the product identification process in step S3 on the converted text; If the product still cannot be identified, return an empty result. < &​​​​(2)Identify the character "各": Use re.finditer('各', input_string) to extract the positions of all "各" characters; (3)Record the positions of product placeholders: Use the regular expression r'-?\d+' to extract all numbers and their positions; Use re.finditer('#', input_string) to extract the positions of all product placeholders; Arrange all elements in order of position to form an analysis sequence; (4)Handling special cases: ① Zero-value order detection: If the number 0 is detected and there are only two digits and one product, it is determined as a cancelled order, and a special format ['#', 0] is returned to indicate cancellation; ② Filtering of interrogative messages: If only one product is recognized and the remaining text contains "几" or "多少", it is determined as an interrogative message, and an empty result is returned and not processed as an order; ③ Judgment of non-order messages: Calculate the ratio of the remaining text length to the number of products. If the ratio is greater than 15, it is determined as a non-order message, and an empty result is returned.

[0018] Step S7: Parsing the quantity of products (1)Segment by the character "各": Split the element sequence by the character "各" into multiple segments; Each "各" character and the first digit after it form a complete segment; The remaining elements form the last segment; (2)Association between numbers and products: ① Segments containing "各": The numbers before "各" correspond to the product list one by one according to the position; If the number of product types before "各" is greater than the number of digits, the remaining products use the number after "各" as their quantity; ② Segments not containing "各": The numbers and the product list correspond to each other in order of position; If the number of digits is insufficient, the quantity of the remaining products is 0; (3)Calculate the final quantity: For the input "蘋果5份,香交和梨各2份": ① After preprocessing: "苹果5份,香交和梨各2份"; ② Product identification: ["苹果", "梨"]; ③ After enhanced fuzzy matching by pinyin: ["苹果", "香蕉", "梨"]; ④ Extraction of numbers and identifiers: "#5份#和#各2份"; ⑤ Element extraction: ["#", 5, "#", "#", "各", 2]; ⑥ Segmentation result: [["#", 5], ["#", "#", "各", 2]]; ⑦ Analysis of the first segment: 苹果 → 5份, Analysis of the second segment: 香蕉 → 2份, 梨 → 2份.

[0019] Step S8: Output structured order information Finally, output the standardized order information in JSON format, such as: { "张三": { "苹果": 5, "香蕉": 2,"梨": 2}}。

[0020] Technical effects and performance indicators This embodiment verifies the technical effect of the method through a large number of actual tests: (1) Recognition accuracy: Standard product name recognition: 100%; Pinyin fuzzy matching: 96%; Serial number order conversion: 100%; Complex quantity expression analysis: 98%; (2) Robustness: Supports mixed texts of Traditional and Simplified Chinese; handles typos and dialect expressions with tolerance; automatically filters non-order information; supports multiple ways of expressing quantities; This embodiment fully demonstrates the technical advantages of the present invention in processing complex natural language order information. Through the branch processing strategy and multi-level matching algorithm, the accuracy of order information extraction and the practicality of the system are significantly improved.

[0021] Example 2 Examples of application scenarios of the method of the present invention: (1) Example of proxy delivery detection: Input text: "On behalf of Xiao Wang = 3 boxes of strawberries, 1 orange", output result: {"Xiao Wang":{"Strawberries": 3, "Orange": 1}}; (2) Example of a serial order: Assume that item 1 is an apple, item 2 is a mango, and item 3 is an orange. Input text: "123, two each", output result: {"Xiao Wang": {"Apple": 2, "Mango": 2, "Orange": 2}}; (3) Example of canceling an order: Input text: "I don't want apples anymore, I want 2 more bananas", output result: {"username":{"apples": 0, "bananas": 2}}; (4) Example of reducing products: Input text: "I need 3 less apples", output result: {"username": {"apples": -2}}; The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Those skilled in the art should understand that various equivalent modifications or replacements may be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and such modifications or replacements shall fall within the scope of protection of the present invention.

Claims

1. A method for intelligently identifying commodity order information based on natural language processing, characterized in that: It includes the following steps: (1) Receive unstructured text containing commodity order information; (2) Preprocess the text, including converting traditional Chinese characters to simplified Chinese characters, removing spaces and specific keywords; (3) Extract commodity names from the preprocessed text, including: ① Perform exact matching based on a preset commodity name list; ② Replace the matched commodity names in the text with specific markers; ③ Remove duplicates from the extracted commodity names and retain the commodity that appears first; (4) For commodity names that cannot be exactly matched, use a pinyin fuzzy matching algorithm for identification, including: ① Segment the text and use the sliding window technique; ② Calculate the pinyin similarity between the text segment and the commodity name; ③ When the similarity exceeds the preset threshold, confirm it as a successful match; (5) Extract numbers and quantity identifiers from the preprocessed text, including: ① Identify Arabic numerals and Chinese numerals; ② Identify the character "each" as a quantity allocation identifier; ③ Identify the positions of commodity placeholders; (6) Based on the extracted numbers, quantity identifiers and commodity placeholders, parse the quantity of each commodity, including: ① Segment the text by the character "each"; ② Associate the numbers with the commodities according to the segmentation situation and the number positions; ③ Calculate the final quantity of each commodity; (7) Output structured order information containing commodity names and their corresponding quantities.

2. The method according to claim 1, characterized in that The preprocessing also includes detecting and processing drop-shipping orders, specifically including: (1) Identify text starting with "drop-shipping"; (2) Extract the drop-shipping object and order content from the text; (3) Use the drop-shipping object as the order user name and the order content as the text to be processed.

3. The method according to claim 1, characterized in that The preprocessing also includes detecting and processing serial number orders, specifically including: (1) When no commodity name is extracted from the text, try to parse the text as a serial number order; (2) Match the serial number with a preset commodity ID; (3) Replace the matched commodity ID with the corresponding commodity name.

4. The method according to claim 1, wherein The pinyin fuzzy matching algorithm specifically includes: (1) Set the minimum and maximum sliding window sizes; (2) Calculate the pinyin similarity between each possible window in the text and the commodity name; (3) Record and return the text segment with the highest similarity that exceeds the preset threshold.

5. The method according to claim 1, wherein The steps of parsing the quantity of each commodity also include: (1) Process quantity expressions with the character "each", and apply the number after "each" to all commodities that have not been assigned a quantity mentioned before; (2) Process quantity expressions without the character "each", and directly associate the numbers with the commodities at the corresponding positions; (3) Process reduction orders represented by negative numbers and cancellation orders represented by "0".

Images