Formula compliance detection method, system and storage medium

Abstract

The present application discloses a formula compliance detection method and system, comprising the following steps: identifying text content in an image; determining whether the text content includes ingredient information according to the word segmentation result, and determining each candidate word in the ingredient information; searching the candidate words in the ingredient information in a preset database to determine whether they are component words; if some candidate words are neither component words nor numerical words, they are marked as words to be corrected; determining whether product information can be obtained, and if so, determining the corresponding product category according to the product information; searching the knowledge graph for associated components of the product category according to the product category; matching the associated components with the words to be corrected to determine the correct representation of the words to be corrected; mapping all the component words after the corrected representation into an editable container for the user to adjust the ingredients and content; and checking whether the formula is compliant when the user triggers a compliance check.

Description

Technical Field

[0001] The present application relates to natural language processing technology, and in particular to a formula compliance detection method, system and storage medium. Background Art

[0002] In the fields of cosmetics, daily chemical products, etc., the compliance of formulas is a concern for relevant product developers and consumers. At present, to detect whether a formula is compliant, users need to enter the corresponding ingredients into the relevant software, and the ingredient list is generally long. If you need to enter the ingredients of each product, it is more troublesome. Although text can be recognized by using OCR technology as a text input method, it is found in practice that the recognition error rate is high due to the complexity of the words involved in the ingredients. If the user hopes to use a ready-made product formula as the basic input information for ingredient adjustment or compliance analysis, more modifications are required. Summary of the invention

[0003] The present invention aims to solve at least one of the technical problems existing in the prior art. To this end, the present invention provides a formula compliance detection method, system and storage medium.

[0004] On the one hand, an embodiment of the present application provides a method for formula compliance detection, comprising the following steps:

[0005] Get the image taken by the user;

[0006] Identify text content in the image and perform word segmentation based on the content of the text;

[0007] Determine whether the text content includes component information according to the word segmentation result, and determine candidate words in the component information;

[0008] Searching the candidate words in the component information in a preset database to determine whether they are component words;

[0009] If some candidate words are neither component words nor numerical words, they are marked as words to be corrected;

[0010] Determine whether the product information can be obtained, and if so, determine the corresponding product type based on the product information;

[0011] Retrieve the associated components of product categories in the knowledge graph based on the product categories;

[0012] Matching the obtained associated components with the words to be corrected to determine the correct representation of the words to be corrected;

[0013] Map all the corrected ingredient words into editable containers, so that users can adjust the ingredients and contents;

[0014] When the user triggers a compliance check, the recipe is checked for compliance.

[0015] In some embodiments, matching the obtained associated components with the words to be corrected to determine the correct representation of the words to be corrected specifically includes:

[0016] Classifying the fonts of the words to be corrected, and determining a target commonly used font similar to the font used by the words to be corrected according to the font classification;

[0017] Obtaining image features of the word to be corrected and each associated component in the target commonly used font;

[0018] Screen out candidate associated components whose difference from the word to be corrected is less than N, where N is a positive integer greater than or equal to 1;

[0019] Compare the image features of each different word in the word to be corrected and the candidate associated components one by one;

[0020] The candidate associated component whose similarities corresponding to all the different characters are greater than the first threshold and whose number of different characters is the least is taken as the correct representation of the word to be corrected; when there are multiple candidate associated components with the same number of different characters and the least number of different characters, the associated component with the largest average value is selected as the correct representation of the word to be corrected based on the average value of the similarities corresponding to the different characters.

[0021] In some embodiments, determining whether the text content includes component information according to the word segmentation result and determining each candidate word in the component information specifically includes:

[0022] Determine whether there is a feature word representing component information in the word segmentation result. If so, determine that the paragraph corresponding to the component word is component information; if not, determine the component information according to the distribution characteristics of the text and symbols, wherein when the distribution density of a specific symbol in the text is greater than a preset value, determine that the corresponding paragraph is component information;

[0023] Determine each candidate word in the component information based on the word segmentation result.

[0024] In some embodiments, the identifying text content in the image includes:

[0025] When extracting text from an image, extract text blocks by line, and determine paragraph relationships according to the height and position of the text blocks; wherein two text blocks belonging to the same paragraph have the same height or a height difference less than a second threshold, and the two text blocks are adjacent;

[0026] The determining whether the product information can be obtained specifically includes:

[0027] Determine whether it contains barcode information. If so, obtain the product information through the barcode information; if not, select the text in some text blocks in descending order according to the height of the text block when recognizing the text, and determine whether the corresponding product or product category can be retrieved. If so, get the product type corresponding to the text to be recognized.

[0028] In some embodiments, mapping the corrected component words into the editable container specifically includes:

[0029] Generate a container matrix according to the number of component words, wherein the container matrix includes at least two columns, wherein the first column of containers each includes a name container of a component word, and the second column of containers includes a component quantity container corresponding to each component word;

[0030] Each container in the container matrix can be changed according to the user's input instruction.

[0031] In some embodiments, the following steps are also included:

[0032] Before the user triggers the compliance check, input options are displayed to allow the user to select the commodity category and the commodity usage area, wherein the default value of the commodity category option is determined according to the acquired commodity category.

[0033] In some embodiments, the following steps are also included:

[0034] Based on the user's selection instruction, the shooting mode is converted into the recipe input mode;

[0035] The recipe input mode creates a specified number of containers according to the user's instructions, and each container and the content input into the container by the user can be deleted together.

[0036] In some embodiments, the following steps are also included:

[0037] When a product name is identified, if the specific ingredient content of the corresponding product exists in the database, the ingredient content is directly filled in the second column container corresponding to the ingredient by default.

[0038] On the other hand, an embodiment of the present application provides a formula compliance detection system, including:

[0039] Memory, used to store programs;

[0040] A processor is used to load the program to execute the formula compliance detection method.

[0041] On the other hand, an embodiment of the present application provides a computer-readable storage medium storing a program, which implements the formula compliance detection method as described above when the program is executed.

[0042] When the embodiment of the present application uses OCR technology to recognize text to assist users in inputting recipe information, the paragraph where the ingredient information is located can be determined based on the word segmentation results, and which words are component words and which are not component words in the database can be determined based on the word segmentation results, so as to determine whether there is a problem of individual component word recognition errors. At the same time, the product type is determined based on other information about the product, and then possible related ingredients are found from the knowledge graph based on the product type, and then the component words that may have recognition errors are corrected based on the related components; the corrected component words are then mapped to an editable container and handed over to the user for manual editing and modification before compliance testing; it can be understood that in the scenario of recipe ingredient recognition, the user's input can be simplified and the user's efficiency can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments.

[0044] Figure 1 is a flow chart of a method provided by an embodiment of the present application;

[0045] Figure 2 is a schematic diagram of recognizing characters in an image provided by an embodiment of the present application;

[0046] Figure 3 is a schematic diagram of a recognition result interface provided in an embodiment of the present application;

[0047] Figure 4 It is a schematic diagram of a system framework provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described through the implementation method with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application.

[0049] Reference Figure 1 , the embodiment of the present application discloses a formula compliance detection method, comprising the following steps:

[0050] S1. Obtain an image taken by a user. In this step, the user can directly take a picture of the recipe information of a product by calling the mobile phone camera. The user can also upload and identify the relevant pictures.

[0051] S2, identifying the text content in the image, and performing word segmentation according to the content of the text. In this step, the content of the text can be extracted from the image by OCR technology. Then, the content of the text can be segmented by a word segmentation model.

[0052] The identifying text content in the image includes:

[0053] Reference Figure 2 When extracting text from an image, text blocks are extracted row by row, and the paragraph relationship is determined based on the height and position of the text blocks; wherein, two text blocks belonging to the same paragraph have the same height or a height difference less than a second threshold, and the two text blocks are adjacent. It is understandable that in commodity information, trademarks or commodity names may be marked in different sizes, and usually the text in the same paragraph is adjacent, has the same text size, and generally has at least one side aligned. The paragraph relationship can be determined by the size and position characteristics of these text blocks.

[0054] S3. Determine whether the text content includes ingredient information based on the word segmentation result, and determine the candidate words in the ingredient information. It is understandable that in the formula, the ingredient information may have certain characteristic words, such as ingredients, formula, ingredients, etc. These contents can determine whether the current paragraph is ingredient information. However, given that the labels of goods are usually small and the printing colors are different, there are many cases of recognition errors, or even no recognition without these words. Therefore, step S3 specifically includes:

[0055] It is determined whether there are characteristic words representing ingredient information in the word segmentation result. If so, it is determined that the paragraph corresponding to the component word is ingredient information; if not, the component information is determined according to the distribution characteristics of the text and symbols, wherein when the distribution density of a specific symbol in the text is greater than a preset value, the corresponding paragraph is determined to be ingredient information; it is understandable that a recipe is usually described by multiple ingredients, and the ingredient name is not particularly long, and there are usually symbols such as commas and spaces between ingredients. Therefore, it is possible to determine whether it is ingredient information by the distribution characteristics of the text and symbols. Among them, the recognition rules can be set, for example, the average length of a single word is less than a certain number of words, and the words are separated by spaces, commas, and semicolons. In addition, the density of the above symbols in the paragraph is greater than a certain value. The candidate words in the ingredient information are determined based on the word segmentation results. It is understandable that since the content shot by the user cannot be controlled, the content in the image may also include other information in addition to the ingredient information, and the other information is not necessarily the information required for related business. Through intelligent recognition, the user's operation can be reduced.

[0056] S4. Search the candidate words in the component information in a preset database to determine whether they are component words.

[0057] The data may store component words and synonyms of component words in related fields. Whether the related candidate words are component words can be determined by searching the database.

[0058] S5. If some candidate words are neither component words nor numerical words, they are marked as words to be corrected.

[0059] It is understandable that some ingredient tables are set with corresponding ingredient ratios or weights. Therefore, if there are words that are neither ingredient words nor numerical words, there is a high probability that they are not correctly identified. Therefore, the relevant object words can be used as words to be corrected. Of course, for those that ultimately fail to meet the correction conditions, they can also be discarded or specially marked in the results.

[0060] S6. Determine whether the product information can be obtained. If so, determine the corresponding product type based on the product information.

[0061] Specifically, the determining whether the commodity information can be obtained includes:

[0062] Determine whether it contains barcode information. If so, obtain product information through the barcode information; if not, select the text in some text blocks in descending order according to the height of the text block when recognizing the text, and determine whether the corresponding product or product category can be retrieved. If so, obtain the product type corresponding to the text to be recognized. It is understandable that the product name or product name in the product label is usually marked with different font sizes. Therefore, it is easier to obtain product information by searching for different font sizes, which can be used as a supplementary identification method when there is no barcode.

[0063] Of course, if the corresponding product category cannot be identified, the product categories associated with each component word can be analyzed based on the currently known component words through the knowledge graph. In the knowledge graph, the relationship between the component and the product category can be described by distance. When the distances between all known components and a certain product category are relatively close, the smaller the average distance between each component and the product category.

[0064] In addition, in the case where the product type cannot be confirmed, you can also choose to abandon the word to be corrected. One way is to not use it as ingredient information, and another way is to indicate in the container that the ingredient information may be misidentified.

[0065] S7. Retrieve related components of product categories in the knowledge graph based on the product categories.

[0066] In this embodiment, a knowledge graph can be constructed based on the relationship between ingredients and commodity types. The knowledge graph includes a multi-layer triple relationship network. In the knowledge graph, the relationship between ingredients and commodity types, the relationship between commodity types and regulations, etc. can be added. Through the knowledge graph, relevant information can be retrieved. The knowledge graph can be constructed through a large amount of triples extracted from existing data and model algorithms.

[0067] S8. Match the obtained associated components with the words to be corrected to determine the correct representation of the words to be corrected.

[0068] Specifically, this step includes:

[0069] The words to be corrected are classified into fonts, and a target commonly used font similar to the font used by the words to be corrected is determined based on the font classification. In this solution, since the number of fonts is theoretically infinite, the features of commonly used fonts can be learned and refined. By classifying and identifying fonts, many different fonts can be classified into commonly used fonts.

[0070] Obtain the image features of the word to be corrected and each associated component in the target commonly used font. It is understandable that for different fonts, the similarity between related words will be different. Therefore, this solution uses image features for comparison instead of matching based on the meaning of the text content.

[0071] The candidate associated components whose difference from the word to be corrected is less than N are selected, where N is a positive integer greater than or equal to 1.

[0072] Compare the image features of each different word in the word to be corrected and the candidate associated components one by one;

[0073] The candidate associated component with the least number of difference characters and the similarities corresponding to all difference characters are greater than the first threshold value is used as the correct representation of the word to be corrected; when there are multiple candidate associated components with the same number of difference characters and the least number of difference characters, the associated component with the largest average value is selected as the correct representation of the word to be corrected according to the average value of the similarities corresponding to the difference characters. Since the probability of misidentifying fewer characters at the same time is less than the probability of identifying more characters, the ones with smaller differences are given priority. Although it is a case of recognition error, there is still a relatively large similarity between the misidentified character and the correct character, but in the recognition process, the character is recognized as another character with higher similarity. Therefore, in this scheme, the above method is adopted. After the similarity condition of each difference character is met, it is judged according to the number of difference characters. If there are still multiple candidate associated components with the same number of difference characters, the judgment can be made according to the average value of the similarity of the difference characters.

[0074] S9. Map all the component words after the correction into an editable container so that the user can adjust the components and contents.

[0075] Reference Figure 3 , the mapping of the corrected component words into the editable container specifically includes:

[0076] According to the number of component words, a container matrix is ​​generated, and the container matrix includes at least two columns. The first column of containers contains a name container of a component word, and the second column of containers contains a component quantity container corresponding to each component word. It can be understood that the above container refers to a component such as an input box. In this step, a container matrix is ​​generated according to the number of recognized components. The container matrix mainly includes two columns of parameters, one for the component name and the other for the component ratio.

[0077] from Figure 3 As can be seen in the figure, before the user triggers the compliance check, an input option is displayed to allow the user to select the product type and the product usage area, where the default value of the product type option is determined according to the obtained product type. Since regulations in different regions may differ, the user is allowed to select regulations in different regions for testing. At the same time, the type of product can also be edited by the user, such as the cosmetic formula.

[0078] When a product name is identified, if the specific ingredient content of the corresponding product exists in the database, the ingredient content is directly filled into the second column container corresponding to the ingredient by default. Whether to enter the corresponding ingredient content into the container can be set by the user. The user is allowed to switch according to different user habits.

[0079] Each container in the container matrix can be changed according to the user's input instruction.

[0080] S10. When the user triggers a compliance check, check whether the formula is compliant, where the formula refers to a formula composed of the identified ingredients.

[0081] It is understandable that compliance detection can retrieve and analyze the relevant formula based on the knowledge graph or other formatted database to see if there is any violation of the prohibition regulations. If so, it means that the formula is not compliant.

[0082] In other embodiments, the shooting mode may be converted to the recipe input mode based on a user's selection instruction;

[0083] The recipe input mode creates a specified number of containers according to the user's instructions, and each container and the content input by the user into the container can be deleted together. Based on the user's selection instruction, the shooting mode is converted to the recipe input mode.

[0084] It is understandable that in this mode, the user is allowed to freely add containers and then input relevant ingredients and values ​​into the containers. That is, this solution also allows the user to switch to a non-image input mode.

[0085] Reference Figure 4 , the embodiment of the present application provides a formula compliance detection system, including:

[0086] Memory, used to store programs;

[0087] A processor is used to load the program to execute the formula compliance detection method.

[0088] An embodiment of the present application provides a computer-readable storage medium storing a program, which implements the formula compliance detection method as described above when the program is executed.

[0089] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0090] Note that the above are only preferred embodiments of the present application and the technical principles used. Those skilled in the art will understand that the present application is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present application. Therefore, although the present application is described in more detail through the above embodiments, the present application is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present application, and the scope of the present application is determined by the scope of the appended claims.

Claims

1. A method for detecting formulation compliance, characterized in that: The following steps are involved: Get the image taken by the user; Identify text content in the image and perform word segmentation based on the content of the text; Determine whether the text content includes component information according to the word segmentation result, and determine candidate words in the component information; Searching the candidate words in the component information in a preset database to determine whether they are component words; If some candidate words are neither component words nor numerical words, they are marked as words to be corrected; Determine whether the product information can be obtained, and if so, determine the corresponding product type based on the product information; Retrieve the associated components of product categories in the knowledge graph based on the product categories; Matching the obtained associated components with the words to be corrected to determine the correct representation of the words to be corrected; Map all the corrected ingredient words into editable containers, so that users can adjust the ingredients and contents; When the user triggers a compliance check, the recipe is checked for compliance.

2. A method for detecting formulation compliance according to claim 1, characterized in that: The step of matching the obtained associated components with the words to be corrected to determine the correct representation of the words to be corrected specifically includes: Classifying the fonts of the words to be corrected, and determining a target commonly used font similar to the font used by the words to be corrected according to the font classification; Obtaining image features of the word to be corrected and each associated component in the target commonly used font; Screen out candidate associated components whose difference from the word to be corrected is less than N, where N is a positive integer greater than or equal to 1; Compare the image features of each different word in the word to be corrected and the candidate associated components one by one; The candidate associated component whose similarities corresponding to all the different characters are greater than the first threshold and whose number of different characters is the least is taken as the correct representation of the word to be corrected; when there are multiple candidate associated components with the same number of different characters and the least number of different characters, the associated component with the largest average value is selected as the correct representation of the word to be corrected based on the average value of the similarities corresponding to the different characters.

3. A method for detecting formulation compliance according to claim 2, characterized in that: Determining whether the text content includes component information according to the word segmentation result and determining each candidate word in the component information specifically includes: Determine whether there is a feature word representing component information in the word segmentation result. If so, determine that the paragraph corresponding to the component word is component information; if not, determine the component information according to the distribution characteristics of the text and symbols, wherein when the distribution density of a specific symbol in the text is greater than a preset value, determine that the corresponding paragraph is component information; Determine each candidate word in the component information based on the word segmentation result.

4. A method for detecting formulation compliance according to claim 3, characterized in that: The identifying text content in the image includes: When extracting text from an image, extract text blocks by line, and determine paragraph relationships according to the height and position of the text blocks; wherein two text blocks belonging to the same paragraph have the same height or a height difference less than a second threshold, and the two text blocks are adjacent; The determining whether the product information can be obtained specifically includes: Determine whether it contains barcode information. If so, obtain the product information through the barcode information; if not, select the text in some text blocks in descending order according to the height of the text block when recognizing the text, and determine whether the corresponding product or product category can be retrieved. If so, get the product type corresponding to the text to be recognized.

5. A method for detecting formulation compliance according to claim 1, characterized in that: The mapping of all component words after the correction expression into the editable container specifically includes: Generate a container matrix according to the number of component words, wherein the container matrix includes at least two columns, wherein the first column of containers each includes a name container of a component word, and the second column of containers includes a component quantity container corresponding to each component word; Each container in the container matrix can be changed according to the user's input instruction.

6. A method for formula compliance detection according to claim 5, characterized in that: The following steps are also included: Before the user triggers the compliance check, input options are displayed to allow the user to select the commodity category and the commodity usage area, wherein the default value of the commodity category option is determined according to the acquired commodity category.

7. A method for detecting formulation compliance according to claim 1, characterized in that: The following steps are also included: Based on the user's selection instruction, the shooting mode is converted into the recipe input mode; The recipe input mode creates a specified number of containers according to the user's instructions, and each container and the content input into the container by the user can be deleted together.

8. A method for detecting formulation compliance according to claim 5, characterized in that: The following steps are also included: When a product name is identified, if the specific ingredient content of the corresponding product exists in the database, the ingredient content is directly filled in the second column container corresponding to the ingredient by default.

9. A formula compliance detection system, characterized in that: include: Memory, used to store programs; A processor, configured to load the program to execute the formula compliance detection method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: It stores a program, and when the program is executed, it implements the formula compliance detection method as described in any one of claims 1-7.

Images