Material similarity retrieval method and system

Abstract

The invention provides a material similarity retrieval method and system. The method comprises the following steps: S1, constructing a material duplicate checking model; s2, performing duplicate checking comparison on the target material and the existing material in the current material pool by using the material duplicate checking model, and when the similarity is higher than a preset value, considering that the current target material is a repeated material; the material duplicate checking model is used for checking repeated material codes in the code library by calculating the similarity of material names, specifications and material length descriptions.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a material similarity retrieval method and system. Background Art

[0002] In the prior art, material data duplication checking uses a configuration table to dynamically configure material filter fields to detect duplicate material data; fields that are not allowed to be repeated are configured through the configuration table in the duplication checking method, and are automatically checked when the material data is imported to ensure the uniqueness of the material data. The current defect of material duplication checking is that the actual content of the check field must be completely consistent to be determined as duplicate data. In fact, for super-large group companies, the addition of material data is independently added by the material administrators of the group and its subordinate organizations. For the same material, the material name, model specifications and other descriptions entered by different material declarants only need to be slightly different. Under the duplication checking rules of the prior art, it is impossible to identify that the same material has caused the material to be entered repeatedly. At the same time, the duplication checking scenario is limited to the new addition stage, and the existing data cannot be checked for duplication.

[0003] Patent document CN109885571A (application number: 201910122124.X) discloses a material data duplicate checking method and system based on configurable rules, which belongs to the field of computer technology. It uses a configuration table to dynamically configure material filter fields to detect duplicate data of materials; through the configuration table in the duplicate checking method, the fields that are not allowed to be repeated are configured, and the material data is automatically detected when it is imported to ensure the uniqueness of the material data. A material data duplicate checking system based on configurable rules includes a configuration table, a material setting module and a duplicate checking module. The configuration table is established in the information system database; the material setting module stores the set material attribute fields in the created configuration table; when the material is added or imported, the duplicate checking module combines SQL into a where condition to check for duplicates and automatically screen the data. Summary of the invention

[0004] In view of the defects in the prior art, the object of the present invention is to provide a material similarity retrieval method and system.

[0005] A material similarity retrieval method provided by the present invention includes:

[0006] Step S1: Build a material duplicate checking model;

[0007] Step S2: Use the material duplication checking model to check and compare the target material with the existing materials in the current material pool. When the similarity is higher than a preset value, the current target material is considered to be a duplicate material;

[0008] The material duplicate checking model is to check for duplicate material codes in the code library by calculating the similarity of the material name, specification and material long description.

[0009] Preferably, step S2 comprises:

[0010] Step S2.1: Obtain material information based on the material data table, the material information including: material name, model specification and material long description;

[0011] Step S2.2: preprocessing the acquired material information to obtain preprocessed material information;

[0012] Step S2.3: input the preprocessed material name into the material name similarity model to obtain the material name similarity between the target material and the existing materials in the current material pool;

[0013] Step S2.4: input the preprocessed model specifications into the model specification similarity model to obtain the model specification similarity between the target material and the existing materials in the current material pool;

[0014] Step S2.5: input the preprocessed material long description into the long description similarity model to obtain the material long description similarity between the target material and the existing materials in the current material pool;

[0015] Step S2.6: Based on the obtained material name similarity, model specification similarity and material long description similarity, the similarity between the target material and the existing materials in the material pool is calculated through the configured preset weights and adjustment parameters;

[0016] The material name similarity model is to obtain the material name similarity by calculating the semantic similarity of the material names;

[0017] The long description similarity model is to obtain the long description similarity by calculating the technical attribute encoding similarity;

[0018] The model specification similarity model obtains the model specification similarity by calculating the w2v encoding similarity.

[0019] Preferably, the step S2.2 comprises:

[0020] For the acquired material names, remove the special characters that meet the preset requirements;

[0021] For the model specifications obtained, remove non-English characters and numbers;

[0022] For the long description of the obtained material, the key technical attributes are extracted through the large language model, and the extracted key technical attributes are structured.

[0023] Preferably, step S2.3 includes: converting the material name into a semantic vector through a fine-tuned BERT model, constructing a vector index based on the semantic vector, using an L2 index to calculate the similarity between the target material name and the names of existing materials in the material pool, and normalizing the similarity.

[0024] Preferably, the step S2.4 comprises: training w2v with a model specification vocabulary for the preprocessed model specifications, and encoding with the trained w2v.

[0025] Preferably, step S2.5 includes: for the structured long description of the material, obtaining the technical attribute value root by the TF-IDF method, and normalizing the attribute value; then performing onehot encoding and normalizing the numerical value; and then calculating the vector similarity for the long description of the material and the model specification respectively, and normalizing the similarity.

[0026] Preferably, the step S2.6 comprises:

[0027] SIM=A*SIM1+B*SIM2+C*SIM3+D

[0028] Among them, A, B, and C represent the weights of the configurations; SIM1 represents the similarity of material names; SIM2 represents the similarity of material long descriptions; SIM3 represents the similarity of model specifications; and D represents the adjustment parameter.

[0029] A material similarity retrieval system provided according to the present invention includes:

[0030] Module M1: Build a material duplicate checking model;

[0031] Module M2: Use the material duplication check model to check and compare the target material with the existing materials in the current material pool. When the similarity is higher than the preset value, the current target material is considered to be a duplicate material;

[0032] The material duplicate checking model is to check for duplicate material codes in the code library by calculating the similarity of the material name, specification and material long description.

[0033] Preferably, the module M2 comprises:

[0034] Module M2.1: Obtain material information based on the material data table, the material information includes: material name, model specification and material long description;

[0035] Module M2.2: pre-process the acquired material information to obtain pre-processed material information;

[0036] Module M2.3: Input the preprocessed material name into the material name similarity model to obtain the material name similarity between the target material and the existing materials in the current material pool;

[0037] Module M2.4: Input the preprocessed model specifications into the model specification similarity model to obtain the model specification similarity between the target material and the existing materials in the current material pool;

[0038] Module M2.5: Input the preprocessed material long description into the long description similarity model to obtain the material long description similarity between the target material and the existing materials in the current material pool;

[0039] Module M2.6: Based on the obtained material name similarity, model specification similarity and material long description similarity, the similarity between the target material and the existing materials in the material pool is calculated through the configured preset weights and adjustment parameters;

[0040] The material name similarity model is to obtain the material name similarity by calculating the semantic similarity of the material names;

[0041] The long description similarity model is to obtain the long description similarity by calculating the technical attribute encoding similarity;

[0042] The model specification similarity model obtains the model specification similarity by calculating the w2v encoding similarity.

[0043] Preferably, the module M2.2 comprises:

[0044] For the acquired material names, remove the special characters that meet the preset requirements;

[0045] For the model specifications obtained, remove non-English characters and numbers;

[0046] For the acquired long description of the material, the key technical attributes are extracted through the large language model, and the extracted key technical attributes are structured;

[0047] The module M2.3 includes: converting the material name into a semantic vector through a fine-tuned BERT model, constructing a vector index based on the semantic vector, using an L2 index to calculate the similarity between the target material name and the name of existing materials in the material pool, and normalizing the similarity;

[0048] The module M2.4 includes: training w2v using a model specification vocabulary for the preprocessed model specifications, and encoding using the trained w2v;

[0049] The module M2.5 includes: for the structured material long description, obtaining the technical attribute value root by the TF-IDF method, and normalizing the attribute value; then performing onehot encoding and normalizing the numerical value; then calculating the vector similarity for the material long description and model specification respectively, and normalizing the similarity;

[0050] The module M2.6 includes:

[0051] SIM=A*SIM1+B*SIM2+C*SIM3+D

[0052] Among them, A, B, and C represent the weights of the configurations; SIM1 represents the similarity of material names; SIM2 represents the similarity of material long descriptions; SIM3 represents the similarity of model specifications; and D represents the adjustment parameter.

[0053] Compared with the prior art, the present invention has the following beneficial effects:

[0054] 1. The present invention provides a material similarity search method and system, which constructs an overall similarity model of the material based on the material name, model specification, and long description of the material data, optimizes the efficiency and scope of material duplicate checking, and realizes that when adding new materials, it prompts and recommends duplicate materials to reduce the number of newly added duplicate materials; or when searching in an existing material pool, it identifies the possible duplicate relationship of existing materials, thereby further handling the system function and further reducing the material duplication rate;

[0055] 2. The present invention can assist people in identifying scenarios where, although the material names, models, specifications, and other descriptions are different, they are essentially the same material at any time. This greatly reduces the scope of duplicate checking and confirmation of material codes, improves confirmation efficiency, and further reduces the material duplication rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0057] Figure 1 This is a schematic diagram of material duplication checking.

[0058] Figure 2 Build a flow chart for the material duplication check model.

[0059] Figure 3 Apply the flow chart for the material duplicate checking model. DETAILED DESCRIPTION

[0060] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0061] Example 1

[0062] According to a material similarity retrieval method and system provided by the present invention, the material name, model specification, and long description of the material data are preprocessed and the Embedding model is trained. By respectively constructing a material name similarity model and a model specification and long description similarity model, the overall similarity model of the material is obtained by presetting weights and adjusting parameters. When new material data is added or existing materials are queried, the material duplicate checking model is called, and the system calculates the similarity between the new material and the existing material, or the similarity between the existing materials, and provides a list of materials with a similarity of more than 0.9 for the material declaration personnel to view and determine whether the newly added material already exists; wherein, the similarity threshold interval is between 0-1, 0 is the lowest possibility, and 1 is the highest possibility.

[0063] The material similarity retrieval method comprises:

[0064] Step S1: Build a material duplicate checking model;

[0065] Step S2: Use the material duplication checking model to check and compare the target material with the existing materials in the current material pool. When the similarity is higher than a preset value, the current target material is considered to be a duplicate material;

[0066] The material duplicate checking model is to check for duplicate material codes in the code library by calculating the similarity of the material name, specification and material long description.

[0067] Specifically, the step S2 includes:

[0068] Step S2.1: Obtain material information based on the material data table, the material information including: material name, model specification and material long description;

[0069] Step S2.2: preprocessing the acquired material information to obtain preprocessed material information;

[0070] Step S2.3: input the preprocessed material name into the material name similarity model to obtain the material name similarity between the target material and the existing materials in the current material pool;

[0071] Step S2.4: input the preprocessed model specifications into the model specification similarity model to obtain the model specification similarity between the target material and the existing materials in the current material pool;

[0072] Step S2.5: input the preprocessed material long description into the long description similarity model to obtain the material long description similarity between the target material and the existing materials in the current material pool;

[0073] Step S2.6: Based on the obtained material name similarity, model specification similarity and material long description similarity, the similarity between the target material and the existing materials in the material pool is calculated through the configured preset weights and adjustment parameters;

[0074] The material name similarity model is to obtain the material name similarity by calculating the semantic similarity of the material names;

[0075] The long description similarity model is to obtain the long description similarity by calculating the technical attribute encoding similarity;

[0076] The model specification similarity model obtains the model specification similarity by calculating the w2v encoding similarity.

[0077] Specifically, the step S2.2 includes:

[0078] For the acquired material names, remove the special characters that meet the preset requirements;

[0079] For the model specifications obtained, remove non-English characters and numbers;

[0080] For the long description of the obtained material, the key technical attributes are extracted through the large language model, and the extracted key technical attributes are structured.

[0081] Specifically, step S2.3 includes: converting the material name into a semantic vector through the fine-tuned BERT model, building a vector index based on the semantic vector, using the L2 index to calculate the similarity between the target material name and the name of the existing materials in the material pool, and normalizing the similarity.

[0082] Specifically, the step S2.4 includes: training w2v with the model specification vocabulary according to the preprocessed model specification, and encoding with the trained w2v.

[0083] Specifically, the step S2.5 includes: for the structured material long description, obtaining the technical attribute value root by the TF-IDF method, and normalizing the attribute value; then performing onehot encoding and normalizing the numerical value; and then calculating the vector similarity for the material long description and model specifications respectively, and normalizing the similarity.

[0084] Specifically, the step S2.6 includes:

[0085] SIM=A*SIM1+B*SIM2+C*SIM3+D

[0086] Among them, A, B, and C represent the weights of the configurations; SIM1 represents the similarity of material names; SIM2 represents the similarity of material long descriptions; SIM3 represents the similarity of model specifications; and D represents the adjustment parameter.

[0087] The present invention also provides a material similarity retrieval system, which can be implemented by executing the process steps of the material similarity retrieval method, that is, those skilled in the art can understand the material similarity retrieval method as a preferred implementation of the material similarity retrieval system.

[0088] Example 2

[0089] Embodiment 2 is a preferred embodiment of Embodiment 1

[0090] A material similarity retrieval method provided by the present invention includes:

[0091] Step 1: Build a material duplication checking model, such as Figure 2 As shown;

[0092] The construction of the material duplication checking model mainly includes key steps such as data preprocessing, model training, sub-model construction, and comprehensive model construction.

[0093] Material duplication check model application process, such as Figure 3 As shown;

[0094] The application of the material duplication check model is divided into two processes: duplication check and comparison between new materials and existing materials, and duplication check and comparison between existing materials.

[0095] Material duplication check application functions:

[0096] The application of the material duplicate checking model requires the support of the following functions: duplicate checking task creation, duplicate checking task result viewing, and duplicate checking result handling.

[0097] Duplicate check task creation: When creating a task, you can filter the task materials. The main filtering conditions for task materials are as follows:

[0098] Category: The category here can be large, medium, or leaf, and multiple selections are supported.

[0099] Region: The place where the code is created. Multiple selections are supported.

[0100] Material status: Material status supports multiple selections.

[0101] Material retrieval: Material retrieval supports fuzzy search of material name, long description, model specification.

[0102] After the task materials are screened, call the "Check for duplicate content based on query results" service, the duplicate checking task is created successfully, and the duplicate checking calculation begins.

[0103] View the results of duplicate checking tasks: On the historical task viewing page, you can see detailed information about the task, including the task number, the filtering conditions for task materials, the quantity of task materials, the quantity processed, the execution status of the task, and the button and status for processing the task.

[0104] Disposal of duplicate check results: If the status of the button for processing the task is to go to process, click the button to enter the duplicate check disposal page. Set the similarity threshold of similar materials in the "Similarity" box, for example, set it to 0.7. After clicking "Query", the task materials whose maximum similarity in the calculation results of the task materials does not meet the threshold will be filtered out. View the materials that meet the similarity, view the details, and process the similar materials. There are two major types of processing methods supported: deactivate and maintain the original status. They represent the modification of the code status (i.e. changing the duplicate code from valid to invalid) and retaining the existing status without processing (generally used when there is no duplication).

[0105] Those skilled in the art know that, in addition to realizing the system and its various devices, modules, and units provided by the present invention in a purely computer-readable program code, it is entirely possible to realize the same functions in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered as a hardware component, and the devices, modules, and units included therein for realizing various functions can also be regarded as structures within the hardware component; the devices, modules, and units for realizing various functions can also be regarded as both software modules for realizing the method and structures within the hardware component.

[0106] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A material similarity retrieval method, characterized in that: include: Step S1: Build a material duplicate checking model; Step S2: Use the material duplication checking model to check and compare the target material with the existing materials in the current material pool. When the similarity is higher than a preset value, the current target material is considered to be a duplicate material; The material duplicate checking model is to check for duplicate material codes in the code library by calculating the similarity of the material name, specification and material long description.

2. The material similarity retrieval method according to claim 1, characterized in that: The step S2 comprises: Step S2.1: Obtain material information based on the material data table, the material information including: material name, model specification and material long description; Step S2.2: preprocessing the acquired material information to obtain preprocessed material information; Step S2.3: input the preprocessed material name into the material name similarity model to obtain the material name similarity between the target material and the existing materials in the current material pool; Step S2.4: input the preprocessed model specifications into the model specification similarity model to obtain the model specification similarity between the target material and the existing materials in the current material pool; Step S2.5: input the preprocessed material long description into the long description similarity model to obtain the material long description similarity between the target material and the existing materials in the current material pool; Step S2.6: Based on the obtained material name similarity, model specification similarity and material long description similarity, the similarity between the target material and the existing materials in the material pool is calculated through the configured preset weights and adjustment parameters; The material name similarity model is to obtain the material name similarity by calculating the semantic similarity of the material names; The long description similarity model is to obtain the long description similarity by calculating the technical attribute encoding similarity; The model specification similarity model obtains the model specification similarity by calculating the w2v encoding similarity.

3. The material similarity retrieval method according to claim 2, characterized in that: The step S2.2 comprises: For the acquired material names, remove the special characters that meet the preset requirements; For the model specifications obtained, remove non-English characters and numbers; For the long description of the obtained material, the key technical attributes are extracted through the large language model, and the extracted key technical attributes are structured.

4. The material similarity retrieval method according to claim 2, characterized in that: The step S2.3 includes: converting the material name into a semantic vector through the fine-tuned BERT model, constructing a vector index based on the semantic vector, using the L2 index to calculate the similarity between the target material name and the name of the existing materials in the material pool, and normalizing the similarity.

5. The material similarity retrieval method according to claim 2, characterized in that: The step S2.4 includes: training w2v using a model specification vocabulary for the preprocessed model specifications, and encoding using the trained w2v.

6. The material similarity retrieval method according to claim 3, characterized in that: The step S2.5 includes: for the structured material long description, obtaining the technical attribute value root by the TF-IDF method, and normalizing the attribute value; then performing onehot encoding and normalizing the numerical value; and then calculating the vector similarity for the material long description and model specifications respectively, and normalizing the similarity.

7. The material similarity retrieval method according to claim 2, characterized in that: The step S2.6 comprises: SIM=A*SIM1+B*SIM2+C*SIM3+D Among them, A, B, and C represent the weights of the configurations; SIM1 represents the similarity of material names; SIM2 represents the similarity of material long descriptions; SIM3 represents the similarity of model specifications; and D represents the adjustment parameter.

8. A material similarity retrieval system, characterized in that: include: Module M1: Build a material duplicate checking model; Module M2: Use the material duplication check model to check and compare the target material with the existing materials in the current material pool. When the similarity is higher than the preset value, the current target material is considered to be a duplicate material; The material duplicate checking model is to check for duplicate material codes in the code library by calculating the similarity of the material name, specification and material long description.

9. The material similarity retrieval system according to claim 8, characterized in that: The module M2 comprises: Module M2.1: Obtain material information based on the material data table, the material information includes: material name, model specification and material long description; Module M2.2: pre-process the acquired material information to obtain pre-processed material information; Module M2.3: Input the preprocessed material name into the material name similarity model to obtain the material name similarity between the target material and the existing materials in the current material pool; Module M2.4: Input the preprocessed model specifications into the model specification similarity model to obtain the model specification similarity between the target material and the existing materials in the current material pool; Module M2.5: Input the preprocessed material long description into the long description similarity model to obtain the material long description similarity between the target material and the existing materials in the current material pool; Module M2.6: Based on the obtained material name similarity, model specification similarity and material long description similarity, the similarity between the target material and the existing materials in the material pool is calculated through the configured preset weights and adjustment parameters; The material name similarity model is to obtain the material name similarity by calculating the semantic similarity of the material names; The long description similarity model is to obtain the long description similarity by calculating the technical attribute encoding similarity; The model specification similarity model obtains the model specification similarity by calculating the w2v encoding similarity.

10. The material similarity retrieval system according to claim 9, characterized in that: The module M2.2 includes: For the acquired material names, remove the special characters that meet the preset requirements; For the model specifications obtained, remove non-English characters and numbers; For the acquired long description of the material, the key technical attributes are extracted through the large language model, and the extracted key technical attributes are structured; The module M2.3 includes: converting the material name into a semantic vector through a fine-tuned BERT model, constructing a vector index based on the semantic vector, using an L2 index to calculate the similarity between the target material name and the name of existing materials in the material pool, and normalizing the similarity; The module M2.4 includes: training w2v using a model specification vocabulary for the preprocessed model specifications, and encoding using the trained w2v; The module M2.5 includes: for the structured material long description, obtaining the technical attribute value root by the TF-IDF method, and normalizing the attribute value; then performing onehot encoding and normalizing the numerical value; then calculating the vector similarity for the material long description and model specification respectively, and normalizing the similarity; The module M2.6 includes: SIM=A*SIM1+B*SIM2+C*SIM3+D Among them, A, B, and C represent the weights of the configurations; SIM1 represents the similarity of material names; SIM2 represents the similarity of material long descriptions; SIM3 represents the similarity of model specifications; and D represents the adjustment parameter.

Images