A product standard document duplicate checking method, device and equipment

Abstract

The application relates to the technical field of text duplication checking, and discloses a product standard document duplication checking method, device and equipment, which comprises the following steps: for each relevant product standard document, performing text extraction on multiple parts of the relevant product standard document to obtain relevant part texts; based on the relevant part texts of each relevant product standard document, multiple duplication checking databases are constructed; performing text extraction on multiple parts of a target product standard document to obtain target part texts; and based on the target part texts of the target product standard document, the multiple duplication checking databases are used to perform duplication checking to obtain a duplication checking result. According to the application, the duplication checking is performed on parts, the key parts are focused on, the pertinence of the duplication checking is significantly improved, the design of the multiple duplication checking databases can more accurately identify the core differences between the product standard documents, the whole duplication checking process is designed according to the characteristics of the product standard documents, actual demands can be met, and the duplication checking accuracy is improved.

Description

Technical Field

[0001] This invention relates to the field of text plagiarism detection technology, specifically to a method, apparatus, and equipment for plagiarism detection of product standard documents. Background Technology

[0002] Product standard documents, as documents that standardize the technical requirements, testing methods, and quality assessment procedures of specific products, are written in accordance with a strict standard system. The structure includes multiple parts such as cover, preface, scope, normative references, terms and definitions, main text (covering raw materials, production requirements, production equipment, personnel, production process, flow, storage and transportation, etc.), and appendices.

[0003] In the field of product standard document plagiarism detection, currently used text feature plagiarism detection, vector plagiarism detection, and artificial intelligence semantic plagiarism detection technologies have certain limitations when dealing with product standard documents. For example, for the cover page, only the standard title has plagiarism detection value; for the preface and normative references, due to their inherent characteristics, they are not suitable as plagiarism detection content; for the scope and main text, since most product standards are written according to a fixed framework, existing text feature plagiarism detection methods struggle to accurately identify key differences.

[0004] Based on the above, existing plagiarism detection technologies cannot effectively focus on the core differences in product standard documents and cannot meet the actual needs of product standard plagiarism detection, resulting in poor targeting and accuracy. Summary of the Invention

[0005] In view of this, the present invention provides a method, apparatus and equipment for plagiarism detection of product standard documents, in order to solve the problem that existing plagiarism detection technologies cannot effectively focus on the core differences of product standard documents and cannot meet the actual needs of product standard plagiarism detection, resulting in poor targeting and accuracy.

[0006] In a first aspect, the present invention provides a method for deduplication of product standard documents, the method comprising:

[0007] For each relevant product standard document, text extraction is performed on multiple parts of the relevant product standard document to obtain the relevant text.

[0008] Multiple plagiarism detection databases are constructed based on relevant text sections of each related product standard document.

[0009] Text extraction is performed on multiple parts of the target product's standard document to obtain the target text.

[0010] Multiple plagiarism detection databases are used to perform plagiarism checks on the target text portion of the target product's standard document, and the plagiarism detection results are obtained.

[0011] This invention improves the accuracy of plagiarism detection by selectively extracting text from different parts of product standard documents. Based on the extracted text from each relevant product standard document, multiple plagiarism databases are constructed. Data can be stored and managed separately for the characteristics of different parts, making the plagiarism detection process more refined and better focusing on core differences. When checking a target product standard document for plagiarism, text is first extracted from multiple parts to obtain the target text, which is then compared with the corresponding plagiarism databases to comprehensively assess the degree of duplication in the target product standard document. By processing text in parts, the focus of plagiarism detection can be placed on key parts, significantly improving the targeting of the detection. The design of multiple plagiarism databases can more accurately identify the core differences between product standard documents, and the entire plagiarism detection process is designed according to the characteristics of product standard documents to ensure that it meets practical needs while improving detection accuracy.

[0012] In one alternative implementation, each relevant product standard document includes a cover section, a scope section, a terms and definitions section, and a body section;

[0013] For each relevant product standard document, text extraction is performed on multiple parts of the relevant product standard document to obtain the corresponding partial text, including:

[0014] Extract the standard name from the cover page of the relevant product standard document;

[0015] Extract a scope summary from the scope section of the relevant product standard document;

[0016] Extract term definitions from the terminology and definition section of relevant product standard documents;

[0017] Extract the main text summary, product methods, and product indicators from the main text of relevant product standard documents;

[0018] The standard name, scope summary, term definitions, main text summary, product methods, and product indicators of the relevant product standard documents shall be included as the relevant parts of the text of the relevant product standard documents.

[0019] This invention extracts key information from the cover, scope, terms and definitions, and main text of relevant product standard documents, enabling it to focus on differences in key parts of the documents and ensuring that the plagiarism detection results are more targeted and practical.

[0020] In one optional implementation, the plagiarism detection database is a text database;

[0021] Based on relevant text sections of each related product standard document, multiple plagiarism detection databases are constructed, including:

[0022] The standard name in each relevant part of the text is segmented into words to obtain multiple first word texts corresponding to the standard name, and an inverted index is set.

[0023] The range summary in each relevant part of the text is segmented into words to obtain multiple second word texts corresponding to the range summary, and inverted index items are set.

[0024] The main text summary in each relevant part of the text is segmented into words to obtain multiple third-word texts corresponding to the main text summary, and inverted index items are set.

[0025] Based on multiple first-word texts and their inverted index entries, multiple second-word texts and their inverted index entries, and multiple third-word texts and their inverted index entries, construct name inverted index table, range inverted index table, and text inverted index table respectively;

[0026] A text database is constructed based on the name inverted index table, the range inverted index table, and the text inverted index table.

[0027] This invention achieves document deduplication from multiple dimensions by segmenting the standard names, scopes, and text summaries of the extracted relevant text parts and constructing corresponding inverted index tables. This further ensures the comprehensiveness and reliability of the deduplication results, and the text database built based on the inverted index has good scalability and query speed.

[0028] In one optional implementation, the deduplication database is a vector database;

[0029] Based on relevant text portions of each related product standard document, multiple plagiarism detection databases are constructed, including:

[0030] By using a word vector model, the term definitions in each relevant part of the text are transformed into vectors, resulting in multiple term definition vectors;

[0031] By using a word vector model, the product methods in each relevant part of the text are transformed into vectors, resulting in multiple product method vectors;

[0032] By using a word vector model, product indicators in each relevant part of the text are transformed into vectors, resulting in multiple product indicator vectors.

[0033] A vector database is constructed based on multiple terminology definition vectors, multiple product method vectors, multiple product indicator vectors, and the relevant product standard documents corresponding to each vector.

[0034] This invention transforms terminology definitions, product methods, and product metrics into vectors. Based on word vector models, it can construct a vector database. When performing plagiarism checks based on this vector database, it can more accurately capture semantic similarity compared to traditional text matching methods, thereby improving the accuracy of plagiarism check results. This is particularly suitable for fields such as product standard documents, which have high requirements for semantic accuracy. Furthermore, the vector database also has good scalability and query speed.

[0035] In one alternative implementation, the target text includes the standard name, scope summary, term definitions, main text summary, product method, and product metrics;

[0036] Multiple plagiarism detection databases are used to perform plagiarism checks on the target text portion of the target product's standard document, yielding the following results:

[0037] Using a text database, plagiarism checks were performed based on the standard names, scope summaries, and main text summaries in the target text, and the corresponding similarity rates and similarity levels were obtained.

[0038] Using a vector database, we performed plagiarism checks based on the terminology definitions, product methods, and product metrics in the target text, and obtained the corresponding similarity scores and similarity levels.

[0039] Determine the weight of each part in the target text;

[0040] The overall similarity score is determined based on the similarity rates of the standard names, scope summaries, and main text summaries in the target text, the similarity of the term definitions, product methods, and product indicators in the target text, and the weight of each part of the target text.

[0041] The similarity rate and similarity level of the standard name, scope summary, and main text summary in the target text, the similarity rate and similarity level of the terminology definition, product method, and product indicator in the target text, and the overall similarity rate are used as the plagiarism check results of the target product standard document.

[0042] This invention achieves comprehensive plagiarism detection of target product standard documents from multiple dimensions by performing duplication rate checks on standard names, scope summaries, and main text summaries in the target text, as well as similarity checks on terminology definitions, product methods, and product indicators, thereby providing more comprehensive and accurate plagiarism detection results.

[0043] In one optional implementation, a text database is used to perform plagiarism checks based on standard names in the target text, obtaining the corresponding repetition rate and repetition rate level, including:

[0044] The standard names in the target text are segmented into multiple target word texts.

[0045] For each target word text, the relevant standard document corresponding to the target word text is determined based on the name inverted index table of the text database.

[0046] Based on the relevant standard documents corresponding to multiple target word texts, determine the number of intersections and the number of unions;

[0047] The ratio of the number of intersections to the number of unions is determined as the repetition rate of the standard names in the target product's standard documents;

[0048] The repetition rate level is determined based on the range to which the repetition rate falls.

[0049] This invention utilizes a name inverted index table to quickly identify the relevant standard documents corresponding to each target word text in the standard name of the target product standard document, thereby determining the number of intersections and unions, and calculating the comparison repetition rate. This efficiently achieves accurate matching of standard names and determines the repetition rate level, making the plagiarism detection results more intuitive and easier to understand.

[0050] In one optional implementation, a vector database is used to perform plagiarism checks based on term definitions in the target text, obtaining corresponding similarity scores and similarity levels, including:

[0051] For each term definition vector in the vector database, based on the term definitions and term definition vectors in the target text, determine the vector dot product, the first vector modulus, and the second vector modulus.

[0052] Determine the product of the first vector modulus and the second vector modulus, and determine the ratio of the vector dot product to the product as the comparison similarity between the terminology definition and the terminology definition vector in the target product standard document;

[0053] The maximum comparative similarity between the terminology definitions in the target product standard document and all terminology definition vectors is determined as the similarity of the terminology definitions in the target product standard document.

[0054] The similarity level is determined based on the range to which the similarity belongs.

[0055] This invention calculates the comparative similarity by using vector dot product, first vector modulus, and second vector modulus. It can evaluate the similarity between term definitions in the target product standard document and term definition vectors in the vector database from a semantic level. Compared with traditional text matching methods, it captures the deep association between term definitions more accurately, determines the maximum comparative similarity as the final similarity, and determines the similarity level, making the plagiarism detection results more intuitive and easier to understand.

[0056] In one alternative implementation, the method further includes:

[0057] A plagiarism check report is generated based on the plagiarism check results of the target product's standard documents.

[0058] This invention generates plagiarism detection reports based on the results, making the results more intuitive and easier to understand, which helps users understand the results and provides support for subsequent decision-making.

[0059] Secondly, the present invention provides a device for checking the duplication of product standard documents, the device comprising:

[0060] The first extraction module is used to extract text from multiple parts of each relevant product standard document to obtain the relevant text.

[0061] The building module is used to construct multiple plagiarism detection databases based on relevant text sections of each related product standard document;

[0062] The second extraction module is used to extract text from multiple parts of the target product standard document to obtain the target text.

[0063] The plagiarism detection module uses multiple plagiarism databases to perform plagiarism checks on the target text portion of the target product's standard document and obtain the plagiarism detection results.

[0064] Thirdly, the present invention provides a computer device, including: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to perform the product standard document deduplication method described in the first aspect or any corresponding embodiment.

[0065] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to execute the deduplication method for product standard documents described in the first aspect or any corresponding embodiment thereof.

[0066] Fifthly, the present invention provides a computer program product, including computer instructions, which are used to cause a computer to execute a method for deduplication of product standard documents described in the first aspect or any corresponding embodiment. Attached Figure Description

[0067] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0068] Figure 1 This is a flowchart of a method for deduplication of product standard documents according to an embodiment of the present invention;

[0069] Figure 2 This is a structural block diagram of a product standard document plagiarism detection device according to an embodiment of the present invention;

[0070] Figure 3 This is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. Detailed Implementation

[0071] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0072] In the field of product standard document plagiarism detection, commonly used text feature plagiarism detection, vector plagiarism detection, and artificial intelligence semantic plagiarism detection technologies have certain limitations when dealing with product standard documents. They suffer from poor targeting and accuracy because they cannot effectively focus on the core differences between product standard documents and thus cannot meet the actual needs of plagiarism detection. This invention addresses this issue by using a segmented processing approach, which focuses plagiarism detection on key parts, significantly improving targeting. The design of multiple plagiarism databases enables more accurate identification of core differences between product standard documents. Furthermore, the entire plagiarism detection process is designed according to the characteristics of product standard documents, ensuring that it meets actual needs while improving plagiarism detection accuracy.

[0073] According to an embodiment of the present invention, a method for deduplication of product standard documents is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0074] This embodiment provides a method for deduplication of product standard documents, which can be used on terminals such as computers. Figure 1This is a flowchart of a method for deduplication of product standard documents according to an embodiment of the present invention, such as... Figure 1 As shown, the process includes the following steps:

[0075] Step S101: For each relevant product standard document, text extraction is performed on multiple parts of the document to obtain the relevant text. Specifically, as existing documents that uniformly regulate one or more aspects of a product, relevant product standard documents are essential for ensuring the quality and innovation of new product standard documents and avoiding content duplication when developing new documents. Product standard documents follow a unified format and include multiple parts such as cover, preface, scope, normative references, terms and definitions, main text, and appendices. However, different parts contribute differently to the duplication checking process. If all parts are checked for duplication, or if all parts are checked using the same method, the results will lack specificity and accuracy. Therefore, for each relevant product standard document, text extraction is performed on specific parts to form the relevant text, providing basic data for subsequent duplication checking processes.

[0076] Step S102: Based on the relevant text portions of each related product standard document, construct multiple plagiarism detection databases. Specifically, different plagiarism detection databases are constructed according to different parts of the relevant text. During plagiarism detection, targeted plagiarism checks can be performed on different parts, and using databases significantly improves detection efficiency. When new related product standard documents appear, the constructed plagiarism detection databases can be expanded and updated to ensure that the plagiarism detection process is based on the latest information, further improving the accuracy of the plagiarism detection results.

[0077] Step S103 involves extracting text from multiple parts of the target product standard document to obtain the target text. Specifically, the target product standard document is the document to be checked for plagiarism. Referring to the text extraction process for related product standard documents in step S101, text extraction is also performed on it to form the target text. The target text will serve as the core data for subsequent plagiarism checks, used to perform plagiarism checks against the established plagiarism database.

[0078] Step S104 involves using multiple plagiarism detection databases to perform plagiarism checks on the target portion of the target product standard document, thereby obtaining the plagiarism detection results. Specifically, different plagiarism detection databases are used for targeted plagiarism checks on different content within the target portion of the text. This allows for more accurate identification of core differences between product standard documents, improving plagiarism detection accuracy and effectively enhancing the overall plagiarism detection process.

[0079] This invention improves the accuracy of plagiarism detection by selectively extracting text from different parts of product standard documents. Based on the extracted text from each relevant product standard document, multiple plagiarism databases are constructed. Data can be stored and managed separately for the characteristics of different parts, making the plagiarism detection process more refined and better focusing on core differences. When checking a target product standard document for plagiarism, text is first extracted from multiple parts to obtain the target text, which is then compared with the corresponding plagiarism databases to comprehensively assess the degree of duplication in the target product standard document. By processing text in parts, the focus of plagiarism detection can be placed on key parts, significantly improving the targeting of the detection. The design of multiple plagiarism databases can more accurately identify the core differences between product standard documents, and the entire plagiarism detection process is designed according to the characteristics of product standard documents to ensure that it meets practical needs while improving detection accuracy.

[0080] This embodiment provides a method for deduplication of product standard documents, which can be used in the aforementioned terminal. The method specifically includes the following steps:

[0081] Step S201: For each relevant product standard document, extract the text from multiple parts of the relevant product standard document to obtain the relevant text. Each relevant product standard document includes a cover section, a scope section, a terminology and definition section, and a main text section.

[0082] Specifically, step S201 includes:

[0083] Step S2011: Extract the standard name from the cover section of the relevant product standard document. Specifically, since product standard documents follow a uniform format, the standard name is generally located in a specific position on the cover, such as the first line of the title bar, and the font is large and the format is prominent. Therefore, by using text parsing technology and writing a program to set specific text positioning rules, the standard name can be extracted from the cover section. More specifically, using a text processing language, such as Python's regular expression library `re`, assuming the cover section is stored in the variable `cover_text`, the following program is written to extract the standard name:

[0084]

[0085] The code above is just an example and is not intended to be limiting. In actual applications, it can be flexibly adjusted according to specific needs.

[0086] Step S2012 involves extracting a scope summary from the scope section of the relevant product standard document. Specifically, using natural language processing technology, regular expression rules are constructed based on common phrases in the scope section, such as "This standard applies to..." or "This standard covers...". Then, a string matching algorithm is used to traverse and match the text in the scope section to extract the scope summary. For example, for the text "This standard applies to the manufacturing of electronic products, covering raw material procurement, production processes, quality inspection, etc.", rule matching can extract "Applies to the manufacturing of electronic products, covering raw material procurement, production processes, quality inspection, etc." as the scope summary. In practice, the following code can be written using Python's `re` library:

[0087]

[0088] The code above is just an example and is not intended to be limiting. In actual applications, it can be flexibly adjusted according to specific needs.

[0089] Step S2013: Extract term definitions from the terminology and definition section of the relevant product standard documents. Specifically, firstly, use a part-of-speech tagging tool, such as the part-of-speech tagger in NLTK (Natural Language Toolkit), to tag each word in the terminology and definition section with its part of speech. Then, use a deep learning-based named entity recognition model, such as the BiLSTM-CRF (Bidirectional Long Short-Term Memory-Conditional Random Field) model, to identify the terms and extract their corresponding definitions. Taking "Term: Semiconductor material; Definition: Material used to manufacture semiconductor devices and integrated circuits" as an example, the model processing can accurately extract "semiconductor material" and its definition. More specifically, the following code can be used when using NLTK's part-of-speech tagger to tag the text with part-of-speech tags:

[0090] import nltk

[0091] from nltk.tokenize import word_tokenize

[0092] Terminology: Semiconductor materials; Definition: Materials used in the manufacture of semiconductor devices and integrated circuits.

[0093] tokens = word_tokenize(text)

[0094] pos_tags=nltk.pos_tag(tokens)

[0095] The code above is just an example and is not intended to be limiting. In actual applications, it can be flexibly adjusted according to specific needs.

[0096] Step S2014 involves extracting the main text summary, product methods, and product indicators from the main body of the relevant product standard documents. Specifically, the main text is divided into different sections based on specific keywords such as "raw materials," "production requirements," and "production equipment." For each section, the TextRank algorithm is used to extract the main text summary. This algorithm constructs a word graph model of the text, establishing connections between words through co-occurrence relationships, with edge weights determined by the frequency of word co-occurrence. By iteratively calculating the importance score of each word, several key sentences with high scores are selected to form the main text summary. When extracting product methods and product indicators, for product methods, common method expression patterns are pre-defined, such as "using [specific method name]" or "performed through [operation method]," and text fragments matching these patterns are identified by combining part-of-speech tagging and syntactic analysis. For product indicators, a pre-trained indicator recognition model is used. This model, trained on a large amount of standard text data, can identify indicators such as "product qualification rate ≥ 98%" and "production energy consumption ≤ 50 kW·h / piece." Taking text extraction as an example, the TextRank algorithm is implemented using Python's textrank4zh library:

[0097]

[0098] The code above is just an example and is not intended to be limiting. In actual applications, it can be flexibly adjusted according to specific needs.

[0099] Step S2015 involves using the standard name, scope summary, terminology definitions, text summary, product methods, and product indicators from relevant product standard documents as relevant text portions of the relevant product standard documents. Specifically, the standard name, scope summary, terminology definitions, text summary, product methods, and product indicators extracted from relevant product standard documents reflect the core parts of the document. Using this relevant text portion in the subsequent plagiarism detection process significantly improves the targeting and accuracy of the plagiarism check.

[0100] In some optional implementations, while the preface and normative references sections of relevant product standard documents are not included in plagiarism checks, they can still be accurately extracted during the text extraction process. The preface typically contains information such as the purpose and background of the standard's development; the normative references section lists other standards or documents cited during the standard's development. Accurately defining these sections helps in understanding the overall structure of the product standard document. Specifically, these sections can be extracted from the document using specific text identifiers, such as keywords like "preface" and "normative references," employing string search algorithms.

[0101] Step S202: Based on the relevant text sections of each related product standard document, construct multiple plagiarism detection databases.

[0102] Specifically, when the plagiarism detection database is a text database, step S202 above includes:

[0103] Step S2021 involves segmenting the standard name in each relevant text section into multiple first-word texts corresponding to the standard name, and setting up an inverted index entry. Specifically, when the deduplication database is a text database, the database construction is achieved through steps S2021 to S2025. For the standard name in each relevant text section, taking "smart home appliance product standard" as an example, it will be split into first-word texts: "smart", "home appliance", and "product standard". For each such first-word text, a corresponding inverted index entry is set. Assume the set of all relevant standard documents is D = {d1, d2, ..., d...} n}, where d n This represents the index of the nth relevant standard document. For any first word text w in any standard name, its inverted index entry is represented as I(w) = {d i1 ,d i2 ,…,d im} represents the ID (Identifier) ​​of all relevant standard documents containing the first word text w. More specifically, in Python, a simple inverted index can be built using a dictionary data structure:

[0104]

[0105] The code above is just an example and is not intended to be limiting. In actual applications, it can be flexibly adjusted according to specific needs.

[0106] Step S2022 involves segmenting the range summary in each relevant text portion into words, obtaining multiple second-word texts corresponding to the range summary, and setting up inverted index entries. Specifically, referring to step S2022, the range summary is segmented into words, and an inverted index entry is set up for each second-word text. During word segmentation, the word segmentation strategy and index entry recording method need to be adjusted according to the text characteristics of the range summary.

[0107] Step S2023 involves segmenting the main text summary in each relevant section into words, obtaining multiple third-word texts corresponding to the main text summary, and setting up inverted index entries. Specifically, referring to step S2022, the main text summary is segmented into words, and an inverted index entry is set up for each third-word text. During word segmentation, the word segmentation strategy and index entry recording method need to be adjusted according to the text characteristics of the main text summary.

[0108] Step S2024 involves constructing a name inverted index table, a range inverted index table, and a text inverted index table based on multiple first-word texts and their inverted index entries, multiple second-word texts and their inverted index entries, and multiple third-word texts and their inverted index entries, respectively. Specifically, an inverted index table is constructed for each first-word text and its inverted index entry obtained by segmenting the standard name. The table records the ID of the relevant standard document to which each word belongs. During plagiarism detection, the standard name to be checked is segmented, and then the relevant matching standard document is quickly located by querying the inverted index table. The construction process of the range inverted index table and the text inverted index table is described above and will not be repeated here.

[0109] Step S2025 involves constructing a text database based on the name inverted index table, the range inverted index table, and the text inverted index table. Specifically, by segmenting the standard names, ranges, and text summaries of the extracted relevant text parts into words and constructing corresponding inverted index tables, data support for document plagiarism detection is provided from multiple dimensions, further ensuring the comprehensiveness and reliability of the plagiarism detection results. Furthermore, the text database built based on the inverted indexes not only possesses good scalability, facilitating timely data updates as new documents are added, but also offers significant advantages in query speed, efficiently meeting the needs of plagiarism detection work.

[0110] Specifically, when the deduplication database is a vector database, step S202 above includes:

[0111] Step S2026 involves using a word vector model to convert the term definitions in each relevant text section into vectors, resulting in multiple term definition vectors. Specifically, when the plagiarism detection database is a vector database, the database is constructed through steps S2026 to S2029. For each term definition in the relevant text section, the Word2Vec word vector model is used to convert the text into vector form. More specifically, assume that any term definition is T = {t1, t2, ..., t...} m}, using the trained Word2Vec model, each word it includes is mapped to a k-dimensional vector, and the term definition vector corresponding to each term definition is obtained by the average pooling method shown in the following formula (1).

[0112]

[0113] in, This represents the term definition vector corresponding to term definition T; m represents the number of words included in term definition T; t i This represents the i-th word in the term definition; Let represent the vector corresponding to the i-th word.

[0114] In some alternative implementations, the Python gensim library is used to train the Word2Vec model and generate vectors:

[0115] from gensim.models import Word2Vec

[0116] sentences = [["Terminology","Semiconductor Materials","Definition","Used in","Manufacturing","Semiconductor Devices","And","Integrated Circuits","Materials"]]

[0117]

[0118] The code above is just an example and is not intended to be limiting. In actual applications, it can be flexibly adjusted according to specific needs.

[0119] Step S2027 involves using a word vector model to transform the product methods in each relevant part of the text into vectors, resulting in multiple product method vectors. Specifically, referring to step S2026, the text of product methods is vectorized to obtain product method vectors. The text preprocessing and vector combination methods can be adjusted according to the actual situation.

[0120] Step S2028 involves using a word vector model to convert product indicators in each relevant part of the text into vectors, resulting in multiple product indicator vectors. Specifically, referring to step S2026, the text of the product indicators is vectorized to obtain product indicator vectors. The text preprocessing and vector combination methods can be adjusted according to the actual situation.

[0121] Step S2029 involves constructing a vector database based on multiple terminology definition vectors, multiple product method vectors, multiple product indicator vectors, and the corresponding relevant product standard documents for each vector. Specifically, a vector database, such as Faiss, is constructed based on all vectors obtained after vectorizing terminology definitions, product methods, and product indicators, recording the text content corresponding to each vector and the ID of its associated standard document. By converting terminology definitions, product methods, and product indicators into vectors and constructing a vector database, semantic similarity can be captured more accurately compared to traditional text matching methods when performing plagiarism detection based on the vector database. This improves the accuracy of plagiarism detection results, making it particularly suitable for fields like product standard documents where semantic accuracy is crucial. Furthermore, the vector database also boasts good scalability and query speed.

[0122] Step S203 involves extracting text from multiple sections of the target product standard document to obtain the target text. For details, please refer to [link to relevant documentation]. Figure 1 Step S103 of the illustrated embodiment will not be described again here.

[0123] Step S204: Using multiple plagiarism detection databases, plagiarism detection is performed on the target part of the target product standard document to obtain the plagiarism detection results. The target part of the text includes the standard name, scope summary, terminology definition, main text summary, product method, and product indicators.

[0124] Specifically, step S204 includes:

[0125] Step S2041: Using a text database, perform plagiarism checks based on the standard name, scope summary, and main text summary in the target text to obtain the corresponding similarity rate and similarity rate level.

[0126] In some optional implementations, step S2041 above uses a text database to perform plagiarism checks based on standard names in the target text, obtaining the corresponding repetition rate and repetition rate level, including:

[0127] Step a1 involves segmenting the standard names in the target text into multiple target word texts. Specifically, a word vector model is used to segment the standard names in the target text. For example, "glass cup quality product standard" is split into three target word texts: "glass cup", "quality", and "product standard".

[0128] Step a2: For each target word text, determine the relevant standard documents corresponding to the target word text based on the name inverted index table of the text database. Specifically, since the name inverted index table records the ID of each word text and its corresponding relevant standard document, for the target word text "glass", the ID of the relevant standard document containing "glass" is determined from the name inverted index table as [1,3,5]. Similarly, the ID of the relevant standard document containing "quality" is [2,3,5], and the ID of the relevant standard document containing "product standard" is [3,5].

[0129] Step a3: Based on the relevant standard documents corresponding to the multiple target word texts, determine the number of intersections and the number of unions. Specifically, based on the above assumptions, the intersection of the relevant standard documents corresponding to the three target word texts is [3,5], and the union is [1,2,3,5]. Therefore, the number of intersections is determined to be 2, and the number of unions is determined to be 4.

[0130] Step a4: The ratio of the number of intersections to the number of unions is determined as the repetition rate of the standard names in the target product standard documents. Specifically, determining the ratio of the number of intersections to the number of unions, which is 2 / 4 = 50%, yields a repetition rate of 50% for the standard names in the target product standard documents.

[0131] Step a5: Determine the repetition rate level based on the range of repetition rates. Specifically, assuming the repetition rate of the standard name falls within [5%, 10%], the repetition rate level is low; if the repetition rate falls within (10%, 20%), the repetition rate level is medium; and if the repetition rate is greater than or equal to 20%, the repetition rate level is high. It should be noted that the above correspondence between the repetition rate range and the repetition rate level is only an example and not a limitation. By utilizing an inverted index table, the relevant standard documents corresponding to the target product standard document are quickly identified to determine the number of intersections and unions, thereby determining the comparison repetition rate and repetition rate level, achieving efficient and accurate matching.

[0132] In some optional implementations, steps a1 to a5 are the process of determining the repetition rate and repetition rate level of the standard name. The repetition rate and repetition rate level corresponding to the scope summary and the main text summary can be determined by referring to the above steps, which will not be repeated here.

[0133] Step S2042: Using a vector database, perform plagiarism checks based on the terminology definitions, product methods, and product indicators in the target text to obtain the corresponding similarity scores and similarity levels.

[0134] In some optional implementations, step S2042 above uses a vector database to perform plagiarism checks based on the term definitions in the target text, obtaining the corresponding similarity and similarity level, including:

[0135] Step b1: For each term definition vector in the vector database, determine the vector dot product, the first vector modulus, and the second vector modulus based on the term definitions in the target text and their corresponding term definition vectors. Specifically, each term definition in the target text is vectorized, and its dot product, the first vector modulus, and the second vector modulus are determined by comparing it with each term definition vector in the vector database. The first vector modulus is the modulus of the vector corresponding to the term definition in the target text, and the second vector modulus is the modulus of the term definition vector in the vector database.

[0136] Step b2: Determine the product of the first vector modulus and the second vector modulus, and determine the ratio of the vector dot product to the product as the comparison similarity between the terminology definition and the terminology definition vector in the target product standard document. Specifically, the comparison similarity between the two vectors is calculated using the following formula (2).

[0137]

[0138] Where S represents the comparison similarity; This represents a vector corresponding to the term definitions in the target text. Vectors represent the terminology defined in the vector database; Denotes the magnitude of the first vector; This represents the magnitude of the second vector.

[0139] Step b3: The maximum comparative similarity between the terminology definition of the target product standard document and all terminology definition vectors is determined as the similarity of the terminology definition of the target product standard document. Specifically, the comparative similarity between the terminology definition of the target product standard document and all terminology definition vectors in the vector database can be obtained through the above formula (2). The maximum value of these comparative similarities is taken as the similarity of the terminology definition of the target product standard document.

[0140] Step b4: Determine the similarity level based on the range of similarity. Specifically, assuming the similarity of the terms falls within [20%, 60%], the similarity level is low; if the similarity falls within (60%, 70%), the similarity level is medium; and if the similarity is greater than or equal to 70%, the similarity level is high. It should be noted that the above correspondence between the similarity range and the similarity level is only an example and not a limitation. By using vector dot product, the first vector modulus, and the second vector modulus to calculate comparative similarity, similarity can be evaluated at the semantic level. Compared with traditional text matching methods, this method more accurately captures deep connections and improves the accuracy of plagiarism detection.

[0141] In some optional implementations, steps b1 to b4 are the process of determining the similarity and similarity level of the term definitions. The similarity and similarity level of the product method and product indicators can be determined by referring to the above steps, which will not be repeated here.

[0142] Step S2043: Determine the weight corresponding to each part of the target text. Specifically, for each part of the target text, its weight can be determined according to actual needs. For example, the standard name has a weight of 3, the scope summary has a weight of 2, the main text summary has a weight of 2, the terminology definition has a weight of 1, the product indicator has a weight of 1, and the product method has a weight of 1, with a total weight of 10 for all parts.

[0143] Step S2044: Based on the similarity rates corresponding to the standard name, scope summary, and main text summary in the target text, the similarity rates corresponding to the terminology definitions, product methods, and product indicators in the target text, and the weight corresponding to each part in the target text, the overall similarity score is determined. Specifically, by calculating multiple similarity rates and multiple similarities based on the target text of the target product standard document, and the weight corresponding to each part, the overall similarity score of the target product standard document can be determined by the following formula (3), realizing a comprehensive and targeted plagiarism check of the target product standard document from multiple dimensions. Optionally, the corresponding level can also be determined according to the range of the overall similarity score value.

[0144]

[0145] Where D represents the overall repetition rate; Q1 represents the weight of the standard name; R1 represents the repetition rate of the standard name; Q2 represents the weight of the scope summary; R2 represents the repetition rate of the scope summary; Q3 represents the weight of the main text summary; R3 represents the repetition rate of the main text summary; Q4 represents the weight of the terminology definition; S1 represents the similarity of the terminology definition; Q5 represents the weight of the product method; S2 represents the similarity of the product method; Q6 represents the weight of the product indicator; and S3 represents the similarity of the product indicator.

[0146] Step S2045 involves using the duplication rate and duplication rate level corresponding to the standard name, scope summary, and main text summary in the target text, the similarity and similarity level corresponding to the terminology definitions, product methods, and product indicators in the target text, and the overall duplication rate as the duplication check results for the target product standard document. Specifically, the duplication rate and its level, or similarity and its level, and the overall duplication rate for each part of the target text are used together as the duplication check results for the target product standard document, ensuring the comprehensiveness and accuracy of the duplication check results.

[0147] Step S205: Generate a plagiarism report based on the plagiarism detection results of the target product standard document. Specifically, a detailed plagiarism report is generated based on the plagiarism detection results. This report can be presented in the form of a table as shown in Table 1 below, allowing users to intuitively understand the duplication situation of the target product standard document and providing support for subsequent decisions. Optionally, when calculating the duplication rate or similarity for each part of the target text, multiple related standard documents are usually involved. When generating the plagiarism report, a duplication rate threshold or similarity threshold can be set, and only the IDs of related standard documents that meet the above thresholds can be displayed. Table 1 below is only an example and is not a limitation.

[0148] Table 1

[0149] Standard name repetition rate 15％ middle d001, d005 Range summary repetition rate 25％ high d001 Abstract similarity rate 35％ middle d001 Terminology definition similarity 75％ high d003 Product indicator similarity 80％ high d003 Product Method Similarity 78％ high d003 Overall repeatability 39.8％ middle -

[0150] This invention improves the accuracy of plagiarism detection by selectively extracting text from different parts of product standard documents. Based on the extracted text from each relevant product standard document, multiple plagiarism databases are constructed. Data can be stored and managed separately for the characteristics of different parts, making the plagiarism detection process more refined and better focusing on core differences. When checking a target product standard document for plagiarism, text is first extracted from multiple parts to obtain the target text, which is then compared with the corresponding plagiarism databases to comprehensively assess the degree of duplication in the target product standard document. By processing text in parts, the focus of plagiarism detection can be placed on key parts, significantly improving the targeting of the detection. The design of multiple plagiarism databases can more accurately identify the core differences between product standard documents, and the entire plagiarism detection process is designed according to the characteristics of product standard documents to ensure that it meets practical needs while improving detection accuracy.

[0151] This embodiment also provides a device for checking the duplication of product standard documents. This device is used to implement the above embodiments and preferred embodiments, and will not be repeated as already described. As used below, the term "module" can be a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0152] This embodiment provides a device for checking the duplication of product standard documents, such as... Figure 2 As shown, it includes:

[0153] The first extraction module 201 is used to extract text from multiple parts of each relevant product standard document to obtain the relevant text.

[0154] Module 202 is used to build multiple plagiarism detection databases based on relevant text sections of each relevant product standard document.

[0155] The second extraction module 203 is used to extract text from multiple parts of the target product standard document to obtain the target part text.

[0156] The plagiarism detection module 204 is used to perform plagiarism checks on the target portion of the target product's standard document using multiple plagiarism databases, and obtain the plagiarism detection results.

[0157] In some alternative implementations, each relevant product standard document includes a cover section, a scope section, a terms and definitions section, and a body section;

[0158] The first extraction module 201 includes:

[0159] The first extraction unit is used to extract the standard name from the cover section of the relevant product standard document.

[0160] The second extraction unit is used to extract a scope summary from the scope section of the relevant product standard document.

[0161] The third extraction unit is used to extract term definitions from the terminology and definition section of relevant product standard documents.

[0162] The fourth extraction unit is used to extract the text summary, product methods, and product indicators from the main body of the relevant product standard documents.

[0163] The first determining unit is used to take the standard name, scope summary, term definitions, text summary, product methods, and product indicators of the relevant product standard document as the relevant part of the text of the relevant product standard document.

[0164] In some optional implementations, the plagiarism detection database is a text database;

[0165] Module 202 includes:

[0166] The first word segmentation unit is used to segment the standard name in each relevant part of the text, obtain multiple first word texts corresponding to the standard name, and set up inverted index items.

[0167] The second word segmentation unit is used to segment the range summary in each relevant part of the text into words, obtain multiple second word texts corresponding to the range summary, and set up inverted index entries.

[0168] The third word segmentation unit is used to segment the main text summary in each relevant part of the text to obtain multiple third-word texts corresponding to the main text summary, and set up inverted index items.

[0169] The first construction unit is used to construct a name inverted index table, a range inverted index table, and a text inverted index table based on multiple first-word texts and their inverted index items, multiple second-word texts and their inverted index items, and multiple third-word texts and their inverted index items, respectively.

[0170] The second building unit is used to construct a text database based on the name inverted index table, the range inverted index table, and the text inverted index table.

[0171] In some optional implementations, the deduplication database is a vector database;

[0172] Module 202 also includes:

[0173] The first transformation unit is used to use a word vector model to transform the term definitions in each relevant part of the text into vectors, resulting in multiple term definition vectors.

[0174] The second transformation unit is used to use a word vector model to transform the product methods in each relevant part of the text into vectors, resulting in multiple product method vectors.

[0175] The third conversion unit is used to use a word vector model to convert product indicators in each relevant part of the text into vectors, resulting in multiple product indicator vectors.

[0176] The third building unit is used to construct a vector database based on multiple terminology definition vectors, multiple product method vectors, multiple product indicator vectors, and the relevant product standard documents corresponding to each vector.

[0177] In some optional implementations, the target text includes the standard name, scope summary, term definitions, main text summary, product method, and product metrics;

[0178] The plagiarism detection module 204 includes:

[0179] The first plagiarism detection unit uses a text database to perform plagiarism checks based on the standard names, scope summaries, and main text summaries in the target text, and obtains the corresponding similarity rate and similarity rate level.

[0180] The second plagiarism detection unit uses a vector database to perform plagiarism checks based on terminology definitions, product methods, and product indicators in the target text, and obtains the corresponding similarity scores and similarity levels.

[0181] The first determining unit is used to determine the weight corresponding to each part of the target text.

[0182] The second determining unit is used to determine the overall repetition rate based on the repetition rates corresponding to the standard names, scope summaries, and main text summaries in the target text, the similarity rates corresponding to the term definitions, product methods, and product indicators in the target text, and the weight corresponding to each part in the target text.

[0183] The third determining unit is used to take the duplication rate and duplication rate level corresponding to the standard name, scope summary and main text summary in the target part of the text, the similarity and similarity level corresponding to the term definitions, product methods and product indicators in the target part of the text, and the overall duplication rate as the duplication check results of the target product standard document.

[0184] In some optional implementations, the first plagiarism detection unit includes:

[0185] The word segmentation subunit is used to segment the standard names in the target text to obtain multiple target word texts.

[0186] The first determining subunit is used to determine the relevant standard document corresponding to each target word text based on the name inverted index table of the text database.

[0187] The second determining subunit is used to determine the number of intersections and unions based on the relevant standard documents corresponding to multiple target word texts.

[0188] The third determining sub-unit is used to determine the repetition rate of the standard name of the target product standard document by the ratio of the number of intersections to the number of unions.

[0189] The fourth determination sub-unit is used to determine the repetition rate level based on the range to which the repetition rate belongs.

[0190] In some optional implementations, the second plagiarism detection unit includes:

[0191] The fifth determining subunit is used to determine the vector dot product, the first vector modulus, and the second vector modulus for each term definition vector in the vector database, based on the term definitions and term definition vectors in the target part of the text.

[0192] The sixth determining subunit is used to determine the product of the first vector modulus and the second vector modulus, and the ratio of the vector dot product to the product is determined as the comparison similarity between the terminology definition and the terminology definition vector in the target product standard document.

[0193] The seventh determination subunit is used to determine the maximum comparative similarity between the terminology definitions in the target product standard document and all terminology definition vectors as the similarity of the terminology definitions in the target product standard document.

[0194] The eighth determination subunit is used to determine the similarity level based on the range to which the similarity belongs.

[0195] In some alternative embodiments, the device further includes:

[0196] The generation module is used to generate a plagiarism report based on the plagiarism detection results of the target product's standard documents.

[0197] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0198] In this embodiment, the product standard document plagiarism detection device is presented in the form of a functional unit. Here, a unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0199] This invention also provides a computer device having the above-described features. Figure 2 The device shown is for checking the duplication of product standard documents.

[0200] Please see Figure 3 , Figure 3 This is a schematic diagram of the structure of a computer device provided in an optional embodiment of the present invention, such as... Figure 3 As shown, the computer device includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 3 Take a processor 10 as an example.

[0201] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GDA), or any combination thereof.

[0202] The memory 20 stores instructions executable by at least one processor 10 to cause at least one processor 10 to perform the method shown in the above embodiments.

[0203] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the computer device. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0204] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0205] The computer device also includes an input device 30 and an output device 40. The processor 10, memory 20, input device 30, and output device 40 can be connected via a bus or other means. Figure 3 Taking the example of a connection between China and Israel via a bus.

[0206] Input device 30 can receive input numerical or character information, and generate key signal inputs related to user settings and function control of the computer device, such as a touchscreen, keypad, mouse, trackpad, touchpad, joystick, one or more mouse buttons, trackball, joystick, etc. Output device 40 may include display devices, auxiliary lighting devices (e.g., LEDs), and haptic feedback devices (e.g., vibration motors). The aforementioned display devices include, but are not limited to, liquid crystal displays, light-emitting diodes, displays, and plasma displays. In some alternative embodiments, the display device may be a touchscreen.

[0207] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods shown in the above embodiments.

[0208] A portion of this invention can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the invention through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0209] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A method for detecting duplicates in product standard documents, characterized in that, The method includes: For each relevant product standard document, text extraction is performed on multiple parts of the relevant product standard document to obtain relevant text, including: extracting the standard name from the cover page of the relevant product standard document; extracting the scope summary from the scope section of the relevant product standard document; extracting terminology definitions from the terminology and definition section of the relevant product standard document; extracting the text summary, product method, and product indicator from the main body of the relevant product standard document; and using the standard name, scope summary, terminology definition, main body summary, product method, and product indicator of the relevant product standard document as the relevant text of the relevant product standard document. Multiple plagiarism detection databases are constructed based on relevant text sections of each related product standard document. Text extraction is performed on multiple parts of the target product's standard document to obtain the target text. The plagiarism detection results are obtained by using multiple plagiarism databases to perform plagiarism checks on the target portion of the target product standard document. This includes: using a text database to perform plagiarism checks on the standard name, scope summary, and main text summary in the target portion of the text, obtaining the corresponding similarity rate and similarity level; using a vector database to perform plagiarism checks on the terminology definitions, product methods, and product indicators in the target portion of the text, obtaining the corresponding similarity and similarity level; determining the weight corresponding to each part of the target portion of the text; determining the overall similarity score based on the similarity rates corresponding to the standard name, scope summary, and main text summary in the target portion of the text, the similarity scores corresponding to the terminology definitions, product methods, and product indicators in the target portion of the text, and the weight corresponding to each part of the target portion of the text; and using the similarity rates and similarity levels corresponding to the standard name, scope summary, and main text summary in the target portion of the text, the similarity scores corresponding to the terminology definitions, product methods, and product indicators in the target portion of the text, and the overall similarity score as the plagiarism detection results for the target product standard document.

2. The method according to claim 1, characterized in that, The plagiarism detection database is a text database; The method involves constructing multiple plagiarism detection databases based on relevant text portions of each related product standard document, including: The standard name in each relevant part of the text is segmented into words to obtain multiple first word texts corresponding to the standard name, and an inverted index is set. The range summary in each relevant part of the text is segmented into words to obtain multiple second word texts corresponding to the range summary, and inverted index items are set. The main text summary in each relevant part of the text is segmented into words to obtain multiple third-word texts corresponding to the main text summary, and inverted index items are set. Based on the multiple first-word texts and their inverted index entries, the multiple second-word texts and their inverted index entries, and the multiple third-word texts and their inverted index entries, a name inverted index table, a range inverted index table, and a text inverted index table are constructed respectively; The text database is constructed based on the name inverted index table, the range inverted index table, and the text inverted index table.

3. The method according to claim 2, characterized in that, The plagiarism detection database is a vector database; The construction of multiple plagiarism detection databases based on relevant text portions of each relevant product standard document also includes: By using a word vector model, the term definitions in each relevant part of the text are transformed into vectors, resulting in multiple term definition vectors; By using a word vector model, the product methods in each relevant part of the text are transformed into vectors, resulting in multiple product method vectors; By using a word vector model, product indicators in each relevant part of the text are transformed into vectors, resulting in multiple product indicator vectors. The vector database is constructed based on the multiple terminology definition vectors, the multiple product method vectors, the multiple product indicator vectors, and the relevant product standard documents corresponding to each vector.

4. The method according to claim 1, characterized in that, The text database is used to perform plagiarism checks based on standard names in the target text, obtaining the corresponding repetition rate and repetition rate level, including: The standard names in the target text are segmented into words to obtain multiple target word texts; For each target word text, the relevant standard document corresponding to the target word text is determined based on the name inverted index table of the text database; Based on the relevant standard documents corresponding to the multiple target word texts, determine the number of intersections and the number of unions; The ratio of the number of intersections to the number of unions is determined as the repetition rate of the standard names of the target product standard documents; The repetition rate level is determined based on the range to which the repetition rate belongs.

5. The method according to claim 1, characterized in that, The method employs a vector database to perform plagiarism detection based on the terminology definitions in the target text, obtaining the corresponding similarity score and similarity level, including: For each term definition vector in the vector database, the vector dot product, the first vector modulus, and the second vector modulus are determined based on the term definition in the target text and the term definition vector. The product of the first vector modulus and the second vector modulus is determined, and the ratio of the vector dot product to the product is determined as the comparison similarity between the terminology definition of the target product standard document and the terminology definition vector. The maximum comparative similarity between the terminology definitions in the target product standard document and all terminology definition vectors is determined as the similarity of the terminology definitions in the target product standard document. The similarity level is determined based on the range to which the similarity belongs.

6. The method according to claim 1, characterized in that, The method further includes: Based on the plagiarism detection results of the target product standard document, a plagiarism detection report is generated.

7. A device for checking the duplication of product standard documents, characterized in that, The device includes: The first extraction module is used to extract text from multiple parts of each relevant product standard document to obtain relevant text, including: a first extraction unit for extracting the standard name from the cover of the relevant product standard document; a second extraction unit for extracting a scope summary from the scope section of the relevant product standard document; a third extraction unit for extracting terminology definitions from the terminology and definition section of the relevant product standard document; a fourth extraction unit for extracting a text summary, product method, and product indicator from the main body of the relevant product standard document; and a first determination unit for using the standard name, scope summary, terminology definition, main body summary, product method, and product indicator of the relevant product standard document as the relevant text of the relevant product standard document. The building module is used to construct multiple plagiarism detection databases based on relevant text sections of each related product standard document; The second extraction module is used to extract text from multiple parts of the target product standard document to obtain the target text. The plagiarism detection module is used to perform plagiarism checks on the target portion of the target product standard document using the multiple plagiarism detection databases, and obtain plagiarism results. It includes: a first plagiarism detection unit, used to perform plagiarism checks on the standard name, scope summary, and main text summary in the target portion of the text using a text database, obtaining the corresponding similarity rate and similarity level; a second plagiarism detection unit, used to perform plagiarism checks on the terminology definition, product method, and product indicator in the target portion of the text using a vector database, obtaining the corresponding similarity and similarity level; a second determination unit, used to determine the weight corresponding to each part of the target portion of the text; and a third determination unit. The first determination unit is used to determine the overall similarity based on the similarity rates corresponding to the standard name, scope summary, and main text summary in the target text, the similarity rates corresponding to the terminology definitions, product methods, and product indicators in the target text, and the weight corresponding to each part in the target text; the second determination unit is used to take the similarity rates and similarity rate levels corresponding to the standard name, scope summary, and main text summary in the target text, the similarity rates and similarity levels corresponding to the terminology definitions, product methods, and product indicators in the target text, and the overall similarity as the plagiarism detection result of the target product standard document.

8. A computer device, characterized in that, include: A memory and a processor are interconnected, the memory stores computer instructions, and the processor executes the computer instructions to perform the deduplication method for product standard documents as described in any one of claims 1 to 6.

Images