Content comparison method, device and system based on artificial intelligence and medium

Abstract

The invention discloses a content comparison method, device and system based on artificial intelligence and a medium, and the method comprises the steps: obtaining clause information of a file, carrying out word segmentation processing on the clause information, obtaining word segmentation data, training the word segmentation data into word vectors, and obtaining a word vector model, and generating corresponding document vectors for two files needing to be compared through the word vector model, and comparing the two document vectors with a similarity threshold to obtain a comparison result of the two files. According to the technical scheme, differential content identification can be carried out, the identification is not limited to a fixed format and sequence, semantic similarity comparison can be carried out, the content comparison accuracy can be improved, a large amount of labor cost can be saved, and the comparison efficiency can be improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an artificial intelligence-based content comparison method, device, system and medium. Background Art

[0002] AI-based content comparison usually refers to the use of AI technology to automatically detect and display the differences between different versions of documents. This technology is widely used in many fields such as contract review, document revision tracking, bidding document comparison, etc., which can significantly improve work efficiency and accuracy. The AI ​​content comparison tool can process a variety of formats including text, PDF, Word, scanned copies, etc. Through high-precision document structure analysis technology, it can intelligently identify the differences in text paragraphs, tables, etc., and display the comparison results in a visual way.

[0003] For example, in the process of developing insurance clauses, it is usually necessary to compare the similarities or differences between two clauses, or to extract important information from multiple documents to expand business. Currently, the available technologies are to compare the content line by line. When the file formats are quite different or the same meaning is expressed in a different way, the comparison results are unsatisfactory. Since each clause has a large amount of information and the amount of data of the same type of clauses is large, manual comparison is very inefficient. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the prior art, the purpose of the present invention is to provide an artificial intelligence-based content comparison method, device, system and medium that can be applied to financial technology or other related fields. Its main purpose is to identify differential content, which is not limited to a fixed format and sequence, and can perform semantic similarity comparison, thereby improving the accuracy of content comparison, saving a lot of manpower costs, and improving comparison efficiency.

[0005] The technical solution of the present invention is as follows:

[0006] The first aspect of the present invention provides a comparison method based on artificial intelligence, comprising:

[0007] Get information about the terms of the document;

[0008] Performing word segmentation processing on the clause information to obtain word segmentation data;

[0009] Training the word segmentation data into word vectors to obtain a word vector model;

[0010] The two files to be compared are respectively generated with corresponding document vectors through the word vector model, and the two document vectors are compared with the similarity threshold to obtain the comparison result of the two files.

[0011] In one embodiment, the step of obtaining the clause information of the file includes:

[0012] Obtain information on insurance terms from the China Insurance Association, the Aviation Insurance Association, and the insurance terms disclosure information on the official websites of various insurance companies;

[0013] Use text analysis to read and parse the terms and conditions, generate structured data and store it in an Oracle database.

[0014] In one embodiment, the step of performing word segmentation processing on the clause information to obtain word segmentation data includes:

[0015] Clean the clause information and preprocess it into a txt document, and use the txt document as a corpus for model training;

[0016] The corpus is segmented, POS tagged, named entity recognized, dependency parsed, and keyword extracted by a word segmentation tool, and the word parts are marked and word frequencies are counted to obtain word segmentation data.

[0017] In one embodiment, the step of training the word segmentation data into word vectors to obtain a word vector model includes:

[0018] Remove stop words from the word segmentation data and ignore high-frequency words to obtain accurate word segmentation data;

[0019] The precise word segmentation data is trained into word vectors through the Word2VecTrainer of the hanlp package to obtain a word vector model.

[0020] In one embodiment, the step of generating corresponding document vectors for the two files to be compared respectively through the word vector model, and comparing the two document vectors with a preset similarity threshold to obtain the comparison result of the two files includes:

[0021] Generate corresponding document vectors for each key paragraph of the two files to be compared through the word vector model;

[0022] Assigning a weight ratio to each key paragraph of the two files, summing the document vectors of each key paragraph of the two files according to the weight ratio, and obtaining the total similarity of the two files;

[0023] The total similarity of the two files is compared with a preset similarity threshold to obtain a comparison result of the two files.

[0024] In one embodiment, the step of assigning a weight ratio to each key paragraph of the two files, summing the document vectors of each key paragraph of the two files according to the weight ratio, and obtaining the total similarity of the two files includes:

[0025] Calculate the cosine similarity between two document vectors;

[0026] assigning a weight ratio to each key paragraph of both said documents;

[0027] The cosine similarity is multiplied by the weight ratio of each key paragraph, and the products are summed to obtain the total similarity.

[0028] In one embodiment, the step of comparing the total similarity of the two files with a preset similarity threshold to obtain a comparison result of the two files includes:

[0029] Comparing the total similarity of the two files with a preset similarity threshold;

[0030] When the total similarity is greater than or equal to the preset similarity threshold, the comparison result is that the two files are similar; when the total similarity is less than the preset similarity threshold, the comparison result is that there are differences between the two files.

[0031] A second aspect of the present invention provides a content comparison device based on artificial intelligence, comprising:

[0032] The acquisition module is used to obtain the clause information of the document;

[0033] A word segmentation module, used to perform word segmentation processing on the clause information to obtain word segmentation data;

[0034] A training module, used to train the word segmentation data into word vectors to obtain a word vector model;

[0035] The judgment module is used to generate corresponding document vectors for the two files to be compared through the word vector model, compare the two document vectors with the similarity threshold, and obtain the comparison result of the two files.

[0036] A third aspect of the present invention provides an artificial intelligence-based content comparison system, the system comprising at least one processor; and

[0037] a memory communicatively connected to the at least one processor; wherein,

[0038] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the above-mentioned artificial intelligence-based content comparison method.

[0039] A fourth aspect of the present invention provides a non-volatile computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are executed by one or more processors, the one or more processors can execute the above-mentioned artificial intelligence-based content comparison method.

[0040] Beneficial effects: The present invention discloses a content comparison method, device, system and medium based on artificial intelligence. Compared with the prior art, the embodiment of the present invention obtains the clause information of a file, performs word segmentation processing on the clause information, obtains word segmentation data, trains the word segmentation data into a word vector, obtains a word vector model, generates corresponding document vectors for the two files to be compared through the word vector model, compares the two document vectors with a similarity threshold, and obtains the comparison result of the two files. According to the technical solution of the present invention, it is possible to identify differential content, not limited to a fixed format and order, and can perform semantic similarity comparison, which can improve the accuracy of content comparison, save a lot of manpower costs, and improve comparison efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0042] Figure 1 A flow chart of a content comparison method based on artificial intelligence provided by an embodiment of the present invention;

[0043] Figure 2 A flowchart of step S100 in the artificial intelligence-based content comparison method provided in an embodiment of the present invention;

[0044] Figure 3 A flowchart of step S200 in the artificial intelligence-based content comparison method provided in an embodiment of the present invention;

[0045] Figure 4 A flowchart of step S300 in the artificial intelligence-based content comparison method provided in an embodiment of the present invention;

[0046] Figure 5 A flowchart of step S400 in the artificial intelligence-based content comparison method provided in an embodiment of the present invention;

[0047] Figure 6 A flowchart of step S420 in the artificial intelligence-based content comparison method provided in an embodiment of the present invention;

[0048] Figure 7 A flowchart of step S430 in the artificial intelligence-based content comparison method provided in an embodiment of the present invention;

[0049] Figure 8A schematic diagram of functional modules of an artificial intelligence-based content comparison device provided in an embodiment of the present invention;

[0050] Fig. 9 A schematic diagram of the hardware structure of an artificial intelligence-based content comparison system provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail below. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. The embodiments of the present invention are described below in conjunction with the accompanying drawings.

[0052] AI-based content comparison refers to the use of artificial intelligence technology, especially machine learning and natural language processing (NLP), to automatically analyze and identify the differences between different documents or text contents. It is not limited to fixed formats and sequences, and semantic similarity comparison can be performed through AI models, which can improve the accuracy of content comparison, save a lot of labor costs, and improve comparison efficiency, such as the development of insurance clauses.

[0053] In the process of developing insurance clauses, it is usually necessary to compare the similarities or differences between two clauses, or to extract important information from multiple documents to expand business. Currently, the available technologies are to compare the content line by line. When the file formats are quite different or the same meaning is expressed in a different way, the comparison results are unsatisfactory. Since each clause has a large amount of information and the amount of data of the same type of clauses is large, manual comparison is very inefficient.

[0054] In order to solve the above problems, the present invention proposes a content comparison method based on artificial intelligence. Figure 1 , Figure 1 A flowchart of an embodiment of the content comparison method based on artificial intelligence provided by the present invention. The content comparison method based on artificial intelligence provided in this embodiment is applied to a system consisting of a terminal device, a network and a server, wherein the network is a medium for providing a communication link between the terminal device and the server, which may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.; the operating system on the terminal device may include a handheld device operating system (iPhone operating system, iOS system), Android system or other operating system, and the terminal device is connected to the server through the network to achieve interaction, thereby performing operations such as receiving or sending data, and specifically may be various electronic devices with a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, portable computers and desktop servers, etc. Figure 1 As shown, the method specifically comprises the following steps:

[0055] S100, obtaining clause information of the document;

[0056] S200, performing word segmentation processing on the clause information to obtain word segmentation data;

[0057] S300, training the word segmentation data into word vectors to obtain a word vector model;

[0058] S400: Generate corresponding document vectors for the two files to be compared respectively through the word vector model, compare the two document vectors with a similarity threshold, and obtain a comparison result of the two files.

[0059] In this embodiment, the content comparison technology based on artificial intelligence is a method that uses AI algorithms to process and analyze text data to identify and compare the differences between different documents. This technology is widely used in many fields such as contract review, document revision tracking, bidding document comparison, etc., and improves the efficiency and accuracy of the review work through automation.

[0060] First, extract key terms information from the documents that need to be compared. The terms information may include contract terms, policy regulations or any other form of text agreements. Perform word segmentation on the extracted terms information, decompose the text into separate words, and obtain the word segmentation data that can be used for subsequent processing, that is, decompose continuous text strings into separate words or phrases. Word segmentation is the basis of text analysis and helps with subsequent text processing and feature extraction. Use the word segmentation data to train the word vector model to convert words into vector form, such as Word2Vec or GloVe. The word vector model can convert words into numerical vectors. These vectors can capture the semantic relationship between words so that computers can process and analyze language data more efficiently.

[0061] The two files that need to be compared are converted into corresponding document vectors respectively through the trained word vector model. Finally, the system will compare the two document vectors with the preset similarity threshold. For the two files that need to be compared, the trained word vector model is used to generate their respective document vectors. The document vector is obtained by aggregating (such as averaging, summing) the word vectors of all words in the document. The two document vectors are compared with the preset similarity threshold. The similarity can be calculated by metrics such as cosine similarity. If the similarity between the two document vectors exceeds the threshold, the two files are considered to be similar in content; if it is lower than the threshold, it is considered that there is a difference, and finally the comparison result of the two files is obtained. This similarity threshold is a key parameter that determines the acceptable range of differences between files.

[0062] Among them, obtaining the terms information may involve automated text extraction tools or manual document review processes to ensure that the information obtained is accurate and comprehensive. Word segmentation can be implemented using natural language processing libraries that are capable of processing multiple languages ​​and support advanced functions such as stop word filtering and stem extraction. When training the word vector model, different model parameters such as vector dimension, window size, and number of iterations can be selected to obtain the best performance. The file comparison results can be used in a variety of applications, such as contract review, copyright detection, content duplication detection, etc. The comparison results can be presented in the form of a visual report, highlighting the differences or similar parts. Through these steps, in-depth analysis and comparison of document terms can be achieved, helping users quickly identify differences and similarities between documents.

[0063] Technical effect: AI-based content comparison technology can efficiently identify minor differences between documents through a refined process. This technology is not limited to literal comparison of text, but also extends to changes in multiple levels such as format and semantics. For example, in the legal industry, it can accurately identify subtle changes in contract terms; in the financial field, it can promptly detect data deviations in financial reports; in the medical industry, it can help review the accuracy and consistency of medical records. This comparison technology is not limited to a fixed format or sequence, which means it can adapt to various types of document structures, providing a wider range of applicability. Through semantic similarity comparison by AI models, the technology can deeply understand the meaning of the text, rather than just superficial keyword matching. This advancement significantly improves the accuracy of content comparison, especially when dealing with complex or professional documents, and can capture details that human reviewers may overlook. Integrated into a document management system or content creation platform, this AI-based content comparison technology can serve users' real-time needs, such as instant feedback on differences in contract modifications, automatic detection of duplicate content in research reports, etc. Such applications not only save a lot of human resources, but also significantly improve the efficiency of review and management work. They can identify differential content, are not limited to fixed formats and sequences, and can perform semantic similarity comparisons, which can improve the accuracy of content comparisons, save a lot of labor costs, and improve comparison efficiency.

[0064] In one embodiment, Figure 2 As shown, in the step of obtaining the clause information of the file in step S100, it includes:

[0065] S110. Obtain information on insurance terms from the China Insurance Association, the Aviation Insurance Association, and the insurance terms disclosure information on the official websites of various insurance companies;

[0066] S120, using text analysis to specifically read and parse the clause information, generate structured data and store it in an Oracle database.

[0067] In this embodiment, obtaining the insurance clause information from the insurance clause information disclosed by the China Insurance Association, China Banking and Insurance Information Technology Management Co., Ltd. and other institutions is a process involving data collection, processing and storage. First, the original text information of the insurance clauses is collected from the official websites, official public service accounts and other channels of the above-mentioned institutions through automated tools or manual methods. Subsequently, these texts are read and parsed using natural language processing technology to extract key information and convert it into structured data, such as converting key information such as clause content, insurance liability, liability exemption, insurance period, etc. into a format that can be recognized by the database.

[0068] Next, the structured data is stored in the Oracle database through data import tools or programming interfaces (APIs). Oracle database is known for its high reliability, scalability and security, and is very suitable for storing and managing structured data. During the storage process, Oracle's advanced features, such as partitioning, indexing and data encryption, can be used to optimize data access efficiency and ensure data security.

[0069] In addition, to improve the accessibility and analyzability of data, views and stored procedures can be created in the database to quickly query and analyze insurance policy data. In this way, the structured storage and management of insurance terms not only helps to achieve data standardization and consistency, but also supports complex queries and report generation.

[0070] Oracle Database is a widely used commercial relational database management system (RDBMS). Oracle Database is known for its high reliability and data integrity. It is suitable for critical business applications that require 24 / 7 uninterrupted operation. It can support data processing needs from small to large scale. Oracle provides powerful security features, including data encryption, user access control and audit functions. Oracle Database optimizes multiple performance aspects, including query processing, concurrency control and caching mechanisms. Oracle 12c and later versions introduce multi-tenant architecture, allowing multiple independent databases to share the same database instance, improving resource utilization. Oracle Database supports SQL (Structured Query Language) and PL / SQL (Procedural Extensions to SQL). Oracle Database supports cloud computing deployment and can run on Oracle Cloud or other cloud platforms. Oracle Database is widely used in finance, telecommunications, government, education, and healthcare. Many industries.

[0071] In one embodiment, Figure 3As shown, the step of performing word segmentation processing on the clause information in step S200 to obtain word segmentation data includes:

[0072] S210, cleaning and preprocessing the clause information into a txt document, and using the txt document as a corpus for model training;

[0073] S220, performing word segmentation, part-of-speech tagging, named entity recognition, dependency syntax analysis, and keyword extraction on the corpus through a word segmentation tool, marking the part of speech and counting word frequency to obtain word segmentation data.

[0074] In this embodiment, the original text information of the insurance clauses is obtained from official channels, and data cleaning is performed to remove useless formats, special characters and irrelevant content to ensure the accuracy and consistency of the text. The cleaned clause text is preprocessed, including unified text encoding format, paragraphing, adding necessary tags, etc., and then the processed text is saved as a TXT document. The preprocessed TXT document is used as the corpus to provide basic data for the model training, ensuring that the corpus has a wide coverage and contains various types of insurance clauses.

[0075] TXT documents contain only unformatted text information and do not contain any images, tables or non-text elements. Due to their simplicity, TXT documents can be opened and read in almost all operating systems and text editors, including Windows, macOS, Linux, etc. Due to their simple format, TXT documents can be easily processed by scripts and programs, making them suitable for batch processing and automated tasks.

[0076] Use the word segmentation tool to segment the text in the corpus and divide the continuous text into meaningful vocabulary units, laying the foundation for subsequent text analysis. Perform part-of-speech tagging on the word segmentation results and identify the part of speech of each word (such as noun, verb, adjective, etc.), which helps to understand the grammatical structure of the text. Identify named entities in the text (such as names of people, places, and institutions, etc.). These entities are crucial to understanding the specific content of the text. Perform dependency syntactic analysis to reveal the dependency relationship between words in the text and understand the grammatical structure and semantic relationship of the sentence. Extract keywords or phrases from the text. These keywords can represent the main content and theme of the text. Perform word frequency statistics on the data after word segmentation to identify words with high frequency of occurrence, providing a basis for further analysis of the text content.

[0077] Through this refinement process, we ensure that every step from the original insurance clause text to the corpus for model training is carefully processed and optimized so that an efficient and accurate natural language processing model can be built.

[0078] In one embodiment, Figure 4As shown, the step of training the word segmentation data into word vectors to obtain a word vector model in step S300 includes:

[0079] S310. Remove the stop words in the word segmentation data and ignore the high-frequency words to obtain accurate word segmentation data;

[0080] S320. Train the accurate word segmentation data into word vectors through the Word2VecTrainer of the hanlp package to obtain a word vector model.

[0081] In this embodiment, first, remove those words that frequently appear in the language but usually do not carry important information from the accurate word segmentation data, such as "de", "he", "shi", etc. These words are called stop words, and removing them can reduce the error rate in text analysis. Then, identify and ignore those words with extremely high frequencies that may contribute little to the meaning of the text, such as "company name", "geographical name", "fixed title", etc. Although these words may carry certain information, in some analyses, their universality may reduce their importance. After cleaning the data, perform part-of-speech tagging to assign a part-of-speech label to each word, such as noun, verb, etc. This helps to understand the grammatical structure and lexical function in the text. Conduct word frequency statistics on the cleaned and tagged words to identify the words with higher usage frequencies, providing a basis for subsequent text analysis. By analyzing the remaining words, extract keywords or key phrases, which can facilitate the understanding and analysis of the theme and content of the text.

[0082] Use the processed accurate word segmentation data as the training set and prepare to input it into the Word2VecTrainer tool. These data include the words in the text and their context information, providing the necessary input for training. Use Word2VecTrainer or other similar tools, such as the Word2Vec model in the gensim library, to start the training process. During the training process, the model will learn the semantic relationships between words and encode these relationships into the word vectors. After training is completed, each word will be assigned a word vector to generate the word vector model, and the word vector can capture the semantic features and context information of the word. The dimension of the word vector is usually a preset parameter, which can affect the expressive ability and training complexity of the model. The generated word vector model can be used for various downstream tasks, such as text classification, sentiment analysis, named entity recognition, etc.

[0083] In practical applications, the similarity or relevance of words can be evaluated by calculating the distance between word vectors (such as cosine similarity). According to the training results and application effects, the parameters of the word vector model can be adjusted, such as the learning rate, window size, number of iterations, etc., to optimize the performance of the word vector model.

[0084] Word2VecTrainer in the HanLP package is a module for training word vectors. It generates word embeddings based on deep learning technology. Word2VecTrainer can receive a large amount of text data as input and output a vector representation of each word. These vectors can capture the semantic relationship between words. Word2Vec is a widely used word embedding technology that can convert words into vector form and capture the semantic relationship between words. HanLP is a feature-rich natural language processing toolkit that supports text processing tasks in multiple languages, including word segmentation, part-of-speech tagging, named entity recognition, etc. Using HanLP's Word2VecTrainer, the word vector model can be efficiently trained. Through the above steps, we can obtain an accurate word segmentation dataset and a word vector model that can reflect the semantic information of the vocabulary.

[0085] In one embodiment, Figure 5 As shown, the step S400 of generating corresponding document vectors for the two files to be compared by using the word vector model, comparing the two document vectors with a preset similarity threshold, and obtaining the comparison results of the two files includes:

[0086] S410, generating corresponding document vectors for each key paragraph of the two files to be compared through the word vector model;

[0087] S420, assigning a weight ratio to each key paragraph of the two files, summing the document vectors of each key paragraph of the two files according to the weight ratio, and obtaining the total similarity of the two files;

[0088] S430: Compare the total similarity of the two files with a preset similarity threshold to obtain a comparison result of the two files.

[0089] In this embodiment, each key paragraph is identified and extracted from the two files to be compared. The key paragraph may contain the main point, argument or conclusion of the document, such as insurance liability, liability exemption, insurance amount, etc. Each key paragraph is converted into a document vector using a pre-trained word vector model. This step involves aggregating the word vectors of each word in the paragraph to form a comprehensive vector representation.

[0090] Assign weight ratios to the key paragraphs of each file. The weight ratios can be determined based on the importance, length, or other criteria of the paragraphs. Based on the assigned weight ratios, the document vectors of the key paragraphs of each file are weighted summed to obtain the total document vector for each file. Calculate the similarity between the total document vectors of the two files. The similarity can be calculated using cosine similarity, Euclidean distance, or other similarity metrics.

[0091] Compare the calculated total similarity with the preset similarity threshold. The similarity threshold is a predefined standard used to determine whether two files are similar enough. Based on the comparison results of the similarity and the threshold, the comparison results of the two files are output. If the similarity is higher than the threshold, the two files are considered to have similar contents; if it is lower than the threshold, the contents are considered to be dissimilar.

[0092] The comparison results have a wide range of applications, covering copyright detection, document duplication checking, contract review and other fields. These application scenarios require the rapid and accurate identification of differences and similarities between documents for subsequent processing. For example, in copyright detection, the comparison results can help users quickly discover potential infringements and protect original content; in document duplication checking, it can effectively avoid duplication of work and improve work efficiency; in contract review, it can ensure the accuracy and consistency of contract terms.

[0093] In order to improve the accuracy and adaptability of the comparison, the similarity threshold can be dynamically adjusted according to the feedback from actual applications. This means that the system will continuously optimize the comparison algorithm based on the user's usage and feedback to make it more in line with actual needs. In addition, the comparison results are not limited to a fixed format and order, and have strong flexibility. Whether it is structured data or unstructured data, satisfactory results can be obtained through comparison.

[0094] The AI ​​model plays an important role in the comparison process. Through semantic similarity comparison, the AI ​​model can understand the meaning of the text, not just the literal similarity. This makes the comparison results more accurate, especially when dealing with complex texts. At the same time, the AI ​​model can also self-learn and optimize based on user feedback to continuously improve the comparison effect.

[0095] In short, the comparison results play a key role in multiple application scenarios, helping users quickly identify differences and similarities between documents. By dynamically adjusting the similarity threshold, not being limited to fixed formats and sequences, and using AI models for semantic similarity comparison, the accuracy and adaptability of the comparison results have been significantly improved. This not only saves a lot of manpower costs, but also improves comparison efficiency, bringing real convenience to users.

[0096] In one embodiment, Figure 6As shown, the step of assigning a weight ratio to each key paragraph of the two files in step S420, summing the document vectors of each key paragraph of the two files according to the weight ratio, and obtaining the total similarity of the two files includes:

[0097] S421, calculating the cosine similarity of two document vectors;

[0098] S422, assigning a weight ratio to each key paragraph of the two files;

[0099] S423: multiply the cosine similarity by the weight ratio of each key paragraph respectively, and sum the products to obtain the total similarity.

[0100] In this embodiment, first, the two documents to be compared are converted into a numerical vector form, i.e., a document vector. These vectors are usually obtained by converting the words in the document into word vectors and aggregating them (such as averaging, summing, etc.). For each pair of key paragraphs, the cosine similarity formula is used to calculate the similarity between them. The cosine similarity calculation formula is the ratio of the dot product of the two vectors to the product of their modulus length (i.e., vector length), and the expression is: Among them, A and B are two document vectors, A·B is their dot product, and ‖A‖ and ‖B‖ are their Euclidean norms.

[0101] Assign a weight ratio to each key paragraph. These weights can be determined based on the importance of the paragraph, the professionalism of the content, the information richness, or other relevant criteria. Multiply the cosine similarity of each key paragraph by its corresponding weight ratio, and then sum these products to get the total similarity of the entire document. This step can be expressed as: (Cosine similarity i *Weight Ratio i )where n is the number of key paragraphs.

[0102] It can be understood that i represents the i-th paragraph in the document. For example, when i=2, it means that there are two key paragraphs in the document. The total similarity can be obtained by simply multiplying the cosine similarities of the two paragraphs by the corresponding weight ratio and then adding them together.

[0103] Set a similarity threshold to determine whether two documents are similar. This threshold can be determined according to the needs of the specific application scenario, and the calculated total similarity is compared with the preset similarity threshold. If the total similarity exceeds the threshold, the two documents are considered similar in content; if it is lower than the threshold, the content is considered dissimilar.

[0104] The comparison results have a wide range of applications, covering copyright detection, document duplication checking, contract review and other fields. These application scenarios require the rapid and accurate identification of differences and similarities between documents for subsequent processing. For example, in copyright detection, the comparison results can help users quickly discover potential infringements and protect original content; in document duplication checking, it can effectively avoid duplication of work and improve work efficiency; in contract review, it can ensure the accuracy and consistency of contract terms.

[0105] In order to improve the accuracy and efficiency of the comparison, the weight allocation strategy and similarity threshold setting in the comparison algorithm are dynamically adjusted according to feedback and actual application effects. This means that the system will continuously optimize the comparison algorithm based on user usage and feedback to make it more in line with actual needs. In addition, the comparison results are not limited to a fixed format and order, and have strong flexibility. Whether it is structured data or unstructured data, satisfactory results can be obtained through comparison.

[0106] The AI ​​model plays an important role in the comparison process. Through semantic similarity comparison, the AI ​​model can understand the meaning of the text, not just the literal similarity. This makes the comparison results more accurate, especially when dealing with complex texts. At the same time, the AI ​​model can also self-learn and optimize based on user feedback to continuously improve the comparison effect.

[0107] Integrate the above method into the document management system or content creation platform to provide users with real-time comparison services. This method can effectively quantify the similarity between two documents and provide data support for document analysis and decision-making. Through the real-time comparison service, users can instantly understand the similarity of documents and make more informed decisions.

[0108] In short, the comparison results play a key role in multiple application scenarios, helping users quickly identify differences and similarities between documents. By dynamically adjusting the weight distribution strategy and similarity threshold setting, not being limited to fixed formats and sequences, and using AI models for semantic similarity comparison, the accuracy and efficiency of the comparison results have been significantly improved. Integrating this method into a document management system or content creation platform to provide users with real-time comparison services can effectively quantify the similarity between two documents and provide data support for document analysis and decision-making. This not only saves a lot of manpower costs, but also improves the comparison efficiency, bringing real convenience to users.

[0109] In one embodiment, Figure 7 As shown, the step of comparing the total similarity of the two files with a preset similarity threshold to obtain a comparison result of the two files in step S430 includes:

[0110] S431, comparing the total similarity of the two files with a preset similarity threshold;

[0111] S432: When the total similarity is greater than or equal to the preset similarity threshold, the comparison result is that the two files are similar; when the total similarity is less than the preset similarity threshold, the comparison result is that there are differences between the two files.

[0112] In this embodiment, before the comparison, a similarity threshold needs to be set, and the similarity threshold is a predetermined value used to evaluate the similarity level between two documents. The setting of the similarity threshold may be based on business requirements, industry standards or historical data analysis.

[0113] After completing the document vectorization and summing the weighted cosine similarity, the total similarity of the two files is obtained. The total similarity reflects the degree of similarity between the two documents in terms of content. The total similarity is compared with the preset similarity threshold. If the total similarity is greater than or equal to the preset similarity threshold, the two files are determined to be similar in content, which means that the two documents have a high degree of consistency in key information, themes or expressions. If the total similarity is less than the preset similarity threshold, it is determined that there are differences between the two files, which may mean that the two documents have obvious differences in some key paragraphs, term usage or expressions.

[0114] The comparison results are output in the form of a report, which should include the specific value of the total similarity, the similarity threshold and the final comparison conclusion. The comparison results can be applied to a variety of scenarios, such as contract review, academic duplication check, content originality check, etc. Users can make corresponding decisions or further analysis based on the comparison results. Depending on the actual application effect of the comparison results, it may be necessary to adjust the similarity threshold to ensure the accuracy and applicability of the comparison results.

[0115] Through this refinement process, the accuracy and effectiveness of document comparison can be ensured, providing users with reliable decision support. At the same time, this method also improves the automation level of document analysis and reduces the workload of manual review.

[0116] Another embodiment of the present invention provides a content comparison device based on artificial intelligence, such as Figure 8 As shown, the device 1 comprises:

[0117] An acquisition module 11 is used to acquire the clause information of a document;

[0118] A word segmentation module 12, used to perform word segmentation processing on the clause information to obtain word segmentation data;

[0119] A training module 13 is used to train the word segmentation data into word vectors to obtain a word vector model;

[0120] The processing module 14 is used to generate corresponding document vectors for the two files to be compared through the word vector model, compare the two document vectors with a similarity threshold, and obtain a comparison result of the two files.

[0121] The module referred to in the present invention refers to a series of computer program instruction segments that can perform specific functions, which is more suitable for describing the content comparison execution process based on artificial intelligence than a program. For the specific implementation of each module, please refer to the corresponding method embodiment above, which will not be repeated here.

[0122] In one embodiment, the acquisition module 11 includes:

[0123] The crawling unit is used to obtain the insurance terms information from the China Insurance Association, the Aviation Insurance Association, and the insurance terms disclosure information on the official websites of various insurance companies;

[0124] The parsing unit is used to read and parse the clause information using text analysis, generate structured data and store it in an Oracle database.

[0125] In one embodiment, the word segmentation module 12 includes:

[0126] A cleaning unit, used for cleaning and preprocessing the clause information into a txt document, and using the txt document as a corpus for model training;

[0127] The screening unit is used to perform word segmentation, part-of-speech tagging, named entity recognition, dependency syntax analysis, keyword extraction, and part-of-speech marking and word frequency statistics on the corpus through a word segmentation tool to obtain word segmentation data.

[0128] In one embodiment, the training module 13 includes:

[0129] A removal unit, used to remove stop words in the word segmentation data and ignore high-frequency words to obtain accurate word segmentation data;

[0130] The training unit is used to train the precise word segmentation data into word vectors through the Word2VecTrainer of the hanlp package to obtain a word vector model.

[0131] In one embodiment, the device 1 further comprises:

[0132] A conversion module, used for generating corresponding document vectors for each key paragraph of two files to be compared through the word vector model;

[0133] A product module assigns a weight ratio to each key paragraph of the two files, and sums the document vectors of each key paragraph of the two files according to the weight ratio to obtain the total similarity of the two files;

[0134] The judgment module is used to compare the total similarity of the two files with a preset similarity threshold to obtain a comparison result of the two files.

[0135] In one embodiment, the product module includes:

[0136] A calculation unit, used to calculate the cosine similarity of two document vectors;

[0137] an allocating unit, used for allocating a weight ratio to each key paragraph of the two files;

[0138] The synthesis unit is used to multiply the cosine similarity by the weight ratio of each key paragraph respectively, and sum the products to obtain the total similarity.

[0139] In one embodiment, the judgment module includes:

[0140] A comparison unit, used for comparing the total similarity of the two files with a preset similarity threshold;

[0141] The output unit is used to determine that when the total similarity is greater than or equal to a preset similarity threshold, the comparison result is that the two files are similar; when the total similarity is less than the preset similarity threshold, the comparison result is that the two files are different.

[0142] Another embodiment of the present invention provides a content comparison system based on artificial intelligence, such as Fig. 9 As shown, the system 10 includes:

[0143] One or more processors 110 and memory 120, Fig. 9 A processor 110 is used as an example for description. The processor 110 and the memory 120 may be connected via a bus or other means. Fig. 9 The example of connecting through bus is taken in the following.

[0144] The processor 110 is used to complete various control logics of the system 10, and it can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a single-chip microcomputer, an ARM (Acorn RISC Machine) or other programmable logic device, a discrete gate or transistor logic, a discrete hardware component, or any combination of these components. In addition, the processor 110 can also be any traditional processor, microprocessor or state machine. The processor 110 can also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP and / or any other such configuration.

[0145] The memory 120 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions corresponding to the content comparison method based on artificial intelligence in the embodiment of the present invention. The processor 110 executes various functional applications and data processing of the system 10 by running the non-volatile software programs, instructions and units stored in the memory 120, that is, implementing the content comparison method based on artificial intelligence in the above method embodiment.

[0146] The memory 120 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required by at least one function; the data storage area may store data created according to the use of the system 10, etc. In addition, the memory 120 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 120 may optionally include a memory remotely arranged relative to the processor 110, and these remote memories may be connected to the system 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0147] One or more units are stored in the memory 120, and when executed by one or more processors 110, the artificial intelligence-based content comparison method in any of the above method embodiments is executed, for example, the following steps are executed:

[0148] Get information about the terms of the document;

[0149] Performing word segmentation processing on the clause information to obtain word segmentation data;

[0150] Training the word segmentation data into word vectors to obtain a word vector model;

[0151] The two files to be compared are respectively generated with corresponding document vectors through the word vector model, and the two document vectors are compared with the similarity threshold to obtain the comparison result of the two files.

[0152] An embodiment of the present invention provides a non-volatile computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are executed by one or more processors, the artificial intelligence-based content comparison method in any of the above method embodiments is executed, for example, the following steps are performed:

[0153] Get information about the terms of the document;

[0154] Performing word segmentation processing on the clause information to obtain word segmentation data;

[0155] Training the word segmentation data into word vectors to obtain a word vector model;

[0156] The two files to be compared are respectively generated with corresponding document vectors through the word vector model, and the two document vectors are compared with the similarity threshold to obtain the comparison result of the two files.

[0157] As an example, non-volatile storage media can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) as external cache memory. By way of illustration and not limitation, RAM can be obtained in many forms such as synchronous RAM (SRAM), dynamic RAM, (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and direct Rambus RAM (DRRAM). The disclosed memory components or memories of the operating environment described herein are intended to include one or more of these and / or any other suitable types of memory.

[0158] In summary, the present invention discloses a method, device, system and medium for content comparison based on artificial intelligence, the method includes: obtaining the clause information of the file, performing word segmentation processing on the clause information, obtaining word segmentation data, training the word segmentation data into word vectors, obtaining a word vector model, respectively generating corresponding document vectors for the two files to be compared through the word vector model, comparing the two document vectors with a similarity threshold, and obtaining a comparison result of the two files. According to the technical solution of the present invention, it is possible to identify difference content, not limited to a fixed format and order, and can perform semantic similarity comparison, which can improve the accuracy of content comparison, save a lot of manpower costs, and improve comparison efficiency.

[0159] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium, and the computer program can include the processes of the above-mentioned method embodiments when executed. The storage medium can be a memory, a disk, a floppy disk, a flash memory, an optical storage device, etc.

[0160] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A content comparison method based on artificial intelligence, characterized in that: include: Get information about the terms of the document; Performing word segmentation processing on the clause information to obtain word segmentation data; Training the word segmentation data into word vectors to obtain a word vector model; The two files to be compared are respectively generated with corresponding document vectors through the word vector model, and the two document vectors are compared with the similarity threshold to obtain the comparison result of the two files.

2. The content comparison method based on artificial intelligence according to claim 1, characterized in that: The step of obtaining the clause information of the document includes: Obtain information on insurance terms from the China Insurance Association, the Aviation Insurance Association, and the insurance terms disclosure information on the official websites of various insurance companies; Use text analysis to read and parse the terms and conditions, generate structured data and store it in an Oracle database.

3. The content comparison method based on artificial intelligence according to claim 2, characterized in that: The step of performing word segmentation processing on the clause information to obtain word segmentation data includes: Clean the clause information and preprocess it into a txt document, and use the txt document as a corpus for model training; The corpus is segmented, POS tagged, named entity recognized, dependency parsed, and keyword extracted by a word segmentation tool, and the word parts are marked and word frequencies are counted to obtain word segmentation data.

4. The content comparison method based on artificial intelligence according to claim 3 is characterized in that: The step of training the word segmentation data into word vectors to obtain a word vector model includes: Remove stop words from the word segmentation data and ignore high-frequency words to obtain accurate word segmentation data; The precise word segmentation data is trained into word vectors through the Word2VecTrainer of the hanlp package to obtain a word vector model.

5. The content comparison method based on artificial intelligence according to claim 4 is characterized in that: The step of generating corresponding document vectors for the two files to be compared respectively through the word vector model, and comparing the two document vectors with a preset similarity threshold to obtain the comparison results of the two files includes: Generate corresponding document vectors for each key paragraph of the two files to be compared through the word vector model; Assigning a weight ratio to each key paragraph of the two files, summing the document vectors of each key paragraph of the two files according to the weight ratio, and obtaining the total similarity of the two files; The total similarity of the two files is compared with a preset similarity threshold to obtain a comparison result of the two files.

6. The content comparison method based on artificial intelligence according to claim 5 is characterized in that: The step of assigning a weight ratio to each key paragraph of the two files, summing the document vectors of each key paragraph of the two files according to the weight ratio, and obtaining the total similarity of the two files comprises: Calculate the cosine similarity between two document vectors; assigning a weight ratio to each key paragraph of both said documents; The cosine similarity is multiplied by the weight ratio of each key paragraph, and the products are summed to obtain the total similarity.

7. The content comparison method based on artificial intelligence according to claim 5 is characterized in that: The step of comparing the total similarity of the two files with a preset similarity threshold to obtain a comparison result of the two files includes: Comparing the total similarity of the two files with a preset similarity threshold; When the total similarity is greater than or equal to the preset similarity threshold, the comparison result is that the two files are similar; when the total similarity is less than the preset similarity threshold, the comparison result is that there are differences between the two files.

8. A content comparison device based on artificial intelligence, characterized in that: include: The acquisition module is used to obtain the clause information of the document; A word segmentation module, used to perform word segmentation processing on the clause information to obtain word segmentation data; A training module, used to train the word segmentation data into word vectors to obtain a word vector model; The judgment module is used to generate corresponding document vectors for the two files to be compared through the word vector model, compare the two document vectors with the similarity threshold, and obtain the comparison result of the two files.

9. A content comparison system based on artificial intelligence, characterized in that: The system includes at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the artificial intelligence-based content comparison method described in any one of claims 1-7.

10. A non-volatile computer-readable storage medium, characterized in that: The non-volatile computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by one or more processors, the one or more processors can execute the artificial intelligence-based content comparison method described in any one of claims 1-7.

Images