A real-time semantic similarity matching design method based on cosine similarity algorithm

Abstract

The present invention relates to a real-time semantic similarity matching design method based on a cosine similarity algorithm. A Java Web application end synchronizes data from an application database MySQL to a Kafka message queue cluster in real time, and completes real-time extraction and conversion of incremental data from the application database MySQL; a Python Web consumer obtains data through client consumption, and uses a cosine similarity algorithm to retrieve the semantic similarity of texts. The advantages of the present invention are: the text vector generated by the sentence embedding method contains the semantic relationship between words, can effectively measure the semantic similarity between texts, solves the problem of repeated verification of test questions, designs and implements a test question entry and architecture of a test paper system based on text semantic similarity, and has low computational complexity; the sentence similarity adopts a cosine similarity algorithm, the dynamic capture of change data adopts a Debezium low-latency streaming processing platform, and the Kafka message queue cluster realizes the extraction and storage of the test question stem stream data information.

Description

Technical Field

[0001] The present invention relates to the technical field of machine learning, and in particular to a real-time semantic similarity matching design method based on a cosine similarity algorithm. Background Art

[0002] CDC (Change Data Capture) is a change data capture technology. In a broad sense, any technology that can capture data changes can be called CDC. The CDC technology that is commonly described at present is mainly oriented towards database changes. It is a technology used to capture data changes in the database.

[0003] Kafka Connect focuses on the data flow between Kafka, making it easier to write connector plug-ins. Each Connector deployed in the Kafka Connect service monitors an upstream database server and writes the captured data to the corresponding Topic in the Kafka message queue cluster.

[0004] Semantic similarity is a measure of the similarity between two texts, identifying whether two texts are semantically similar. Semantic similarity is an important research direction in the field of natural language processing, and also plays an important role in question-answering systems, article search, and news recommendation.

[0005] Sentence embedding is an important component of the natural language processing (NLP) system based on deep learning. The sentence vector constructed by the sentence embedding method contains the semantic relationship between words, and can calculate the semantic relevance between words in the sentence. The commonly used relevance calculation method is cosine similarity.

[0006] The core issues of the question bank management system are the arrangement of test questions, entry and repeatability verification of the question bank, and intelligent test paper composition algorithm. With the increasing number of test questions in the question bank, automatically identifying and removing similar test questions has become the key to establishing a high-quality question bank. However, there is currently little research on the identification of test question similarity at home and abroad. A reasonable question bank structure and a complete question management system are the basis for establishing a high-quality question bank and successfully compiling test papers. The key issue of the question entry module in the question bank management system is the repeatability verification of the test questions. Only by identifying repeated or similar test questions can the purpose of establishing a high-quality question bank be achieved.

[0007] Elasticsearch (ES) is a distributed, free, open source search and analytics engine for all types of data, including text, numbers, geospatial, structured, and unstructured data. Elasticsearch uses a data structure called an inverted index, which is designed to allow very fast full-text searches. Using sentence embeddings and Elasticsearch's dense_vector type, the cosine similarity algorithm can support text similarity searches, i.e., the identification of similar test questions in the question bank management system.

[0008] With the development of machine learning and deep learning, various pre-training models (PTM) have been introduced. A pre-training model is to first train a model through a batch of corpus, and then continue training based on this preliminarily trained model. At present, pre-training is a very common method in both the CV field and the natural language processing (NLP) field. Universal Sentence Encoder is a model proposed by Google in 2018 that encodes sentences into embedded vectors, that is, the question encoding model in the question bank management system. These embedded vectors are specifically used for NLP tasks such as sentence similarity research.

[0009] The publication number is CN115344645A, which discloses a method and tool for synchronizing Oracle data to GreenPlum based on Debezium. Although it has the advantages of real-time and fast data synchronization, compared with other databases, although Oracle provides a LogMiner interface, it can only perform CDC on the main database and cannot be used for the ADG backup database; since the original design of LogMiner was a diagnostic tool for Redo logs rather than a dedicated CDC tool, it does not perform any efficiency cost optimization for continuous operation. Running directly on the main database will inevitably affect the normal operation of the main database business; the extraction performance is limited, and large delays and memory problems will be encountered when there are hundreds of transactions per second.

[0010] Publication number CN115391577A discloses an electronic archive management method and system based on a machine learning algorithm. This invention patent performs weighted processing on matching keywords of text files to obtain the weights of matching keywords, but there is a dimensionality explosion problem, and text similarity is difficult to calculate effectively; in addition, the words in the text set are independent of each other, so the semantic relationship between words is missing in the calculation, affecting the performance of text semantic similarity. Summary of the invention

[0011] In order to overcome the shortcomings of the prior art, the purpose of the present invention is to provide a real-time semantic similarity matching design method based on the cosine similarity algorithm. The JavaWeb application end synchronizes data from the application database MySQL to the Kafka message queue in real time, and completes the real-time extraction and conversion of incremental data from the application database MySQL; the Python Web consumer obtains data through client consumption and uses the cosine similarity algorithm to retrieve the semantic similarity of the text.

[0012] To achieve the above object, the present invention is implemented through the following technical solutions:

[0013] A real-time semantic similarity matching design method based on cosine similarity algorithm specifically includes the following contents:

[0014] 1) Upload text information to the application database MySQL on the JavaWeb application side;

[0015] 2) Dynamically capture CDC real-time data based on the Debezium low-latency streaming processing platform;

[0016] 3) Based on the Kafka message queue cluster, monitor an upstream database server through the Connector connector of the Kafka Connect message middleware plug-in, capture all changes of the application database MySQL configured by Debezium, and write them to the corresponding topic of the Kafka message queue cluster;

[0017] 4) Configure the Kafka Connector transforms filter in the Kafka Connect message middleware plug-in for lightweight message modification and filtering;

[0018] 5) The captured data is lightly filtered by the Kafka Connect message middleware plug-in and then written to the Kafka message queue cluster;

[0019] 6) Use Python Consumer to consume data in the Kafka message queue cluster, embed and vectorize sentences, and write the sentence text information and sentence vectors into the ElasticSearch full-text search engine;

[0020] 7) By sending a request of HTTP protocol, the successfully written text information is returned to the Java Web application end, and the application database MySQL is updated;

[0021] 8) In the Browser / Server network structure mode, the Python Web backend receives the user input text, connects to the ElasticSearch full-text search engine, and uses the cosine similarity algorithm to perform semantic similarity matching, which specifically includes the following:

[0022] Based on the cosine similarity algorithm and CDC technology of text embedding technology, data is synchronized from the application database MySQL in real time, and the text data of each row of data is converted into a vector using the Universal Sentence Encoder pre-trained model and stored in the ElasticSearch full-text search engine;

[0023] The client user inputs text data, and the Universal Sentence Encoder pre-trained model outputs a 512-dimensional text embedding vector. The feature data of the text is converted into a vector, and the vector is put into the query statement in the ElasticSearch full-text search engine. By using the cosine similarity algorithm, the TopN results with the highest similarity in the ElasticSearch full-text search engine are obtained, and the scores are sorted from high to low for semantic similarity matching.

[0024] The cosine similarity algorithm specifically includes the following contents:

[0025] 1) Based on the text embedding sentence embedding technology, the information of the text semantic layer is used to rank the documents;

[0026] 2) Based on the Universal Sentence Encoder pre-trained model, use the text embedding technology to output the sentence embedding vector;

[0027] 3) The vector dimension of text embedding sentence embedding technology is 512; Text embedding sentence embedding technology encodes text into a low-dimensional space vector;

[0028] 4) Cosine similarity measures the similarity between two vectors by calculating the cosine value of the angle between them. If the two vectors have the same direction, the cosine similarity value is 1, if the angle between the two vectors is 90°, the cosine similarity value is 0, and if the two vectors point in completely opposite directions, the cosine similarity value is -1; the closer the angle between the two vectors is to 0, the closer the cosine value is to 1, indicating that the two vectors are more similar;

[0029] The formula for calculating the cosine of the vector angle is as follows:

[0030]

[0031] In the formula, taking a two-dimensional vector as an example, the cosine similarity of vector A and vector B is calculated. Vector A starts from the origin (0, 0) and ends at (x1, y1). Vector B starts from the origin (0, 0) and ends at (x2, y2). α is the angle between the two vectors, and cosα is the cosine value between the two vectors, ranging from -1 to 1.

[0032] The Debezium low-latency streaming processing platform includes the CDC configuration of the application database MySQL, monitoring of the backup database of the application database MySQL, real-time capture of MySQL change data based on the Kafka Connect message middleware plug-in, and writing to the kafka message queue cluster; the CDC configuration of the application database MySQL includes creating a new MySQL user, authorizing the MySQL user, setting parameters, and enabling log-bin.

[0033] The upstream database server is used to store and upload text data of the local application database MySQL of the JavaWeb application side in the Browser / Server network structure mode.

[0034] Python Consumer is used to obtain Topic data. It can consume every row-level addition and change of the application database MySQL, process the data to generate sentence embedding vectors and load them into the ElasticSearch full-text search engine. The client can stop consumption at any time, then restart and continue consumption from the last stop point.

[0035] The Java Web application and the ElasticSearch full-text search engine are both connected to the FastAPI Web service. FastAPI Web is a Web framework based on Python technology.

[0036] This method is implemented based on the dual-stack architecture of Python and Java, which includes the application database MySQL, Kafka message queue cluster, Connector connector of Kafka Connect message middleware plug-in, Debezium low-latency streaming processing platform, FastApi framework, Python Consumer consumer, cosine similarity algorithm based on sentence embedding, and ElasticSearch full-text search engine.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] 1. The text vector generated by the sentence embedding method contains the semantic relationship between words, which can effectively measure the semantic similarity between texts and solve the problem of duplicate test questions. The present invention designs and implements a test question entry and duplicate checking system based on text semantic similarity, which has a lower computational complexity reduction;

[0039] 2. The cosine similarity algorithm is used for sentence similarity, the Debezium low-latency streaming processing platform is used for dynamic capture of change data, and the Kafka message queue cluster is used to extract and store the test question stream data information;

[0040] 3. Based on Debezium, synchronize the MySQL data of the application database and run it on the standby database. CDC runs on the standby database to avoid performance impact on the main database.

[0041] 4. As a search and analysis engine, ElasticSearch has stable performance in search performance and repeatability verification. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a schematic diagram of the real-time semantic similarity matching architecture based on CDC technology and sentence embedding algorithm.

[0043] Figure 2 It is a flowchart of the cosine similarity algorithm based on text embedding.

[0044] Figure 3 It is the business process diagram of the test question management module in the question bank management system. DETAILED DESCRIPTION

[0045] The present invention is described in detail below in conjunction with the accompanying drawings, but it should be noted that the implementation of the present invention is not limited to the following embodiments.

[0046] The following examples are implemented on the premise of the technical solution of the present invention, and provide detailed implementation methods and specific operation processes, but the protection scope of the present invention is not limited to the following examples. The methods used in the following examples are conventional methods unless otherwise specified.

[0047] [Example 1]

[0048] CDC-based technology is a way to back up databases, and is often used to back up large amounts of data. The non-invasive backup method is log-based backup. The log-based CDC of the application database MySQL requires turning on the MySQLbin log. CDC is mainly based on Debezium technology, which provides a low-latency streaming processing platform for capturing change data, and uses the Kafka message queue cluster and Kafka Connect message middleware plug-in to achieve persistence, reliability, and fault tolerance.

[0049] See Figure 1 , a real-time semantic similarity matching design method based on cosine similarity algorithm, dynamic capture of real-time data based on CDC and dual-stack architecture of Python and Java; the dual-stack architecture of Python and Java includes application database MySQL, Kafka message queue cluster, Connector connector of Kafka Connect message middleware plug-in, Debezium low-latency streaming processing platform, FastApi framework, Python Consumer consumer, cosine similarity algorithm based on sentence embedding, ElasticSearch cluster; the Connector connector of Kafka Connect message middleware plug-in is based on Kafka message queue cluster and connects to upstream database server; the upstream database server is used for storage and upload of text data of local application database of JavaWeb application side in Browser / Server network structure mode, and provides data addition, deletion, modification and query operations in local application database; Kafka message queue connects Python Consumer consumer, Python contains Web framework, such as FastAPI Web, FastAPI Web service connects Java Web application side and ElasticSearch cluster server communication, wherein JavaWeb application side is used to call the service of FastAPI Web framework on Web server side to obtain cosine similarity result value; Python Consumer is used to obtain Topic data. It can consume every row-level addition and change of the application database, process the data to generate sentence embedding vectors and load them into the ElasticSearch full-text search engine. The client can stop consumption at any time, then restart and continue consumption from the last stop point.

[0050] The JavaWeb application synchronizes data from the application database MySQL to the Kafka message queue in real time, completing the real-time extraction and conversion of incremental data from the application database MySQL. Java Web uses Java technology to solve the technology stack in the related Web Internet field. The Web includes two parts: the Web server and the Web client. Python Web consumers obtain data through client consumption and use the cosine similarity algorithm to retrieve the semantic similarity of the text. Finally, the full decoupling and closed-loop drive of the business system and the AI ​​system are achieved.

[0051] The specific contents of the method include:

[0052] 1) The Java Web application uploads the text to the application database MySQL and stores the text information in the application database MySQL;

[0053] 2) Dynamic capture of CDC real-time data, based on the Debezium low-latency streaming processing platform. The Debezium low-latency streaming processing platform includes CDC configuration of the application database MySQL, monitoring of the backup database of the application database MySQL, real-time capture of MySQL change data based on the KafkaConnect message middleware plug-in, and writing to the Kafka message queue cluster. The CDC configuration of the application database MySQL includes creating a new MySQL user, authorizing the MySQL user, setting parameters, and enabling log-bin.

[0054] 3) Based on the Kafka message queue cluster, the Connector of the Kafka Connect message middleware plug-in monitors an upstream database server, captures all changes of the application database MySQL configured by Debezium, and writes them to the corresponding topic of the Kafka message queue. It has a fault-tolerant mechanism, and the recovery of the job realizes breakpoint retransmission, without data loss or duplicate data generation;

[0055] 4) Configure the Kafka Connector transforms filter in the Kafka Connect message middleware plug-in to perform lightweight message modification and message filtering, and facilitate data modification and event routing;

[0056] 5) The captured data is lightly filtered by the Kafka Connect message middleware plug-in and then written to the Kafka cluster;

[0057] 6) Use Python Consumer to consume data in the Kafka message queue cluster, embed and vectorize sentences, and write the sentence text information and sentence vectors into the ElasticSearch full-text search engine;

[0058] 7) By sending a request of HTTP protocol, the successfully written text information is returned to the Java Web application end, and the application database MySQL is updated;

[0059] 8) In the subsequent semantic similarity matching process, it is only necessary to match the vector input by the user with the vector in the ElasticSearch full-text search engine to find the relevant vector;

[0060] 9) In the Browser / Server network structure mode, the Python Web backend receives user input text, connects to the ElasticSearch cluster server, and uses the cosine similarity algorithm to implement the semantic similarity matching process; the cosine similarity algorithm based on text embedding synchronizes data from the database in real time based on the CDC technology, and uses the UniversalSentence Encoder pre-trained model to convert the text data of each row of data into a vector and save it in the ElasticSearch full-text search engine; the JavaWeb client user inputs text data, and through the pre-trained model, a 512-dimensional text embedding vector is output, the feature data of the text is converted into a vector, and the vector is put into the query statement in the ElasticSearch full-text search engine. By using the cosine similarity algorithm, the TopN results with the highest similarity in the ElasticSearch full-text search engine are obtained, and the scores are sorted from high to low for semantic similarity matching.

[0061] See Figure 2 , the cosine similarity algorithm specifically includes the following contents:

[0062] a) Based on the text embedding sentence embedding technology, the information of the text semantic layer is used to rank the documents;

[0063] b) Sentence embedding technology is based on the Universal Sentence Encoder pre-training model and uses text embedding technology to output text embedding sentence embedding vectors;

[0064] c) The vector dimension of text embedding is 512. Text embedding encodes text into low-dimensional space vectors. Synonyms and phrases are represented in a very similar way in the new vector space.

[0065] d) Cosine similarity measures the similarity between two vectors by calculating the cosine value of the angle between them. If the two vectors have the same direction, the cosine similarity value is 1, if the angle between the two vectors is 90°, the cosine similarity value is 0, and if the two vectors point in completely opposite directions, the cosine similarity value is -1; the closer the angle between the two vectors is to 0, the closer the cosine value is to 1, indicating that the two vectors are more similar;

[0066] The cosine of the vector angle is calculated using the following formula:

[0067]

[0068] In the formula, taking a two-dimensional vector as an example, the cosine similarity between vector A and vector B is calculated. Vector A starts from the origin (0, 0) and ends at (x1, y1); vector B starts from the origin (0, 0) and ends at (x2, y2); α is the angle between the two vectors, in degrees; cosα is the cosine value between the two vectors, ranging from -1 to 1;

[0069] As a full-text search engine, ElasticSearch provides full-text search capabilities, and word vector search provides a computing platform for text similarity matching.

[0070] [Example 2]

[0071] In this embodiment, a real-time semantic similarity matching design method based on the cosine similarity algorithm is the same as that in Embodiment 1, on the basis of which a working process description is added.

[0072] See Figure 1 ,See Figure 2 , the working process steps are as follows:

[0073] S1. The Java Web application uploads the text to the application database MySQL and stores the text information in the application database MySQL.

[0074] S2, CDC real-time data capture, based on the Debezium low-latency streaming processing platform; basic configuration of the application database MySQL is required, the specific steps are as follows:

[0075] S21. Create a new MySQL user:

[0076] CREATE USER 'debezium'@'%' IDENTIFIED BY 'debeziumPassword'; Create a new user with the username debezium and the password debeziumPassword;

[0077] S22. After creating a new user account, the user has no permissions; grant permissions to the user, including "SELECT", "RELOAD", "SHOW DATABASES", "REPLICATION SLAVE", and "REPLICATION CLIENT":

[0078] GRANT SELECT,RELOAD,SHOW DATABASES,REPLICATION SLAVE,REPLICATIONCLIENT ON*.*TO'debezium'IDENTIFIED BY'debeziumPassword'; Use the GRANT statement to grant permissions. Specify one or more permissions after the GRANT keyword; Grant multiple permissions to user debezium, with each permission separated by a comma;

[0079] S23. Set the query log events parameters:

[0080] binlog_rows_query_log_events = ON; turn on binlog_rows_query_log_events, print SQL statements into the binlog log, and view the transaction log;

[0081] S24. Check whether log-bin is enabled:

[0082] select variable_value from global_variables

[0083] where variable_name='log_bin'; check whether log_bin is turned on, "ON" means turned on, "OFF" means turned off;

[0084] S25. Enable Global transaction identifiers (GTIDs, consistency between master and slave databases):

[0085] gtid_mode = ON; modify the consistency parameters of the master database and the slave database, "ON" means enable;

[0086] enforce_gtid_consistency = ON; modify the consistency parameters of the master database and the slave database, "ON" means enabled.

[0087] S3. Start the Kafka message queue cluster and configure the Connector of Kafka Connect. The specific steps are as follows:

[0088] S31. Configure the connect-standalone.properties file, which contains the plugin path plugin.path, key.converter, and value.converter information of the plugin.

[0089] S32, connector.class is io.debezium.connector.mysql.MySqlConnector; Debezium MySQL connector configuration properties, connector.class specifies the name of the connector Java class;

[0090] S33. Configure the connection information of the application database MySQL and the whitelist of databases, tables, and fields to be monitored for changes.

[0091] In S4 and Kafka Connect message middleware plug-ins, you need to configure the Kafka Connector transforms filter to perform lightweight message modification and message filtering. The specific steps are as follows:

[0092] S41. Set the transformation name to filter and the transformation type to io.debezium.transforms.Filter.

[0093] S42. The newly added data in the application database MySQL sets FLAG2 to 0. After the data is written into ES, it is set to 1 when it is written back.

[0094] S43, when updating, FLAG2 is set to 2, that is, only the data that FLAG2 is 1 before the change and 2 after the change is captured;

[0095] S44, capture the deleted data and delete it in the ES database after judgment;

[0096] S45, the update logic is to update ES synchronously. After the update, it is set to 1 when writing back.

[0097] The conversion condition, the expression is as follows:

[0098]

[0099] "||" is a logical operator. Expression ① is expressed as "expression 1||expression 2||expression 3". The operation rule of the expression is that as long as any expression is true, the operation result of the entire expression is true; "expression 1" represents the newly added data; "expression 2" represents the modified data, and before the modification, FLAG2 is 1, and after the modification, FLAG is 2; "expression 3" represents the deleted data.

[0100] S5. After light filtering by the Kafka Connect message middleware plug-in, the captured data is written to the Kafka cluster. The topic naming rules are as follows:

[0101] topicPrefix.databaseName.tableName.

[0102] S6. Use Python Consumer to consume data in the Kafka message queue cluster, and after sentence embedding and vectorization, write the sentence text information and sentence vectors into ES.

[0103] S7. By sending a request of the HTTP protocol, the successful writing information is returned to the Java Web application end, and the application database MySQL is updated. When updating, FLAG2 is set to 1.

[0104] S8. In the subsequent semantic similarity matching process, it is only necessary to match the vector input by the user with the vector in ES to find the relevant vector.

[0105] S9, see Figure 2 As shown in the figure, the cosine similarity algorithm based on sentence embedding has the following specific steps:

[0106] S91, a pre-trained model based on TensorFlow Hub, universal-sentence-encoder-multilingual, which takes long text such as sentences, phrases or short paragraphs as input and outputs a 512-dimensional sentence embedding vector;

[0107] S92, duplicate checking Python backend, the user inputs the query text, the model outputs the sentence embedding vector as query_vector, queries ES to index the questions_vector field, and obtains the sentence embedding vector as doc['questions_vector']. The cosineSimilarity function is used to calculate the semantic similarity TopN result. The query statement is as follows:

[0108] "script":{

[0109] "source":"cosineSimilarity(params.query_vector,doc['question_vector'])+1.0",②"params":{"query_vector":query_vector}

[0110] }

[0111] Statement ② is a JSON format string, which is a set of name / value pairs separated by commas "," . The value of "params" query_vector is a 512-dimensional vector; the value of "source" is the cosine similarity function cosineSimilarity, and cosineSimilarity passes in two parameters for calculating cosine similarity, namely params.query_vector (i.e. the value of "params" query_vector), and the index value of the questions_vector field in ES doc['question_vector']; in order to avoid negative scores, add 1 to the result of the cosine similarity function value for weighting.

[0112] S10, see Figure 2 , the process of user duplicate checking has been optimized. During the process of user duplicate checking, the Java Web application client can upload text in batches, and after uploading, the text storage ID of the application database MySQL can be obtained; therefore, when checking for duplicates in batches, including the uploaded text, the sentence embedding of the text is directly obtained by querying from ES, and the model output is no longer used. The model output is run through the sentence embedding model once, which is the S91 execution process. The similarity matching time of each text can be optimized by 1s.

[0113] [Example 3]

[0114] In this embodiment, a real-time semantic similarity matching design method based on the cosine similarity algorithm is the same as that in Embodiment 1, on the basis of which a description of the business process of the question management module of the question bank management system is added.

[0115] See Figure 3 , the specific steps are as follows:

[0116] S1. Set the test question verification scope and verification standard;

[0117] The user sets the question stem similarity verification rules on the Java Web application side, and the similarity ranges are 100%, 90%, 80%, and 70% respectively.

[0118] S2, test question input;

[0119] There are two ways to enter test questions. The first is to import test questions in batches (only supports excel2003 and above versions, wps files). At the same time, there is a duplicate check for test questions of the same type of work. Test questions with completely repeated stem information are considered duplicate test questions, and there are information prompts for duplicate test questions (when importing test questions, the three parts of professional skill evaluation, professional technical qualifications, and professional ability model test questions can be imported from a unified set of templates). The system supports manual addition, deletion (logical deletion), and modification of test questions.

[0120] S3, manually create test questions;

[0121] There are two ways to enter test questions. The second way is to manually create test questions on the Java Web application side and enter the test question stem and other information.

[0122] S4. Save historical records;

[0123] On the Java Web application side, after manually creating a test question, the user can add, delete, and modify the test question. The system records the operation log, which includes the user's name, work number, operation date, operation category, operation question number, question type, difficulty, knowledge points, and application scope (public questions, training exercises, and internal questions). The test question modification log should include the content before and after the modification.

[0124] S5, test question repeated judgment;

[0125] The user clicks "Question Duplicate Check" on the application side, and the Java Web application side calls the service of the FastAPI Web framework to perform question duplicate check based on the similarity percentage value selected by the S1 user, obtain the cosine similarity result value, and display the duplicate question list on the application side based on the sorted TopN result values.

[0126] S6, offline adjustment Excel;

[0127] According to the repeated judgment results of S5 test questions, check the repeated test questions list, adjust Excel offline, adjust Excel, and re-upload the test questions to perform repeated judgment of S5 test questions.

[0128] S7. Import the question bank;

[0129] Adjust the test questions without duplication and import them into the application database.

[0130] The text vector generated by the sentence embedding method of the present invention contains the semantic relationship between words, can effectively measure the semantic similarity between texts, solves the problem of test question duplication verification, designs and implements a test question entry and duplicate checking system based on text semantic similarity, and has low computational complexity; sentence similarity adopts the cosine similarity algorithm, and the dynamic capture of change data adopts the Debezium low-latency streaming processing platform, and the Kafka message queue cluster realizes the extraction and storage of test question stem stream data information.

Claims

1. A real-time semantic similarity matching design method based on cosine similarity algorithm, characterized in that: Specifically include the following: 1) Upload text information to the application database MySQL on the JavaWeb application side; 2) Dynamically capture CDC real-time data based on the Debezium low-latency streaming processing platform; 3) Based on the Kafka message queue cluster, monitor an upstream database server through the Connector connector of the Kafka Connect message middleware plug-in, capture all changes of the application database MySQL configured by Debezium, and write them to the corresponding topic of the Kafka message queue cluster; 4) Configure the Kafka Connector transforms filter in the Kafka Connect message middleware plug-in for lightweight message modification and filtering; 5) The captured data is lightly filtered by the Kafka Connect message middleware plug-in and written to the Kafka message queue cluster; 6) Python Consumer is used to obtain Topic data. It can consume every row-level addition and change of the application database MySQL, process the data to generate sentence embedding vectors and load them into the ElasticSearch full-text search engine. The client can stop consumption at any time and then restart it to continue consumption from the place where it stopped last time. After sentence embedding vectorization processing, the sentence text information and sentence vectors are written into the ElasticSearch full-text search engine; 7) By sending a request of HTTP protocol, the successfully written text information is returned to the Java Web application end, and the application database MySQL is updated; 8) In the Browser / Server network structure mode, the Python Web backend receives the user input text, connects to the ElasticSearch full-text search engine, and uses the cosine similarity algorithm to perform semantic similarity matching, which specifically includes the following: Based on the cosine similarity algorithm and CDC technology of text embedding technology, data is synchronized from the application database MySQL in real time, and the text data of each row of data is converted into a vector using the Universal Sentence Encoder pre-trained model and stored in the ElasticSearch full-text search engine; The client user inputs text data, and the Universal Sentence Encoder pre-trained model outputs a 512-dimensional text embedding vector. The feature data of the text is converted into a vector, and the vector is put into the query statement in the ElasticSearch full-text search engine. By using the cosine similarity algorithm, the TopN results with the highest similarity in the ElasticSearch full-text search engine are obtained, and the scores are sorted from high to low for semantic similarity matching.

2. The real-time semantic similarity matching design method based on the cosine similarity algorithm according to claim 1, characterized in that: The cosine similarity algorithm specifically includes the following contents: 1) Based on the text embedding sentence embedding technology, the information of the text semantic layer is used to rank the documents; 2) Based on the Universal Sentence Encoder pre-trained model, use the text embedding technology to output the sentence embedding vector; 3) The vector dimension of text embedding sentence embedding technology is 512; Text embedding sentence embedding technology encodes text into a low-dimensional space vector; 4) Cosine similarity measures the similarity between two vectors by calculating the cosine value of the angle between them. If the two vectors have the same direction, the cosine similarity value is 1, if the angle between the two vectors is 90°, the cosine similarity value is 0, and if the two vectors point in completely opposite directions, the cosine similarity value is -1; the closer the angle between the two vectors is to 0, the closer the cosine value is to 1, indicating that the two vectors are more similar; The formula for calculating the cosine of the vector angle is as follows: In the formula, taking a two-dimensional vector as an example, the cosine similarity of vector A and vector B is calculated. Vector A starts from the origin (0, 0) and ends at (x1, y1). Vector B starts from the origin (0, 0) and ends at (x2, y2). α is the angle between the two vectors, and cosα is the cosine value between the two vectors, ranging from -1 to 1.

3. The real-time semantic similarity matching design method based on the cosine similarity algorithm according to claim 1, characterized in that: The Debezium low-latency streaming processing platform includes CDC configuration of the application database MySQL, monitoring of the backup database of the application database MySQL, real-time capture of MySQL change data based on the Kafka Connect message middleware plug-in, and writing to the kafka message queue cluster; the CDC configuration of the application database MySQL includes creating a new MySQL user, authorizing the MySQL user, setting parameters, and opening log-bin.

4. The real-time semantic similarity matching design method based on cosine similarity algorithm according to claim 1, characterized in that: The upstream database server is used for storing and uploading text data of the local application database MySQL of the JavaWeb application end in the Browser / Server network structure mode.

5. The real-time semantic similarity matching design method based on cosine similarity algorithm according to claim 1, characterized in that: The Python Consumer is used to obtain Topic data. It can consume every row-level addition and change of the application database MySQL, process the data to generate sentence embedding vectors and load them into the ElasticSearch full-text search engine. The client can stop consumption at any time, and then restart and continue consumption from the place where consumption was stopped last time.

6. A real-time semantic similarity matching design method based on cosine similarity algorithm according to claim 1, characterized in that: The JavaWeb application and the ElasticSearch full-text search engine are both connected to the FastAPI Web service, and FastAPI Web is a Web framework based on Python technology.

7. A real-time semantic similarity matching design method based on cosine similarity algorithm according to claim 1, characterized in that: The method is implemented based on a dual-stack architecture of Python and Java, wherein the dual-stack architecture of Python and Java includes an application database MySQL, a Kafka message queue cluster, a Connector connector of a Kafka Connect message middleware plug-in, a Debezium low-latency streaming processing platform, a FastApi framework, a Python Consumer consumer, a cosine similarity algorithm based on sentence embedding, and an ElasticSearch full-text search engine.

Images