System for automatically updating website articles, tags and category matches with a model

Abstract

This invention discloses an automatic update system for website article, tag, and category matching models, relating to the field of model update technology. It includes a data input module, a recommendation query unit, and a training unit. The data input module includes inputting query data and matching keyword data with data obtained from the database by the recommendation query unit to obtain matching data and corresponding articles. The database used is the Milvus database. This application optimizes the model and its performance, accelerating the training speed. Furthermore, this application uses the Milvus database, the Roformer model, and the prompt technique to solve the problem of requiring multiple manual training and restarts for a single site. Model updates are convenient and do not affect business operations. The same model can be adapted to data groups from multiple sites, and the problem of repeated training across multiple sites is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of model updating, in particular to a model automatic updating system for matching website articles, tags and categories. BACKGROUND

[0002] In the automatic matching of article and tag text semantics, many industrial applications measure the similarity of text in semantics, collectively known as semantic matching. According to the length of the text, semantic matching can be divided into three categories: short text-short text semantic matching, short text-long text semantic matching and long text-long text semantic matching. For some sites, the data of tags and articles are more and need to be managed in detail. The time and effort consumed by manual management of tag data and matching is large. The tag data exists in the form of words, sentence breaks and long sentences, such as: interview skills, short stories with reason, how to effectively communicate, what are the duties and tasks of accountants, etc. In the above application scenarios, the matching scheme of short text-long text can be selected as follows: when calculating the similarity, the theme mapping of short text is avoided, and the probability of generating short text from the theme distribution of long text is calculated as their similarity, and the optimal multiple matching results are selected as the final result.

[0003] In the automatic identification of article tag classification, common text classification is to divide the text into a specified category or several categories in a given classification system. The classification objects are short text (sentence / title / product review) and long text (article). The classification system is generally constructed by humans. The classification modes include binary classification problem, multi-class problem and multi-label problem. The multi-label problem can be defined as a text belonging to multiple categories, which is a difficult problem of text classification. First, the classification method using machine learning includes feature engineering + algorithm (Native Bayes / SVM / LR / KNN). Second, the classification method using deep learning includes word vector + model, FastText, TextCNN, TextRNN, TextRCNN, DPCNN and BERT. The process of classifying text includes text preprocessing, feature extraction (such as word frequency feature, secondary feature, syntax feature and TFIDF feature), text representation (such as word2vec, Glove and Bert), and classification model processing (machine learning, deep learning, CNN and RNN). For a site, the number of tag classifications and the coverage range of tag content gradually increase. Manual maintenance of the machine has problems such as lack of rationality of subjective classification, repeated classification, confusion of tag classification division standards, and the need for continuous addition and refinement of classification. Manual maintenance of classification is very difficult.

[0004] For applications that only need to meet the above requirements for a single site, the goal can be achieved by using local model training, a Redis database, and local caching. The algorithm model does not need to be deployed separately, and it can be updated regularly based on the updates to the source data. Redis is an open-source key-value database written in C that supports network interaction and can be either memory-based or persistent.

[0005] When implementing the above solutions, Redis is used as a caching and data exchange tool. However, when a single site becomes a site cluster, the local caching method is no longer suitable. When adding a new site, if distributed deployment is required, data synchronization issues will inevitably arise.

[0006] The basic approach is to store all the data on every node, which leads to serious memory waste. Assuming three nodes are deployed, the storage size is N*4 (three machines + Redis). This is because matrix multiplication is performed directly when calculating similarity, so all the data needs to be put into the matrix for calculation and needs to be stored in memory. If each node in a cluster stores a part of the data, the code will be very complex during the final merging process, and data skew and data comparison issues need to be manually resolved.

[0007] Furthermore, when multiple machines are synchronized, there will be issues of network waste and usage, as well as data consistency problems. When synchronization fails or is delayed, the results of multiple distributed outputs will be inconsistent. Summary of the Invention

[0008] The purpose of this invention is to provide an automatic update system for a model that matches website articles, tags, and categories.

[0009] The technical problem solved by this invention is to address the issues of network waste and occupancy when multiple machines are synchronized, as well as data consistency problems, where inconsistent results occur when synchronization fails or is delayed. It also addresses the problems of subjective and unreasonable classification, duplicate categorization, and chaotic label classification standards in manually maintaining machine hierarchical relationships, which requires continuous addition and refinement of classifications, making manual classification very difficult.

[0010] This invention can be achieved through the following technical solution: an automatic update system for website articles, tags and categories matching models, including a data input module, a recommendation query unit and a training unit. The data input module includes inputting query data, obtaining query data through the recommendation query unit, processing the query data to obtain keyword data, and matching the keyword data with the data in the database obtained by the recommendation query unit to obtain matching data and the corresponding articles.

[0011] The database used is Milvus, which employs a shared architecture. Milvus retrieves article information data from the website, processes the data, and then stores it.

[0012] The training unit adaptively trains the data processing model in the recommendation query unit. The training content of the training unit includes request incremental training, so that the data processing model can adapt to the processing of business data when business data increases.

[0013] A further technical improvement of the present invention is that: the data input module includes an offline unit and an online unit; the offline unit is used to set a timed data acquisition unit in an offline state, and by setting the time interval for data acquisition, the data in the site is processed by an algorithm model, and the data obtained after processing by the algorithm model is stored in the Milvus database;

[0014] The online unit is used in an online mode where the site backend directly transmits data, processes the data within the site using an algorithm model, and stores the data obtained after the algorithm model processing in the Milvus database.

[0015] A further technical improvement of the present invention is that: the recommendation query unit includes an input data processing unit and a comparison unit. The input data processing unit is used to obtain the query data, process the input data to obtain vector mapping data embedding, and store the vector mapping data embedding in the Milvus database. The comparison unit obtains the vector mapping data embedding and compares it with the data in the Milvus database to obtain matching data and the corresponding articles.

[0016] A further technical improvement of the present invention is that the processing steps of the data processing unit include:

[0017] Step 1: Input article data and tag data;

[0018] Step 2: The Roformer model obtains the input article data and tag data. After processing the article data and tag data within the Roformer model, the embedding data is obtained.

[0019] Step 3: Store the obtained embedding data in the Milvus database.

[0020] A further technical improvement of the present invention is that it also includes a timed synchronization module for correcting data sync issues within the site. The synchronization steps of the synchronization module include:

[0021] Step 1: Set a timer to periodically retrieve updated data from the site;

[0022] Step 2: Extract the embedding data from the updated data within the site using the Roformer model and store it in the Milvue database.

[0023] A further technical improvement of the present invention is that the training unit includes a model training unit and an automatic update unit. The model training unit is used to train the modified model by inputting data for model training, modifying the values ​​by inputting the data into the original model, obtaining the training result, and updating the model with the training result through the automatic update unit to modify the model.

[0024] A further technical improvement of the present invention is that the working steps of the model training unit include:

[0025] Step 1: The model training unit receives the request for incremental training of the model, and the offline unit obtains the training time and training labels of the samples.

[0026] Step 2: The model training unit trains the model according to the training time and training labels of the samples, and uploads the trained model to the file server.

[0027] A further technical improvement of the present invention is that the working steps of the automatic update unit include:

[0028] Step 3: The online unit receives the signal from the site requesting an algorithm update, and downloads the latest training result model from the file server;

[0029] Step 4: Retrieve the version number of the original model, define the version number of the latest training result model based on the version number of the original model, and store the original model in the file server.

[0030] A further technical improvement of the present invention is that it also includes a deployment module, which includes a public deployment unit and a private deployment unit. The public deployment unit adopts a public algorithm interface, and the private deployment unit adopts a separate interface.

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] 1. This application processes the input articles, extracting embeddings from the article and tag data using the Roformer model and storing them in the Milvue database. Then, by comparing the input article and tag data with existing embeddings in the Milvue database, it matches the input articles and tag data with the same or similar embeddings, retrieves the articles to which the matched embeddings belong, and sends the resulting articles. This process optimizes the model and its performance, accelerating training. Specifically, this application uses the Roformer model for classification tasks with limited data, primarily improving training and prediction speed through structural simplification. Furthermore, it utilizes prompts, unsupervised pre-training, and few-shot training. This application combines fine-tuning and data set analysis to address the issue of poor model performance caused by insufficient data, achieving a "win-win" situation for both speed and effectiveness. Furthermore, the use of the Roformer model simplifies the structure, accelerating the retrieval of embedding data. The combination of unsupervised and supervised training further enhances performance, achieving a win-win situation for both speed and effectiveness. Moreover, the application employs the Milvus database, the Roformer model, and prompt technology to eliminate the need for multiple manual training and restarts at a single site. Model updates are convenient and do not impact business operations. The same model can be adapted to data sets from multiple sites, resolving the issue of repetitive training across multiple sites. Even with real-time changes and continuous additions to business and classification data, it can still meet the long-term needs of business stakeholders.

[0033] 2. Furthermore, this application achieves the separation of data flow and control flow by using a Milvus database. During use, data from any site is stored in and retrieved from the Milvus database. Using a single database saves code, reduces space utilization, and minimizes memory waste. When using the same Milvus database, data processing becomes more consistent and rational. Moreover, when matching data, calling data from the same database ensures data processing speed and stability, automatically maintains the system, and avoids problems such as the difficulty of manual maintenance and classification.

[0034] 3. The Milvus database used in this application has the ability to perform fast matrix calculations and store embedding data. It solves the problem of synchronizing data and independently calculating similar articles across multiple computing nodes in a site cluster, simplifies the steps of developing data synchronization and calculation using code, and reduces data storage requirements, since the number of computing nodes in a site cluster is often greater than the number of storage nodes in a Milvus cluster. Attached Figure Description

[0035] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to the accompanying drawings.

[0036] Figure 1 This is a system principle block diagram of the present invention;

[0037] Figure 2 This is a schematic diagram of the data processing module of the present invention;

[0038] Figure 3 This is a schematic diagram illustrating the function of the recommendation query unit of the present invention;

[0039] Figure 4 This is a schematic diagram of the timed synchronization data of the present invention;

[0040] Figure 5 This is a schematic diagram illustrating the model training and automatic update process of the present invention;

[0041] Figure 6 This is a schematic diagram illustrating the algorithm deployment method of the present invention. Detailed Implementation

[0042] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided.

[0043] Please see Figures 1-6 As shown, the website article, tag, and category matching automatic update system includes a functional database. The functional database used in this application is the Milvus database, a cloud-native vector database. In this application, Milvus employs a shared storage architecture, with complete separation of storage and computation, and computation nodes supporting horizontal scaling. Specifically, Milvus follows the separation of data flow and control flow, and is divided into four layers: access layer, coordinator service, worker node, and storage layer. Each layer is independent, independently scalable, and disaster recovery-compatible. It can identify storage characteristics, provide ID query features, and provide direct similarity calculation based on the Faiss algorithm.

[0044] Milvus database features fast matrix calculation and embedding data storage capabilities. It solves the problem of synchronizing data and independently calculating similar articles across multiple compute nodes in a site cluster, simplifying the steps of data synchronization and calculation using code, and reducing data storage requirements, since the number of compute nodes in a site cluster is often greater than the number of storage nodes in a Milvus cluster.

[0045] The overall architecture of the model update system in this application consists of three parts: offline updating of new data, online automatic model training, and an online algorithm interface. This application enables cache synchronization, replacing Redis's storage function with Milvus, thus solving the problems of data consistency and cache waste.

[0046] The model update system in this application includes a data processing module, which comprises an offline unit, an online unit, a recommendation query unit, and a training unit.

[0047] Firstly, the offline unit is used to synchronize data periodically in an offline state to ensure eventual data consistency. In this embodiment, incremental data is synchronized every night. That is, even when the data is offline, it is stored in the local database. Then, MySQL data is retrieved periodically. Here, MySQL data refers to data in the relational database management system. In this application, it refers to data in the local database. This data refers to the data that is being updated. Then, the algorithm is called from the algorithm model to process the data and store the processed data in the Milvue database.

[0048] Secondly, there is the online unit, which is used to directly receive article and tag data and store it in the Milvue database. The specific process is as follows: First, the backend directly transmits the data. Then, the algorithm is called from the algorithm model to process the received data and store the processed data in the Milvue database, and also cache it in the local database.

[0049] For the recommendation query unit, the specific process is as follows: First, query data needs to be input. The recommendation query unit then checks if corresponding data exists in the Milvue database. That is, it compares the input data with the data in the Milvue database to determine if the data exists. If it exists, it returns the specific vector mapping data, i.e., the embedding, to directly locate the specific data. If no matching data exists in the Milvue database, it is determined that the data does not exist. In this case, an algorithm is called to calculate the similarity between the input data, the algorithm, and the data in the Milvue database to determine if similar data exists. The similar data is then returned as the result.

[0050] The training unit is used to implement online model updates. First, the training unit receives the data to be trained and trains the model to input the data into the model. Then, it periodically replaces the original model to achieve model training.

[0051] The specific usage process is as follows: First, articles are matched with other articles, then articles are matched with tags, and finally tags are matched with other tags to achieve comprehensive matching and retrieve a sufficient number of similar samples. The enterprise first uses the Roformer model to extract embeddings from all its articles and tag data. The Roformer model is a machine learning model, and embeddings are related vector mapping data. Essentially, key data is extracted and stored in the Milvue database. When new articles or tags need matching, the same method is used to extract embeddings from the new articles, resulting in several discrete embeddings. These embeddings are then matched to matching data with identical or similar embeddings, and the articles to which the matched embeddings belong are retrieved and sent. This process supports batch processing, directly obtaining several related articles.

[0052] In this process, the model and performance were optimized. For classification tasks with limited data and the desire to accelerate training, the Roformer was adopted. It primarily improves training and prediction speed by simplifying the structure, and addresses the issue of poor model performance due to insufficient data by combining prompts, unsupervised pre-training, and few-shot fine-turns, thus achieving a "win-win" situation for both speed and performance.

[0053] When inputting articles, the input content is: article ID + title + article content. The output content is: the distance between the first N article IDs corresponding to each article ID and the matrix space. The specific process is as follows: First, the synchronized data from the receiving site is used to extract the embedding through the roformer model and stored in the Milvue database. This embedding is then compared with the embedding data obtained from the site matching data based on the ID, and the site matching data is compared with the site synchronized data to determine if there is a matching data. If not, the site matching data is used to extract the embedding through the roformer model and stored in the Milvue database. The obtained embedding data is then compared with the embedding data obtained from the site synchronized data to obtain several groups of identical or similar articles. The comparison results and the obtained articles are then sent to the site location.

[0054] The specific data code is as follows.

[0055] First, the code for synchronizing data:

[0056]

[0057]

[0058]

[0059] The following is the matching code:

[0060]

[0061]

[0062] Furthermore, this application includes a timed synchronization module to address the issue of data synchronization and unification when new or modified data is added to the site without using the site's interface for real-time synchronization. The specific process involves setting up a timed task, which then sets a clock to periodically retrieve site data, process the data, extract embedding data using the Roformer model, and store it in the Milvue database. During this process, the site data is periodically stored in the Milvue database, achieving synchronization between the site data and the data within the Milvue database and ensuring data consistency.

[0063] The code process of the timed synchronization module includes

[0064]

[0065]

[0066]

[0067]

[0068] This application then includes a model training and automatic update module to address the issue that, with the continuous addition of new site tag categories, the traditional model approach requires algorithm engineers to manually incrementally train the data, then replace the model, restart the system, and finally complete the update. In this process, the training steps are cumbersome, require manual processing, and the system will be in a paused state. To solve the above problems, this application proposes an improvement.

[0069] Specifically, the interface first receives a request for incremental model training. Then, the offline unit obtains the training time (train_data) and training labels for each sample, starts training, and uploads the trained model to the file server. Subsequently, the online unit downloads the latest trained model and replaces the original model. During this process, it can automatically read incremental data, automatically train, and update the model without restarting the system. This application uses the popular prompt, a method in JavaScript, whose main purpose is to display a prompt dialog box. Therefore, the number of categories can be changed at any time. If a new category needs to be added, the newly added category can be trained directly without training all categories at once, saving training time and workload.

[0070] The specific process is as follows: The offline unit sends a request for incremental training to the site. The site sends the incremental training request to the Roformer model. The Roformer model needs to obtain training data, including incremental data, from the site to train the model and achieve normal model training. If the trained model is ready for use, it is initially stored in a file server. Subsequently, the online unit receives a signal from the site requesting an algorithm update. The online unit downloads the latest model from the file server and replaces the current model with the new one. The replacement process includes: first, obtaining the version number; if the current version number is 0, then using model 0 for prediction; replacing model 0 with the latest model and defining the latest model as 1; switching the system version number to 1 to maintain consistency with the latest model's positioning; during this process, model 0 continues to be used while model 1 replaces model 0 until model 1 is usable. Then, model 0 is paused and stored in the file server. This application ensures that model 0 can be used normally during the replacement process by replacing model 1 with model 0 during normal use, guaranteeing the normal operation of the system. The update is performed during operation, avoiding time delays or system pauses caused by updates, demonstrating a high degree of intelligence.

[0071] Training code includes

[0072]

[0073]

[0074]

[0075] Hot update code includes

[0076]

[0077]

[0078] Furthermore, this application provides two algorithm deployment methods, including public deployment and private deployment. Public deployment allows enterprises to directly use public algorithm interfaces, store website content in partitions, and use a general model, which is more suitable when the amount of data is insufficient.

[0079] However, private deployment allows enterprises to deploy services independently, with no limit on the number of sites or computing power, and can have private algorithm models that can be shared by multiple sites within the enterprise.

[0080] Furthermore, this application focuses on the semantic matching of short text to long text, aiming to help users create content quickly across multiple fields.

[0081] This application employs technologies such as Milvus, Roofomer, and Prompt, combined with an online hot-update model approach. This solves the problem of manual training and restarting at a single site. Model updates are convenient and do not impact business operations. Furthermore, the same model can be adapted to multiple site datasets, resolving the issue of repetitive training across multiple sites. Even with real-time changes and continuous additions to business / classification data, it can still meet the long-term needs of business stakeholders.

[0082] In use, this invention firstly achieves the separation of data flow and control flow by employing a Milvus database. During use, data from any site is stored in and retrieved from the Milvus database. Using a single database can save code, reduce space utilization, and reduce memory waste. Furthermore, when using the same Milvus database, data processing will be consistent and reasonable. When matching data retrievals, calling data from the same database ensures the speed and stability of data processing.

[0083] When matching data, this application includes offline and online units, which can extract and store all updated data of the site location into the Milvus database to achieve data unification. The application also uses a recommendation query unit to match data, including matching content that is the same or highly similar, and returns the matching results. Finally, this application uses a training unit to train the model and update the model according to the current situation.

[0084] In the recommendation query unit, the input articles are first processed. Embeddings of the articles and tags are extracted using the Roformer model and stored in the Milvue database. Then, by comparing the input articles and tags with existing embeddings in the Milvue database, matching them with identical or similar embeddings, and retrieving the articles to which the matched embeddings belong. This process optimizes the model and improves performance, accelerating training. Furthermore, the use of the Roformer model further simplifies the structure, speeding up embedding data retrieval. The combination of unsupervised and supervised training enhances performance, achieving a win-win situation for both speed and effectiveness. Moreover, the use of the Milvue database, Roformer model, and prompt technology solves the problem of multiple manual training and restarts required for a single site. Model updates are convenient and do not impact business operations. The same model can adapt to data sets from multiple sites, resolving the issue of repetitive training across multiple sites. Even with real-time changes and continuous additions to business and classification data, it can meet the long-term needs of business stakeholders.

[0085] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A website article, tag, and category matching model automatic update system, characterized by: It includes a data input module, a recommendation query unit, and a training unit. The data input module includes inputting query data, obtaining query data through the recommendation query unit, processing the query data to obtain keyword data, and matching the keyword data with data in the database obtained by the recommendation query unit to obtain matching data and the corresponding articles. The database used is the Milvus database, which adopts a shared architecture. The Milvus database obtains article information data from the site, processes the article information data from the site, and then stores it. The training unit performs adaptive training on the data processing model in the recommendation query unit. The training content of the training unit includes request incremental training, which is used to enable the data processing model to adapt to the processing of business data when business data increases. The recommendation query unit includes an input data processing unit and a comparison unit. The input data processing unit is used to obtain the query data, process the input data to obtain vector mapping data embedding, and store the vector mapping data embedding in the Milvus database. The comparison unit obtains the vector mapping data embedding and compares it with the data in the Milvus database to obtain matching data and the corresponding articles. The processing steps of the data processing unit include: Step 1: Input article data and tag data; Step 2: The Roformer model obtains the input article data and tag data. After processing the article data and tag data within the Roformer model, the embedding data is obtained. Step 3: Store the obtained embedding data in the Milvus database.

2. The website article, tag, and category matching model automatic update system according to claim 1, characterized in that, The data input module includes an offline unit and an online unit; the offline unit is used to set up a timed data collection unit in an offline state, and by setting the time interval for data collection, the data in the site is processed by an algorithm model, and the data obtained after processing by the algorithm model is stored in the Milvus database. The online unit is used in an online mode where the site backend directly transmits data, processes the data within the site using an algorithm model, and stores the data obtained after algorithm model processing in the Milvus database.

3. The website article, tag, and category matching model automatic update system according to claim 1, characterized in that, It also includes a timed synchronization module, used to correct data that is not synchronized within the site. The synchronization steps of the module include: Step 1: Set a timer to periodically retrieve updated data from the site; Step 2: Extract the embedding data from the updated data within the site using the Roformer model and store it in the Milvue database.

4. The website article, tag, and category matching model automatic update system according to claim 2, characterized in that, The training unit includes a model training unit and an automatic update unit. The model training unit is used to train the modified model by inputting data into the original model and changing the values ​​of the data, thereby obtaining the training result. The training result is then updated into the model through the automatic update unit to modify the model.

5. The website article, tag, and category matching model automatic update system according to claim 4, characterized in that, The working steps of the model training unit include: Step 1: The model training unit receives a request for incremental training of the model, and the offline unit obtains the training time and training labels of the samples. Step 2: The model training unit trains the model according to the training time and training labels of the samples, and uploads the trained model to the file server.

6. The website article, tag, and category matching model automatic update system according to claim 5, characterized in that, The automatic update unit's operating steps include: Step 3: The online unit receives the signal from the site requesting an algorithm update, and the online unit downloads the latest training result model from the file server; Step 4: Retrieve the version number of the original model, define the version number of the latest training result model based on the version number of the original model, and store the original model in the file server.

7. The website article, tag, and category matching model automatic update system according to claim 1, characterized in that, It also includes a deployment module, which comprises a public deployment unit and a private deployment unit. The public deployment unit uses a public algorithm interface, while the private deployment unit uses a separate interface.

Images