Vector database-based government affair service new and old matter standardization fusion processing method

Abstract

The invention relates to the technical field of government affair services, and particularly discloses a government affair service new and old matter standardization fusion processing method based on a vector database, which is used for solving the problem that the potential relevance of new and old government affair service matters is difficult to comprehensively identify and process by the existing rule matching and manual adjustment method. The method solves the problems that a large amount of manual adjustment is caused, and manpower consumption and error risks are increased due to the fact that semantic similarity cannot be effectively captured depending on keyword matching, and comprises the steps of data preprocessing, data enhancement and self-training, vector representation generation, vector database construction, similarity calculation and matching, dynamic weight adjustment and item priority adjustment. Through vector representation and similarity calculation, the problem of deep semantic association identification of new and old items is solved, the matching accuracy and efficiency are improved, through dynamic weight adjustment and priority ranking, data standardization is optimized, manual intervention is reduced, and the automatic processing capacity and overall efficiency of government affair services are improved.

Description

Technical Field

[0001] The present invention relates to the field of government service technology, and more specifically, to a method for standardized fusion processing of new and old government service items based on a vector database. Background Art

[0002] In digital government services, the standardized processing of various government service items is a key step to achieve efficient government operation and optimize the public service experience. The standardization of government service items not only includes the standardized processing of existing items, but also needs to face the problem of integrating the new and old items. With the development of society, government service items are constantly updated, adjusted and expanded, resulting in the increasingly prominent problem of inconsistent standards of new and old items, which not only affects the efficiency of government services, but also brings challenges to data sharing and cross-departmental collaboration. Vector database is a new type of data storage and query technology that can efficiently process high-dimensional data and complex queries. It performs well in the fields of natural language processing, image retrieval and recommendation systems. It can represent the characteristics of data through vectors, realize efficient similarity calculation and corresponding data retrieval. The standardized fusion processing method of new and old government service items based on vector database, using vector representation and similarity calculation, can effectively solve the problem of inconsistent standards of new and old items and realize data standardization and consistency processing. At present, the standardized processing of government service items includes item classification, rule matching and manual adjustment. First, the new and old items are classified and preliminarily matched according to predefined rules, and then keywords and pattern recognition technology are used to classify the new and old items. New and old items are matched to try to find similar or related items. For problems that cannot be solved by rule matching, professionals will perform manual adjustments and verifications to ensure the consistency and accuracy of the items. However, in the existing rule matching and manual adjustment methods, the potential correlation between new and old items is difficult to be fully identified and processed. The description methods of new and old items are different. Simply relying on keyword matching is often difficult to fully cover all potential related items. Existing methods are difficult to effectively identify the similarities between new and old items at the semantic level, and cannot fully capture the deep semantic correlation between new and old items. The results of rule matching often require a lot of manual adjustments, which not only consumes human resources, but also easily introduces human errors, affecting the standardization and consistency of items. Summary of the invention

[0003] In order to overcome the above-mentioned defects of the prior art, the present invention provides a standardized fusion processing method for new and old items of government service based on a vector database. Through vector representation and similarity calculation, the difficult problem of deep semantic association identification of new and old items is solved, and the matching accuracy and efficiency are improved. Through dynamic weight adjustment and priority sorting, data standardization is optimized, manual intervention is reduced, and the automated processing capability and overall efficiency of government services are improved.

[0004] To achieve the above object, the present invention provides the following technical solutions:

[0005] A standardized fusion processing method for new and old government service items based on a vector database includes:

[0006] Step 1: Data preprocessing: Collect basic information of new and old government service items, clean the collected text data, use fluency improvement learning strategies, conduct multiple rounds of error correction on the text, and perform word segmentation and part-of-speech tagging on the text;

[0007] Step 2: Data enhancement and self-training: Use synonym replacement, random insertion, random deletion, and random exchange techniques to enhance the data of existing government service item descriptions to generate diversified text data. Use the trained model to automatically annotate the unlabeled government service item descriptions to generate pseudo-labeled data. Combine the pseudo-labeled data with the original annotated data for joint training.

[0008] Step 3: Vector representation generation: Select a suitable pre-trained word embedding model, use the collected government service item description text to fine-tune the model, and use the selected word embedding model to convert the text into a vector representation;

[0009] Step 4: vector database construction: store the generated text vectors into the vector database and build an index for the stored vectors;

[0010] Step 5: Similarity calculation and matching: Use the optimized fusion similarity calculation formula to measure the similarity between the new and old government service items, search for similar items in the vector database, and find the old items that are most similar to the new items;

[0011] Step 6, dynamic weight adjustment: dynamically adjust the weight coefficient in phase velocity calculation according to the length and importance of the description text;

[0012] Step 7: Adjust the priority of items: Adjust the priority of items based on the similarity and importance of the items.

[0013] As a further solution of the present invention, in step 1, the specific process of using the fluency improvement learning strategy to perform multiple rounds of error correction on the text is as follows:

[0014] Step 11, first round of improvement: input the original text into the pre-trained BERT model to generate improved text and optimize the syntactic structure;

[0015] Step 12, second round of improvement: input the text after the first round of improvement into the language model again to further improve the fluency and coherence of the text;

[0016] Step 13, multiple rounds of iterations: Repeat the above process and evaluate the text improvement effect until the improvement reaches the expected quality standard.

[0017] As a further solution of the present invention, in step 13, the improvement effect of the text is obtained by the weighted sum of the inverse of the language model Perplexity value of the improved text, the sentence fluency score of the improved text, and the text consistency score of the improved text, and then subtracting the regularization term. The improvement effect evaluation formula of the text is:

[0018]

[0019] Where: E is the text improvement effect score, k PPL is the language model Perplexity value of the improved text, which is calculated by the pre-trained language model, k FS is the fluency score of the improved text, calculated using the fluency scoring tool, k CS is the text consistency score, calculated by the text consistency scoring tool, L 2 is the regularization term, where L is the text length, α, β, γ, δ are weight coefficients, which are set through experience and verification data.

[0020] As a further solution of the present invention, in step 5, the optimized fusion similarity calculation formula measures the similarity between the new and old government service items by fusion similarity and attention mechanism. The optimized fusion similarity calculation formula is:

[0021]

[0022] Where: i, j are the indexes of government service items, v i 、v j are the vector representations of government service items i and j respectively. Attention(·) is the attention mechanism function, which performs a weighted combination of different similarity metrics. ε is the attention weight parameter used to adjust the Gaussian kernel similarity, which is obtained through training and tuning. σ is the bandwidth parameter of the Gaussian kernel function, which is obtained through verification data tuning. i ,t j are the description texts of government service items i and j respectively, θ is the attention weight parameter used to adjust the word overlap similarity, obtained through training and tuning, Word Overlap(t i ,t j ) is the word overlap measure, which is obtained by calculating the vocabulary overlap in the description text of government service items i and j.

[0023] As a further solution of the present invention, in the optimized fusion similarity calculation formula of step 5, the word overlap is equal to the ratio of the number of words that appear together in the description texts of government service items i and j to the total number of words in the description texts of government service items i and j. The formula for word overlap is:

[0024]

[0025] In the formula: Words(t i )、Words(t j ) are the vocabulary sets in the description text of government service items i and j, respectively, |Words(t i )∩Words(t j )| is the number of words that appear in the description texts of government service items i and j, |Words(t i )∪Words(t j )| is the total number of words in the description text of government service items i and j.

[0026] As a further solution of the present invention, in step 5, the specific process of using the optimized fusion similarity to measure the similarity between the new and old government service items includes:

[0027] Step 501, data preparation: obtain the description text t of government service items i and j i ,t j , use the pre-trained word embedding model to convert the description text into a vector representation v i 、v j ;

[0028] Step 502, calculate Gaussian kernel similarity: calculate vector v i 、v j The Euclidean distance between i -v j ‖, calculate Gaussian kernel similarity

[0029] Step 503, calculate word overlap similarity: i ,t j Perform word segmentation to obtain the vocabulary set Words(t i )、Words(t j ), calculate the word overlap measure Word Overlap(t i ,t j );

[0030] Step 504, attention mechanism weighting: Calculate the attention weighted value of Gaussian kernel similarity Attention weighted value of word overlap similarity Attention(θ,WordOverlap(t i ,t j ));

[0031] Step 505, fusion similarity calculation: Combine the Gaussian kernel similarity and word overlap similarity to calculate the final fusion similarity H FS (v i ,v j ).

[0032] As a further solution of the present invention, in step 5, the specific implementation process of searching similar items in the vector database to find the old items that are most similar to the new items includes:

[0033] Step 511, new item vectorization: input the description text of the new government service item new , use the pre-trained word embedding model to embed t new Convert to vector representation v new , v new =BERT(t new );

[0034] Step 512, vectorization of old items: input the description text of all old government service items stored in the vector database old,n , where n is the index of the description text of the old government service items, n = 1, 2, ..., N, is a natural number, and N is the total number of description texts of the old government service items stored in the vector database. These old item description texts have been vectorized in the vector database to obtain the vector representation v of the old items. old,n ;

[0035] Step 513, fusion similarity calculation: input new event vector v new And each old event vector v old,n , use the optimized fusion similarity calculation formula to calculate the similarity H between new and old events FS (v new ,v old,n );

[0036] Step 514, similar item search: input the calculated similarity H between the new and old items FS (v new ,v old,n ), in the vector database, based on the similarity H FS (v new ,v old,n ) from high to low, and use the approximate nearest neighbor search retrieval algorithm to retrieve the most similar old items;

[0037] Step 515, determination of the most similar old items: find the corresponding old items according to the index with the highest similarity, match the most similar old items with the new items, and further process them.

[0038] As a further solution of the present invention, in step 513, the similarity H between the new and old items is calculated using the optimized fusion similarity calculation formula. FS (v new ,v old,n ) is in the form of the specific application formula:

[0039]

[0040] Where: is the attention weighted value of the Gaussian kernel function measuring the vector similarity of new and old items, Attention(θ, Word Overlap(t new ,t old,n )) is the attention weighted value of the vocabulary overlap between new and old items.

[0041] As a further solution of the present invention, in step six, the process of dynamically adjusting the weight coefficient in the phase velocity calculation according to the length and importance of the description text includes:

[0042] Step 61, text length and importance calculation: Enter the description text of the government service item i , calculate the length of the text L i , L i To describe the text t i The number of words contained in , enter the description text of government service items t i , calculate the importance of the text based on the frequency of keyword occurrence and the priority of the items in the historical items. i ;

[0043] Step 62, dynamic weight adjustment: according to the length and importance of the text, the weight coefficient in the similarity calculation is adjusted through the dynamic weight adjustment formula. where α i , β i is the dynamically adjusted weight coefficient, λ1, λ2, λ3, and λ4 are model parameters used to adjust the mass coefficient, which are obtained by tuning the validation data, and m is the traversal index of the description text of the government service items. i , β i Normalize and obtain the adjusted weight α′ i , β′ i ;

[0044] Step 63, applying dynamic weights: in the fusion similarity calculation formula, use the normalized adjusted weight coefficient α′ i , β′ i Weighting of Gaussian kernel similarity and vocabulary overlap;

[0045] Step 64, adjusted similarity calculation and matching: use the adjusted weight coefficient to calculate the fusion similarity between the new and old government service items, and use the updated fusion similarity formula to retrieve the old items most similar to the new items in the vector database.

[0046] As a further solution of the present invention, in step seven, the specific implementation process of adjusting the priority of the matters in combination with the similarity and the importance of the matters is as follows:

[0047] Step 71, obtain similarity calculation result: input the calculated similarity H between the new and old items FS (v new ,v old,n ), obtain the similarity calculation results of all old items and new items, and obtain the numerical set of similarity calculation results of old government service items and new government service items, which set is {H FS (v new ,v old,1 ),H FS (v new ,v old,2 ),…,H FS (v new ,v old,N )};

[0048] Step 72, obtain the importance score of government service items: enter the description text of the old government service items old,n , use the method in step 61 to calculate the importance of government service items I n ;

[0049] Step 73, calculate the priority score: Combine the similarity and the importance of the government service items, and use the formula Get the priority score of each old item, where P n Score the priority of old task n, η, The weight coefficient used to adjust the similarity and importance impact is set according to application requirements;

[0050] Step 74, priority sorting: sort all old items from high to low according to the priority scores to obtain the priority sorting result, which is used to guide resource allocation and task scheduling;

[0051] Step 75, priority adjustment and decision support: According to business needs and external conditions, further manual adjustments are made to the sorting results, and processing suggestions are automatically generated based on the priority sorting results, and finally an adjusted priority list of government service items is output.

[0052] Compared with the prior art, the technical effect of the present invention to solve the technical problem is as follows: the present invention solves the problem in the prior art that it is difficult to comprehensively identify the deep semantic associations between new and old matters by introducing vector representation and similarity calculation. Compared with traditional rule matching and manual adjustment methods, this method can effectively capture the potential associations between matters under different description methods, reduce the dependence on keyword matching, and thus improve the matching accuracy and efficiency. Through dynamic weight adjustment and matter priority adjustment, the similarity calculation results can be flexibly adjusted according to the text features and importance of the matters, further optimizing the data standardization and consistency processing process, significantly reducing the possibility of manual intervention and human errors, and improving the automated processing capabilities and overall efficiency of government services. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the present invention to clearly and completely describe the technical solutions in the present invention. Obviously, the described technical solutions are only part of the present invention but not all of it. Based on the content of the present invention, all other technical solutions obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0055] Compared with the prior art, the standardized fusion processing method for new and old government service items based on vector database proposed in the present invention solves the problem that it is difficult to fully identify the deep semantic associations between new and old items in the prior art by introducing vector representation and similarity calculation. Compared with the traditional rule matching and manual adjustment methods, this method can effectively capture the potential associations between items under different descriptions, reduce the reliance on keyword matching, and thus improve the accuracy and efficiency of matching. Through dynamic weight adjustment and item priority adjustment, the similarity calculation results can be flexibly adjusted according to the text features and importance of the items, further optimizing the data standardization and consistency processing process, significantly reducing the possibility of manual intervention and human errors, and improving the automated processing capabilities and overall efficiency of government services. Figure 1 As shown, the method proposed by the present invention includes:

[0056] Step 1: Data preprocessing: Collect basic information of new and old government service items, clean the collected text data, use fluency improvement learning strategies, conduct multiple rounds of error correction on the text, and perform word segmentation and part-of-speech tagging on the text;

[0057] Step 2: Data enhancement and self-training: Use synonym replacement, random insertion, random deletion, and random exchange techniques to enhance the data of existing government service item descriptions to generate diversified text data. Use the trained model to automatically annotate the unlabeled government service item descriptions to generate pseudo-labeled data. Combine the pseudo-labeled data with the original annotated data for joint training.

[0058] Step 3: Vector representation generation: Select a suitable pre-trained word embedding model, use the collected government service item description text to fine-tune the model, and use the selected word embedding model to convert the text into a vector representation;

[0059] Step 4: vector database construction: store the generated text vectors into the vector database and build an index for the stored vectors;

[0060] Step 5: Similarity calculation and matching: Use the optimized fusion similarity calculation formula to measure the similarity between the new and old government service items, search for similar items in the vector database, and find the old items that are most similar to the new items;

[0061] Step 6, dynamic weight adjustment: dynamically adjust the weight coefficient in phase velocity calculation according to the length and importance of the description text;

[0062] Step 7: Adjust the priority of items: Adjust the priority of items based on the similarity and importance of the items.

[0063] It should be noted that in step 1, the specific process of using the fluency improvement learning strategy to perform multiple rounds of error correction on the text is as follows:

[0064] Step 11, first round of improvement: input the original text into the pre-trained BERT model to generate improved text and optimize the syntactic structure;

[0065] Step 12, second round of improvement: input the text after the first round of improvement into the language model again to further improve the fluency and coherence of the text;

[0066] Step 13, multiple rounds of iterations: Repeat the above process and evaluate the text improvement effect until the improvement reaches the expected quality standard.

[0067] This multi-round error correction process gradually optimizes the syntactic structure and fluency of the text, ensuring that the description text is more standardized and consistent, thereby improving the accuracy of subsequent vector representation and similarity calculation, reducing noise and ambiguity, making the matching of new and old government affairs more accurate, and ultimately improving the overall efficiency of data standardization and fusion processing.

[0068] It should be noted that in step 13, the improvement effect of the text is obtained by the weighted sum of the inverse of the language model Perplexity value of the improved text, the sentence fluency score of the improved text, and the text consistency score of the improved text, minus the regularization term. The improvement effect evaluation formula of the text is:

[0069]

[0070] Where: E is the text improvement effect score, k PPL is the language model Perplexity value of the improved text, which is calculated by the pre-trained language model, k FS is the fluency score of the improved text, calculated using the fluency scoring tool, k CS is the text consistency score, calculated by the text consistency scoring tool, L 2 is the regularization term, where L is the text length, α, β, γ, δ are weight coefficients, which are set through experience and verification data.

[0071] This formula setting effectively evaluates the improvement effect of the text by comprehensively considering multiple quality indicators of the text, thereby solving the problem that it is difficult to quantify the quality of improvement of the text during the standardization process. Specifically, the Perplexity value of the language model in the formula reflects the grammatical rationality of the text, and its reciprocal ensures that the better the grammar, the higher the score; the fluency score measures the natural fluency of the text; and the consistency score evaluates the coherence and logic of the text. These indicators are combined through a weighted sum, so that the evaluation can comprehensively cover all key quality dimensions of the text. In addition, the introduction of a regularization term (based on the square of the text length) is used to prevent overfitting problems caused by too long texts, ensuring that the evaluation is more objective and fair. By reasonably setting the weight coefficient, the formula can flexibly adapt to the needs of different application scenarios, and ultimately achieve accurate measurement and optimization of text quality, thereby improving the matching accuracy of new and old government service items, reducing manual intervention, and improving overall data processing efficiency.

[0072] It should be noted that in step 5, the optimized fusion similarity calculation formula measures the similarity between the new and old government service items by fusion similarity and attention mechanism. The optimized fusion similarity calculation formula is:

[0073]

[0074] Where: i, j are the indexes of government service items, v i 、v jare the vector representations of government service items i and j respectively. Attention(·) is the attention mechanism function, which performs a weighted combination of different similarity metrics. ε is the attention weight parameter used to adjust the Gaussian kernel similarity, which is obtained through training and tuning. σ is the bandwidth parameter of the Gaussian kernel function, which is obtained through verification data tuning. i ,t j are the description texts of government service items i and j respectively, θ is the attention weight parameter used to adjust the word overlap similarity, obtained through training and tuning, Word Overlap(t i ,t j ) is the word overlap measure, which is obtained by calculating the vocabulary overlap in the description text of government service items i and j.

[0075] This formula is set up by comprehensively using vector similarity and word overlap metrics to accurately measure the similarity between new and old government service items, thereby effectively solving the problem of difficulty in capturing deep semantic associations in existing technologies. Specifically, the Gaussian kernel function is used in the formula to calculate vector similarity, which can capture subtle differences between government items in high-dimensional space, and the introduction of the attention mechanism (Attention) adjusts the influence of different similarity metrics by weighting to ensure that key information is given greater weight in similarity calculation. The attention weight parameters of Gaussian kernel similarity and word overlap similarity are obtained through training and tuning, so that the model can dynamically adapt to different types of text data according to specific application scenarios, thereby improving the accuracy of similarity calculation. The word overlap metric further enhances the ability to capture semantic overlap by calculating common words in the description text. This optimization formula can more comprehensively reflect the potential association between new and old items, reduce the reliance on single keyword matching, improve the accuracy and efficiency of matching, and ultimately provide strong support for the standardized processing of government service items.

[0076] It should be noted that in the optimized fusion similarity calculation formula in step 5, the word overlap is equal to the ratio of the number of words that appear together in the description texts of government service items i and j to the total number of words in the description texts of government service items i and j. The formula for word overlap is:

[0077]

[0078] In the formula: Words(t i )、Words(t j ) are the vocabulary sets in the description text of government service items i and j, respectively, |Words(t i )∩Words(t j )| is the number of words that appear in the description texts of government service items i and i, |Words(t i)∪Words(t j )| is the total number of words in the description text of government service items i and j.

[0079] This formula setting effectively solves the problem that it is difficult to capture the semantic similarity between new and old items in the prior art by quantifying the degree of vocabulary overlap between the description texts of government service items. Specifically, the word overlap formula directly reflects the similarity between the two at the vocabulary level by calculating the proportion of common vocabulary in the description texts of government service items i and j. The formula calculates the ratio of the number of common words (i.e., intersection) to the total vocabulary (i.e., union), so that the calculation of similarity not only considers the common vocabulary between the two texts, but also considers the overall vocabulary of the text, thereby avoiding the deviation caused by the difference in text length. In this way, the formula can more accurately measure the overlap between the description texts of new and old items, especially when the description methods are different but the substantive content is similar, the word overlap formula can capture the potential correlation between the two. In addition, the word overlap calculation is combined with the vector similarity, so that the similarity calculation not only considers the semantic information of the text, but also reflects the vocabulary structure of the text, enhancing the comprehensiveness and accuracy of the similarity evaluation. Ultimately, this setting improves the accuracy of matching new and old government service items, reduces the necessity of manual intervention, and significantly improves the efficiency and effectiveness of standardized processing of government service items.

[0080] It should be noted that in step 5, the specific process of using the optimized fusion similarity to measure the similarity between the new and old government service items includes:

[0081] Step 501, data preparation: obtain the description text t of government service items i and j i ,t j , use the pre-trained word embedding model to convert the description text into a vector representation v i 、v j ;

[0082] Step 502, calculate Gaussian kernel similarity: calculate vector v i 、v j The Euclidean distance between i -v j ‖, calculate Gaussian kernel similarity

[0083] Step 503, calculate word overlap similarity: i ,t j Perform word segmentation to obtain the vocabulary set Words(t i )、Words(t j ), calculate the word overlap measure Word Overlap(t i ,t j);

[0084] Step 504, attention mechanism weighting: Calculate the attention weighted value of Gaussian kernel similarity Attention weighted value of word overlap similarity Attention(θ,WordOverlap(t i ,t j ));

[0085] Step 505, fusion similarity calculation: Combine the Gaussian kernel similarity and word overlap similarity to calculate the final fusion similarity H FS (v i ,v j ).

[0086] This process setting effectively solves the problem of deep semantic association identification in the standardized processing of new and old government service items by systematically combining semantic similarity and lexical similarity. Specifically, first, through the data preparation step, the description text is converted into a vector representation using a pre-trained word embedding model to ensure that the semantic information of the text is accurately expressed in the high-dimensional vector space. Then, by calculating the Gaussian kernel similarity, the Euclidean distance between the two item description vectors is measured to capture the subtle differences in the text at the semantic level. At the same time, the calculation of word overlap similarity is through word segmentation, analyzing the proportion of words that appear together in the text, providing a direct lexical level similarity measure. This step can effectively reflect the surface similarity of the text, especially when the description method is different but the content is similar, which enhances the accuracy of the similarity calculation. Subsequently, the attention mechanism weighting step further weights the two similarities, so that the final fusion similarity can dynamically adjust the weights of Gaussian kernel similarity and word overlap similarity in the overall similarity according to the actual situation. This weighting mechanism ensures that important information has a greater influence in the similarity calculation, thereby improving the accuracy of the match. Through this fusion similarity calculation method, it is possible to more comprehensively and accurately identify potential connections between new and old matters, reduce reliance on manual intervention, improve the efficiency and accuracy of standardized processing, and ultimately provide technical support for the efficient operation of government services and cross-departmental collaboration.

[0087] It should be noted that in step 5, the specific implementation process of searching similar items in the vector database to find the old items that are most similar to the new items includes:

[0088] Step 511, new item vectorization: input the description text of the new government service item new , use the pre-trained word embedding model to embed t new Convert to vector representation v new , v new =BERT(t new );

[0089] Step 512, vectorization of old items: input the description text of all old government service items stored in the vector database old,n , where n is the index of the description text of the old government service items, n = 1, 2, ..., N, is a natural number, and N is the total number of description texts of the old government service items stored in the vector database. These old item description texts have been vectorized in the vector database to obtain the vector representation v of the old items. old,n ;

[0090] Step 513, fusion similarity calculation: input new event vector v new And each old event vector v old,n , use the optimized fusion similarity calculation formula to calculate the similarity H between new and old events FS (v new ,v old,n );

[0091] Step 514, similar item search: input the calculated similarity H between the new and old items FS (v new ,v old,n ), in the vector database, based on the similarity H FS (v new ,v old,n ) from high to low, and use the approximate nearest neighbor search retrieval algorithm to retrieve the most similar old items;

[0092] Step 515, determination of the most similar old items: find the corresponding old items according to the index with the highest similarity, match the most similar old items with the new items, and further process them.

[0093] This setting combines complex similarity calculations with efficient vector database retrieval through a systematic process, effectively solving the accuracy and efficiency problems in matching new and old government service items. First, in the new item vectorization step, the pre-trained BERT model is used to convert the description text of the new government service item into a high-dimensional vector representation v new , ensuring that the semantic information of the text is accurately captured. Next, in the old item vectorization step, the description text of all old government service items has been pre-converted into vectors and stored in the vector database, ensuring that all old items in the database have vector representation v old,n In the fusion similarity calculation step, the optimized fusion similarity calculation formula H is used. FS (v new ,v old,n), combining the semantic similarity and lexical similarity between new and old items, thus obtaining a more comprehensive similarity measurement result. This fusion similarity calculation method can capture the deep-level associations in the description texts of new and old items and improve the matching accuracy. Then, in the similar item retrieval step, based on the calculated similarity H FS (v new ,v old,n ), sort and retrieve old items using the approximate nearest neighbor search algorithm to ensure that the most similar items to the new items are quickly found in the massive amount of old items. Finally, the most similar old items are determined by the index with the highest similarity, and the matching results are further processed. This process not only improves the accuracy and speed of matching, but also reduces the need for manual intervention, significantly improves the efficiency of standardized processing of government service items, and provides strong technical support for the efficient operation of digital government services.

[0094] It should be noted that in step 513, the optimized fusion similarity calculation formula is used to calculate the similarity H between the new and old items. FS (v new ,v old,n ) is in the form of the specific application formula:

[0095]

[0096] Where: is the attention weighted value of the Gaussian kernel function measuring the vector similarity of new and old items, Attention(θ, Word Overlap(t new ,t old,n )) is the attention weighted value of the vocabulary overlap between new and old items.

[0097] This formula combines the vector similarity measured by the Gaussian kernel function with the word overlap and introduces the attention mechanism for weighting, which significantly improves the accuracy and efficiency of matching new and old government service items. Specifically, the Gaussian kernel function similarity captures the semantic similarity of text in high-dimensional space by measuring the Euclidean distance between the vectors of new and old items, while the attention mechanism This enables the model to dynamically adjust the influence of Gaussian kernel similarity according to the actual situation, ensuring that semantic information is appropriately weighted in similarity calculations. At the same time, word overlap directly reflects the surface semantic similarity of texts by calculating the proportion of common words in the descriptions of new and old events. The introduced attention mechanism Attention(θ,Word Overlap(t new ,t old,n))Further weights the word overlap so that in the similarity calculation, important word overlap information can have a greater impact on the final similarity value. This setting effectively solves the problem of insufficient semantic level understanding caused by relying solely on keyword or rule matching in traditional matching methods. By combining Gaussian kernel similarity and word overlap, and using the attention mechanism to flexibly adjust their contributions in similarity calculation, the formula can more comprehensively and accurately reflect the potential correlation between new and old items, reduce the probability of matching errors, and improve the matching accuracy, ultimately providing more reliable and efficient technical support for the standardized processing of government service items.

[0098] It should be noted that in step 6, the process of dynamically adjusting the weight coefficient in the phase velocity calculation according to the length and importance of the description text includes:

[0099] Step 61, text length and importance calculation: Enter the description text of the government service item i , calculate the length of the text L i , L i To describe the text t i The number of words contained in , enter the description text of government service items t i , calculate the importance of the text based on the frequency of keyword occurrence and the priority of the items in the historical items. i ;

[0100] Step 62, dynamic weight adjustment: according to the length and importance of the text, the weight coefficient in the similarity calculation is adjusted through the dynamic weight adjustment formula. where α i , β i is the dynamically adjusted weight coefficient, λ1, λ2, λ3, and λ4 are model parameters used to adjust the mass coefficient, which are obtained by tuning the validation data, and m is the traversal index of the description text of the government service items. i , β i Normalize and obtain the adjusted weight α′ i , β′ i ,in

[0101] Step 63, applying dynamic weights: in the fusion similarity calculation formula, use the normalized adjusted weight coefficient α′ i , β′ i Weighting of Gaussian kernel similarity and vocabulary overlap;

[0102] Step 64, adjusted similarity calculation and matching: use the adjusted weight coefficient to calculate the fusion similarity between the new and old government service items, and use the updated fusion similarity formula to retrieve the old items most similar to the new items in the vector database.

[0103] In step 61, the importance of the text is obtained by analyzing the frequency of occurrence of keywords in the descriptive text and the processing priority of similar matters in the historical data. The analysis of the frequency of keyword occurrence adopts the TF-IDF (term frequency-inverse document frequency) method to calculate the importance score of each word. A keyword with a higher TF-IDF indicates that the word is more important in the current text and more unique in the entire corpus. The formula is:

[0104]

[0105] Where: i is the index of the vocabulary, TF(ω i ,t i ) is the vocabulary ω i In the text i The word frequency in , Z is the total number of texts in the corpus, DF(ω i ) is the word containing the word ω i The amount of text.

[0106] For the analysis of the priority of historical matters, we first analyze the priority of similar matters in the past based on historical data, and give higher weights to matters with higher priorities. The priority is determined based on the completion time, influence, and resource allocation factors of the matters. The final text importance is the weighted sum of the keyword importance and the priority of historical matters, and the weights of the keyword importance and the priority of historical matters are adjusted according to actual needs. The formula for text importance is:

[0107] I i =θ×TF-IDF(ω i )+τ×K his

[0108] Where: K his is the priority of historical matters, θ and τ are the weights of keyword importance and historical matters priority, respectively.

[0109] This setting solves the problem of ignoring the difference in text length and importance in similarity calculation by dynamically adjusting the weight coefficient. First, by calculating the text length L i and text importance I i, accurately measuring the text characteristics of each government service item. Using these characteristics, the dynamic weight adjustment formula assigns appropriate weights to different metrics in the similarity calculation based on the length and importance of the text, ensuring that longer and more important texts are fully considered in the similarity calculation. The normalized weight coefficient α′ i , β′ i The influence of Gaussian kernel similarity and vocabulary overlap in the fusion similarity calculation is further balanced, making the final similarity more accurate and representative. This method not only improves the accuracy of matching new and old items, reduces the deviation caused by differences in text length or importance, but also optimizes the efficiency and reliability of the matching process. Through this dynamic weight adjustment, the system can handle government service items of different types and characteristics more intelligently, significantly improving the overall effect of standardized processing.

[0110] It should be noted that in step seven, the specific implementation process of adjusting the priority of the items based on the similarity and the importance of the items is as follows:

[0111] Step 71, obtain similarity calculation result: input the calculated similarity H between the new and old items FS (v new ,v old,n ), obtain the similarity calculation results of all old items and new items, and obtain the numerical set of similarity calculation results of old government service items and new government service items, which set is {H FS (v new ,v old,1 ),H FS (v new ,v old,2 ),…,H FS (v new ,v old,N )};

[0112] Step 72, obtain the importance score of government service items: enter the description text of the old government service items old,n , use the method in step 61 to calculate the importance of government service items I n ;

[0113] Step 73, calculate the priority score: Combine the similarity and the importance of the government service items, and use the formula Get the priority score of each old item, where P n Score the priority of old task n, η, The weight coefficient used to adjust the similarity and importance impact is set according to application requirements;

[0114] Step 74, priority sorting: sort all old items from high to low according to the priority scores to obtain the priority sorting result, which is used to guide resource allocation and task scheduling;

[0115] Step 75, priority adjustment and decision support: According to business needs and external conditions, further manual adjustments are made to the sorting results, and processing suggestions are automatically generated based on the priority sorting results, and finally an adjusted priority list of government service items is output.

[0116] By combining the similarity H between new and old events FS (v new ,v old,n ) and the importance score of the matter n ,formula Generate a composite priority score P for each old item n . This approach ensures that not only the historical importance of the matter is taken into account, but also the relevance to the current new matter, making the scoring more comprehensive and accurate. Sorting according to the priority score effectively guides resource allocation and task scheduling, so that resources can be used to deal with the most important and relevant matters first. This sorting result directly improves the response speed and processing efficiency of government services and optimizes the use of government resources. In the priority adjustment and decision support link, it is allowed to manually adjust the sorting results according to business needs and external conditions, combined with automatically generated processing suggestions, to ensure that the system has sufficient flexibility in dealing with actual scenarios. This setting not only improves the accuracy of automated decision-making, but also ensures that changes can be responded to quickly in special circumstances. Through automated priority scoring and sorting, the necessity of manual intervention is reduced and the risk of human error is reduced. The final output priority list is based on a scientific and reasonable scoring mechanism and practical considerations of business needs, which improves the efficiency and accuracy of overall decision-making.

[0117] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0118] Finally: The above description is only the technical solution of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A standardized fusion processing method for new and old government service items based on a vector database, characterized in that: include: Step 1: Data preprocessing: Collect basic information of new and old government service items, clean the collected text data, use fluency improvement learning strategies, conduct multiple rounds of error correction on the text, and perform word segmentation and part-of-speech tagging on the text; Step 2: Data enhancement and self-training: Use synonym replacement, random insertion, random deletion, and random exchange techniques to enhance the data of existing government service item descriptions to generate diversified text data. Use the trained model to automatically annotate the unlabeled government service item descriptions to generate pseudo-labeled data. Combine the pseudo-labeled data with the original annotated data for joint training. Step 3: Vector representation generation: Select a suitable pre-trained word embedding model, use the collected government service item description text to fine-tune the model, and use the selected word embedding model to convert the text into a vector representation; Step 4: vector database construction: store the generated text vectors into the vector database and build an index for the stored vectors; Step 5: Similarity calculation and matching: Use the optimized fusion similarity calculation formula to measure the similarity between the new and old government service items, search for similar items in the vector database, and find the old items that are most similar to the new items; Step 6, dynamic weight adjustment: dynamically adjust the weight coefficient in phase velocity calculation according to the length and importance of the description text; Step 7: Adjust the priority of items: Adjust the priority of items based on the similarity and importance of the items.

2. The standardized fusion processing method for new and old items of government services based on vector database according to claim 1 is characterized in that: In step 1, the specific process of using the fluency improvement learning strategy to conduct multiple rounds of error correction on the text is as follows: Step 11, first round of improvement: input the original text into the pre-trained BERT model to generate improved text and optimize the syntactic structure; Step 12, second round of improvement: input the text after the first round of improvement into the language model again to further improve the fluency and coherence of the text; Step 13, multiple rounds of iterations: Repeat the above steps and evaluate the text improvement effect until the improvement reaches the expected quality standard.

3. The standardized fusion processing method for new and old items of government services based on vector database according to claim 2 is characterized in that: In step 13, the improvement effect of the text is obtained by the weighted sum of the inverse of the language model Perplexity value of the improved text, the sentence fluency score of the improved text, and the text consistency score of the improved text, minus the regularization term. The improvement effect evaluation formula of the text is: Where: E is the text improvement effect score, k PPL is the language model Perplexity value of the improved text, which is calculated by the pre-trained language model, k FS is the fluency score of the improved text, calculated using the fluency scoring tool, k CS is the text consistency score, calculated by the text consistency scoring tool, L 2 is the regularization term, where L is the text length, α, β, γ, δ are weight coefficients, which are set through experience and verification data.

4. The standardized fusion processing method for new and old items of government services based on vector database according to claim 1 is characterized in that: In step five, the optimized fusion similarity calculation formula measures the similarity between the new and old government service items by fusion similarity and attention mechanism. The optimized fusion similarity calculation formula is: Where: i, j are the indexes of government service items, v i 、v j are the vector representations of government service items i and j respectively. Attention(·) is the attention mechanism function, which performs a weighted combination of different similarity metrics. ε is the attention weight parameter used to adjust the Gaussian kernel similarity, which is obtained through training and tuning. σ is the bandwidth parameter of the Gaussian kernel function, which is obtained through verification data tuning. i ,t j are the description texts of government service items i and j respectively, θ is the attention weight parameter used to adjust the word overlap similarity, obtained through training and tuning, Word Overlap(t i ,t j ) is the word overlap measure, which is obtained by calculating the vocabulary overlap in the description text of government service items i and j.

5. The standardized fusion processing method for new and old items of government services based on vector database according to claim 4 is characterized in that: In the optimized fusion similarity calculation formula of step 5, the word overlap is equal to the ratio of the number of words that appear together in the description texts of government service items i and j to the total number of words in the description texts of government service items i and j. The formula for word overlap is: In the formula: Words(t i )、Words(t j ) are the vocabulary sets in the description text of government service items i and j, respectively, |Words(t i )∩Words(t j )| is the number of words that appear in the description texts of government service items i and j, |Words(t i )∪Words(t j )| is the total number of words in the description text of government service items i and j.

6. The standardized fusion processing method for new and old items of government services based on vector database according to claim 5 is characterized in that: In step 5, the specific process of using the optimized fusion similarity to measure the similarity between the new and old government service items includes: Step 501, data preparation: obtain the description text t of government service items i and j i ,t j , use the pre-trained word embedding model to convert the description text into a vector representation v i 、v j ; Step 502, calculate Gaussian kernel similarity: calculate vector v i 、v j The Euclidean distance between i -v j ‖, calculate Gaussian kernel similarity Step 503, calculate word overlap similarity: i ,t j Perform word segmentation to obtain the vocabulary set Words(t i )、Words(t j ), calculate the word overlap measure WordOverlap(t i ,t j ); Step 504, attention mechanism weighting: Calculate the attention weighted value of Gaussian kernel similarity Attention weighted value of word overlap similarity Attention(θ,WordOverlap(t i ,t j )); Step 505, fusion similarity calculation: Combine the Gaussian kernel similarity and word overlap similarity to calculate the final fusion similarity H FS (v i ,v j ).

7. The standardized fusion processing method for new and old items of government services based on vector database according to claim 6 is characterized in that: In step 5, the specific implementation process of searching similar items in the vector database and finding the old items that are most similar to the new items includes: Step 511, new item vectorization: input the description text of the new government service item new , use the pre-trained word embedding model to embed t new Convert to vector representation v new , v new =BERT(t new ); Step 512, vectorization of old items: input the description text of all old government service items stored in the vector database old,n , where n is the index of the description text of the old government service items, n = 1, 2, ..., N, is a natural number, and N is the total number of description texts of the old government service items stored in the vector database. These old item description texts have been vectorized in the vector database to obtain the vector representation v of the old items. old,n ; Step 513, fusion similarity calculation: input new event vector v new And each old event vector v old,n , use the optimized fusion similarity calculation formula to calculate the similarity H between new and old events FS (v new ,v old,n ); Step 514, similar item search: input the calculated similarity H between the new and old items FS (v new ,v old,n ), in the vector database, based on the similarity H FS (v new ,v old,n ) from high to low, and use the approximate nearest neighbor search retrieval algorithm to retrieve the most similar old items; Step 515, determination of the most similar old items: find the corresponding old items according to the index with the highest similarity, match the most similar old items with the new items, and further process them.

8. The standardized fusion processing method for new and old items of government services based on vector database according to claim 7 is characterized in that: In step 513, the similarity H between the new and old items is calculated using the optimized fusion similarity calculation formula. FS (v new ,v old,n ) is in the form of the specific application formula: Where: is the attention weighted value of the Gaussian kernel function measuring the vector similarity of new and old items, Attention(θ, Word Overlap(t new ,t old,n )) is the attention weighted value of the vocabulary overlap between new and old items.

9. The standardized fusion processing method for new and old items of government services based on vector database according to claim 8 is characterized in that: In step six, the process of dynamically adjusting the weight coefficient in phase velocity calculation according to the length and importance of the description text includes: Step 61, text length and importance calculation: Enter the description text of the government service item i , calculate the length of the text L i , L i To describe the text t i The number of words contained in , enter the description text of government service items t i , calculate the importance of the text based on the frequency of keyword occurrence and the priority of the items in the historical items. i ; Step 62, dynamic weight adjustment: according to the length and importance of the text, the weight coefficient in the similarity calculation is adjusted through the dynamic weight adjustment formula. where α i , β i is the dynamically adjusted weight coefficient, λ1, λ2, λ3, and λ4 are model parameters used to adjust the mass coefficient, which are obtained by tuning the validation data, and m is the traversal index of the description text of the government service items. i , β i Normalize and obtain the adjusted weight α′ i , β′ i ; Step 63, applying dynamic weights: in the fusion similarity calculation formula, use the normalized adjusted weight coefficient α′ i , β′ i Weighting of Gaussian kernel similarity and vocabulary overlap; Step 64, adjusted similarity calculation and matching: use the adjusted weight coefficient to calculate the fusion similarity between the new and old government service items, and use the updated fusion similarity formula to retrieve the old items most similar to the new items in the vector database.

10. The standardized fusion processing method for new and old items of government services based on vector database according to claim 9 is characterized in that: In step seven, the specific implementation process of adjusting the priority of items based on similarity and importance is as follows: Step 71, obtain similarity calculation result: input the calculated similarity H between the new and old items FS (v new ,v old,n ), obtain the similarity calculation results of all old items and new items, and obtain the numerical set of similarity calculation results of old government service items and new government service items, which is {H FS (v new ,v old,1 ),H FS (v new ,v old,2 ),…,H FS (v new ,v old,N )}; Step 72, obtain the importance score of government service items: enter the description text of the old government service items old,n , using the method in step 61 to calculate the importance In of the government service item; Step 73, calculate the priority score: Combine the similarity and the importance of the government service items, and use the formula Get the priority score of each old item, where P n Score the priority of old task n, η, The weight coefficient used to adjust the similarity and importance impact is set according to application requirements; Step 74, priority sorting: sort all old items from high to low according to the priority scores to obtain the priority sorting result, which is used to guide resource allocation and task scheduling; Step 75, priority adjustment and decision support: According to business needs and external conditions, further manual adjustments are made to the sorting results, and processing suggestions are automatically generated based on the priority sorting results, and finally an adjusted priority list of government service items is output.

Images