Data detection method and system based on ElasticSearch

Abstract

The invention relates to a data detection method and system based on ElasticSearch, and relates to the technical field of data process.The method comprises the steps that a composite density tree index is constructed based on ElasticSearch, and a similar data set is pre-screened based on the composite density tree index to obtain a target document and candidate documents; performing anti-interference reconstruction on the target document and the candidate document based on the reversible feature transformation matrix to obtain a reconstructed target document and a reconstructed candidate document; constructing a self-adaptive dynamic weight; calculating the similarity between the target document and the candidate document; determining a dynamic weight of incremental updating; and dynamically selecting an optimal calculation path based on the incremental updated dynamic weight, and performing repeatability detection on the target document and the candidate document again based on the optimal calculation path to obtain a repeatability detection result. According to the invention, the accuracy and efficiency of data repeatability detection can be improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data detection method, system, electronic device and non-transient computer-readable storage medium based on ElasticSearch. Background Art

[0002] Nowadays, data duplication detection methods mainly rely on traditional techniques based on string matching, hash value comparison or similarity algorithms. Common methods include calculating text similarity to determine whether two data items are duplicated, which is widely used in text data processing, information retrieval and data deduplication. Specifically, many applications use the full-text search function provided by ElasticSearch and similarity calculation based on TF-IDF, combined with algorithms such as cosine similarity to perform duplication detection. These methods can effectively extract documents with high similarity from massive data, and then determine whether they are duplicate content.

[0003] However, traditional duplication detection based on full-text retrieval and similarity algorithms is often constrained by performance bottlenecks. As the amount of data increases, the computational complexity and storage requirements will grow rapidly, resulting in reduced retrieval efficiency. In addition, existing methods are less effective in detecting fuzzy or inexact duplicate data, especially when faced with noisy or mutated texts. Traditional similarity algorithms find it difficult to efficiently and accurately identify all types of duplicate data. Summary of the invention

[0004] In view of the technical problems existing in the prior art, the present invention provides a data detection method, system, electronic device and non-transitory computer-readable storage medium based on ElasticSearch, which can improve the efficiency and accuracy of data repeatability detection.

[0005] The technical solution of the present invention to solve the above technical problems is as follows: The present invention provides a data detection method based on ElasticSearch, the method comprising: Building a composite density tree index based on ElasticSearch, and pre-screening similar data sets based on the composite density tree index to obtain target documents and candidate documents; A decomposable attention mechanism is used to generate a reversible feature transformation matrix, and based on the reversible feature transformation matrix, the target document and the candidate document are subjected to interference-resistant reconstruction to obtain a reconstructed target document and a reconstructed candidate document; Constructing an adaptive dynamic weight based on the entropy difference of each dimension between the reconstructed target document and the reconstructed candidate document; Calculating the similarity between the target document and the candidate document based on the asymptotic weighted cosine similarity function of the dynamic weight; Incremental parameter optimization using frequency vectorization mechanism to determine dynamic weights for incremental updates; The optimal calculation path is dynamically selected based on the incrementally updated dynamic weight, and the target document and the candidate document are again tested for repeatability based on the optimal calculation path to obtain a repeatability test result.

[0006] Optionally, the adopting of a decomposable attention mechanism to generate a reversible feature transformation matrix includes: Determine a query mapping for each original feature according to a position autocorrelation tensor representing correlations between different positions in the input data and a decomposable query head matrix; Combining each of the original features with each of the query mappings to obtain a combined feature; The difference between each of the combined features and each of the target features is determined and minimized to generate the reversible feature transformation matrix.

[0007] Optionally, the reversible feature transformation matrix is ​​expressed as: ; in, is the reversible feature transformation matrix, is the decomposable query head matrix, and is the position autocorrelation tensor, is the original feature, is the target feature after removing noise interference, is the total number of samples.

[0008] Optionally, constructing an adaptive dynamic weight based on the entropy difference of each dimension between the reconstructed target document and the reconstructed candidate document includes: Determining the global entropy of the similar data set and the local entropy of each feature dimension in the similar data set; Normalizing the difference between the global entropy and the local entropy to obtain a normalized difference; The normalized difference is processed by exponential operation of mutual information strength to obtain the dynamic weight.

[0009] Optionally, the dynamic weight is expressed as: ; in, is the dynamic weight, is the sensitivity control parameter, is the sliding window variance, is the third-order mutual information strength measure, is the attenuation coefficient, is the global entropy, is the feature dimension The local entropy of is the smoothing term.

[0010] Optionally, the calculating the similarity between the target document and the candidate document based on the asymptotic weighted cosine similarity function of the dynamic weight includes: Obtaining a first feature value of each dimension in the set of target documents and a second feature value of each dimension in the set of candidate documents; Performing power transformation on the first eigenvalue and the second eigenvalue to obtain a first eigenvalue after power transformation and a second eigenvalue after power transformation; Calculating the distance between the first eigenvalue and the second eigenvalue; Normalizing the first eigenvalue after the power transformation and the second eigenvalue after the power transformation to obtain normalized eigenvalues; The similarity between the target document and the candidate document is determined based on the distance between the first eigenvalue and the second eigenvalue, the first eigenvalue after power transformation, the second eigenvalue after power transformation, and the normalized eigenvalue.

[0011] Optionally, the performing incremental parameter optimization by using a frequency vectorization mechanism to determine a dynamic weight of an incremental update includes: Determine a basic update amount of the dynamic weight according to a learning rate of the dynamic weight update and a gradient of a loss function of the similarity to the dynamic weight update; Performing feature calibration processing according to the feature value of the dimension corresponding to the dynamic weight to obtain a corresponding feature calibration value; The similarity normalizes the L1.5 norm of the gradient vector of the dynamic weight to obtain a gradient normalization value; Performing momentum adjustment on the change of the reversible characteristic transformation matrix to obtain a dynamic adjustment amount; A dynamic weight of the incremental update is determined according to the basic update amount, the feature calibration value, the gradient normalization value and the dynamic adjustment amount.

[0012] Optionally, the optimal computing path is calculated based on a hybrid sharding architecture, and the hybrid sharding architecture adopts a heterogeneous computing unit sharding strategy, wherein: The CPU shard processes the CDTI pre-screening task, the GPU shard performs the AWCS similarity calculation, and low-latency communication is achieved between shards through RDMA.

[0013] Optionally, the method further includes determining a sharding efficiency of the unit sharding strategy, including: Calculate the sharding efficiency impact value based on the communication delay time of sending and receiving data between the shards and the sharding cost of each shard in computing and storage; Calculate the weight difference between the master server's weight vector and the shard server's weight vector; Based on the sharding efficiency impact value and the weight difference, a sharding efficiency of the unit sharding strategy is determined.

[0014] The present invention also provides a data detection system based on ElasticSearch, the system comprising: A data screening module is used to build a compound density tree index based on ElasticSearch, and pre-screen similar data sets based on the compound density tree index to obtain target documents and candidate documents; A data reconstruction module, used to generate a reversible feature transformation matrix by using a decomposable attention mechanism, and to perform interference-resistant reconstruction on the target document and the candidate document based on the reversible feature transformation matrix to obtain a reconstructed target document and a reconstructed candidate document; A weight determination module, used to construct an adaptive dynamic weight based on the entropy difference of each dimension between the reconstructed target document and the reconstructed candidate document; A preliminary detection module, configured to calculate the similarity between the target document and the candidate document based on the asymptotic weighted cosine similarity function of the dynamic weight; A weight update module, which is used to perform incremental parameter optimization using a frequency vectorization mechanism and determine the dynamic weights of incremental updates; The target detection module is used to dynamically select an optimal calculation path based on the incrementally updated dynamic weight, and perform repeatability detection on the target document and the candidate document again based on the optimal calculation path to obtain a repeatability detection result.

[0015] In addition, to achieve the above-mentioned purpose, the present invention also proposes an electronic device, comprising: a memory for storing a computer software program; a processor for reading and executing the computer software program, thereby implementing a data detection method based on ElasticSearch as described above.

[0016] In addition, to achieve the above-mentioned purpose, the present invention also proposes a non-transitory computer-readable storage medium, in which a computer software program is stored. When the computer software program is executed by a processor, it implements a data detection method based on ElasticSearch as described above.

[0017] The beneficial effects of the present invention are: (1) The present invention adopts a dynamic weighted cosine similarity calculation method, combined with information entropy density ratio and asymptotic weighting, to more accurately calculate the similarity between data sets. By dynamically adjusting the weights of feature dimensions, the key features in the data can be effectively highlighted, thereby improving detection accuracy, especially when facing deformed repetitions (such as machine translation repetitions, segmented plagiarism, etc.).

[0018] (2) The present invention can significantly reduce computational complexity while ensuring accuracy by introducing dimensionality compression and reversible feature transformation matrix. Specifically, dimensionality compression reduces data redundancy and maps high-dimensional features to low-dimensional space through nonlinear transformation, thereby reducing computational and storage overhead.

[0019] (3) The gradient reversal layer (GRL) of the present invention constrains the stability of feature transformation, ensuring that important feature information is not lost during the reconstruction of the feature space. The stability of features helps to reduce the interference of data noise and ensure the reliability of subsequent similarity calculations, which is particularly important when the data set is of low quality or contains noise.

[0020] In summary, the present invention has high accuracy, low computational complexity, efficient feature optimization capability, and good scalability and adaptability in data repeatability detection, especially in deformed repetition and complex repetition modes. Through the combination of dynamic weighted cosine similarity, incremental learning optimization, distributed computing and other technologies, it can significantly improve the efficiency of large-scale data processing, and improve the accuracy of repeatability detection through refined feature processing, meeting the data processing requirements in different scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A flow chart of a data detection method based on ElasticSearch provided by the present invention; Figure 2 A structural schematic diagram of a data detection system based on ElasticSearch provided by the present invention; Figure 3 A schematic diagram of the hardware structure of a possible electronic device provided by the present invention; Figure 4 A schematic diagram of the hardware structure of a possible computer-readable storage medium provided by the present invention. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0023] In the description of the present invention, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0024] In the description of the present invention, the term "for example" is used to mean "used as an example, illustration or explanation". Any embodiment described as "for example" in the present invention is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is given to enable any technician in the field to implement and use the present invention. In the following description, details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art can recognize that the present invention can be implemented without using these specific details. In other examples, well-known structures and processes will not be elaborated in detail to avoid obscuring the description of the present invention with unnecessary details. Therefore, the present invention is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed in the present invention.

[0025] See also Figure 1 , provides a flow chart of a data detection method based on ElasticSearch of the present invention, comprising the following steps: Step 201: construct a compound density tree index based on ElasticSearch, and pre-screen similar data sets based on the compound density tree index to obtain target documents and candidate documents.

[0026] In ElasticSearch, a composite inverted index based on spatial partitioning (such as CDTI) is established. The index decomposes spatial data (document features, queries, etc.) into multiple subspaces and uses a dynamic critical density function ( ) to locate potential similar data sets. This means that this method can quickly locate potential similar document sets with high density in the feature space through the composite index structure without the need for a complete linear scan of all data.

[0027] The dynamic critical density function plays a role in screening similar data in the data set. Through this function, the threshold of similarity detection can be dynamically adjusted so that documents with similar feature space density can be selected first, thereby reducing the number of candidate documents in subsequent similarity calculations. This screening method is more efficient, especially when facing large-scale data sets, and can significantly reduce the computational burden.

[0028] Based on CDTI index and dynamic density function, ElasticSearch can pre-screen documents first during document retrieval, so that only those potential and highly similar document sets are used as candidate documents for subsequent feature space reconstruction and similarity calculation. This pre-screening process reduces the amount of calculation, especially when the number of documents is very large, which can improve the retrieval speed and accuracy.

[0029] In short, the composite density tree index built on ElasticSearch divides the data feature space into multiple subspaces and locates potential similar data sets through dynamic critical density functions, thus pre-screening the target documents and candidate documents. This method can significantly improve the efficiency of similarity detection and is particularly suitable for scenarios with large-scale data sets.

[0030] Step 202: A decomposable attention mechanism is used to generate a reversible feature transformation matrix, and based on the reversible feature transformation matrix, the target document and the candidate documents are reconstructed in an anti-interference manner to obtain a reconstructed target document and a reconstructed candidate document.

[0031] In some embodiments, step 202 may include: Determine a query mapping for each original feature according to a position autocorrelation tensor representing correlations between different positions in the input data and a decomposable query head matrix; Combining each of the original features with each of the query mappings to obtain a combined feature; The difference between each of the combined features and each of the target features is determined and minimized to generate the reversible feature transformation matrix.

[0032] In some embodiments, the reversible feature transformation matrix is ​​expressed as: ; in, is the reversible eigentransformation matrix, is the decomposable query head matrix, and is the position autocorrelation tensor, is the original feature, is the target feature after removing noise interference, is the total number of samples.

[0033] In the specific implementation, It is the target matrix obtained by the optimization algorithm, which transforms the original features to obtain new feature representations. Through this transformation, the feature vector will be re-encoded, noise interference will be removed, and effective features will be enhanced.

[0034] and is the position autocorrelation tensor, which is used to represent the correlation between different positions in the input data. In the self-attention mechanism, and Matrices are usually used to capture local features or global information. They reflect the structural characteristics of the data by calculating weights at different positions.

[0035] It is a decomposable query head matrix, which is usually used to represent the embedding or mapping of the query in a specific task. It is used to generate the query vector in the self-attention mechanism. It can make the calculation more efficient by decomposing it, and enable the model to learn the dependencies between different dimensions.

[0036] are the original features, which are the original features in the input data, and may be obtained through some encoding or representation learning method. These features will be passed as input to the subsequent transformation and optimization steps.

[0037] It is the target feature after removing noise interference. After optimization, the desired features will be more representative and discriminative after removing noise and interference. The goal is to use the feature transformation matrix The original feature Mapped to this new feature space.

[0038] is an integer representing the number of samples involved in the optimization process during the feature space reconstruction. Each sample corresponds to a feature vector (original feature), and its denoised target feature , they all participate in the calculation of the loss function.

[0039] The goal in the formula is to minimize a loss function that measures the original features and the transformed target features The difference between . The form of the loss function is the square of the Euclidean distance, that is: , the square of the Euclidean distance (L2 norm) is used to calculate the difference, aiming to make the target feature as close as possible to the transformed feature.

[0040] Functions are applied to Softmax is usually used for normalization to convert the input into a probability distribution, which emphasizes the characteristics of certain dimensions or positions, so that the model can better focus on important features.

[0041] here, The product of represents a certain relationship or mapping of the features, converting the original features Combined with this mapping, a new feature representation is generated.

[0042] The goal of the loss function is to make the transformed features As close as possible to the target feature after denoising The process of minimizing the loss function is to solve the optimal reversible feature transformation matrix through optimization algorithms (such as back propagation, BPTT, etc.) , so that the feature mapping is optimized and adjusted.

[0043] BPTT is a back-propagation algorithm used for optimization, especially in sequence data (such as time series or recurrent neural networks). Here BPTT is used to optimize the feature transformation matrix , ensuring that in sequence tasks, feature transformations remain valid over time or steps.

[0044] The goal is to learn a reversible feature transformation matrix , so that the features transformed by this matrix With target features The difference between them is minimal. In this way, the noise in the feature space can be removed, and what remains are more effective features. Through this optimization process, the feature representation will be strengthened, and the model will learn how to extract more useful information from the original features and reduce the impact of noise, thereby improving the results of subsequent tasks (such as similarity calculation, classification, etc.).

[0045] In summary, the present invention uses a reversible feature transformation matrix To optimize and reconstruct the feature space, the transformed features can better represent the data, while removing noise to achieve feature enhancement. Through the application of Softmax and back propagation optimization (BPTT), the model can gradually learn the optimal transformation matrix, thereby improving the quality of features and the performance of the model.

[0046] Step 203: construct an adaptive dynamic weight based on the entropy difference of each dimension between the reconstructed target document and the reconstructed candidate document.

[0047] In some embodiments, step 203 may include: Determining the global entropy of the similar data set and the local entropy of each feature dimension in the similar data set; Normalizing the difference between the global entropy and the local entropy to obtain a normalized difference; The normalized difference is processed by exponential operation of mutual information strength to obtain the dynamic weight.

[0048] In some embodiments, the dynamic weight is expressed as: ; in, is the dynamic weight, is the sensitivity control parameter, is the sliding window variance, is the third-order mutual information strength measure, is the attenuation coefficient, is the global entropy, is the feature dimension The local entropy of is a smoothing term.

[0049] In the specific implementation, is a dynamic weight, indicating the feature dimension In this solution, the dynamic weight The importance of each feature dimension in similarity calculation is adjusted by comprehensively considering global information and local information. The dynamic weight mechanism helps to automatically adjust the influence of feature dimensions to highlight more important features and reduce the weight of irrelevant features, thereby improving the accuracy of similarity calculation.

[0050] is a sensitivity control parameter used to adjust the global entropy and local entropy This parameter controls how much this difference affects the dynamic weight calculation. Larger values ​​of 𝜆 increase the effect of the difference between global and local entropy on the weight calculation, while smaller values ​​of 𝜆 reduce this effect.

[0051] is the sliding window variance, which is used to measure the feature dimension The magnitude of change within the sliding window. By calculating the variance of the data within the window, the volatility of the dimension in the local range can be reflected. The larger the variance, the greater the change in the features of the dimension, and the weight will also be adjusted, making the model pay more attention to features with high variability. Conversely, features with smaller variance are considered more stable and have less impact on similarity calculations.

[0052] is the third-order mutual information strength measurement value, indicating the feature dimension The mutual information between the feature dimensions and other feature dimensions quantifies the importance of the feature by measuring the information correlation between different dimensions. A higher mutual information value means that the feature dimension The information sharing between the dimensions and other feature dimensions is strong, so its weight in the similarity calculation should also be increased. Dimensions with high mutual information strength can provide more relevant information about the data, which helps to judge the similarity of the data more accurately.

[0053] is the attenuation coefficient, which is used to adjust the strength of the third-order mutual information The strength of its role in weight calculation. The value will increase The impact in the calculation means that feature dimensions with higher mutual information strength will receive greater dynamic weights; A small value will weaken the influence of mutual information on the weight.

[0054] is the global entropy, which is the entropy value of the entire data set, measuring the uncertainty or information content of the data as a whole. Higher entropy indicates that the data has higher uncertainty or diversity at the global level. As a quantitative indicator of the overall complexity of the data set, the global entropy affects the weight distribution of the feature dimensions. When the global entropy is high, the information of the data set is more dispersed, and more sophisticated local weight adjustments may be required to capture the regularity of the data.

[0055] is the feature dimension The local entropy of The entropy value within a certain local range (such as a subset selected by a sliding window) measures the amount of information and uncertainty of the dimension on the local data. The local entropy reflects the degree of change of the feature within a specific local data range. If a feature dimension shows a large information change in the local range, its local entropy is large, which means that the feature is more important in this range and requires a higher dynamic weight.

[0056] is a smoothing term, often used to avoid division by zero or instability in numerical calculations. The smoothing term ensures that when calculating the local variance, division by zero errors are avoided when the variance is zero. By smoothing, the model is able to operate stably in all cases.

[0057] In the first part, the difference between global entropy and local entropy is normalized (by variance smoothing), and the weight is limited using the tanh function to keep it within a reasonable range. Specifically, when the difference is large, the weight is increased, and when the difference is small, the weight is reduced.

[0058] Part, through the mutual information strength The exponential operation further strengthens the feature dimension with high mutual information strength and enhances its role in similarity calculation.

[0059] Step 204: Calculate the similarity between the target document and the candidate document based on the asymptotic weighted cosine similarity function with dynamic weights.

[0060] In some embodiments, step 204 may include: Obtaining a first feature value of each dimension in the set of target documents and a second feature value of each dimension in the set of candidate documents; Performing power transformation on the first eigenvalue and the second eigenvalue to obtain a first eigenvalue after power transformation and a second eigenvalue after power transformation; Calculating the distance between the first eigenvalue and the second eigenvalue; Normalizing the first eigenvalue after the power transformation and the second eigenvalue after the power transformation to obtain normalized eigenvalues; The similarity between the target document and the candidate document is determined based on the distance between the first eigenvalue and the second eigenvalue, the first eigenvalue after power transformation, the second eigenvalue after power transformation, and the normalized eigenvalue.

[0061] In some embodiments, the similarity between the target document and the candidate document can be expressed as: ; in, is the weighted cosine similarity, It is Dynamic weights of dimensions, is the first dimensional eigenvalue, is the harmonic coefficient (corresponding to the CDTI The values ​​are negatively correlated). is the power transformation factor, is the nonlinear compensation parameter, Eigenvalues ​​after power transformation.

[0062] In the specific implementation, this function effectively solves the shortcomings of traditional cosine similarity in deformation duplication detection through nonlinear dimension weighting and hybrid similarity measurement mechanism. The following is the working principle and collaborative working mechanism of each component: Molecular structure: A composite similarity fuser.

[0063] In the semantic similarity reinforcement , power transformation factor : The calculation formula is ,in For Dimension The information entropy of is the maximum entropy value of the system.

[0064] When the dimension The entropy value Higher (features are evenly distributed), , degenerates into a linear product, retaining the original semantic similarity; when Low (high feature concentration), , strengthening the multiplicative effect of important dimensions.

[0065] In application scenarios, when detecting machine translation repetitions, the feature dimension of keywords (such as named entities) is improved. value, enhancing the matching capability of semantically equivalent but different expressions.

[0066] Local tampering compensation , nonlinear compensation parameters Constrained in the interval (0,1 / 3), ensure Small differences (such as paragraph reordering) will be magnified ( ); significant difference ( ) is suppressed.

[0067] In application scenarios, the negative index can be used to capture the correlation of small differences in local word replacement (such as synonym replacement) in segmented plagiarism.

[0068] Harmonic coefficient , with the density function pre-screened by CDTI The value is negatively correlated. When the candidate document and the target document are in a high-density area ( The value is large, and semantic items are used first ( Reduced), focusing on overall semantic matching; in low-density areas ( value is small), increase To strengthen local difference compensation.

[0069] Dynamic adjustment example: When detecting translation duplication, if CDTI determines that the semantics are highly overlapped ( value is large), then = 0.3, focusing on semantic items; if structural variation is detected ( value is small), then =0.7, strengthen local compensation.

[0070] Denominator structure: Anti-skew normalizer .

[0071] The eigenvalues After amplification by the power, weighted summation is achieved: Suppress long tail distribution: High Dimensional features are squared and dominate the normalization process; Dimension Selectivity Enhancement: Low The noise characteristics of the dimension are weakened in normalization.

[0072] when( =2), the denominator becomes , extreme value dominated normalization, suitable for detecting plagiarism patterns with mutations in key features.

[0073] Entropy difference drive

[0074] Entropy Difference Identify dimension specificity: Dimensions with low global entropy but high local entropy (abnormally concentrated features) are given high weights; for example, in cross-language plagiarism, the local entropy of the proper noun dimension drops suddenly, and the weight increases.

[0075] Mutual information term: Strengthening the associated dimension: When dimension Strongly correlated with plagiarism patterns (such as syntactic structure dimensions), third-order mutual information As it increases, the weight increases exponentially.

[0076] Detection scenario: Identify duplicate documents for Chinese-English machine translation.

[0077] CDTI Pre-Screening: Pass Function locates high semantic overlap candidate sets ( Automatically adjusted to 0.3); Feature reconstruction: M matrix filters translation noise (such as differences in prepositional structure); Weight distribution: Named Entity Dimension: =1.8 (low entropy, power amplification), =0.9 (high mutual information strength); Word order dimension: =0.25, compensating for the slight difference caused by word order adjustment.

[0078] Calculation results: Traditional cosine similarity = 0.72 (underestimated due to word order disturbance); AWCS similarity = 0.89 (local difference term contribution +0.17, semantic term contribution +0.62).

[0079] Through the above mechanism, this method achieved an F1 value of 92.3% for deformed repetitions in the TREC2022 duplicate detection benchmark test, an improvement of 19.5% over the baseline model, and reduced the false alarm rate to 2.7%.

[0080] Step 205: Perform incremental parameter optimization using the frequency vectorization mechanism to determine the dynamic weight of the incremental update.

[0081] In some embodiments, step 205 may include: Determine a basic update amount of the dynamic weight according to a learning rate of the dynamic weight update and a gradient of a loss function of the similarity to the dynamic weight update; Performing feature calibration processing according to the feature value of the dimension corresponding to the dynamic weight to obtain a corresponding feature calibration value; The similarity normalizes the L1.5 norm of the gradient vector of the dynamic weight to obtain a gradient normalization value; Performing momentum adjustment on the change of the reversible characteristic transformation matrix to obtain a dynamic adjustment amount; A dynamic weight of the incremental update is determined according to the basic update amount, the feature calibration value, the gradient normalization value and the dynamic adjustment amount.

[0082] In some embodiments, the incrementally updated dynamic weight may be expressed as: ; in, is the dynamic weight of incremental update, is the learning rate, is the loss function for similarity The gradient of is the correction of the eigenvalue after gradient calibration, Is similarity The L1.5 norm of the gradient, is the momentum adjustment function, are the matrix reconstruction coefficients (constrained by Lebesgue integration).

[0083] In the specific implementation, Indicates Dynamic weight increment update of dimension features, that is, during the training process, the weight of the feature dimension The amount of adjustment in each iteration. This incremental update reflects the extent of the modification of the weights in the current iteration. Incremental updates reflect the adaptive learning process of the model. It adjusts the weight of each feature dimension according to the current gradient and historical changes through gradient descent, ensuring that the model can be gradually optimized as the training process progresses.

[0084] is the learning rate, which controls the step size of weight update. It determines the amplitude of parameter adjustment of the model at each gradient update. Too large a learning rate may lead to unstable training, while too small a learning rate may lead to slow convergence. The learning rate determines the strength of each update step and is a key parameter in the gradient descent algorithm. It directly affects the convergence speed and stability of the training process. Too large a learning rate will lead to gradient oscillation, while too small a learning rate may cause the model to fail to learn effectively.

[0085] is the loss function for similarity The gradient of , which indicates the degree of influence of similarity on the loss function. It reveals how changes in the loss function affect the similarity , which is the direction of similarity change. The gradient of the loss function is used to guide how to adjust the weights so that the similarity 𝑆 is closer to the target value (for example, improving similarity and reducing loss). Through the chain rule, this gradient is eventually fed back into the weight update of the feature.

[0086] It is The eigenvalue of the dimension is corrected after gradient calibration. Gradient calibration coefficient is a dynamic adjustment factor, which indicates the adjustment of the feature value according to the change of learning rate in each iteration. When the learning rate changes, the calibration factor The eigenvalues ​​are scaled to avoid gradient oscillations caused by learning rate fluctuations. The gradient calibration coefficient can be adaptively adjusted at different learning rates to ensure the smoothness of the training process and avoid update instability caused by sudden changes in the learning rate.

[0087] Is similarity The L1.5 norm of the gradient is a measure of the gradient vector, which is between the L1 norm (sparseness) and the L2 norm (smoothness), taking into account both feature selection and convergence smoothness. The L1.5 norm can effectively limit the amplitude of the gradient and avoid gradient explosion during training (that is, extremely large gradients lead to excessive weight updates). This norm strikes a balance between sparsity and smoothness, helping to reduce the impact of unimportant features while retaining important features. By normalizing the gradient, the impact of excessive gradients on weight updates is suppressed, making the weight update process more stable and preventing extreme fluctuations during training.

[0088] middle, is the momentum adjustment function, which adjusts the current learning momentum through the change of historical state. , through the Lebesgue integral, quantized the matrix In the degree of change in the feature space, when the matrix changes greatly, the momentum decays to avoid the interference of historical gradients on the current gradient; when the matrix changes less, the momentum increases to accelerate convergence. Momentum adjustment adjusts the update speed and amplitude according to the change of parameters, making the training process more stable, and intelligently reduces or increases the contribution of momentum according to historical changes to improve training efficiency.

[0089] Step 206: dynamically select the optimal calculation path based on the incrementally updated dynamic weight, and perform repeatability detection on the target document and the candidate documents again based on the optimal calculation path to obtain a repeatability detection result.

[0090] In some embodiments, step 206 may include determining an optimal computing path based on a hybrid sharding architecture that employs a heterogeneous computing unit sharding strategy, wherein: The CPU shard processes the CDTI pre-screening task, the GPU shard performs the AWCS similarity calculation, and low-latency communication is achieved between shards through RDMA.

[0091] In some embodiments, the sharding efficiency of the unit sharding strategy is determined by: Calculate the sharding efficiency impact value based on the communication delay time of sending and receiving data between the shards and the sharding cost of each shard in computing and storage; Calculate the weight difference between the master server's weight vector and the shard server's weight vector; Based on the sharding efficiency impact value and the weight difference, a sharding efficiency of the unit sharding strategy is determined.

[0092] Among them, the sharding efficiency can be expressed as: ; in, is the sharding efficiency, is the inter-shard communication factor, is the communication delay time, is the hardware acceleration coefficient, is the sharding cost, is the weight vector of the primary server, is the weight vector of the shard servers, is the weight difference threshold.

[0093] In the specific implementation, Sharding efficiency is an indicator used to evaluate the data processing efficiency between different shards in distributed computing. It takes into account factors such as communication delay, sharding cost, hardware acceleration capability, and sharding weight differences. Sharding efficiency 𝐸 can measure whether the computing path of a shard is optimal in a distributed computing framework, and help optimize the resource allocation of the distributed system, thereby improving overall performance.

[0094] It is the inter-shard communication factor, which represents the communication cost when different computing nodes (i.e., between shards) exchange data. Affects the efficiency of data transmission between shards. Higher A value of 0 indicates that the communication delay between shards is large, which may affect the computing efficiency of the system. In actual deployment, reducing communication delay can usually significantly improve the overall computing efficiency.

[0095] is the communication delay, which is the time required to send and receive data from one shard to another. Communication delay is a key factor affecting the performance of distributed systems. Longer delays will increase the waiting time during the calculation process, thereby reducing the overall efficiency. Indicates the impact of communication delay on sharding efficiency. The greater the delay, the lower the efficiency.

[0096] is the hardware acceleration coefficient, which is used to indicate the computing speed improvement brought by hardware acceleration (such as using GPU, TPU, etc.) in distributed computing. Hardware acceleration can significantly improve the computing speed and reduce the computing time of each shard. Therefore, the larger 𝑣 is, the more significant the effect of hardware acceleration is, thereby improving the sharding efficiency 𝐸.

[0097] It is the sharding cost, which indicates the resource consumption of each shard in terms of computing and storage, usually involving the resources required for computing, memory, and storage. The sharding cost is an important factor affecting the efficiency of sharding. A higher sharding cost means more computing resources and time consumption, which may lead to a decrease in overall efficiency.

[0098] is the weight vector of the master server, which indicates the weight of each feature dimension in the master server. The weight vector is usually used to indicate the priority or resource allocation of the server when processing a specific task. The weight vector of the master server represents the importance of the master server in the task scheduling and calculation process. A larger weight value means that the server plays a more important role in a specific task.

[0099] is the weight vector of the shard server, and Similarly, it indicates the priority or resource allocation of the shard server in the distributed system. The weight vector of the shard server determines its importance and workload in the distributed computing task. If the weight difference of the shard is too large, it may cause load imbalance, thus affecting the efficiency of the shard.

[0100] is the weight difference threshold, which is used to limit the weight difference between the master server and the shard server. If the weight difference between the two exceeds this threshold, it may cause unbalanced calculation or excessive load. , which can avoid excessive mismatch between the master server and the shard server and ensure reasonable allocation of resources. The value indicates that a larger weight difference is allowed, and a larger A large value can avoid performance issues caused by large weight differences.

[0101] The role of is to calculate the impact of inter-shard communication and hardware acceleration, the impact of communication delay: inter-shard communication factor and communication delays The product of is processed by a logarithmic function, indicating that as the communication delay increases, the sharding efficiency decreases. The greater the communication delay, the lower the sharding efficiency. The lower the value, the lower the impact of hardware acceleration: Hardware acceleration coefficient and sharding costs The ratio of indicates that the computing cost is reduced under hardware acceleration, thereby improving the sharding efficiency.

[0102] The role of this part is to consider the weight difference between the master server and the shard server, calculate the weight difference between the master server and the shard server (through the L1 norm), and pass If the weight difference is large, it means that the resource allocation is unbalanced, which may lead to reduced efficiency. Therefore, this item is subtracted to make the weight difference larger, and the sharding efficiency lower. By setting an appropriate threshold , the impact of weight differences on sharding efficiency can be controlled to avoid efficiency degradation caused by uneven load.

[0103] The present invention optimizes the efficiency of sharding in a distributed computing environment. It comprehensively considers communication delay, hardware acceleration, sharding cost, and the weight difference between the main server and the sharding server to calculate the efficiency of each shard in the overall computing task. By properly configuring these parameters, the utilization of resources can be improved in a distributed system, the computing path can be optimized, and ultimately the performance of the system can be improved.

[0104] Just as an example, the masternode weight With shard nodes The L1 norm difference exceeds the threshold Path redistribution is triggered when the GPU shard weight differs from the master node by more than , it is determined to be a calculation path mismatch.

[0105] Penalize shard combinations with high communication latency, Parameters are dynamically assigned based on device type, such as CPU sharding =1.0, GPU sharding = 2.5. CDTI pre-screening smoothing parameter , nonlinear compensation parameter of the similarity function .

[0106] In some embodiments, when detecting cross-language duplication, the system automatically selects the path: CPU sharding (CDTI pre-screening) → GPU sharding 1 (AWCS calculation) → GPU sharding 3 (result aggregation). Compared with static sharding, latency is reduced by 38% and throughput is increased by 2.1 times.

[0107] In summary, the present invention realizes the intelligent closed loop of "detection → optimization → re-detection" through the deep coupling of incremental update and calculation path selection, which improves the recall rate while maintaining low resource consumption, and is particularly suitable for scenarios that require rapid response to new plagiarism variants.

[0108] See also Figure 2 , Figure 2 A structural schematic diagram of a data detection system based on ElasticSearch provided by the present invention.

[0109] like Figure 2 As shown, an ElasticSearch-based data detection system proposed in an embodiment of the present invention includes: The data screening module 301 is used to construct a composite density tree index based on ElasticSearch, and pre-screen similar data sets based on the composite density tree index to obtain target documents and candidate documents; A data reconstruction module 302 is used to generate a reversible feature transformation matrix using a decomposable attention mechanism, and to perform interference-resistant reconstruction on the target document and the candidate document based on the reversible feature transformation matrix to obtain a reconstructed target document and a reconstructed candidate document; A weight determination module 303, configured to construct an adaptive dynamic weight based on the entropy difference of each dimension between the reconstructed target document and the reconstructed candidate document; A preliminary detection module 304, configured to calculate the similarity between the target document and the candidate document based on the asymptotic weighted cosine similarity function of the dynamic weight; A weight updating module 305, for performing incremental parameter optimization using a frequency vectorization mechanism and determining a dynamic weight for incremental update; The target detection module 306 is used to dynamically select an optimal calculation path based on the incrementally updated dynamic weight, and perform a repeatability detection on the target document and the candidate document again based on the optimal calculation path to obtain a repeatability detection result.

[0110] See also Figure 3 , Figure 3 Schematic diagram of an electronic device provided by an embodiment of the present invention. Figure 3 As shown, an embodiment of the present invention provides an electronic device 400, including a memory 410, a processor 420, and a computer program 411 stored in the memory 410 and executable on the processor 420. When the processor 420 executes the computer program 411, the following steps are implemented: Building a composite density tree index based on ElasticSearch, and pre-screening similar data sets based on the composite density tree index to obtain target documents and candidate documents; A decomposable attention mechanism is used to generate a reversible feature transformation matrix, and based on the reversible feature transformation matrix, the target document and the candidate document are subjected to interference-resistant reconstruction to obtain a reconstructed target document and a reconstructed candidate document; Constructing an adaptive dynamic weight based on the entropy difference of each dimension between the reconstructed target document and the reconstructed candidate document; Calculating the similarity between the target document and the candidate document based on the asymptotic weighted cosine similarity function of the dynamic weight; Incremental parameter optimization using frequency vectorization mechanism to determine dynamic weights for incremental updates; The optimal calculation path is dynamically selected based on the incrementally updated dynamic weight, and the target document and the candidate document are again tested for repeatability based on the optimal calculation path to obtain a repeatability test result.

[0111] See also Figure 4 , Figure 4 A schematic diagram of an embodiment of a computer-readable storage medium provided in an embodiment of the present invention. Figure 4 As shown, this embodiment provides a computer-readable storage medium 500, on which a computer program 411 is stored. When the computer program 411 is executed by a processor, the following steps are implemented: Building a composite density tree index based on ElasticSearch, and pre-screening similar data sets based on the composite density tree index to obtain target documents and candidate documents; A decomposable attention mechanism is used to generate a reversible feature transformation matrix, and based on the reversible feature transformation matrix, the target document and the candidate document are subjected to interference-resistant reconstruction to obtain a reconstructed target document and a reconstructed candidate document; Constructing an adaptive dynamic weight based on the entropy difference of each dimension between the reconstructed target document and the reconstructed candidate document; Calculating the similarity between the target document and the candidate document based on the asymptotic weighted cosine similarity function of the dynamic weight; Incremental parameter optimization using frequency vectorization mechanism to determine dynamic weights for incremental updates; The optimal calculation path is dynamically selected based on the incrementally updated dynamic weight, and the target document and the candidate document are again tested for repeatability based on the optimal calculation path to obtain a repeatability test result.

[0112] It should be noted that in the above embodiments, the description of each embodiment has its own emphasis, and for parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0113] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0114] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded computer, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A system that specifies the functions of a box or multiple boxes.

[0115] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction system, which is implemented in the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0116] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0117] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0118] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A data detection method based on ElasticSearch, characterized in that: The method comprises: Building a composite density tree index based on ElasticSearch, and pre-screening similar data sets based on the composite density tree index to obtain target documents and candidate documents; A decomposable attention mechanism is used to generate a reversible feature transformation matrix, and based on the reversible feature transformation matrix, the target document and the candidate document are subjected to interference-resistant reconstruction to obtain a reconstructed target document and a reconstructed candidate document; Constructing an adaptive dynamic weight based on the entropy difference of each dimension between the reconstructed target document and the reconstructed candidate document; Calculating the similarity between the target document and the candidate document based on the asymptotic weighted cosine similarity function of the dynamic weight; Incremental parameter optimization using frequency vectorization mechanism to determine dynamic weights for incremental updates; The optimal calculation path is dynamically selected based on the incrementally updated dynamic weight, and the target document and the candidate document are again tested for repeatability based on the optimal calculation path to obtain a repeatability test result.

2. The data detection method based on ElasticSearch according to claim 1, characterized in that: The method of using a decomposable attention mechanism to generate a reversible feature transformation matrix includes: Determine a query mapping for each original feature according to a position autocorrelation tensor representing correlations between different positions in the input data and a decomposable query head matrix; Combining each of the original features with each of the query mappings to obtain a combined feature; The difference between each of the combined features and each of the target features is determined and minimized to generate the reversible feature transformation matrix.

3. The data detection method based on ElasticSearch according to claim 2, characterized in that: The reversible feature transformation matrix is ​​expressed as: ； in, is the reversible feature transformation matrix, is the decomposable query head matrix, and is the position autocorrelation tensor, is the original feature, is the target feature after removing noise interference, is the total number of samples.

4. The data detection method based on ElasticSearch according to claim 3 is characterized in that: The step of constructing an adaptive dynamic weight based on the entropy difference of each dimension between the reconstructed target document and the reconstructed candidate document includes: Determining the global entropy of the similar data set and the local entropy of each feature dimension in the similar data set; Normalizing the difference between the global entropy and the local entropy to obtain a normalized difference; The normalized difference is processed by exponential operation of mutual information strength to obtain the dynamic weight.

5. The data detection method based on ElasticSearch according to claim 4, characterized in that: The dynamic weight is expressed as: ； in, is the dynamic weight, is the sensitivity control parameter, is the sliding window variance, is the third-order mutual information strength measure, is the attenuation coefficient, is the global entropy, is the feature dimension The local entropy of is the smoothing term.

6. The data detection method based on ElasticSearch according to claim 5, characterized in that: The calculating the similarity between the target document and the candidate document based on the asymptotic weighted cosine similarity function of the dynamic weight includes: Obtaining a first feature value of each dimension in the set of target documents and a second feature value of each dimension in the set of candidate documents; Performing power transformation on the first eigenvalue and the second eigenvalue to obtain a first eigenvalue after power transformation and a second eigenvalue after power transformation; Calculating the distance between the first eigenvalue and the second eigenvalue; Normalizing the first eigenvalue after the power transformation and the second eigenvalue after the power transformation to obtain normalized eigenvalues; The similarity between the target document and the candidate document is determined based on the distance between the first eigenvalue and the second eigenvalue, the first eigenvalue after power transformation, the second eigenvalue after power transformation, and the normalized eigenvalue.

7. The data detection method based on ElasticSearch according to claim 6, characterized in that: The method of performing incremental parameter optimization by using a frequency vectorization mechanism and determining a dynamic weight of an incremental update includes: Determine a basic update amount of the dynamic weight according to a learning rate of the dynamic weight update and a gradient of a loss function of the similarity to the dynamic weight update; Performing feature calibration processing according to the feature value of the dimension corresponding to the dynamic weight to obtain a corresponding feature calibration value; The similarity normalizes the L1.5 norm of the gradient vector of the dynamic weight to obtain a gradient normalization value; Performing momentum adjustment on the change of the reversible characteristic transformation matrix to obtain a dynamic adjustment amount; A dynamic weight of the incremental update is determined according to the basic update amount, the feature calibration value, the gradient normalization value and the dynamic adjustment amount.

8. The data detection method based on ElasticSearch according to claim 7, characterized in that: The optimal computing path is calculated based on a hybrid sharding architecture, which adopts a heterogeneous computing unit sharding strategy, wherein: The CPU shard processes the CDTI pre-screening task, the GPU shard performs the AWCS similarity calculation, and low-latency communication is achieved between shards through RDMA.

9. The data detection method based on ElasticSearch according to claim 8, characterized in that: Also included is determining a sharding efficiency of the unit sharding strategy, including: Calculate the sharding efficiency impact value based on the communication delay time of sending and receiving data between the shards and the sharding cost of each shard in computing and storage; Calculate the weight difference between the master server's weight vector and the shard server's weight vector; Based on the sharding efficiency impact value and the weight difference, a sharding efficiency of the unit sharding strategy is determined.

10. A data detection system based on ElasticSearch, characterized in that: The system comprises: A data screening module is used to build a compound density tree index based on ElasticSearch, and pre-screen similar data sets based on the compound density tree index to obtain target documents and candidate documents; A data reconstruction module, used to generate a reversible feature transformation matrix by using a decomposable attention mechanism, and to perform interference-resistant reconstruction on the target document and the candidate document based on the reversible feature transformation matrix to obtain a reconstructed target document and a reconstructed candidate document; A weight determination module, used to construct an adaptive dynamic weight based on the entropy difference of each dimension between the reconstructed target document and the reconstructed candidate document; A preliminary detection module, configured to calculate the similarity between the target document and the candidate document based on the asymptotic weighted cosine similarity function of the dynamic weight; A weight update module, which is used to perform incremental parameter optimization using a frequency vectorization mechanism and determine the dynamic weights of incremental updates; The target detection module is used to dynamically select an optimal calculation path based on the incrementally updated dynamic weight, and perform repeatability detection on the target document and the candidate document again based on the optimal calculation path to obtain a repeatability detection result.

Images