A text similarity data processing method fusing statistical entropy and multiple factors

Abstract

The application relates to the technical field of electric digital data processing, and discloses a text similarity data processing method fusing statistical entropy and multiple factors, which comprises the following steps: a processor extracts a maximum common sub-string set not containing each other of first and second data sequences, calculates a local statistical entropy by calculating the square sum of sub-string lengths; the maximum common sub-string set is traversed to obtain storage address indexes thereof in the first and second data sequences, and a topological mapping vector of mapping structure displacement relationship is constructed; the total number of reverse order pairs of the topological mapping vector is calculated by using a merge sorting algorithm to generate a normalized topological dissipation index; the local statistical entropy is taken as an information carrier, the topological dissipation index is taken as a structure damping factor, and a nonlinear damping modulation operation is performed to obtain a final similarity score; and the application solves the technical problem that a block-level displacement cannot be recognized by linear scanning logic by increasing the topological entropy of the distribution of quantitative data in a storage space.

Description

Technical Field

[0001] This invention relates to a text similarity data processing method that integrates statistical entropy and multi-factor methods, belonging to the field of electronic digital data processing technology. Background Technology

[0002] In current digital data processing scenarios such as code version control, distributed database deduplication, and AIGC content compliance verification, calculating text sequence similarity is the core foundation for determining content consistency. Existing technologies generally adopt linear scanning logic algorithm models such as Levenshtein edit distance, which quantify the degree of difference by statistically analyzing the minimum number of single-character operations required to convert one string into another. These methods are based on the assumption that text differences mainly originate from local additions, deletions, and modifications at the character level. By accumulating similarity scores through character-by-character comparison, they perform stably in handling spell correction or short text matching tasks.

[0003] With the widespread application of structured data exchange formats and program source code, data content changes often manifest as the overall displacement or sequential exchange of complete data blocks, rather than random modification of discrete characters. Faced with such meaning-preserving reorganizations, traditional algorithms adhering to strict linear alignment constraints treat the shifted data block as if it were completely deleted from its original position and re-inserted into its new position. This results in computational editing costs far exceeding the actual magnitude of the change, producing extremely low similarity scores that deviate from the objective fact that the data content is substantially highly consistent. Some improved solutions attempt to weaken the sequence constraints using statistical entropy or bag-of-words models, assessing content overlap by calculating the frequency distribution of character occurrences. However, discarding positional information fails to distinguish between meaningful structural reorganizations and meaningless random discrete noise, potentially giving incorrect high similarity ratings for completely out-of-order garbled text. To address the problem of rigid linear logic, complex soft computation schemes are introduced, such as authorization... Chinese invention patent CN111737438B discloses a method, apparatus, and electronic device for text similarity data processing. This scheme constructs triplet-based labeled data and uses recurrent neural networks and attention mechanisms for information fusion training to obtain deep semantic representations of text. Such deep neural network-based methods essentially transform deterministic structural comparisons into probabilistic black-box inferences. Although this enhances the generalization matching ability of fuzzy semantics, the accuracy and system overhead are huge: neural network models are prone to overfitting semantic features, and in scenarios sensitive to physical displacement logic and character-level accuracy, such as code refactoring or log auditing, it is difficult to accurately distinguish between structural displacement and discretized noise at the topological level; the method relies on massive labeled data for pre-training, and the huge parameter scale brings huge computing power and time overhead, making it difficult to adapt to the stringent requirements of real-time performance and lightweight design for edge computing nodes or high-concurrency databases.

[0004] Therefore, the technical problem to be solved by this invention is how to overcome the limitation of linear scanning logic on block-level displacement misjudgment, achieve accurate distinction between content reorganization and discrete noise by independently quantifying the dissipation of text storage space topology, and restore the substantive similarity of text in complex data processing scenarios. Summary of the Invention

[0005] To address the problems mentioned in the background art, the technical solution of the present invention is as follows: a text similarity data processing method that integrates statistical entropy and multi-factor analysis, the method being executed by a computer processor, comprising the following steps:

[0006] Obtain the first and second data sequences to be compared. Extract a set of mutually exclusive maximal common substrings from the first and second data sequences using inclusion filtering logic. Square the character length values ​​of each maximal common substring in the set and sum them to generate a local statistical entropy representing the density of substantial overlap in content. ;

[0007] Traverse the set of maximal common substrings, obtain the first storage address index of each maximal common substring in the storage space of the first data sequence and the second storage address index in the storage space of the second data sequence, perform logical sorting on the set of maximal common substrings based on the monotonically increasing order of the first storage address index, and extract the second storage address index corresponding to each maximal common substring in the sorted order, and construct a topological mapping vector that maps the structural displacement relationship between the first data sequence and the second data sequence.

[0008] The merge sort algorithm is called to calculate the total number of inversions within the topological mapping vector. The total number of inversions is then divided by the theoretical maximum number of inversions to obtain the normalized topological dissipation index. The minimum block-level swap operation cost required to transform the first data sequence into the second data sequence is independently quantified using the topological dissipation index.

[0009] Perform nonlinear damping modulation operations with local statistical entropy For information carrier, with topological dissipation index The structural damping factor is used to calculate the final similarity score based on a pre-defined signal modulation model. The signal modulation model satisfies: ,in This is the preset structural stiffness coefficient.

[0010] Preferably, the step of constructing a topological mapping vector that maps the structural displacement relationship between the first data sequence and the second data sequence specifically includes: establishing an index mapping table in computer memory from the address space of the first data sequence to the address space of the second data sequence; locking the physical position index of each maximal common substring in the first data sequence as the reference coordinate, and mapping the physical position index of the corresponding maximal common substring in the second data sequence as the target coordinate; transforming the two-dimensional text block displacement transformation into a sorting transformation of a one-dimensional integer array, generating an integer sequence composed of the target coordinates as the topological mapping vector, wherein the order of elements in the topological mapping vector directly reflects the relative misordered state of the maximal common substring in the two data sequences.

[0011] Preferably, the filtering logic specifically includes: sorting all extracted common substrings in descending order of length; traversing the sorted common substrings and determining whether the character range covered by the current common substring is completely covered by the character range of any long substring already selected into the maximum common substring set; if the determination result is yes, then the current common substring is removed; if the determination result is no, then the current common substring is added to the maximum common substring set to ensure that there is no inclusion relationship between any two substrings in the maximum common substring set.

[0012] Preferably, the rule for calculating the theoretical maximum number of inversions is: obtain the total number of substrings contained in the set of maximally common substrings. ;Calculate the upper limit of the number of inversions in a completely reversed arrangement, the formula is as follows: Multiply The product after subtracting one is then divided by two; the processor uses this upper limit to perform a division operation on the total number in reverse order, and calculates the topological dissipation index. The range of values ​​is strictly limited to a closed interval between zero and one, where zero represents complete order preservation and one represents complete reversal.

[0013] Preferably, the step of performing nonlinear damping modulation calculation further includes introducing a global boundary constraint factor. Final similarity score After correction, the modified signal modulation model satisfies: Global boundary constraint factor The calculation logic is as follows: ,in The Levenshtein edit distance between the first and second data sequences. This is the index of the first character difference position between the first data sequence and the second data sequence. The maximum length value between the first and second data sequences. and The preset micro-correction coefficient is used as the global boundary constraint factor to superimpose the penalty weight of micro-character-level differences on the basis of the macro-structural differences represented by the topological dissipation index.

[0014] Preferably, the structural stiffness coefficient The value is automatically matched based on the type identifier of the data to be processed: in response to the data type identifier indicating that the data to be processed is program source code or script file that is sensitive to execution order, the processor will set the structural rigidity coefficient. Set to the first value to enhance the attenuation effect of the topological dissipation index on the final similarity score; in response to the data type identifier indicating that the data to be processed is a natural language document or log summary, the processor will set the structural stiffness coefficient. Set to the second value, which is less than the first value, to tolerate paragraph-level word order changes.

[0015] Preferably, after obtaining the first data sequence and the second data sequence to be compared, the method further includes: performing denoising and standardization preprocessing on the first data sequence and the second data sequence; the preprocessing includes removing preset meaningless stop word symbols, uniformly converting all letters to lowercase ASCII codes, and compressing consecutive whitespace characters into a single space character; the processor performs extraction of the maximally common substring set based on the preprocessed data sequence to eliminate the interference of format noise on the calculation of local statistical entropy.

[0016] Preferably, the step of calculating and accumulating the character length values ​​of each maximal common substring in the set of maximal common substrings specifically includes: traversing the set of maximal common substrings and reading the character length value of each maximal common substring; performing a square operation on each character length value to obtain the energy value of each substring; accumulating the energy values ​​of all substrings to obtain the total structural energy, and dividing the total structural energy by the shorter sequence length in the first data sequence or the second data sequence to obtain the local statistical entropy. The weights of long continuous substrings are nonlinearly amplified using the square operation.

[0017] Preferably, the method is applied to code version control scenarios. The steps of calculating the total number of inversions specifically include: using a binary indexed tree (BIT) structure or a binary indexed tree as a counting container; inserting each element into the BIT structure in the order of appearance of the elements in the topological mapping vector; and during each insertion operation, querying the number of existing elements in the BIT structure that are greater than the value of the currently inserted element, and accumulating this number to the total number of inversions, thereby quantifying the number of code block movement operations in logarithmic time complexity.

[0018] Preferably, the method also includes a final similarity score. Generate data processing instructions: Calculate the final similarity score. The data sequence is compared with a preset reuse threshold. When the final similarity score is higher than the reuse threshold, the first data sequence and the second data sequence are determined to be substantially duplicated. A deduplication storage instruction is generated and executed for the second data sequence, and only a reference pointer to the first data sequence is retained in the database. When the final similarity score is lower than the reuse threshold, the second data sequence is determined to be new content. An independent index building and full storage instruction is generated and executed for the second data sequence.

[0019] Compared with the prior art, the beneficial effects of the present invention are:

[0020] 1. In statistical entropy and multi-factor text similarity, a dual measurement mechanism of maximum common substring energy aggregation and position index topological dissipation is constructed to solve the technical problem that existing technologies cannot identify block-level topological distortion due to linear scanning logic. The content energy is calculated by using the sum of squares of the length of the maximum common substring, anchoring the actual content overlap density of the text sequence, establishing a position index mapping vector and calculating the total number of inversion pairs, independently quantifying the degree of topological disorder of text blocks in storage space, and decoupling the physical level of content integrity and structural order. This allows the computer system to distinguish between data recombination and random discrete noise based on the objective distribution state of data in memory. When dealing with conditions with a large number of block-level displacements, such as code refactoring and log format changes, it avoids the distortion of similarity calculation caused by traditional edit distance algorithms misjudging position swaps as character replacements.

[0021] 2. A damped modulation model based on topological work nonlinear signal is established to overcome the lack of adaptability of traditional linear weighted summation algorithms when dealing with complex text variations. It abandons the simple numerical superposition of features of different dimensions and regards the basic similarity determined by content statistical entropy as information potential energy. Based on the merge sort algorithm, the inverse pair ratio is derived as a topological dissipation index, which is used as a damping factor to multiplicatively attenuate the information potential energy. The mechanism follows the physical law of signal-to-noise ratio attenuation and automatically adjusts the similarity score penalty according to the actual degree of text structure disorder. When the text content is highly consistent but the word order is reversed, a reasonable score is retained. When the content is fragmented and scattered, a cascading score attenuation is triggered. There is no need for frequent manual adjustment of weight coefficients, so as to achieve robust calculation of the true similarity of text.

[0022] 3. By utilizing integer index array mapping and sorting operations, high-dimensional text structural features can be captured with low computational power. This solves the engineering problem that complex semantic analysis models rely on expensive hardware resources and are difficult to meet real-time requirements. Text structural differences are transformed into memory address index vector representations. The counting logic of basic computer sorting algorithms is reused to obtain the topological inversion number. This maintains high sensitivity to text topology while keeping the time complexity of the core algorithm at the logarithmic linear level. Based on basic electronic digital data processing logic, there is no need to build large floating-point tensors or rely on external pre-trained models. This allows the solution to be directly deployed on edge computing nodes with limited computing power or high-concurrency real-time search backends, ensuring the response speed and operational stability of the data processing system when performing large-scale text comparison tasks. Attached Figure Description

[0023] Figure 1 This is a flowchart illustrating the data processing logic of the present invention, which integrates statistical entropy and topological mapping.

[0024] Figure 2 This is a curve showing the system performance sensitivity analysis under the key correction coefficients of this invention;

[0025] Figure 3 This is a diagram showing the overall system architecture of the present invention, which integrates adaptive evolution and intelligent routing. Detailed Implementation

[0026] The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0027] This invention provides a text similarity data processing method that integrates statistical entropy and multi-factor processing. Executed by an electronic computing device including a processor and memory, the method achieves deep comparison and consistency verification of the content of electronic digital data sequences through physical-level address mapping and nonlinear signal modulation mechanisms. The data processing system transforms the input text data stream into feature vectors in memory space and sequentially performs energy aggregation operations on maximally common substrings, topological dissipation evaluation based on address indexes, and signal modulation calculations coupled with multi-factor processing. Finally, it outputs a normalized similarity score to drive downstream database deduplication or version control instructions. The processor receives the first and second data sequences to be processed and performs denoising and standardization preprocessing, including ASCII code standardization, invalid symbol removal, and whitespace character compression, thereby transforming the unstructured original text into a standard data stream suitable for computation.

[0028] To address the technical challenge of linear scanning algorithms in distinguishing between meaningful structural recombination and meaningless discrete noise, the processor employs a local statistical entropy calculation procedure based on the energy aggregation of maximal common substrings. The processor traverses the first and second data sequences, identifies all matching character segments between them, and eliminates substrings completely covered by longer matching segments based on inclusion filtering logic, thereby generating a set of mutually exclusive maximal common substrings. The processor reads the set sequentially. The character length value of each maximal common substring. The structural energy of each substring is obtained by squaring each length value and summing the structural energies of all substrings. Finally, the total structural energy is divided by the length of the shorter of the two data sequences. Generate local statistical entropy representing the density of overlap in the content. This computational path utilizes the non-linear amplification property of square operations to make the contribution weight of long, continuous data blocks to similarity higher than that of discrete, short data blocks. This allows for the numerical anchoring of the substantial overlap of text content. The process of constructing a maximal common substring set eliminates the uncertainty of computational results caused by the randomness of memory read order. It includes a filtering logic that executes a strict double-keyword sorting protocol. The processor sorts common substrings in descending order based on their length. When substrings of the same length exist, the processor reads the starting storage address index in the first data sequence and sorts them in ascending order based on the index value. An address-first arbitration mechanism ensures that when processing data streams containing a large number of repetitive patterns, the maximal common substring set is accurately determined. and subsequent topology mapping vectors The generated path is unique and reproducible.

[0029] To address the technical flaw of traditional edit distance algorithms that misinterpret the overall displacement of data blocks as large-scale character replacements, the processor executes a topological dissipation evaluation procedure based on memory address index mapping. The processor traverses the aforementioned set of maximal common substrings. Obtain the first storage address index of each substring in the first data sequence storage space. and the second storage address index in the second data sequence storage space The processor indexes the first memory address. Based on the monotonically increasing order, the substring objects in the set are logically rearranged, and the second storage address index corresponding to each maximal common substring is extracted sequentially according to the sorted order. This allows the construction of a topological mapping vector that reflects the relative positional transformations of data blocks. The processor calls a merge sort algorithm or a binary indexed tree (BIT) structure to map the topology vector. Perform an inversion counting operation to calculate the total number of inversion pairs in the vector. And divide the total number of inversions by the theoretical maximum number of inversions. ,in The calculation is done through the formula Obtain, here The operation outputs a normalized topological dissipation index, which represents the total number of maximal common substrings. As a dimensionless physical quantity, this index quantifies the minimum block-level swapping cost required for a data sequence to transition from the first state to the second state, thus intuitively reflecting the degree of topological entropy increase in the data distribution in storage space.

[0030] Based on this, the processor executes comprehensive calculation instructions based on the signal-damping modulation model to achieve a decoupled measurement of content integrity and structural order. The processor reads the structural stiffness coefficient preset in the system register. This coefficient is adaptively matched based on the type identifier of the data to be processed. For program source code data that is highly sensitive to the execution order, the system automatically increases the coefficient. The value is adjusted to increase the penalty for out-of-order processing, while decreasing it for natural language documents. The value is set to tolerate reasonable paragraph swapping, and the processor simultaneously calculates the Levenshtein edit distance between the first and second data sequences. First character difference position index and maximum sequence length Based on this, a global boundary constraint factor is generated. Its computational logic satisfies ,in and The preset micro-correction coefficient is for the structural stiffness coefficient. and minor correction factor and Numerical determination is achieved by using a grid search calibration procedure based on historical benchmark datasets. The processor calls a calibration sample set containing preset similarity ground truth values ​​and performs a full permutation traversal calculation in the preset parameter space with a step size of 0.05. The F1 score corresponding to each parameter combination is recorded, which is the harmonic mean of precision and recall. The parameter combination that makes the F1 score reach the global maximum value is selected and written into the configuration register as a fixed constant at runtime, so that the parameter setting is anchored to the statistical distribution characteristics of the sample set rather than subjective experience values.

[0031] Ultimately, the processor uses the nonlinear damping modulation formula. The final similarity score is calculated. In this model, local statistical entropy As the base carrier carrying content information, the topological dissipation index This serves as a structural damping factor, applying multiplicative attenuation to the carrier signal. This ensures that the calculation results can both identify substantial repetitive content with only block-level displacement and effectively suppress heavily discretized noise data. The processor will then calculate the final similarity score. The similarity of the data is compared with a preset reuse threshold. When the score is higher than the reuse threshold, a deduplication storage instruction is generated, and only a reference pointer to the first data sequence is retained in the database. When the score is lower than the reuse threshold, a full storage instruction is generated, and an independent index is built for the second data sequence. This method avoids the high overhead of floating-point matrix operations by using pure integer operations and logical sorting, enabling efficient similarity determination of large-scale text data on a general computing platform.

[0032] Example 1: In engineering application scenarios of large-scale source code version control and storage optimization, the system faces the challenge of deduplication identification when dealing with two program files that are substantially identical in content but have swapped function module positions. When the original code file represented by the first data sequence is reconstructed, the physical storage order of the two core functional function blocks in the second data sequence is completely swapped. Conventional difference comparison algorithms based on linear scanning strictly follow index alignment constraints and judge such block-level displacements as the complete deletion of the original data block and the re-insertion of data at the new position. This causes the calculated difference cost to exceed the reuse judgment threshold, triggering the system to perform redundant full storage operations.

[0033] After receiving the aforementioned data sequence, the processor of this invention executes a maximal common substring extraction and energy aggregation procedure, identifies two maximal common substrings whose byte-level content remains continuous despite positional changes, and sets their character length values. Perform a square operation to make the structural energy of these two long substrings equal to the total structural energy. The proportion in the middle exhibits a non-linear amplification, generating a high local statistical entropy close to the value of one. This value physically anchors the actual integrity of the file content; the processor constructs a topological mapping vector that reflects the relative positional relationship between the two maximally common substrings in the memory address space. At this point, the elements within the vector are arranged in a simple reverse order. The processor calls the merge sort algorithm to calculate the total number of non-zero inversion pairs. Based on this, a topology dissipation index representing the topology cost of this reconstruction operation is generated. The processor is based on the pre-defined structural rigidity coefficient for the code data. Perform nonlinear damping modulation calculations, utilizing right Applying a multiplicative penalty that conforms to the signal-to-noise ratio attenuation rule, the final output similarity score is obtained. It reflects the structural entropy increase caused by word order changes, and remains above the reuse judgment threshold because it does not touch the content substance. It accurately drives the database to execute deduplication storage instructions, retaining only reference pointers instead of repeatedly storing redundant data blocks, and achieving decoupling judgment of content integrity and structural order at the physical level.

[0034] Example 2: To verify the effectiveness and anti-interference capability of the proposed method for fusing statistical entropy and multi-factor text similarity processing in a real complex data environment, a large-scale code refactoring simulation test platform based on a Linux server cluster was constructed. This platform simulates a distributed version control system containing millions of code files, with the data source being a publicly available dataset from GitHub open-source projects. Noise interference with specific statistical characteristics was actively injected to simulate non-standard operations in actual development. In each commit record of the original code repository, the test platform randomly selected 5% of the files as perturbation samples, applying two typical engineering perturbations to these samples: one is structural reorganization, which randomly swaps the physical order of function modules or class definitions while maintaining their internal logic; the other is discretization noise, which randomly inserts meaningless comment characters or blank lines between code lines. The signal-to-noise ratio was set to 15dB. This mixed perturbation environment aims to recreate the extreme working condition where meaningful reorganization and malicious tampering coexist, thereby stress-testing the effectiveness of the algorithm.

[0035] The experiment selected code deduplication accuracy as the core evaluation index and compared the method of this invention (experimental group) with the traditional algorithm based on Levenshtein edit distance (control group A) and the improved algorithm based on longest common subsequence (LCS) (control group B). The structural rigidity coefficient, a key parameter, was also compared. The settings follow a trade-off logic between signal-to-noise ratio and structural sensitivity: considering the sensitivity of code data to execution order, while also tolerating module-level overall swapping, and after prior gradient calibration experiments, [the following parameters are considered]. The value is set to 0.75, and this value is determined based on: when At times, the system's penalty for out-of-order code is insufficient, resulting in poorly structured code receiving excessively high scores; while when At that time, the tolerance for module swapping was too low, which would inadvertently damage legitimate reconstruction operations. Therefore, 0.75 was determined as the optimal operating point for balancing content consistency and structural order in the current mixed disturbance environment. During the experiment, the processor read each group of samples in sequence, calculated the similarity score, and determined whether it was duplicate content based on the score result. The table below shows the key intermediate data and final judgment results when processing the above mixed disturbance samples. The original similarity in the table refers to the preliminary score without the nonlinear damping modulation of this invention. The topological dissipation index is a structural metric unique to this invention. The final judgment is the binary decision made by the system based on the preset threshold (0.85). See Table 1.

[0036] Table 1: Example of Comparison of Processing Results of Different Algorithms under Mixed Perturbation Environment

[0037]

[0038] As can be observed from the data in Table 1, when faced with typical code refactoring scenarios such as pure module swapping and module swapping plus slight comment noise (samples 1 and 2), the similarity scores given by control group A and control group B are both below 0.5, causing the system to misjudge them as non-repeating new content, resulting in unnecessary full storage. In contrast, although the method of this invention detects drastic changes in physical location (reflected in non-zero similarity scores), the method of this invention can achieve similar results. (in terms of value), but because it is based on the aggregation of maximal common substring energy. Accurately anchor the substantive overlap of content (maintaining it above 0.9), and through reasonable... The value moderately dampes the topological dissipation, so that the final score It remained robustly above the judgment threshold of 0.85, successfully identifying substantial duplicates. Conversely, when faced with severe discretization tampering (sample 3), although some characters still matched, the extremely high topological dissipation index ( This triggers a cascading score decay, reducing the final score to 0.12 and effectively filtering out invalid noise.

[0039] Example 3: This example combines Figures 1 to 3 This paper describes a text similarity data processing method that integrates statistical entropy and multi-factor methods, such as... Figure 1 As shown, the input consists of the first and second data sequences to be compared to obtain the original text data stream. The process then proceeds to the denoising and standardization preprocessing stage, specifically removing stop words, unifying ASCII codes, and compressing whitespace characters. This process is performed in parallel by two branches. The left branch extracts the maximal common substring set based on inclusion filtering logic, and then generates a local statistical entropy that anchors the content density by performing a square aggregation operation on the substring length. The right-hand branch establishes relative positional relationships by obtaining memory address indices to construct a topological mapping vector, and uses merge sort to calculate the total number of normalized inversions to obtain the topological dissipation index. The calculation results from the two branches mentioned above converge to the nonlinear damping modulation calculation module, which also receives the structural stiffness coefficient automatically matched based on the data type. And the global boundary constraint factor calculated based on edit distance and difference location correction. Using local statistical entropy as the carrier and topological dissipation index as the damping factor, modulation calculation is performed, and the final similarity score, representing a comprehensive index of content completeness and structural orderliness, is output. The system then generates data processing instructions based on the comparison between the score and the threshold, pointing to the operation paths for either deduplication storage or independent index construction.

[0040] like Figure 2 As shown, the horizontal axis represents the parameter value, and the vertical axis represents the F1 score. The graph contains three different variation curves, with the solid line representing the penalty coefficient at a fixed edit distance. The penalty coefficient is 0.4 and the position difference is a factor of 0.4. With a value of 0.25, the F1 score varies with the structural stiffness coefficient. The trajectory of change, the dashed line represents the fixed path. It is 0.8 and When the value is 0.25, the F1 score varies with The dotted line corresponds to a fixed change. It is 0.8 and When the value is 0.4, the F1 score varies with The three curves clearly reflect the differences in sensitivity of each correction coefficient to the accuracy of the final similarity determination and the distribution of peak values ​​in different value ranges.

[0041] like Figure 3 As shown, the system's left side features a multi-source heterogeneous data pool responsible for receiving code repository streams and document log streams and performing standardized stream injection. The data streams enter a similarity fusion calculation platform located at the center, which has a built-in high-performance computing cluster and ultra-fast feature cache. This platform is responsible for performing entropy weight analysis, topology mapping, and damping modulation calculations. The top of the system has an adaptive parameter evolution chamber that dynamically generates and pushes optimal configuration parameters using offline calibration and golden sample data. , and The scoring results output by the computing platform are transmitted to the intelligent routing module on the right. This module distributes the data according to the scoring results, pointing duplicate content to the deduplication index library for reference storage, and pointing new content to the full content library for entity storage.

[0042] Example 4: This example aims to provide a systematic offline optimization parameter-finding procedure to address the lack of theoretical support and engineering basis for similarity calculation parameter configuration in specific business scenarios. This ensures that the system achieves optimal performance from the initial deployment stage. In business scenarios oriented towards financial contract risk assessment, due to the extremely high sensitivity of contract text to the order of clauses and the extremely small margin for error, it is necessary to determine the optimal structural rigidity coefficient for this scenario. Edit distance penalty coefficient and position difference penalty coefficient To this end, a golden dataset was constructed, containing no fewer than 2000 pairs of financial contract sample pairs with accurately labeled similarity values ​​by human experts. This dataset covers various typical business variation patterns, such as contract revisions, clause additions and deletions, and paragraph reorganizations. The F1 score (the harmonic mean of precision and recall) was defined as the overall performance index of the system as the objective function for optimization. A parameter optimization algorithm was then initiated, and the following parameters were set... , , The initial search spaces are [0.1,1.0], [0.1,0.8], and [0.1,0.5], respectively, with a step size of 0.05.

[0043] During the traversal search, the processor performs a complete similarity calculation process on the gold dataset for each parameter combination, counts the number of misclassified samples and calculates the corresponding F1 score. By constructing a parameter-performance response surface, the sensitivity of each parameter to system performance and their mutual coupling effects are analyzed. Experimental results show that as... As the value increases, the system's ability to identify structural variations improves, but when... When the value exceeds 0.85, the misjudgment rate for reasonable revisions increases sharply; When the value is around 0.4, the system's sensitivity to minute character-level differences reaches equilibrium. Ultimately, the optimal parameter combination for this financial contract evaluation scenario is determined to be... , , The corresponding F1 score reached 0.96. Through this standardized offline optimization procedure, parameter setting is transformed from empiricism to data-driven engineering decision-making, ensuring the system's accurate adaptation to specific vertical fields.

[0044] Example 5: This example aims to address text similarity processing in large-scale real-time data stream environments. By introducing a multi-level index caching and dynamic threshold adjustment mechanism, it solves the computational efficiency bottleneck and misjudgment risk in high-concurrency scenarios, ensuring system stability and response speed under extreme loads. In large-scale real-time data stream processing scenarios for internet content moderation, the system needs to handle tens of thousands of text submission requests per second, including a large number of short text comments, bullet comments, and long text articles. Conventional similarity calculation methods often suffer from excessive computational overhead due to full comparison when processing such high-concurrency data streams, leading to a sharp increase in system latency, or the inability of fixed thresholds to adapt to texts of different lengths. Due to the characteristics of text, a large number of misjudgments occur. To address this challenge, this embodiment adopts a hierarchical multi-level index caching architecture. A first-level index based on Locality Sensitive Hash (LSH) is built in system memory to quickly filter out the vast majority of obviously dissimilar text data. Only the candidate set with potentially high similarity is sent to the subsequent accurate calculation process. A second-level index is built using a high-performance caching database such as Redis to store recently frequently occurring maximally common substrings and their corresponding statistical entropy values ​​to reduce the overhead of repeated calculations. For candidate text pairs filtered by the first-level index, the system calls the similarity calculation method of the aforementioned embodiment that combines statistical entropy and multi-factor similarity for accurate comparison.

[0045] To further enhance the system's adaptability, this embodiment introduces a dynamic threshold adjustment mechanism. Based on the current system load and the length characteristics of the text to be processed, the reuse threshold for similarity judgment is adjusted in real time. When the system load is low and long texts are being processed, the threshold is appropriately increased to ensure the rigor of the judgment. When the system load is extremely high or short text streams are being processed, the filtering threshold of the first-level index is dynamically reduced to prioritize the system's real-time response capability. Secondary verification is performed through subsequent precise calculations. In addition, the system integrates a sliding statistics module based on time windows to monitor the distribution of text similarity in real time over different time periods. Once an abnormal similarity peak is detected, such as a surge in similar comments caused by a sudden hot event, an early warning mechanism is triggered and the caching strategy is automatically adjusted to cope with the impact of sudden traffic. Through the synergistic effect of the above-mentioned multi-level index caching and dynamic threshold adjustment mechanism, this embodiment improves the system's processing efficiency and stability in a large-scale real-time data stream environment while ensuring high accuracy. Under a stress test with 50,000 requests per second, the system's average response time is controlled at the millisecond level, and the false judgment rate is always maintained at an extremely low level, effectively solving the text similarity processing problem in high-concurrency scenarios.

[0046] Example 6: In the offline preprocessing stage for similarity verification of unstructured long documents, to ensure the accuracy and consistency of the core data table, an offline calibration and data filling procedure based on massive sample learning is executed. The system accesses a full corpus containing multiple document types and extracts a representative standard sample set from it. This set covers various document categories with different structural features, such as legal documents, technical reports, and academic papers. For each standard sample, the system calls the maximal common substring extraction module to generate a complete set of maximal common substrings specific to that sample and calculates the occurrence frequency and distribution density of each substring. Statistical methods are used to perform regression analysis on the substring distribution data of the full sample to determine the structural entropy benchmark value range under different document types. Based on this, the system constructs and fills a globally shared structural feature benchmark database, which records in detail the local statistical entropy threshold range and topological dissipation index distribution characteristics of various documents at different similarity levels. Through this offline calibration process, the system solidifies the abstract similarity judgment logic into a queryable quantitative standard, so that in the subsequent online processing stage, the benchmark database can be directly called for fast and accurate threshold matching, eliminating the calculation uncertainty caused by sample differences.

[0047] To ensure the system's adaptability and accuracy in different deployment environments, a pre-deployment calibration / debugging procedure is implemented. Before deploying the similarity processing system to a new target server or connecting to a new business data stream, an automated environment self-check and parameter fine-tuning process is initiated. The system reads a preset calibration sample package containing test document pairs pre-labeled with true similarity values. The processor sequentially performs a complete similarity calculation process on these test document pairs and compares the calculation results with the labeled true values, calculating the current false positive rate and false negative rate. Based on the comparison results, the system automatically fine-tunes the structural rigidity coefficient using the gradient descent algorithm. Correction coefficients in global boundary constraint factors , This calibration procedure ensures that the system can maintain stable calculation accuracy even when faced with differences in hardware performance or deviations in the distribution of specific business data, until the accuracy of the system's judgment in the current environment reaches the preset acceptance standard. This enables seamless adaptation and performance optimization from general models to specific scenarios.

[0048] Example 7: This example aims to provide a standardized pre-deployment calibration and engineering parameter calibration procedure to address the differentiated needs of various application scenarios and data types. This solves the system performance fluctuation problem caused by fixed parameter settings. In scenarios involving new system deployments or changes in data types, offline benchmark testing based on the Golden Dataset is performed. This Golden Dataset consists of no fewer than 1000 pairs of text pairs with manually precisely labeled ground truth similarity values, covering various typical variation patterns expected in the target business scenario, including plain text reconstruction, character noise introduced by OCR recognition errors, and malicious synonym replacement. Using this dataset, the system initiates a parameter optimization script, traversing the structural rigidity coefficients... Edit distance penalty coefficient and position difference penalty coefficient Given a preset value space, for each parameter combination, the system calculates the F1 score (harmonic mean of precision and recall) on the dataset and plots a parameter-performance response surface. By finding the global optimum on the surface, the optimal parameter configuration for the business scenario is determined. For example, when handling legal contract comparison tasks that are extremely sensitive to logical order, the optimization result usually indicates... The value should be set in the high range of 0.8 to 0.9 to severely punish arbitrary changes to the order of terms; and when dealing with short text clustering tasks on social media, The value could be optimally set between 0.4 and 0.5 to tolerate loose word order resulting from colloquial expressions.

[0049] After the system is deployed and goes live, the built-in confidence monitoring module tracks the distribution histogram of similarity scores in real time. If the score distribution exhibits an abnormal bimodal collapse phenomenon, with the vast majority of scores clustered at either 0 or 1, lacking intermediate transitions, it suggests that the parameter settings are too aggressive or conservative, or that there is an accumulation of intermediate states with a large number of scores hovering near the judgment threshold, leading to decision-making difficulties. In this case, the system will automatically trigger the online fine-tuning logic, adopting a small-step, rapid-adjustment strategy to adjust the score distribution. , , Perform minor adjustments (step size not exceeding 0.05) and compare the change in misclassification rate on a small-volume grayscale test set before and after the adjustment. The new parameter combination will only be fully applied when the misclassification rate shows a statistically significant decrease. For extremely long texts occasionally appearing in the data stream, such as log files exceeding 100,000 characters, to avoid computation timeouts, the procedure stipulates that when the text length exceeds a preset threshold... When the time comes, it automatically switches to the block parallel computing mode, divides the long text into several overlapping data blocks and calculates the similarity of each block. Finally, it obtains the overall score by weighted average, thereby ensuring the timeliness of response while maintaining the reliability of the calculation results. Through the above-mentioned standardized calibration and fine-tuning procedures, it is ensured that the present invention is always in the optimal working state when facing the ever-changing engineering reality.

[0050] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0051] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A text similarity data processing method fusing statistical entropy and multiple factors, characterized in that, The method is executed by a computer processor and includes the following steps: Obtaining a first data sequence and a second data sequence to be compared, extracting a set of maximal common substrings not containing each other from the first data sequence and the second data sequence through containing screening logic, performing square operation on a character length value of each maximal common substring in the set of maximal common substrings and accumulating, and generating local statistical entropy representing content substantial coincidence density ; Traverse the set of maximum common subsequences, respectively obtain the first storage address index of each maximum common subsequence in the first data sequence storage space and the second storage address index in the second data sequence storage space, perform logical sorting on the set of maximum common subsequences based on the monotonically increasing order of the first storage address index, and sequentially extract the second storage address index corresponding to each maximum common subsequence in the sorted order, and construct a topological mapping vector mapping the structural displacement relationship between the first data sequence and the second data sequence; calling a merge sort algorithm to calculate the total number of reverse ordered pairs in the topological mapping vector, dividing the total number of reverse ordered pairs by the theoretical maximum number of reverse orders to obtain a normalized topological dissipation index independently quantifying the minimum block-level exchange operation cost required for transforming the first data sequence into the second data sequence using the topological dissipation index; performing a nonlinear damping modulation operation to locally statistically entropy for an information carrier, to topological dissipation index for a structural damping factor, calculating a final similarity score according to a preset signal modulation model , wherein the step of performing a nonlinear damping modulation operation further comprises introducing a global boundary constraint factor , wherein the final similarity score is modified, and the modified signal modulation model satisfies: ; the calculation logic of the global boundary constraint factor is: , wherein is the Levenshtein edit distance between the first data sequence and the second data sequence, is the first character difference position index of the first data sequence and the second data sequence, is the maximum length value in the first data sequence and the second data sequence, and is a preset trace correction coefficient, and the global boundary constraint factor is used to superimpose the penalty weight of the micro character level difference on the basis of the macro structural difference represented by the topological dissipation index. The calculation rule of the theoretical maximum reverse order number is: obtaining the total number of the sub-strings contained in the maximum common sub-string set ; calculating the upper limit of the reverse order pair number in the completely reverse order arrangement state, the calculation formula is times one after the product is divided by two; the processor performs a division operation on the total number of reverse order pairs by using the upper limit, so that the value range of the topological dissipation index is strictly limited in the closed interval of zero to one, wherein zero represents complete order and one represents complete reverse order, is a preset structural rigidity coefficient. 2.The text similarity data processing method of fusing statistical entropy and multiple factors according to claim 1, characterized in that, The step of constructing the topological mapping vector mapping the structural displacement relationship between the first data sequence and the second data sequence specifically includes: establishing an index mapping relationship table from the address space of the first data sequence to the address space of the second data sequence in the computer memory; locking the physical position index of each maximum common subsequence in the first data sequence as a reference coordinate, and mapping the physical position index of the corresponding maximum common subsequence in the second data sequence as a target coordinate; convert the two-dimensional text block displacement transformation into a one-dimensional integer array sorting transformation, generate an integer sequence composed of target coordinates as a topological mapping vector, and the order of elements in the topological mapping vector directly reflects the relative position disorder state of the maximum common subsequence in the two data sequences.

3. The data processing method for text similarity based on fusion of statistical entropy and multi-factors according to claim 1, characterized in that, The filtering logic specifically includes: arranging all extracted common subsequences in descending order of length; traversing the arranged common subsequences, determining whether the character interval covered by the current common subsequence is completely covered by the character interval of any long subsequence selected into the set of maximum common subsequences; if the result of the determination is yes, the current common subsequence is excluded; if the result of the determination is no, the current common subsequence is added to the set of maximum common subsequences, ensuring that there is no inclusion relationship between any two subsequences in the set of maximum common subsequences.

4. The text similarity data processing method of claim 1, wherein structural rigidity coefficient The value of the structural rigidity coefficient is automatically matched according to the type identification of the data to be processed: in response to the data type identification indicating that the data to be processed is a program source code or a script file sensitive to execution order, the processor sets the structural rigidity coefficient to a first value to enhance the attenuation effect of the topological dissipation index on the final similarity score; in response to the data type identification indicating that the data to be processed is a natural language document or a log summary, the processor sets the structural rigidity coefficient to a second value, which is less than the first value, to tolerate paragraph-level sequence exchange. The value of the structural rigidity coefficient is automatically matched according to the type identification of the data to be processed: in response to the data type identification indicating that the data to be processed is a program source code or a script file sensitive to execution order, the processor sets the structural rigidity coefficient to a first value to enhance the attenuation effect of the topological dissipation index on the final similarity score; in response to the data type identification indicating that the data to be processed is a natural language document or a log summary, the processor sets the structural rigidity coefficient to a second value, which is less than the first value, to tolerate paragraph-level sequence exchange. The value of the structural rigidity coefficient is automatically matched according to the type identification of the data to be processed: in response to the data type identification indicating that the data to be processed is a program 5. The data processing method for text similarity based on fusion of statistical entropy and multi-factors according to claim 1, characterized in that, After the step of obtaining the first data sequence and the second data sequence to be compared, the method further includes: performing denoising and standardization preprocessing on the first data sequence and the second data sequence; the preprocessing includes excluding preset meaningless stop word symbols, uniformly converting all letters to lowercase ASCII codes, and compressing consecutive white space characters into a single space character; the processor performs extraction of the set of maximum common subsequences based on the preprocessed data sequence to eliminate the interference of format noise on local statistical entropy calculation.

6. The text similarity data processing method of claim 1, wherein The step of performing square operation and accumulation on the character length values of each maximal common substring in the maximal common substring set comprises: traversing the maximal common substring set, reading the character length value of each maximal common substring; performing square operation on each character length value to obtain the energy value of each substring; accumulating the energy values of all substrings to obtain the total structure energy, and dividing the total structure energy by the length of the shorter sequence in the first data sequence or the second data sequence to obtain the local statistical entropy The weight of the long continuous substring is nonlinearly amplified by using square operation.

7. The data processing method for text similarity based on fusion of statistical entropy and multi-factors according to claim 1, characterized in that, The method is applied to a code version control scenario, and the step of calculating the total number of reverse order pairs specifically includes: using a tree array structure or a binary index tree as a counting container; according to the order of appearance of the elements in the topological mapping vector, sequentially insert each element into the tree array structure; in each insertion operation, query the number of existing elements in the tree array structure that are greater than the value of the current inserted element, and add the number to the total number of reverse order pairs, thereby completing the quantification of the number of code block movement operations in logarithmic time complexity.

8. The text similarity data processing method of claim 1, wherein, The method further comprises determining a final similarity score based on the similarity scores generating a data processing instruction based on the final similarity score comparing the final similarity score with a preset multiplexing determination threshold; when the final similarity score is higher than the multiplexing determination threshold, determining that the first data sequence and the second data sequence are substantially repetitive content, generating and executing a deduplication storage instruction for the second data sequence, and retaining only a reference pointer pointing to the first data sequence in the database; when the final similarity score is lower than the multiplexing determination threshold, determining that the second data sequence is new content, and generating and executing an independent index construction and full storage instruction for the second data sequence.

Images