Text similarity data processing method fusing statistical entropy and multiple factors

Abstract

The invention relates to the technical field of electrical digital data processing, and discloses a statistical entropy and multi-factor fused text similarity data processing method, which comprises the following steps that: a processor extracts substring sets which do not contain maximum common values of a first data sequence and a second data sequence, and calculates the quadratic sum of the lengths of substrings to generate local statistical entropy; traversing the maximum common substring set to obtain storage address indexes of the maximum common substring set in the first data sequence memory space and the second data sequence memory space, and constructing a topological mapping vector of a mapping structure displacement relationship; calculating the total number of inverted pairs of the topology mapping vector by using a merge sorting algorithm, and generating a normalized topology dissipation index; and by taking the local statistical entropy as an information carrier and taking the topological dissipation index as a structural damping factor, executing nonlinear damping modulation operation to obtain a final similarity score, and solving the technical problem that the block-level displacement cannot be identified by linear scanning logic by quantizing topological entropy increase of data distributed in a storage space.

Description

TECHNICAL FIELD

[0001] The present application relates to a fusion of statistical entropy and multi-factor text similarity data processing method, belonging to the technical field of electronic digital data processing. BACKGROUND

[0002] In current code version control, distributed database deduplication, and AIGC content compliance verification, etc. in the field of electronic digital data processing, calculating the similarity of text sequences is the core basis for determining content consistency. The existing technology generally uses Levenshtein edit distance and other linear scanning logic algorithm models to quantify the difference in the number of single character operations required to convert a string into another string by inserting, deleting, or replacing. This method is based on the assumption that text differences mainly result from character-level local additions, deletions, and modifications. By comparing characters one by one and accumulating similarity scores, it can handle spelling correction or short text matching tasks stably.

[0003] With the large-scale application of structured data exchange formats and program source codes, data content changes often manifest as complete data block displacement or sequential exchange, rather than random modification of discrete characters. In the face of such semantic-preserving reorganization, traditional algorithms that follow strict linear alignment constraints treat displaced data blocks as complete deletions from the original position and reinsertions into the new position, resulting in a much higher cost of editing operations than the actual variation, producing extremely low similarity scores that deviate from the objective fact of high consistency in data content. Some improved solutions attempt to weaken the sequential constraints by using statistical entropy or bag-of-words models to assess the degree of content overlap by calculating the frequency distribution of characters, but discarding position information cannot distinguish meaningful structural reorganization from meaningless random discrete noise. For completely shuffled and generated random text, it may give a false high similarity evaluation. To solve the problem of linear logic rigidity, complex soft computing solutions are introduced, such as the granted patent CN111737438B, which discloses a text similarity data processing method, device, and electronic equipment. This solution constructs triadic form labeled data, uses recurrent neural networks and attention mechanism information fusion training to obtain deep semantic representations of text. This deep neural network-based method essentially converts deterministic structural comparison into probabilistic black box inference, which enhances fuzzy semantic generalization and matching capabilities, but the precision and system overhead are huge: neural network models are prone to overfitting semantic features, making it difficult to accurately distinguish between structural displacement and discrete noise from a topological perspective in sensitive scenarios such as code restructuring or log auditing that require physical displacement logic and character-level precision. The method relies on massive labeled data pre-training, and the large parameter size brings huge computational power and time overhead, making it difficult to meet the real-time and lightweight requirements of edge computing nodes or high-concurrency databases.

[0004] Therefore, how to break through the linear scanning logic to the block level displacement misjudgment limit, dissipate through independent quantization of text storage space topology, realize content reorganization and precise distinction of discrete noise, restore the text substantive similarity in complex data processing scene, become the technical problem to be solved by the present application. SUMMARY

[0005] To solve the problems raised in the background art, the technical solution of the present application is as follows: a text similarity data processing method fusing statistical entropy and multiple factors, which is executed by a computer processor and includes the following steps: Obtain the first data sequence and the second data sequence to be compared, extract a set of maximal common substrings that do not contain each other from the first data sequence and the second data sequence through containing screening logic, perform square operation and accumulation on the character length value of each maximal common substring in the set of maximal common substrings to generate local statistical entropy representing the density of content substantive coincidence ; Traverse the set of maximal common substrings, respectively obtain the first storage address index of each maximal common substring in the first data sequence storage space and the second storage address index in the second data sequence storage space, perform logical sorting on the set of maximal common substrings based on the monotone increasing order of the first storage address index, and extract the second storage address index corresponding to each maximal common substring in turn according to the sorted order to construct a topological mapping vector mapping the structural displacement relationship between the first data sequence and the second data sequence; Call the merge sort algorithm to calculate the total number of reverse order pairs inside the topological mapping vector, divide the total number of reverse order pairs by the theoretical maximum reverse order number to obtain the normalized topological dissipation index , and use the topological dissipation index to independently quantify the minimum block-level exchange operation cost required for the first data sequence to be transformed into the second data sequence; Perform nonlinear damping modulation operation to take the local statistical entropy as the information carrier, take the topological dissipation index as the structural damping factor, and calculate the final similarity score according to the preset signal modulation model , the signal modulation model satisfies: , wherein is a preset structural rigidity coefficient.

[0006] Preferably, the step of constructing the topological mapping vector mapping the structural displacement relationship between the first data sequence and the second data sequence comprises: establishing an index mapping relationship table from the address space of the first data sequence to the address space of the second data sequence in the computer memory; locking the physical position index of each maximal common substring in the first data sequence as a reference coordinate, and mapping the physical position index of the corresponding maximal common substring in the second data sequence as a target coordinate; converting the two-dimensional text block displacement transformation into a one-dimensional integer array sorting transformation, generating an integer sequence composed of the target coordinates as the topological mapping vector, and the order of the elements in the topological mapping vector directly reflects the relative position disorder state of the maximal common substring in the two data sequences.

[0007] Preferably, the filtering logic comprises: arranging all extracted common substrings in descending order of length; traversing the arranged common substrings, judging whether the character interval covered by the current common substring is completely covered by the character interval of any long substring selected into the maximal common substring set; if the judgment result is yes, the current common substring is removed; if the judgment result is no, the current common substring is added to the maximal common substring set, so as to ensure that there is no inclusion relationship between any two substrings in the maximal common substring set.

[0008] Preferably, the calculation rule of the theoretical maximum number of reverse sequences is: obtaining the total number of substrings contained in the maximal common substring set ; calculating the upper limit of the number of reverse sequence pairs in the completely reversed sequence arrangement state, the calculation formula is multiplied by minus one, and then divided by two; the processor performs division operation on the total number of reverse sequence pairs by using the upper limit, so that the value range of the topological dissipation index is strictly limited in the closed interval of zero to one, wherein zero represents complete order, and one represents complete reverse.

[0009] Preferably, the step of performing nonlinear damping modulation operation further comprises introducing a global boundary constraint factor to correct the final similarity score , and the corrected signal modulation model satisfies: ; the calculation logic of the global boundary constraint factor is: , wherein is the Levenshtein edit distance between the first data sequence and the second data sequence, is the first character difference position index of the first data sequence and the second data sequence, is the maximum length value in the first data sequence and the second data sequence, and For the preset micro-correction coefficient, the global boundary constraint factor is used to superimpose the penalty weight of the micro-character level difference on the basis of the macro-structure difference characterized by the topological dissipation index.

[0010] The structural rigidity coefficient is automatically matched according to the type identification of the to-be-processed data: in response to the data type identification indicating that the to-be-processed data is a program source code or a script file sensitive to execution order, the processor sets the structural rigidity coefficient to a first value to enhance the attenuation effect of the topological dissipation index on the final similarity score; in response to the data type identification indicating that the to-be-processed data is a natural language document or a log summary, the processor sets the structural rigidity coefficient to a second value, which is smaller than the first value, to tolerate paragraph-level sequence transposition.

[0011] Preferably, after the step of obtaining the first data sequence and the second data sequence to be compared, the method further comprises: performing denoising and standardization preprocessing on the first data sequence and the second data sequence; the preprocessing includes removing preset meaningless stop word symbols, uniformly converting all letters to lowercase ASCII codes, and compressing consecutive white space characters into a single space character; the processor performs extraction of the maximum common substring set based on the preprocessed data sequence to eliminate the interference of format noise on local statistical entropy calculation.

[0012] Preferably, the step of performing square operation and accumulation on the character length values of the maximum common substrings in the maximum common substring set specifically comprises: traversing the maximum common substring set to read the character length values of each maximum common substring; performing square operation on each character length value to obtain the energy value of each substring; accumulating the energy values of all substrings to obtain the total structural energy, and dividing the total structural energy by the length of the shorter sequence in the first data sequence or the second data sequence to obtain the local statistical entropy , which utilizes square operation to nonlinearly amplify the weight of long continuous substrings.

[0013] Preferably, the method is applied to a code version control scenario, and the step of calculating the total number of reverse order pairs specifically comprises: using a tree array structure or a binary index tree as a counting container; inserting each element in the topological mapping vector into the tree array structure in turn according to the appearance order of the elements; in each insertion operation, querying the number of existing elements greater than the value of the current insertion element in the tree array structure, and accumulating the number to the total number of reverse order pairs, thereby completing the quantification of the number of code block moving operations in logarithmic time complexity.

[0014] Preferably, the method further comprises generating a data processing instruction based on the final similarity score : setting the final similarity score comparing with a preset multiplexing judgment threshold; when the final similarity score is higher than the multiplexing judgment threshold, determining that the first data sequence and the second data sequence are substantially repeated contents, generating and executing a deduplication storage instruction for the second data sequence, and only retaining a reference pointer pointing to the first data sequence in the database; when the final similarity score is lower than the multiplexing judgment threshold, determining that the second data sequence is new content, and generating and executing an independent index construction and full-quantity storage instruction for the second data sequence.

[0015] Compared with the prior art, the beneficial effects of the present application are: 1. In the statistical entropy and multi-factor text similarity, a maximum common substring energy aggregation and position index topology dissipation double measurement mechanism is constructed to solve the technical problem that the linear scanning logic cannot identify block-level topology distortion in the prior art, the content energy is calculated by using the length square sum of the maximum common substring, the text sequence substantive content coincidence density is anchored, the position index mapping vector is established and the total number of inverse order pairs is calculated, the topology confusion degree of the text block in the storage space is independently quantified, the content integrity and the structure order degree are decoupled at the physical level, the computer system is based on the objective distribution state of data in the memory, the data recombination and random discrete noise are distinguished, the code reconstruction, log format change and other conditions with a large number of block-level displacement are processed, and the traditional edit distance algorithm is avoided to misjudge the position exchange as character replacement, thereby avoiding the distortion of the similarity calculation.

[0016] 2. A topology work nonlinear signal damping modulation model is established to overcome the lack of adaptability of the traditional linear weighted sum algorithm in processing complex text variation, and to abandon the simple numerical superposition of different dimension features, the basic similarity determined by the content statistical entropy is regarded as information potential energy, and the inverse order pair proportion mapping is derived based on the merge sorting algorithm as a topology dissipation index, which is used as a damping factor to multiply the information potential energy. The mechanism follows the physical law of signal-to-noise ratio decay, automatically adjusts the similarity score punishment according to the actual confusion degree of the text structure, retains a reasonable score when the text content is highly consistent but the order is reversed, triggers cascading score decay when the content is fragmented and scattered, and does not need frequent manual adjustment of the weight coefficient, thereby realizing robust calculation of the true similarity of the text.

[0017] 3. The method uses integer index array mapping and sorting operation to realize low power capture of high-dimensional text structure characteristics, solves the problem that complex semantic analysis model depends on high hardware resources and is difficult to meet the real-time requirement engineering, converts text structure difference into memory address index vector representation, reuses computer basic sorting algorithm counting logic to obtain topological inverse number, keeps high sensitivity to text topological structure, controls time complexity of core algorithm at logarithmic linear level, based on basic electric digital data processing logic operation mode, does not need to construct large floating point tensor or depend on external pre-training model, so that the scheme can be directly deployed on power limited edge computing node or high concurrency real-time search backend, ensures response speed and running stability of data processing system when performing large-scale text comparison task. BRIEF DESCRIPTION OF DRAWINGS

[0018] Fig. 1 The data processing logic flow chart of the present application fuses statistical entropy and topological mapping; Fig. 2 The system performance sensitivity analysis curve under the key correction coefficient of the present application; Fig. 3 The system overall architecture diagram of the present application integrates adaptive evolution and intelligent routing. DETAILED DESCRIPTION

[0019] The following examples are only used to illustrate the present application, but not to limit the scope of the present application.

[0020] The present application provides a kind of text similarity data processing method fusing statistical entropy and multiple factors, electronic computing equipment containing processor and memory is executed, by physical level address mapping and nonlinear signal modulation mechanism, realize the depth comparison and consistency check of electric digital data sequence content, the data processing system converts the input text data stream into feature vector in memory space, and successively executes the energy aggregation operation of maximum common sub-string, topological dissipation evaluation based on address index and signal modulation calculation of multiple factor coupling, finally outputs normalized similarity score to drive downstream database deduplication or version control instruction, processor receives first data sequence and second data sequence to be processed, and executes denoising and standardization preprocessing including ASCII code standardization, invalid symbol rejection and blank character compression, so as to convert unstructured original text into standard data stream suitable for calculation.

[0021] For the technical problem that linear scanning algorithm is difficult to distinguish meaningful structural reorganization from meaningless discrete noise, the processor adopts a local statistical entropy calculation procedure based on maximum common sub-string energy aggregation, the processor traverses first data sequence and second data sequence, identifies all matching character fragments between them, and removes those sub-strings completely covered by longer matching fragments according to containing screening logic, thereby generating maximum common sub-string set , the processor reads the set of each maximum common substring in turn , the character length value of each maximum common substring , and performs a square operation on each length value to obtain the structure energy of the substring, and accumulates the structure energy of all substrings to obtain the total structure energy , and finally divides the total structure energy by the length value of the shorter sequence in the two data sequences , to generate the local statistical entropy representing the density of the substantial coincidence of the content , the calculation path uses the nonlinear amplification characteristics of the square operation, so that the contribution weight of long continuous data blocks to similarity is higher than that of discrete short data blocks, thereby anchoring the substantial coincidence degree of the text content at the numerical level, and the process of constructing the set of maximum common substrings, to eliminate the uncertainty of the calculation results caused by the randomness of the memory reading order, contains a filtering logic that performs a strict double-key sorting protocol, and the processor arranges the common substrings in descending order according to their lengths, and when there are substrings with the same length value, the processor reads the starting storage address index in the first data sequence, and arranges the indices in ascending order according to the index values, and the address priority arbitration mechanism ensures that when processing data streams containing a large number of repeated patterns, the set of maximum common substrings and the subsequent topological mapping vector is generated in a unique and reproducible manner.

[0022] To solve the technical defect of the traditional edit distance algorithm that misjudges the overall displacement of data blocks as large-scale character replacement, the processor performs a topological dissipation evaluation procedure based on memory address index mapping, the processor traverses the above-mentioned set of maximum common substrings , respectively obtains the first storage address index of each substring in the storage space of the first data sequence and the second storage address index of each substring in the storage space of the second data sequence , the processor rearranges the substring objects in the set in a monotonically increasing order of the first storage address index , and extracts the second storage address index corresponding to each maximum common substring in the order of the rearrangement , thereby constructing a topological mapping vector reflecting the relative positional transformation relationship of the data blocks , the processor calls the merge sort algorithm or the tree array structure to perform the reverse pair counting operation on the topological mapping vector , calculates the total number of reverse pairs in the vector , and divides the total number of reverse pairs by the theoretical maximum number of reverse pairs , wherein the calculation of is obtained by the formula , which is a dimensionless physical quantity, quantifies the minimum block-level exchange operation cost required by the data sequence in transforming from the first state to the second state, that is, intuitively reflects the degree of increase in the topological entropy of the data in the storage space distribution.

[0023] On this basis, the processor executes a comprehensive calculation instruction based on a signal-damping modulation model to realize decoupling measurement of content integrity and structural order, and reads a structural rigidity coefficient preset in a system register , which is adaptively matched according to the type identification of the data to be processed, and for program source code data that is highly sensitive to execution order, the system automatically increases the value to enhance the punishment for disorder, and for natural language documents, the system automatically reduces the value to tolerate reasonable paragraph exchange, and the processor simultaneously calculates the Levenshtein edit distance , the first character difference position index , and the maximum sequence length between the first data sequence and the second data sequence, and generates a global boundary constraint factor according to the above, the calculation logic of which satisfies , wherein and are preset micro correction coefficients, and the values of the structural rigidity coefficient and the micro correction coefficients and are determined by using a grid search calibration procedure based on historical benchmark data sets, the processor calls a preloaded similarity true value calibration sample set, performs full permutation traversal calculation in a preset parameter space with a step size of 0.05, records the F1 score, that is, the harmonic mean of precision and recall, corresponding to each parameter combination, selects the parameter combination that makes the F1 score reach the global maximum, and writes it into a configuration register as a fixed constant at runtime, so that the parameter setting is anchored to the statistical distribution characteristics of the sample set rather than subjective experience values.

[0024] Finally, the processor calculates the final similarity score according to the nonlinear damping modulation formula , in which the local statistical entropy serves as a basic carrier carrying content information, and the topological dissipation index serves as a structural damping factor to apply multiplicative attenuation to the carrier signal, thereby ensuring that the calculation result can not only identify substantive repeated content that only occurs at the block level, but also effectively suppress noise data that is severely discretized; the processor calculates the final similarity score Compared with a preset multiplexing judgment threshold, when the score is higher than the multiplexing judgment threshold, a deduplication storage instruction is generated, and only a reference pointer pointing to the first data sequence is retained in the database; when the score is lower than the multiplexing judgment threshold, a full storage instruction is generated, and an independent index is constructed for the second data sequence, and the method avoids high overhead of floating-point matrix operation through pure integer operation and logical sorting, so that efficient similarity judgment of large-scale text data can be realized on a general computing platform.

[0025] Embodiment 1: In the engineering application scene of large-scale source code version control and storage optimization, the system faces the deduplication identification challenge brought by processing two program files with high consistency in essence but with function module level position interchanging, when the original code file represented by the first data sequence is reconstructed, two core function function blocks in the original code file are physically stored in the second data sequence, and the blocks are completely deleted and inserted in the new position, so that the difference cost calculated by the conventional difference comparison algorithm based on linear scanning is higher than the multiplexing judgment threshold, and the system performs the redundant full storage operation.

[0026] The processor of the application extracts the maximum common substring and energy aggregation procedure after receiving the above-mentioned data sequence, identifies the two maximum common substrings with continuous byte level content although the positions are changed, and constructs a topological mapping vector reflecting the relative position relationship of the two maximum common substrings in the memory address space The square operation is performed, so that the structure energy of the two long substrings accounts for a nonlinear amplification in the total structure energy , and a high local statistical entropy close to the numerical value one is generated , which anchors the substantial integrity of the file content in the physical layer, and the processor constructs a topological mapping vector reflecting the relative position relationship of the two maximum common substrings in the memory address space At this time, the elements in the vector are arranged in a simple reverse order, the processor calculates the total number of non-zero reverse order pairs by calling the merge sorting algorithm , and generates a topological dissipation index representing the topological cost of the reconstruction operation , and the processor performs a nonlinear damping modulation operation according to the preset structure rigidity coefficient of the code data , and uses to apply a multiplicative penalty conforming to the signal-to-noise ratio attenuation rule to , and finally outputs the similarity score , which not only reflects the increase of structural entropy caused by the change of word order, but also remains above the multiplexing judgment threshold because the content substance is not touched, accurately drives the database to execute the deduplication storage instruction, and only retains the reference pointer without storing the redundant data block, so as to realize the decoupling judgment of the content integrity and the structure order in the physical level.

[0027] Example 2: To verify the effectiveness and anti-interference capability of the proposed method for fusing statistical entropy and multi-factor text similarity processing in a real complex data environment, a large-scale code refactoring simulation test platform based on a Linux server cluster was constructed. This platform simulates a distributed version control system containing millions of code files, with the data source being a publicly available dataset from GitHub open-source projects. Noise interference with specific statistical characteristics was actively injected to simulate non-standard operations in actual development. In each commit record of the original code repository, the test platform randomly selected 5% of the files as perturbation samples, applying two typical engineering perturbations to these samples: one is structural reorganization, which randomly swaps the physical order of function modules or class definitions while maintaining their internal logic; the other is discretization noise, which randomly inserts meaningless comment characters or blank lines between code lines. The signal-to-noise ratio was set to 15dB. This mixed perturbation environment aims to recreate the extreme working condition where meaningful reorganization and malicious tampering coexist, thereby stress-testing the effectiveness of the algorithm.

[0028] The experiment selected code deduplication accuracy as the core evaluation index and compared the method of this invention (experimental group) with the traditional algorithm based on Levenshtein edit distance (control group A) and the improved algorithm based on longest common subsequence (LCS) (control group B). The structural rigidity coefficient, a key parameter, was also compared. The settings follow a trade-off logic between signal-to-noise ratio and structural sensitivity: considering the sensitivity of code data to execution order, while also tolerating module-level overall swapping, and after prior gradient calibration experiments, [the following parameters are considered]. The value is set to 0.75, and this value is determined based on: when At times, the system's penalty for out-of-order code is insufficient, resulting in poorly structured code receiving excessively high scores; while when At that time, the tolerance for module swapping was too low, which would inadvertently damage legitimate reconstruction operations. Therefore, 0.75 was determined as the optimal operating point for balancing content consistency and structural order in the current mixed disturbance environment. During the experiment, the processor read each group of samples in sequence, calculated the similarity score, and determined whether it was duplicate content based on the score result. The table below shows the key intermediate data and final judgment results when processing the above mixed disturbance samples. The original similarity in the table refers to the preliminary score without the nonlinear damping modulation of this invention. The topological dissipation index is a structural metric unique to this invention. The final judgment is the binary decision made by the system based on the preset threshold (0.85). See Table 1.

[0029] Table 1: Example of Comparison of Processing Results of Different Algorithms under Mixed Perturbation Environment

[0030] As can be observed from the data in Table 1, when faced with typical code refactoring scenarios such as pure module swapping and module swapping plus slight comment noise (samples 1 and 2), the similarity scores given by control group A and control group B are both below 0.5, causing the system to misjudge them as non-repeating new content, resulting in unnecessary full storage. In contrast, although the method of this invention detects drastic changes in physical location (reflected in non-zero similarity scores), the method of this invention can achieve similar results. (in terms of value), but because it is based on the aggregation of maximal common substring energy. Accurately anchor the substantive overlap of content (maintaining it above 0.9), and through reasonable... The value moderately dampes the topological dissipation, so that the final score It remained robustly above the judgment threshold of 0.85, successfully identifying substantial duplicates. Conversely, when faced with severe discretization tampering (sample 3), although some characters still matched, the extremely high topological dissipation index ( This triggers a cascading score decay, reducing the final score to 0.12 and effectively filtering out invalid noise.

[0031] Example 3: This example combines Figs. 1 to 3 This paper describes a text similarity data processing method that integrates statistical entropy and multi-factor methods, such as... Fig. 1 As shown, the input consists of the first and second data sequences to be compared to obtain the original text data stream. The process then proceeds to the denoising and standardization preprocessing stage, specifically removing stop words, unifying ASCII codes, and compressing whitespace characters. This process is performed in parallel by two branches. The left branch extracts the maximal common substring set based on inclusion filtering logic, and then generates a local statistical entropy that anchors the content density by performing a square aggregation operation on the substring length. The right-hand branch establishes relative positional relationships by obtaining memory address indices to construct a topological mapping vector, and uses merge sort to calculate the total number of normalized inversions to obtain the topological dissipation index. The calculation results from the two branches mentioned above converge to the nonlinear damping modulation calculation module, which also receives the structural stiffness coefficient automatically matched based on the data type. And the global boundary constraint factor calculated based on edit distance and difference location correction. Using local statistical entropy as the carrier and topological dissipation index as the damping factor, modulation calculation is performed, and the final similarity score, representing a comprehensive index of content completeness and structural orderliness, is output. The system then generates data processing instructions based on the comparison between the score and the threshold, pointing to the operation paths for either deduplication storage or independent index construction.

[0032] like Fig. 2As shown, the horizontal axis represents the parameter value, and the vertical axis represents the F1 score. The graph contains three different variation curves, with the solid line representing the penalty coefficient at a fixed edit distance. The penalty coefficient is 0.4 and the position difference is a factor of 0.4. With a value of 0.25, the F1 score varies with the structural stiffness coefficient. The trajectory of change, the dashed line represents the fixed path. It is 0.8 and When the value is 0.25, the F1 score varies with The dotted line corresponds to a fixed change. It is 0.8 and When the value is 0.4, the F1 score varies with The three curves clearly reflect the differences in sensitivity of each correction coefficient to the accuracy of the final similarity determination and the distribution of peak values ​​in different value ranges.

[0033] like Fig. 3 As shown, the system's left side features a multi-source heterogeneous data pool responsible for receiving code repository streams and document log streams and performing standardized stream injection. The data streams enter a similarity fusion calculation platform located at the center, which has a built-in high-performance computing cluster and ultra-fast feature cache. This platform is responsible for performing entropy weight analysis, topology mapping, and damping modulation calculations. The top of the system has an adaptive parameter evolution chamber that dynamically generates and pushes optimal configuration parameters using offline calibration and golden sample data. , and The scoring results output by the computing platform are transmitted to the intelligent routing module on the right. This module distributes the data according to the scoring results, pointing duplicate content to the deduplication index library for reference storage, and pointing new content to the full content library for entity storage.

[0034] Example 4: This example aims to provide a systematic offline optimization parameter-finding procedure to address the lack of theoretical support and engineering basis for similarity calculation parameter configuration in specific business scenarios. This ensures that the system achieves optimal performance from the initial deployment stage. In business scenarios oriented towards financial contract risk assessment, due to the extremely high sensitivity of contract text to the order of clauses and the extremely small margin for error, it is necessary to determine the optimal structural rigidity coefficient for this scenario. Edit distance penalty coefficient and position difference penalty coefficient To this end, a golden dataset was constructed, containing no fewer than 2000 pairs of financial contract sample pairs with accurately labeled similarity values ​​by human experts. This dataset covers various typical business variation patterns, such as contract revisions, clause additions and deletions, and paragraph reorganizations. The F1 score (the harmonic mean of precision and recall) was defined as the overall performance index of the system as the objective function for optimization. A parameter optimization algorithm was then initiated, and the following parameters were set... , , The initial search spaces are [0.1,1.0], [0.1,0.8], and [0.1,0.5], respectively, with a step size of 0.05.

[0035] During the traversal search, the processor performs a complete similarity calculation process on the gold dataset for each parameter combination, counts the number of misclassified samples and calculates the corresponding F1 score. By constructing a parameter-performance response surface, the sensitivity of each parameter to system performance and their mutual coupling effects are analyzed. Experimental results show that as... As the value increases, the system's ability to identify structural variations improves, but when... When the value exceeds 0.85, the misjudgment rate for reasonable revisions increases sharply; When the value is around 0.4, the system's sensitivity to minute character-level differences reaches equilibrium. Ultimately, the optimal parameter combination for this financial contract evaluation scenario is determined to be... , , The corresponding F1 score reached 0.96. Through this standardized offline optimization procedure, parameter setting is transformed from empiricism to data-driven engineering decision-making, ensuring the system's accurate adaptation to specific vertical fields.

[0036] Example 5: This example aims to address text similarity processing in large-scale real-time data stream environments. By introducing a multi-level index caching and dynamic threshold adjustment mechanism, it solves the computational efficiency bottleneck and misjudgment risk in high-concurrency scenarios, ensuring system stability and response speed under extreme loads. In large-scale real-time data stream processing scenarios for internet content moderation, the system needs to handle tens of thousands of text submission requests per second, including a large number of short text comments, bullet comments, and long text articles. Conventional similarity calculation methods often suffer from excessive computational overhead due to full comparison when processing such high-concurrency data streams, leading to a sharp increase in system latency, or the inability of fixed thresholds to adapt to texts of different lengths. Due to the characteristics of text, a large number of misjudgments occur. To address this challenge, this embodiment adopts a hierarchical multi-level index caching architecture. A first-level index based on Locality Sensitive Hash (LSH) is built in system memory to quickly filter out the vast majority of obviously dissimilar text data. Only the candidate set with potentially high similarity is sent to the subsequent accurate calculation process. A second-level index is built using a high-performance caching database such as Redis to store recently frequently occurring maximally common substrings and their corresponding statistical entropy values ​​to reduce the overhead of repeated calculations. For candidate text pairs filtered by the first-level index, the system calls the similarity calculation method of the aforementioned embodiment that combines statistical entropy and multi-factor similarity for accurate comparison.

[0037] To further enhance the adaptive ability of the system, a dynamic threshold adjustment mechanism is introduced in the embodiment, which adjusts the reuse judgment threshold of similarity judgment in real time according to the current system load state and the length characteristics of the text to be processed. When the system load is low and long text is processed, the threshold is appropriately increased to ensure the rigor of the judgment; when the system load is extremely high or short text stream is processed, the filtering threshold of the first-level index is dynamically reduced to prioritize the real-time response capability of the system, and a secondary verification is performed through the subsequent accurate calculation link. In addition, the system also integrates a sliding statistical module based on a time window to monitor the text similarity distribution in different time periods in real time. Once an abnormal similarity peak value such as a surge of similar comments caused by a sudden hot event is found, the early warning mechanism is triggered and the cache strategy is automatically adjusted to cope with the impact of sudden traffic. Through the synergistic effect of the above multi-level index cache and dynamic threshold adjustment mechanism, the embodiment improves the processing efficiency and stability of the system in a large-scale real-time data stream environment while ensuring high accuracy. Under the pressure test of processing 50,000 requests per second, the average response time of the system is controlled within milliseconds, and the misjudgment rate is always maintained at a very low level, effectively solving the problem of text similarity processing in high-concurrency scenarios.

[0038] In the offline preprocessing phase for unstructured long document similarity checking, an offline calibration and data filling procedure based on massive sample learning is performed to ensure the accuracy and consistency of the core data table. The system accesses a full corpus containing various document types and extracts a representative standard sample set from it. This set covers various structured feature difference document categories such as legal documents, technical reports, and academic papers. For each standard sample, the system calls the maximal common substring extraction module to generate a maximal common substring set specific to the sample and calculates the occurrence frequency and distribution density of each substring. Statistical methods are used to perform regression analysis on the substring distribution data of the full sample to determine the structural entropy benchmark value range for different document types. Based on this, the system constructs and fills a globally shared structural feature benchmark database, which records the local statistical entropy threshold interval and topological dissipation index distribution characteristics corresponding to different similarity levels for each type of document. Through this offline calibration process, the abstract similarity judgment logic is solidified into a quantifiable standard that can be queried, enabling fast and accurate threshold matching in the subsequent online processing stage, eliminating computational uncertainty caused by sample differences.

[0039] To ensure the adaptability and accuracy of the system in different deployment environments, a pre-deployment calibration / debugging procedure is also performed. Before deploying the similarity processing system to a new target server or accessing new service data streams, an automated environment self-checking and parameter fine-tuning process is started. The system reads a preset calibration sample package, which contains test document pairs with pre-labeled similarity true values. The processor performs a complete similarity calculation process on these test document pairs in sequence, and compares the calculation results with the labeled true values. The current misjudgment rate and omission rate are calculated. According to the comparison results, the system automatically fine-tunes the structural rigidity coefficient using the gradient descent algorithm and the correction coefficient in the global boundary constraint factor , , until the judgment accuracy of the system in the current environment reaches the preset acceptance standard. This calibration procedure ensures that the system can maintain stable calculation accuracy when facing hardware performance differences or specific business data distribution deviations, and realizes seamless adaptation and performance optimization from general models to specific scenarios.

[0040] Embodiment 7: This embodiment aims to address the differentiated needs of different application scenarios and data types, and provides a standardized pre-deployment calibration and engineering parameter calibration procedure to solve the problem of system performance fluctuations caused by fixed parameter settings. In the scenario of new system deployment or data type change, offline benchmark testing based on a gold data set is performed. The gold data set consists of at least 1000 pairs of text pairs with artificially accurate similarity true values, covering various typical variation patterns expected in the target business scenario, including pure text reorganization, character noise introduced by OCR recognition errors, and malicious synonym replacement, etc. Using this data set, the system starts the parameter optimization script, traverses the preset value space of the structural rigidity coefficient , the edit distance penalty coefficient and the position difference penalty coefficient . For each parameter combination, the system calculates the F1 score (harmonic mean of precision and recall) on the data set, and draws the parameter-performance response surface. By finding the global optimum point on the surface, the optimal parameter configuration for the business scenario is determined. For example, when processing legal contract comparison tasks that are extremely sensitive to logical order, the optimization result usually indicates that the value should be set in the high interval of 0.8 to 0.9 to severely punish arbitrary changes in clause order. When processing social media short text clustering tasks, the value may be optimally set between 0.4 and 0.5 to tolerate the loose syntax caused by colloquial expressions.

[0041] ​After the system is deployed and goes live, the built-in confidence monitoring module tracks the distribution histogram of similarity scores in real time. If the system detects an abnormal bimodal collapse in the score distribution, with the vast majority of scores clustered at either 0 or 1, lacking intermediate transitions, it suggests that the parameter settings are too aggressive or conservative, or that there is an accumulation of intermediate states with a large number of scores hovering near the judgment threshold, leading to decision-making difficulties. In this case, the system will automatically trigger the online fine-tuning logic, adopting a small-step, rapid-adjustment strategy to adjust the score distribution. , , Perform minor adjustments (step size not exceeding 0.05) and compare the change in misclassification rate on a small-volume grayscale test set before and after the adjustment. The new parameter combination will only be fully applied when the misclassification rate shows a statistically significant decrease. For extremely long texts occasionally appearing in the data stream, such as log files exceeding 100,000 characters, to avoid computation timeouts, the procedure stipulates that when the text length exceeds a preset threshold... When the time comes, it automatically switches to the block parallel computing mode, divides the long text into several overlapping data blocks and calculates the similarity of each block. Finally, it obtains the overall score by weighted average, thereby ensuring the timeliness of response while maintaining the reliability of the calculation results. Through the above-mentioned standardized calibration and fine-tuning procedures, it is ensured that the present invention is always in the optimal working state when facing the ever-changing engineering reality.

[0042] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0043] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A text similarity data processing method fusing statistical entropy and multiple factors, characterized in that, The method is executed by a computer processor and includes the following steps: Obtaining a first data sequence and a second data sequence to be compared, extracting a set of maximal common substrings not containing each other from the first data sequence and the second data sequence through containing screening logic, performing square operation on a character length value of each maximal common substring in the set of maximal common substrings and accumulating, and generating local statistical entropy representing content substantial coincidence density ; Traverse the set of maximum common subsequences, respectively obtain the first storage address index of each maximum common subsequence in the first data sequence storage space and the second storage address index in the second data sequence storage space, perform logical sorting on the set of maximum common subsequences based on the monotonically increasing order of the first storage address index, and sequentially extract the second storage address index corresponding to each maximum common subsequence in the sorted order, and construct a topological mapping vector mapping the structural displacement relationship between the first data sequence and the second data sequence; calling a merge sort algorithm to calculate the total number of reverse ordered pairs in the topological mapping vector, dividing the total number of reverse ordered pairs by the theoretical maximum number of reverse orders to obtain a normalized topological dissipation index independently quantifying the minimum block-level exchange operation cost required for transforming the first data sequence into the second data sequence using the topological dissipation index; Performing a nonlinear damping modulation operation to locally statistically entropy For the information carrier, the topological dissipation index For the structural damping factor, the final similarity score is calculated according to the preset signal modulation model , the signal modulation model satisfies: , wherein is a preset structural rigidity coefficient. 2.The text similarity data processing method of fusing statistical entropy and multiple factors according to claim 1, characterized in that, The step of constructing the topological mapping vector mapping the structural displacement relationship between the first data sequence and the second data sequence specifically includes: establishing an index mapping relationship table from the address space of the first data sequence to the address space of the second data sequence in the computer memory; locking the physical position index of each maximum common subsequence in the first data sequence as a reference coordinate, and mapping the physical position index of the corresponding maximum common subsequence in the second data sequence as a target coordinate; convert the two-dimensional text block displacement transformation into a one-dimensional integer array sorting transformation, generate an integer sequence composed of target coordinates as a topological mapping vector, and the order of elements in the topological mapping vector directly reflects the relative position disorder state of the maximum common subsequence in the two data sequences. 3.The text similarity data processing method of fusing statistical entropy and multiple factors according to claim 1, characterized in that, The filtering logic specifically includes: arranging all extracted common subsequences in descending order of length; traversing the arranged common subsequences, determining whether the character interval covered by the current common subsequence is completely covered by the character interval of any long subsequence selected into the set of maximum common subsequences; if the result of the determination is yes, the current common subsequence is excluded; if the result of the determination is no, the current common subsequence is added to the set of maximum common subsequences, ensuring that there is no inclusion relationship between any two subsequences in the set of maximum common subsequences.

4. The text similarity data processing method of claim 1, wherein The calculation rule of the theoretical maximum reverse order number is: obtaining the total number of the substrings contained in the maximum common substring set ; The upper limit of the number of reverse order pairs in the completely reverse order state is calculated, and the calculation formula is The product is multiplied by The product after subtracting one is divided by two; the processor uses the upper limit of the number to perform a division operation on the total number of reverse order pairs, so that the value range of the topological dissipation index is strictly limited to the closed interval of zero to one, where zero represents complete order and one represents complete reverse order.

5. The data processing method for text similarity based on fusion of statistical entropy and multi-factors according to claim 1, characterized in that, The step of performing the nonlinear damping modulation operation further includes introducing a global boundary constraint factor to the final similarity score The corrected signal modulation model satisfies: The calculation logic of the global boundary constraint factor is as follows: wherein is a Levenshtein edit distance between the first data sequence and the second data sequence, is a first character difference position index of the first data sequence and the second data sequence, is a maximum length value in the first data sequence and the second data sequence, and is a preset trace correction coefficient. The global boundary constraint factor is used to superimpose a penalty weight of a micro character level difference on the basis of a macro structure difference represented by a topological dissipation index.

6. The text similarity data processing method of claim 1, wherein structural rigidity coefficient The value of the structural rigidity coefficient is automatically matched according to the type identification of the data to be processed: in response to the data type identification indicating that the data to be processed is a program source code or a script file sensitive to execution order, the processor sets the structural rigidity coefficient to a first value to enhance the attenuation effect of the topological dissipation index on the final similarity score; in response to the data type identification indicating that the data to be processed is a natural language document or a log summary, the processor sets the structural rigidity coefficient to a second value, which is less than the first value, to tolerate paragraph-level sequence exchange. The value of the structural rigidity coefficient is automatically matched according to the type identification of the data to be processed: in response to the data type identification indicating that the data to be processed is a program source code or a script file sensitive to execution order, the processor sets the structural rigidity coefficient to a first value to enhance the attenuation effect of the topological dissipation index on the final similarity score; in response to the data type identification indicating that the data to be processed is a natural language document or a log summary, the processor sets the structural rigidity coefficient to a second value, which is less than the first value, to tolerate paragraph-level sequence exchange. The value of the structural rigidity coefficient is automatically matched according to the type identification of the data to be processed: in response to the data type identification indicating that the data to be processed is a program 7. The data processing method for text similarity based on fusion of statistical entropy and multi-factors according to claim 1, characterized in that, After the step of obtaining the first data sequence and the second data sequence to be compared, the method further includes: performing denoising and standardization preprocessing on the first data sequence and the second data sequence; the preprocessing includes excluding preset meaningless stop word symbols, uniformly converting all letters to lowercase ASCII codes, and compressing consecutive white space characters into a single space character; the processor performs extraction of the set of maximum common subsequences based on the preprocessed data sequence to eliminate the interference of format noise on local statistical entropy calculation.

8. The text similarity data processing method of claim 1, wherein, The step of performing square operation and accumulation on the character length values of each maximal common substring in the maximal common substring set comprises: traversing the maximal common substring set, reading the character length value of each maximal common substring; performing square operation on each character length value to obtain the energy value of each substring; accumulating the energy values of all substrings to obtain the total structure energy, and dividing the total structure energy by the length of the shorter sequence in the first data sequence or the second data sequence to obtain the local statistical entropy The weight of the long continuous substring is nonlinearly amplified by using square operation.

9. The text similarity data processing method of claim 1, wherein, The method is applied to a code version control scenario, and the step of calculating the total number of reverse order pairs specifically includes: using a tree array structure or a binary index tree as a counting container; according to the order of appearance of the elements in the topological mapping vector, sequentially insert each element into the tree array structure; in each insertion operation, query the number of existing elements in the tree array structure that are greater than the value of the current inserted element, and add the number to the total number of reverse order pairs, thereby completing the quantification of the number of code block movement operations in logarithmic time complexity.

10. The text similarity data processing method of claim 1, wherein The method further comprises determining a final similarity score based on the similarity scores generating a data processing instruction based on the final similarity score comparing the final similarity score with a preset multiplexing determination threshold; when the final similarity score is higher than the multiplexing determination threshold, determining that the first data sequence and the second data sequence are substantially repetitive content, generating and executing a deduplication storage instruction for the second data sequence, and retaining only a reference pointer pointing to the first data sequence in the database; when the final similarity score is lower than the multiplexing determination threshold, determining that the second data sequence is new content, and generating and executing an independent index construction and full-quantity storage instruction for the second data sequence.

Images