A cross-language knowledge divergence measurement and knowledge restructuring method and system, electronic equipment and storage medium

Abstract

The application discloses a cross-language knowledge divergence measurement and knowledge reconstruction method and system, electronic equipment and a storage medium, and belongs to the technical field of natural language data processing. Two language corpus sets under the same topic are obtained, a shared narrative framework list covering the two corpus sets is constructed after alignment in a unified semantic space, two sets of emphasis weights are determined and normalized on the same framework axis, and the Jensen-Shannon divergence normalized by ln2 is calculated as a cross-language knowledge divergence index; then, aligned evidence-driven retrieval enhancement generation is carried out, knowledge items of both perspectives are synthesized and integrated, and the balance degree is determined and recalculate according to the confidence weighted source distribution of supporting evidence and the number of conflict opinions. The application eliminates the divergence degeneration caused by independent framework axes, makes the divergence measurement have real discrimination within [0, 1], and makes the balance degree score same input and output, auditable, and improves the stability and reproducibility of cross-language text processing.

Description

Technical Field

[0001] This invention relates to the field of natural language data processing technology, and in particular to a method, system, electronic device, and computer-readable storage medium for cross-language knowledge divergence measurement and knowledge reconstruction. Background Technology

[0002] The same topic often presents different narrative frameworks and emphasis structures in literature, news, and encyclopedic corpora in different languages. Automating the measurement of these cross-linguistic knowledge structure differences and generating knowledge entries that take both perspectives into account is a common requirement in knowledge organization, intelligence analysis, and the construction of multilingual knowledge bases.

[0003] Existing technical approaches suffer from the following problems. First, manual comparison relies on experts reading each article individually, which is not scalable and lacks comparable quantitative metrics. Second, independently modeling topics or extracting frames from two corpora and then comparing the results yields two inconsistent label axes. When the support sets of two distributions are almost non-overlapping, the KL divergence or Jensen-Shannon divergence between them approaches the theoretical limit in each calculation, leading to metric degradation, loss of discriminative power, and inability to support automated comparison. Third, directly outputting divergence or equilibrium scores from large-scale generative language models tends to repeat example values ​​from prompts, and the output drifts when the same input is called multiple times, making the scores unreproducible and difficult to use in engineering systems. Fourth, the text returned by generative language models often contains code fences, prefixes and suffixes, or truncation, resulting in unstable formatting and a high failure rate in automated pipeline parsing.

[0004] Therefore, there is a need for a data processing method that enables cross-linguistic discriminative metrics to have true discriminative power, reproducible scoring results, and robustness to fluctuations in the output of generative models. Summary of the Invention

[0005] The purpose of this invention is to provide a cross-language knowledge divergence measurement and knowledge reconstruction method, system, electronic device and storage medium to solve the technical problems of divergence degradation caused by independent frame axes, unreproducible model self-scoring and fragile model output parsing.

[0006] In a first aspect, the present invention provides a method for cross-linguistic knowledge divergence measurement and knowledge reconstruction, comprising: acquiring a first-language corpus and a second-language corpus under the same topic; calling a generative language model to map the two corpora to a unified semantic space, obtaining a cross-linguistic aligned concept pair and document similarity matrix, and calculating a semantic drift metric; constructing a shared narrative frame list covering the two corpora, determining the weights on both sides of each frame on the same frame axis, setting missing frames to zero, and obtaining two emphasis distributions after normalization; calculating the Jensen-Shannon divergence between the two distributions and normalizing it with ln2 to obtain a cross-linguistic knowledge divergence index with values ​​in [0,1]; performing retrieval enhancement generation with aligned evidence as input, synthesizing and integrating knowledge entries from both perspectives, and obtaining a set of supporting evidence carrying source identifiers and confidence levels, as well as a set of conflicting viewpoints; calculating a balance factor based on the confidence-weighted source distribution of the supporting evidence, determining the conflict factor based on the number of conflicting viewpoints, recalculating and reconstructing the balance, and outputting the result.

[0007] Furthermore, the shared narrative framework list is generated through constrained instructions, including cross-corpus merging identification, quantity range limitation, neutral cross-linguistic naming, and prohibition of separate lists for each corpus. Furthermore, the divergence calculation employs the symmetric KL average form of the intermediate distribution M=(P+Q) / 2, truncated by ln2 normalization. Furthermore, the method includes zero-padding compatibility for key unions of older independent weighted data. Furthermore, the balance factor is weighted by the confidence levels of both sides of the evidence and S. A S B According to E=1−|S A -S B | / (S A +S B The conflict factor is determined according to F = 0.7 + 0.3 × min(N). c ,3) / 3 is determined, and the reconstructed balance is weighted and merged according to B=0.6×E×F+0.4×S0 with an upper limit truncation. Furthermore, the method includes a three-stage fault-tolerant parsing process: fence stripping, outer curly brace extraction, and tail rollback retry. Furthermore, each step is associated with a session identifier, supporting both step-by-step invocation and single-pipeline execution.

[0008] Secondly, this invention provides a cross-language knowledge divergence measurement and knowledge reconstruction system, including a semantic alignment module, a narrative comparison module, a knowledge reconstruction module, a session management unit, and an interface service unit. Thirdly, this invention provides an electronic device, including a memory and a processor, wherein the processor executes a stored computer program to implement the aforementioned method. Fourthly, this invention provides a computer-readable storage medium, wherein the stored computer program, when executed by a processor, implements the aforementioned method.

[0009] Compared with existing technologies, the advantages of this invention are as follows: The shared framework axis design defines the two emphasis distributions on the same support set, preventing the Jensen-Shannon divergence from collapsing to the upper bound in each calculation. The divergence index exhibits true discriminative power within the [0,1] interval, allowing for cross-case horizontal comparisons. The reconstructed equilibrium does not rely on unstable model self-assessments as the final result, but instead uses deterministic recalculation based on the evidence source distribution and conflict quantity. The same input produces the same output, and the calculation basis data is used with the scoring output, making the results auditable and reproducible. The three-stage fault-tolerant parsing significantly reduces sensitivity to fluctuations in the output format of generative models, improving the availability of automated pipelines. The session mechanism avoids redundant calculations at each layer, accommodating both interactive step-by-step analysis and batch pipeline usage. Attached Figure Description

[0010] Figure 1 This is a system architecture block diagram provided for an embodiment of the present invention.

[0011] Figure 2 The above is an overall flowchart of the method provided in the embodiments of the present invention.

[0012] Figure 3 The flowchart for recalculating the reconstructed balance degree is provided in the embodiment of the present invention. Detailed Implementation

[0013] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only for explaining the present invention and are not intended to limit the present invention.

[0014] Example 1 provides a method for cross-language knowledge divergence measurement and knowledge reconstruction. See [link to example]. Figure 2 This includes steps S101 to S107.

[0015] Step S101: Obtain a first-language corpus and a second-language corpus on the same topic. In one implementation, each corpus contains no more than 8 segments, and each segment contains no more than 4000 characters. Input exceeding the limit is rejected and a structured error message is returned. The corpora may come from encyclopedia entries, news reports, or academic literature. The two corpora are written in the first language and the second language, respectively, such as English and Chinese.

[0016] Step S102, Semantic Alignment. A generative language model is invoked to map the two corpora to a unified semantic space, outputting three types of results: 5 to 8 cross-linguistic aligned concept pairs and their alignment confidence scores, a document similarity matrix with dimensions equal to the product of the corpus sizes, and 2 to 3 semantic clustering labels. Simultaneously, a semantic drift metric d is calculated, where d = 0 indicating cross-linguistic equivalence and d = 1 indicating complete separation, with an overall similarity of 1 − d serving as a consistency constraint.

[0017] Step S103: Construct a shared list of narrative frames. Send a constraint instruction to the generative language model, requiring it to identify 4 to 7 main narrative frames across two corpora, each frame named once with a neutral cross-linguistic label, and explicitly prohibiting the generation of separate frame lists for each corpus. This constraint is crucial for preventing subsequent divergence degradation, as both corpora must be scored on identical frame labels.

[0018] Step S104, Shared Axis Weighting and Normalization. For each frame in the list, determine the weights for the first corpus and the second corpus. When a frame is dominant on one side and absent on the other, the weight for that side is explicitly set to zero instead of being omitted. Then, sum and normalize the two sets of weights separately to obtain the first emphasis distribution P and the second emphasis distribution Q defined on the same frame axis. When the input is two sets of independent weights in the old style, the key union is first taken, zeros are padded for alignment, and then normalized to maintain compatibility.

[0019] Step S105: Calculate the cross-linguistic knowledge divergence index. Construct an intermediate distribution M = (P + Q) / 2 on the union of frame labels, and calculate JSD(P,Q) = [KL(P‖M) + KL(Q‖M)] / 2, where KL is the KL divergence calculated as the natural logarithm, summed only for components where both sides are positive. Normalize JSD by dividing by ln2, truncate to [0,1] and retain three decimal places. The resulting value is denoted as the cross-linguistic knowledge divergence index. An index of 0 indicates that the two corpora have consistent emphasis structures, and an index of 1 indicates complete disjointness.

[0020] Step S106, Knowledge Reconstruction. Using aligned evidence containing a shared framework list, two sets of weights, overlapping issues, and conflicting issues as input, a synthesized knowledge entry integrating both perspectives is generated through retrieval enhancement. Generation is controlled by citation patterns, including balanced citations, source labeling, and opinion labeling, corresponding to balanced perspectives, source labeling for each claim, and inline labeling of opposing viewpoints, respectively. The generated result carries both a set of supporting evidence and a set of conflicting viewpoints. Each piece of supporting evidence is accompanied by a source identifier and a confidence level between 0 and 1, with a default confidence level of 0.5.

[0021] Step S107, recalculate the reconstructed balance, see [link / reference] Figure 3 Based on the language or corpus feature words in the source identifier, each supporting piece of evidence is assigned to either the first corpus side or the second corpus side. The confidence weighted sum of both sides is then calculated to obtain S. A With S B Let the balance factor E = 1 − |S A -S B | / (S A +S B Let the number of conflicting viewpoints be N. c The conflict factor F = 0.7 + 0.3 × min(N) cThe formula (3) / 3 treats genuine conflict as a necessary component of equilibrium rather than a bonus, and reduces the value of reconstructions with zero conflict as a tendency to smooth out differences. The model's self-score S0 is obtained; S0 defaults to 0.75 and is used only as a harmonic signal. The final reconstruction equilibrium score B = 0.6 × E × F + 0.4 × S0, truncated at an upper limit of 0.98, and includes S0 along with B in the output. A S B Number of pieces of evidence from both sides, N c The calculation of S0 is based on data that makes the scoring auditable. Since E, F, and the cutoff are all deterministic calculations, the same input will inevitably produce the same score.

[0022] To address the fluctuations in the output format of generative language models, each step performs a three-stage fault-tolerant parsing process on the text returned by the model. First, code fence markers are removed. Then, the content between the outermost curly braces is extracted to discard prefix and suffix description text. If parsing still fails, the parsing process backtracks from the end of the text to the nearest closed curly brace and retryes parsing, thereby recovering the longest valid JSON prefix in end-truncation scenarios.

[0023] Each step of the method is associated with a session identifier. In the stepwise mode, the session identifier returned by semantic alignment is passed to subsequent steps to receive intermediate results; in the pipeline mode, the three stages are executed sequentially within a single request and return a summary result, with each execution taking approximately thirty to ninety seconds. The service side also includes sliding window rate limiting based on the requester and path, as well as structured error classification, returning a prompt with the type, cause, and handling suggestions when an exception occurs.

[0024] In a set of implementation validations, the method was applied to seven bilingual topic cases. The resulting cross-lingual knowledge divergence index exhibited a clear gradient distribution, with approximately 0.04 for AI ethics cases, 0.32 for climate change cases, and 0.53 for Belt and Road Initiative cases, consistent with expert judgments on the degree of narrative divergence for each topic in both language contexts. In contrast, older measures using independent frame axes for each corpus all approached the upper bound on the same batch of cases, failing to distinguish differences between cases. These results demonstrate that the shared frame axis design restores the discriminative power of the measure, and the reconstructed equilibrium remains consistent across multiple executions on the same case, validating reproducibility.

[0025] Example 2 provides a cross-language knowledge divergence measurement and knowledge reconstruction system, see [link to example]. Figure 1System 100 includes a semantic alignment module 110, a narrative comparison module 120, a knowledge reconstruction module 130, a session management unit 140, and an interface service unit 150. Semantic alignment module 110 executes step S102, narrative comparison module 120 executes steps S103 to S105, knowledge reconstruction module 130 executes steps S106 to S107, session management unit 140 associates intermediate results of each module with session identifiers, interface service unit 150 provides step-by-step interfaces and pipeline interfaces, and calls external models via generative language model interface 200, and front-end interactive terminal 300 accesses the system through interface service unit 150.

[0026] This invention also provides an electronic device, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the methods described in any of the foregoing embodiments. This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the methods described in any of the foregoing embodiments.

[0027] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. Those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention should be determined by the appended claims.

Claims

1. A method for cross-linguistic knowledge divergence measurement and knowledge reconstruction, characterized in that, Performed by a computer device, the method includes: acquiring a first language corpus and a second language corpus under the same topic; calling a generative language model to map the first language corpus and the second language corpus to a unified semantic space, obtaining a cross-language aligned concept pair and document similarity matrix, and calculating a semantic drift metric representing the degree of cross-language concept shift; constructing a shared narrative frame list based on the first language corpus and the second language corpus, wherein each frame label in the shared narrative frame list is used simultaneously for the emphasis score of the two corpora; determining a first weight and a second weight for each frame label, wherein the weight of a frame label missing from any corpus is set to zero on the corpus side; normalizing all first weights and all second weights to obtain a first emphasis distribution and a second emphasis distribution defined on the same frame axis; calculating the first... The Jensen-Shannon divergence between the emphasis distribution and the second emphasis distribution is normalized with ln2 to obtain a cross-linguistic knowledge divergence index with values ​​in the interval [0,1]. Using alignment evidence containing the shared narrative framework list and its weights as input, knowledge entries from both corpus perspectives are synthesized and integrated through retrieval enhancement. Supporting evidence sets and conflicting viewpoint sets for each knowledge entry are obtained, with each supporting evidence carrying a source identifier and confidence level. A balance factor is calculated based on the confidence-weighted distribution of the supporting evidence set between the two corpora. A conflict factor is determined based on the number of conflicting viewpoint sets, and the reconstructed balance of the knowledge entry is recalculated at least based on the balance factor and the conflict factor. The cross-linguistic knowledge divergence index, the knowledge entry, and the reconstructed balance are output.

2. The method according to claim 1, characterized in that, The construction of the shared narrative framework list includes: sending a constraint instruction to the generative language model, the constraint instruction requiring the identification of narrative frameworks within a preset range across two corpora, naming each framework with a neutral cross-linguistic label, and prohibiting the generation of a separate framework list for each corpus; receiving the shared framework list and two sets of weights per framework returned by the generative language model.

3. The method according to claim 1, characterized in that, The calculation of Jensen-Shannon divergence includes: mapping the first emphasis distribution and the second emphasis distribution to the union of frame labels, padding missing terms with zeros; constructing an intermediate distribution M=(P+Q) / 2, where P and Q are the two padded emphasis distributions; calculating the KL divergence of P with respect to M and Q with respect to M respectively and taking the average; normalizing the average value with ln2 and truncating it to the interval [0,1].

4. The method according to claim 1, characterized in that, It also includes a compatibility processing step: when the received frame data consists of two independent weight sets for two corpora, the union of the keys of the two weight sets is taken as the shared frame axis, and missing keys in either set are filled with zero weights, and then the normalization is performed.

5. The method according to claim 1, characterized in that, The calculation of the balance factor includes: classifying each supporting piece of evidence into the first corpus side or the second corpus side based on the linguistic feature words or corpus feature words in the source identifier; and performing confidence-weighted summation on the supporting evidence from both sides to obtain the first weighted sum S. A With the second weighted sum S B Let the balance factor E = 1 − |S A -S B | / (S A +S B ).

6. The method according to claim 5, characterized in that, Let the number of conflicting viewpoints be N. c The conflict factor F = 0.7 + 0.3 × min(N) c ,3) / 3; Obtain the self-evaluation score S0 of the generative language model output; The reconstructed balance B = 0.6 × E × F + 0.4 × S0, and is truncated with a preset upper limit; Output B and S0. A S B N c The data used to calculate S0.

7. The method according to claim 1, characterized in that, It also includes fault-tolerant parsing steps for the text returned by the generative language model: stripping code fence markers; extracting the content between the outermost curly braces to discard prefix and suffix text; and backtracking from the end of the text to the nearest closed position and retrying parsing when parsing fails.

8. The method according to claim 1, characterized in that, Each step of the method is associated with a session identifier. Semantic alignment, divergence measurement, and knowledge reconstruction can be called step by step and intermediate results are passed through the session identifier, or they can be executed sequentially within a single request.

9. A cross-linguistic knowledge divergence measurement and knowledge reconstruction system, characterized in that, include: The semantic alignment module is used to perform unified semantic space mapping and semantic drift measurement; The narrative comparison module is used to build a shared list of narrative frameworks and calculate a cross-linguistic knowledge divergence index; The knowledge reconstruction module is used for recalculating the balance between enhanced generation and reconstruction. The session management unit is used to associate and transmit intermediate results from each module using session identifiers; The interface service unit is used to provide step-by-step and pipelined call interfaces to the outside world. The system is configured to perform the method according to any one of claims 1 to 8.

10. An electronic device, characterized in that, The method includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1 to 8.

Images