Digital archive automatic test paper management method and system based on big data analysis

Abstract

The present application belongs to the technical field of data processing, and particularly relates to a digital archive automatic test paper management method and system based on big data analysis, which comprises the following steps: collecting test paper texts and cleaning, and constructing a double-array Trie tree with word frequency weight and semantic label; adopting positive and negative maximum matching bidirectional scanning, and outputting according to the word segmentation number and the ratio of the word frequency weight to the single word number on the segmentation path; using six types of character position conditional random fields containing stroke and pinyin first letter features to identify unregistered words, replacing inverse document frequency with archive category distribution entropy to calculate weighted TF-IDF, and feeding back and adjusting the weight; using genetic algorithm to group test papers based on weighted TF-IDF clustering, introducing distribution entropy disturbance recombination when the iteration tends to be stable, and using keyword hash value and distribution entropy to splice to form a version primary key. The present application can avoid the test paper grouping algorithm falling into local optimum, improve the scientificity and constraint satisfaction ability of the test paper generation, and realize the anti-duplication and traceable management of the test paper archive.

Description

Technical Field

[0001] This invention relates to the field of data processing technology. More specifically, this invention relates to a method and system for automatic compilation and management of digital archives based on big data analysis. Background Technology

[0002] With the deepening of educational informatization, various educational institutions have accumulated massive amounts of digital test question archives in their daily teaching and examination activities. How to efficiently and accurately extract test questions from these unstructured test question texts and automatically arrange them into scientific and reasonable test papers based on multi-dimensional constraints such as knowledge point coverage, difficulty distribution, and question type ratio has become an important research topic in the field of intelligent education.

[0003] Currently, the industry has proposed a variety of technical solutions to achieve automated organization and management of archival materials. For example, Chinese patent application CN114443565A discloses an electronic case file assembly method, system, medium, and terminal. This method assembles electronic case files by acquiring case business information and constructing case file catalog information and legal document information. However, it relies on a pre-built catalog structure and document templates, primarily applicable to structured information processing in law enforcement case files. It does not provide effective text parsing and feature extraction methods for unstructured data such as professional terms, abbreviations, and complex sentence structures found in test question files. When applied to automatic test question assembly scenarios, the underlying word segmentation accuracy is insufficient, making it difficult to accurately identify the core knowledge points and difficulty features of the test questions. Similarly, Chinese patent application CN117131104B discloses an intelligent question extraction and assembly method, device, electronic equipment, and storage medium, further proposing an assembly scheme based on genetic algorithms. This achieves some improvement in assembly efficiency and test paper quality, but its technical starting point presupposes that the test questions have already been correctly segmented, labeled, and characterized. If the accuracy of the underlying word segmentation and feature extraction stages is insufficient, the room for improvement in assembly quality remains limited.

[0004] In summary, existing technologies still have the following shortcomings in the automatic generation of digital test paper archives: First, at the text parsing layer, bidirectional scanning word segmentation lacks scientific evaluation indicators when encountering segmentation ambiguities, and the feature template of the conditional random field model is singular, failing to integrate the semantic distribution information of Chinese character strokes and pinyin initials, resulting in low recognition accuracy of professional terms and out-of-vocabulary words; Second, at the feature extraction layer, the inverse document frequency factor in the traditional word frequency-inverse document frequency algorithm does not consider the distribution differences of words between different test paper archive categories, and the dictionary also lacks a feedback mechanism for adaptive adjustment based on text features, thus limiting the representational ability of feature vectors; Third, at the test paper generation and version management level, the genetic algorithm lacks an effective perturbation and recombination mechanism when fitness stagnates, easily limiting the search to local areas, and the version index of the generated test paper lacks anti-duplicate and traceability strategies associated with the core features of the text, which is not conducive to the large-scale storage and management of the test paper database.

[0005] Therefore, how to form a solution that runs through the entire process of test paper file processing, from underlying text cleaning, word segmentation ambiguity handling, and out-of-vocabulary word recognition enhancement, to feature weight calculation optimization and test paper generation algorithm perturbation and reorganization, thereby improving the accuracy and intelligence level of automatic test paper generation, is an urgent problem to be solved by those skilled in the art. Summary of the Invention

[0006] To address the technical problems in existing automatic digital archive assembly technologies, such as insufficient precision in underlying text parsing leading to a lack of optimal indicators for handling word segmentation ambiguity, low accuracy in identifying out-of-vocabulary words, limited feature vector representation capabilities, and the tendency of assembly algorithms to get trapped in local optima while lacking a traceability and anti-duplicate mechanism for version indexing, this invention provides solutions in the following aspects.

[0007] In a first aspect, the present invention provides an automatic digital archive test paper generation and management method based on big data analysis, comprising: S1, collecting test paper archive text and cleaning it to obtain the text to be segmented, constructing a professional dictionary based on a double-array Trie tree, wherein each entry in the professional dictionary is attached with a word frequency weight and a semantic category label; performing bidirectional scanning of the text to be segmented using forward and reverse maximum matching, and determining the optimal segmentation result based on the evaluation ratio obtained by dividing the product of the word frequency weights of each entry on the segmentation path by the sum of the number of single characters and a preset smoothing constant; S2, using a conditional random field model containing Chinese character stroke features and pinyin initial letter features. The algorithm identifies out-of-vocabulary words in the segmentation results; it calculates a weighted TF-IDF value based on the file category distribution entropy to replace the inverse document frequency, and feeds the distribution entropy back to the dual-array Trie tree to adjust the word frequency weights; S3, it constructs feature vectors and clusters based on the weighted TF-IDF value, uses a genetic algorithm to extract test questions from the clusters to generate test papers, and introduces the distribution entropy as a perturbation factor to reorganize the test papers when the genetic algorithm iterates to a stable state; it writes the generated test papers into the database to establish a version index, and the primary key is composed of the keyword hash value extracted based on the weighted TF-IDF value, the comprehensive value of the test paper file category distribution entropy, and the generation timestamp.

[0008] This invention first stores a weighted professional dictionary using a dual-array Trie tree, combining forward and reverse maximum matching bidirectional scanning. It then optimizes the segmentation results based on the ratio of the product of the number of segmented words and their frequency weights to the sum of the number of individual characters and a preset smoothing constant, thus solving the problem of a lack of scientific evaluation indicators for ambiguous word segmentation. Secondly, it improves the accuracy of out-of-vocabulary word recognition by utilizing a six-category conditional random field model that integrates Chinese character strokes and the initial letter features of pinyin. It replaces the traditional inverse document frequency calculation with archive category distribution entropy to calculate weighted TF-IDF, while simultaneously feeding the distribution entropy back to the dictionary to adaptively adjust the word frequency weights, enhancing the category discrimination ability of the feature vectors. Finally, based on weighted TF-IDF clustering, it employs a genetic algorithm with adaptive crossover mutation and distribution entropy perturbation to generate test papers. This algorithm escapes local optima when it stabilizes, and uses keyword hash values, distribution entropy, and generation timestamps to construct a version index primary key, achieving unique identification, anti-duplicate measures, and traceability management of test papers.

[0009] Preferably, the step of collecting test question archive text and cleaning it to obtain the text to be segmented includes: performing optical character recognition on the digitized scanned image to obtain an initial text sequence; removing invisible control characters, page breaks, and formatting marks from the text using regular expressions and replacing them with standard spaces; converting full-width English letters and punctuation marks to half-width format and uniformly mapping English letters to lowercase to generate standardized text to be segmented.

[0010] Preferably, the construction of a specialized dictionary based on a dual-array Trie tree, wherein each entry in the specialized dictionary is appended with a word frequency weight and a semantic category label, includes: extracting specialized terms from archival classification management specifications and historical dossier documents to form a standard term set; initializing a dual-array Trie tree containing a base address one-dimensional array and a detection one-dimensional array, and mapping and storing the words in the standard term set character by character as tree nodes; calculating the absolute frequency of each entry in a pre-constructed archival corpus, dividing it by the total word frequency of the corpus to obtain the word frequency weight of the entry; extracting the identification code of the archival classification to which the entry belongs as a semantic category label, and appending and storing the category label and word frequency weight together in the end address node of the corresponding entry in the dual-array Trie tree.

[0011] Preferably, the step of determining the optimal segmentation result based on the evaluation ratio obtained by dividing the product of the word segmentation number and the word frequency weights of each word on the segmentation path by the sum of the number of characters and a preset smoothing constant includes: if the word segmentation numbers of the two results obtained from the forward and reverse maximum matching bidirectional scanning are inconsistent, then the result with fewer word segmentation numbers is selected as the segmentation result; if the word segmentation numbers of the two results are consistent but the segmentation paths are different, then the product of the word frequency weights of each word on the two segmentation paths is calculated respectively, and the ratio obtained by dividing the product by the sum of the number of characters and a preset smoothing constant is calculated, and the result corresponding to the largest ratio is selected as the segmentation result.

[0012] By comparing the number of words segmented in forward and reverse scanning, the smaller number is selected when the number is different to reduce fragmented segmentation; when the number is the same but the paths are different, the ratio of the product of word frequency weights on the path to the number of single characters is calculated, and the one with the largest ratio is taken as the final segmentation result. This provides an optimal indicator for word segmentation ambiguity and improves segmentation accuracy.

[0013] Preferably, the step of using a conditional random field model containing Chinese character stroke features and pinyin initial letter features to identify out-of-vocabulary words in the segmentation result includes: expanding the character position markers to six mutually exclusive states, and combining the dependency relationship between the current Chinese character and the characters in the preceding and following windows during feature serial scanning to output an identification sequence; wherein, the Chinese character stroke features are normalized stroke count feature values ​​obtained by dividing the absolute number of strokes of the Chinese character to be identified by the global average stroke constant value; the pinyin initial letter features are extracted from the pinyin initial letters of the Chinese character to be identified and adjacent Chinese characters, statistically distributed in the double-array Trie tree, and the semantic category smoothing probability calculated using the Laplace smoothing algorithm.

[0014] Preferably, the step of calculating the weighted TF-IDF value based on the inverse document frequency replacing the archive category distribution entropy and feeding the distribution entropy back to the dual-array Trie tree to adjust the term frequency weight includes: obtaining the conditional probability that documents containing the target term belong to a certain category under all categories; calculating the archive category distribution entropy of the term according to the Shannon entropy formula; determining whether the archive category distribution entropy is lower than a set threshold; if so, updating the term frequency weight of the term in the dual-array Trie tree to: the original weight multiplied by an amplification factor, wherein the amplification factor is the sum of 1 and the reciprocal of the archive category distribution entropy.

[0015] By calculating the conditional probability of a term in each category, the document category distribution entropy is obtained. The negative exponential function of the distribution entropy is used to replace the inverse document frequency, so that terms with strong category discrimination are given higher weights. The distribution entropy of low-entropy terms is fed back to the dictionary and its word frequency weight is increased by multiplying it by an amplification factor. This forms a positive loop of feature extraction and dictionary update, thereby effectively improving the accuracy of feature vectors in representing professional test questions.

[0016] Preferably, the step of using a genetic algorithm to extract test questions from clusters to generate test papers includes determining the crossover and mutation probabilities based on the number of iterations and the mean fitness of the clusters: monitoring the fitness variance of the population over generations, and linearly increasing the crossover probability of two points when the fitness variance decreases and tends to be homogeneous; calculating the mutation probability, the formula of which is: the mutation probability is equal to the reciprocal of the square root of the current number of iterations of the genetic algorithm, multiplied by the negative exponent of the arithmetic mean of the fitness of all members in the cluster to which the mutated individual belongs, with the natural constant as the base.

[0017] By monitoring the fitness variance, the probability of two-point crossover is linearly increased when the population tends to be homogeneous, thereby increasing the intensity of gene recombination. The mutation probability depends on both the number of iterations and the mean fitness of the clusters. As the number of iterations increases, the global mutation rate decreases, while the mutation probability of low-fitness clusters increases. This achieves a balance between breadth search in the early stage and fine optimization in the later stage, avoiding premature entrapment in local optima.

[0018] Preferably, when the genetic algorithm iteratively stabilizes, the step of introducing the distribution entropy as a perturbation factor to perform test paper reorganization includes: when the fitness variance of the population for three consecutive generations is lower than the set stagnation threshold, it is determined that the iterative process has stabilized, and the chromosomes of the current population are traversed; the corresponding distribution entropy value is multiplied by the randomly generated standard normal distribution coefficient as a perturbation factor, added to the weighted TF-IDF feature corresponding to the gene locus, and the clustering algorithm is called again to perform clustering and draw questions proportionally.

[0019] Preferably, the primary key is composed of a keyword hash value extracted based on a weighted TF-IDF value, a comprehensive value of the test paper archive category distribution entropy, and a generation timestamp, including: sorting the generated test paper text in descending order of weighted TF-IDF value, extracting a preset number of keywords from the top ranks and combining them into a string; calling a hash encryption engine to generate a digest hash code for the string; obtaining the timestamp of the moment the test paper generation instruction is issued, and calculating the comprehensive value of the archive category distribution entropy of the test paper containing the questions; concatenating the digest hash code and the comprehensive value of the distribution entropy to form a unique identifier sequence, which is stored in a two-dimensional relational database as the version index primary key.

[0020] By extracting the top N keywords with the highest weighted TF-IDF from the exam paper to generate a hash code, and concatenating it with the comprehensive value of the exam paper's archive category distribution entropy and the generation timestamp, a unique index primary key is formed that simultaneously associates the core text features and category distribution characteristics, which facilitates subsequent exam paper deduplication, version tracking, and large-scale retrieval management.

[0021] Secondly, the present invention provides an automatic digital archives compilation and management system based on big data analysis, including a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, the above-mentioned automatic digital archives compilation and management method based on big data analysis is implemented.

[0022] By adopting the above technical solution, the above-mentioned automatic digital archive compilation and management method based on big data analysis is generated into a computer program and stored in a memory for loading and execution by a processor. This allows for the creation of a terminal device based on the memory and processor, making it convenient to use.

[0023] The technical solution of the present invention has the following beneficial technical effects: This invention constructs a professional dictionary with word frequency weights and semantic tags using a dual-array Trie tree. It combines forward and reverse maximum matching bidirectional scanning to achieve optimal word segmentation by using the ratio of the product of word frequency weights to the sum of the number of characters and a preset smoothing constant. A six-category conditional random field model integrating stroke and initial letter features of pinyin is employed to improve the accuracy of out-of-vocabulary word recognition. Weighted TF-IDF calculation is improved using archive category distribution entropy with adaptive feedback weighting. A genetic algorithm employing adaptive crossover mutation and distribution entropy perturbation completes the test paper assembly. Simultaneously, a version index primary key is constructed by concatenating keyword hash values, distribution entropy, and timestamps. This effectively improves the word segmentation accuracy and professional vocabulary recognition rate of test texts, optimizes the category discrimination of feature vectors, avoids the test paper assembly algorithm getting trapped in local optima, significantly improves the algorithm's convergence efficiency, enhances the scientific nature and constraint satisfaction of test paper generation, and achieves unique identification, anti-duplicate, and traceable management of test paper archives. Attached Figure Description

[0024] Figure 1 This is a flowchart of a digital archive automatic file assembly and management method based on big data analysis; Figure 2 This is a graph showing the variation of population fitness variance and two-point crossover probability with the number of generations. Figure 3 This is a diagram comparing the performance indicators of different technical solutions. Detailed Implementation

[0025] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0026] This invention discloses an automatic file generation and management method for digital archives based on big data analysis, referring to... Figure 1 This includes the following steps: S1. Collect and clean the text and build a dictionary to perform bidirectional word segmentation.

[0027] This step first obtains a clean character sequence through optical recognition and regularization, then extracts professional features to construct a double-array Trie tree with frequency weights and semantic attributes, and finally outputs the segmentation result through bidirectional matching to provide a noise reduction basis for unstructured text parsing. By using bidirectional verification to resolve structural ambiguities, the word segmentation accuracy of professional terms is improved.

[0028] Specifically, the original data of the test questions to be processed is first obtained. For data from the internet, the test question webpage can be obtained through a web crawler interface, and its DOM tree can be parsed to extract plain text. For images, image preprocessing is performed first, including binarization and tilt angle detection based on Hough transform, and then the CRNN network model is called for text content recognition. The CRNN network model consists of convolutional layers, recurrent layers, and transcription layers connected in sequence. The convolutional layers extract the sequence features of the input image through a multi-layer convolutional neural network, the recurrent layers obtain contextual features based on a bidirectional long short-term memory network, and the transcription layers calculate the output probability and complete the decoding using the connection time classification loss function. The conditional probability calculation formula is as follows: ; in, The probability of decoding and generating a target text sequence given an input feature sequence; The final output is the target text sequence; The input feature sequence; Paths are categorized by connection time; This is the set of alignment paths corresponding to the target text.

[0029] After obtaining the initial text sequence, regular expressions are used to filter out invisible control characters, page breaks, and formatting marks in the text and replace them with standard spaces. Then, character encoding shifting is used to complete full-width and half-width conversion and case unification, and finally, the standardized text to be segmented is output.

[0030] Furthermore, a specialized dictionary is constructed and stored using a double-array Trie tree for indexing. Candidate words are automatically selected from historical records using the TF-IDF algorithm and mutual information metric, supplemented by a domain expert thesaurus to form a standard vocabulary set. When initializing the double-array Trie tree, a one-dimensional base address array and a one-dimensional detection array are set, and its state transition satisfies the following equation: ,and ; In the formula, Index of the current node's state; Encode the input characters to be transferred; The index of the target state node reached after the transfer; For nodes The corresponding base address value; For nodes The predecessor check value.

[0031] Then, the word frequency weight is calculated for each term, and the relationship is as follows: ;in, The basic word frequency weight for the target term; This represents the absolute frequency of the term in a pre-built corpus. This represents the total word frequency statistics for this corpus.

[0032] The aforementioned word frequency weights and the identifier of the file category to which the word belongs are packaged and encapsulated into a combined structure, which is then stored in the reserved data pointer area of ​​the corresponding term's end state node in the Trie tree, so as to ensure that the word segmentation results, weights, and tags can be extracted synchronously with constant time complexity.

[0033] After the dictionary is constructed, forward maximum matching and backward maximum matching are performed sequentially on the text to be segmented to obtain forward segmentation lists and backward segmentation lists, respectively. When the number of elements in the two lists is different, the list with fewer elements is directly selected as the segmentation result. If the number of elements is the same but the segmented fragments are different, the product of the word frequency weights of each word on the two segmentation paths is calculated, and the ratio obtained by dividing the product by the sum of the number of single characters and the preset smoothing constant is used as the evaluation index. The preset smoothing constant is used to prevent the denominator from being zero, for example, it can be set to 1. Finally, the one with the larger ratio is selected as the segmentation result.

[0034] As a specific implementation, the tilt angle detection range in this embodiment is set to -15° to 15°; the invisible control characters matched by the regular expression include common ASCII control characters and page breaks, etc.; during full-width / half-width conversion, full-width characters are converted to their corresponding half-width characters, and English letters are uniformly converted to lowercase. In the standard word set extraction stage, the selection criteria for candidate words are set as a word frequency greater than 50 times and a mutual information value greater than 2.5; the double-array Trie tree is pre-expanded by 1.5 times the total number of words in the vocabulary during initialization. In the word frequency weight calculation example, the total word frequency of the corpus is set as... The absolute frequency of the target term is 10,000,000. The value is 5000, and the calculated weight is 0.0005, which is stored in float32 format. At the same time, the national archive classification identifier code corresponding to the term (such as the integer value 45 for mathematics) is extracted as a semantic category label, and finally encapsulated into a 4-byte and 2-byte combined structure for additional storage.

[0035] As an optional implementation, the aforementioned initial text acquisition process based on OCR is only invoked when the input source is an image file; when the input source is HTML or plain text, plain text can be directly extracted by parsing libraries such as BeautifulSoup, thus eliminating the OCR recognition step.

[0036] S2, CRF out-of-vocabulary word identification and weight feedback.

[0037] This step expands the CRF feature template by subdividing character position states, incorporating normalized stroke features and cross-category smoothing probabilities, and reconstructs TF-IDF weights by replacing traditional inverse document frequency with archive category distribution entropy, thereby improving the boundary recognition recall rate of long-span obscure professional words. At the same time, it strengthens the distinguishability of professional words in the dictionary by using a weight feedback mechanism.

[0038] First, a linear chain conditional random field model is instantiated, and the underlying solver algorithm and overfitting prevention regularization coefficients are specified. Unlike the traditional four-character labeling system, this invention expands the character labeling to six mutually exclusive positional states: B label representing the first Chinese character of a word, E label representing the last Chinese character of a word, S label representing a Chinese character forming an independent word, M1 label representing the first position in a word, M2 label representing the second position in a word, and M3 label representing the third and all subsequent positions in a word. A feature scanning sliding window is set, and the Viterbi algorithm is called during the inference phase to calculate the conditional probability of each possible labeled state sequence; simultaneously, strict constraints are imposed on the state transition matrix to prohibit illegal path combinations that do not conform to the above order, thereby compressing the decoding search space.

[0039] In constructing the feature template, this invention combines normalized stroke features with semantic category smoothing features based on the first letter of the pinyin. Specifically, the absolute number of strokes of the Chinese character to be identified is divided by a preset global average stroke count to obtain normalized stroke feature values, which assist the model in recognizing structurally complex and rare characters. For the pinyin first letter features, conditional probability based on Laplace smoothing is used to calculate feature values ​​for specific categories. The smoothing probability relationship is as follows: ; In the formula, This represents the smoothed conditional probability of the first letter of the current Chinese character's pinyin under a specific semantic category. This represents the number of entries in the double-array Trie tree that belong to this specific category and have the same first letter as the current Chinese character. This represents the total number of entries in the double-array Trie tree that contain this initial letter; This represents the total number of semantic categories in the pre-defined archive corpus. Traditional frequency statistics are prone to zero-probability truncation when dealing with unregistered and rare words. To address this, the above relational expression mitigates the risk of zero probability by adding a smoothing constant to the intra-class matching quantity with the same initial letter and introducing the global category dimension as a buffer term in the normalization operation.

[0040] After the recognition of out-of-vocabulary words, the TF-IDF weights are calculated with improvement. Specifically, the inverse document frequency term in the traditional TF-IDF is replaced by the negative exponential function of the entropy of the document category distribution with the natural constant as the base, and then multiplied by the word frequency to obtain the weighted TF-IDF value; the distribution entropy is calculated based on the conditional distribution probability of the term in the documents of various categories according to the Shannon entropy formula. Further, the distribution entropy is fed back to the double-array Trie tree to update the word frequency weights: when the distribution entropy is lower than the preset threshold, the weight promotion mechanism is triggered, and the original word frequency weight of the term is multiplied by the amplification factor and then written back to the corresponding node of the Trie tree. The amplification factor is the sum of 1 and the reciprocal of the distribution entropy.

[0041] As a specific implementation, in this embodiment, the solver of the conditional random field model selects the quasi-Newton method, the L1 and L2 regularization coefficients are both set to 0.1, and the maximum number of iterations is 100; the size of the feature scanning sliding window is 5, that is, it covers the current Chinese character and 2 adjacent Chinese characters before and after. In terms of the state transition constraint, taking the rules that the M1 tag must follow the B tag immediately, the M2 tag must follow the M1 tag immediately, etc. as typical rules, taking long segment professional terms such as chemical molecular formulas and physical law names as examples, the boundary recognition recall rate can be effectively improved by means of this constraint. In the calculation of the normalized stroke features, the global average Chinese character stroke constant value is 9.8. For the character "繁" with 17 strokes, its normalized feature value is about 1.735. In the example of the smoothing of the first letter of the pinyin, let the total number of terms starting with "f" be 1200, and the total number of system categories be 50, and the number of terms belonging to the physics category and starting with the same first letter "f" be 80. From this, the smoothing probability of this first letter under the physics category is about 0.0648. In the example of the weight feedback, the preset distribution entropy threshold is 1.0: taking the term "calculus" as an example, it only appears in two categories of files, mathematics (accounting for 90%) and physics (accounting for 10%). The calculated result of the distribution entropy is about 0.469, which is lower than the threshold of 1. Therefore, the weight promotion mechanism is triggered, and its original weight of 0.005 is multiplied by the amplification factor , and the finally updated new weight is about 0.0156.

[0042] As an optional implementation, the process of extracting the first letter of the Chinese character pinyin can be implemented by calling the pypinyin library, and the model training and feature template assembly of the conditional random field can be completed with the help of open source toolkits such as sklearn_crfsuite, so as to reduce the system deployment cost.

[0043] S3. Cluster-based preferred test paper generation and construction of a version hash index.

[0044] This step clusters the weighted TF-IDF feature vectors of each test question, selects the best questions using a genetic algorithm with variance monitoring and distribution entropy perturbation recombination mechanism, and generates a unique evidence key by joint hashing of core features and distribution entropy. This avoids the inherent defect of traditional test paper generation algorithms that are prone to premature convergence, ensuring the global scientificity of the generated test paper scheme under multi-dimensional teaching constraints, while realizing the traceability and anti-duplicate storage of digital test paper resources.

[0045] First, the weighted TF-IDF values ​​of each term in each question are concatenated along the dictionary dimension to form a floating-point feature vector. Then, Euclidean distance is used to perform unsupervised iterative clustering on all feature vectors, resulting in multiple clusters. A candidate test paper is abstracted into a chromosome encoding composed of question IDs and input into a genetic algorithm framework for evolutionary extraction. The fitness function comprehensively considers the average difficulty coefficient, the intersection ratio of knowledge point sets, and the differences in the distribution of question types, and outputs a comprehensive fitness score accordingly.

[0046] During the evolutionary iteration process, the fitness scores of each scheme in each generation of the population are recorded in real time, and the overall variance is calculated. When the fitness variance decreases for several consecutive generations and falls below the preset homogenization threshold, an exit mechanism is triggered: on the one hand, the probability of two-point crossover is linearly increased by a set step size to shuffle the ultra-long fixed gene fragments; on the other hand, the distribution entropy of that generation is multiplied by the random standard normal distribution coefficient as a perturbation factor, which is then superimposed on the weighted TF-IDF features corresponding to the gene loci, and clustering is re-executed and topics are re-selected proportionally.

[0047] For individual mutations, this invention constructs the mutation probability as a function based on iterative algebra and cluster fitness, with the following relationship: ; In the formula, This represents the mutation probability of the current chromosome in the individual. Let be the number of iterations in the current genetic algorithm; This is the normalized arithmetic mean of the fitness scores of all members within the cluster to which the individual belongs. This relationship nonlinearly couples the global time evolution scale of the algorithm with the feature quality metric of the local population; it ensures the stability of later convergence by using the decay of the iterative algebra term, while amplifying the mean fitness of the cluster through the negative exponential term, thereby prompting the low-fitness subpopulation to automatically increase the mutation frequency to expand the breadth search.

[0048] After the test paper assembly scheme is generated, the top few core keywords with the highest weighted TF-IDF values ​​in the test paper are extracted in descending order (i.e., the top preset number of keywords are extracted, where N is a positive integer) and concatenated into a feature string. The underlying hash encryption engine is then called to perform a hash operation on the string to obtain a fixed-length digest hash code. At the same time, the precise timestamp of the test paper assembly instruction is obtained, and the distribution entropy comprehensive value of the file categories covered by the entire test paper is calculated. Finally, the digest hash code, the distribution entropy comprehensive value, and the timestamp are concatenated with a concatenation operator to form a unique identifier sequence, which is stored in a two-dimensional relational database as the version index primary key.

[0049] As a specific implementation, this embodiment uses the KMeans algorithm for unsupervised clustering, and the fitness data history queue within the genetic algorithm framework adopts a FIFO structure; the stagnation threshold of fitness variance is set to 0.05, and after the exit mechanism is triggered, the crossover probability of two points increases linearly from the initial value of 0.65 to the upper limit of 0.9 in a step of 0.05 per generation and remains constant.

[0050] In the example of calculating the mutation probability, let the current evolutionary generation be... The value is 25, its square root reciprocal is 0.2, and the mean fitness of its cluster is... The value is 0.8. Substituting this into the formula, we can obtain the mutation probability. Approximately 0.0898. In the version hash index construction phase, the preferred range for the number of core keywords selected (i.e., the aforementioned preset number N) is 5 to 10; the hash encryption engine can use either SHA256 or MD5 algorithms, and the generated digest hash code example is a 32-bit hexadecimal string a1b2c3d4.

[0051] As an optional implementation, the test paper storage instructions, which include the test question ID sequence and the concatenated primary key, can be directly written to the corresponding data table by executing the SQL INSERT statement using the pymysql library, thereby completing the disk archiving operation of the test paper data.

[0052] To objectively evaluate the technical effectiveness of the above implementation methods, the following test benchmark was constructed: A test corpus of 500,000 multi-subject test questions accumulated over the past five years by a municipal education bureau was used, and 5,000 automatic test paper generation tasks were randomly selected as evaluation samples. Hardware deployment employed computing nodes equipped with a 64-core CPU and a dual-path deep learning GPU, while software was based on mainstream tensor computing libraries and a plain text development environment. Three control groups were established: the baseline group used a standard genetic algorithm combining traditional four-bit character labeling with fixed crossover and mutation probabilities to process the test paper generation task; the ablation group, based on the baseline group, introduced six categories of mutually exclusive character position marker states and normalized stroke feature templates; and the complete scheme group, based on the ablation group, further superimposed a mutation probability control mechanism set by the product of the square root of the iteration count and the negative exponent of the cluster fitness mean.

[0053] The experimental results are as follows: The baseline group achieved a recall rate of only 76.5% for identifying long strings of words not yet registered, with an average convergence generation of 125 generations, and the average fitness score of the generated test paper scheme was only 0.72. After introducing fine-grained annotation features, the ablation group saw a rapid increase in the recall rate for identifying long-segment professional vocabulary boundaries to 89.25%, a word segmentation comprehensive index score of 87.6%, a slight decrease in the average convergence generation to 118 generations, and an increase in the average fitness score to 0.78. The complete scheme group maintained a high recall rate of 89.4% for identifying long and difficult words, while significantly reducing the average convergence generation to 68 generations, and the average fitness score of the generated global test paper scheme jumped to an excellent level of 0.91.

[0054] Further analysis in conjunction with the attached figures, such as Figure 2 As shown, the population fitness variance decreases rapidly in the early stages of evolution, gradually stabilizing with increasing generations, reflecting the shrinking individual differences and increasing homogeneity within the population. The two-point crossover probability remains stable in the early stages of evolution, gradually increasing in tandem with the degree of population homogeneity, and then maintaining a constant value in the later stages. This visually demonstrates the regulatory logic of this invention, which adaptively adjusts the two-point crossover probability based on changes in population fitness variance. Figure 3 As shown, the baseline group uses traditional basic technical solutions, the ablation group lacks some key optimization methods of this invention, and the complete solution group is the complete technical solution of this invention. The comparison shows that the complete solution group outperforms both the baseline and ablation groups in terms of out-of-vocabulary word recognition effectiveness, algorithm convergence efficiency, and overall test paper assembly performance. This fully verifies the synergistic optimization effect of multiple technical methods in this invention, including word segmentation ambiguity resolution, enhanced out-of-vocabulary word recognition, adaptive feature weight optimization, and genetic algorithm perturbation recombination.

[0055] This invention also discloses an automatic digital archives file assembly and management system based on big data analysis, including a processor and a memory. The memory stores computer program instructions, and when the computer program instructions are executed by the processor, the automatic digital archives file assembly and management method based on big data analysis according to this invention is implemented.

[0056] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for automatic file generation and management of digital archives based on big data analysis, characterized in that: include: S1. Collect and clean the test question archive text to obtain the text to be segmented, and construct a professional dictionary based on a double-array Trie tree. Each entry in the professional dictionary is attached with a word frequency weight and a semantic category label. The text to be segmented is scanned bidirectionally using forward and reverse maximum matching. The optimal segmentation result is determined based on the number of segments in the scanning result and the evaluation ratio obtained by dividing the product of the word frequency weights of each entry on the segmentation path by the sum of the number of single characters and the preset smoothing constant. S2. Use a conditional random field model that includes Chinese character stroke features and pinyin initial letter features to identify out-of-vocabulary words in the segmentation results; The weighted TF-IDF value is calculated by replacing the inverse document frequency with the distribution entropy of the archive category, and the distribution entropy is fed back to the dual-array Trie tree to adjust the word frequency weights. S3. Based on the weighted TF-IDF values, construct feature vectors and cluster them. Use a genetic algorithm to extract test questions from the clusters to generate test papers. When the genetic algorithm iterates to a stable state, introduce the distribution entropy as a perturbation factor to perform test paper reorganization. Write the generated test papers into the database to establish a version index. The primary key is composed of the keyword hash value extracted based on the weighted TF-IDF values, the comprehensive value of the test paper file category distribution entropy, and the generation timestamp.

2. The method for automatic digital archives compilation and management based on big data analysis according to claim 1, characterized in that, The process of collecting test question archive text and cleaning it to obtain the text to be segmented includes: performing optical character recognition on digitized scanned images to obtain an initial text sequence; Use regular expressions to remove invisible control characters, page breaks, and formatting marks from text and replace them with standard spaces. Convert full-width English letters and punctuation marks to half-width format and uniformly map English letters to lowercase to generate standardized text to be segmented.

3. The method for automatic digital archives compilation and management based on big data analysis according to claim 1, characterized in that, The construction of a specialized dictionary based on a double-array Trie tree, wherein each entry in the specialized dictionary is appended with a word frequency weight and a semantic category label, includes: extracting specialized terms from archival classification management specifications and historical dossier documents to form a standard term set; initializing a double-array Trie tree containing a base address one-dimensional array and a detection one-dimensional array, and mapping and storing the words in the standard term set character by character as tree nodes; calculating the absolute frequency of each entry in the pre-constructed archival corpus, dividing it by the total word frequency of the corpus to obtain the word frequency weight of the entry; extracting the identification code of the archival classification to which the entry belongs as a semantic category label, and appending and storing the category label and word frequency weight together in the end address node of the corresponding entry in the double-array Trie tree.

4. The method for automatic digital archives compilation and management based on big data analysis according to claim 1, characterized in that, The optimal segmentation result is determined by the evaluation ratio obtained by dividing the product of the word segmentation count and the word frequency weights of each word on the segmentation path by the sum of the number of single characters and a preset smoothing constant, including: If the word segmentation counts of the two results obtained from the forward and reverse maximum matching bidirectional scanning are inconsistent, the result with the smaller word segmentation count shall be selected as the segmentation result. If the two results have the same number of words but different segmentation paths, calculate the product of the word frequency weights of each word on the two segmentation paths, and calculate the ratio of the product divided by the sum of the number of single characters and the preset smoothing constant. Take the result corresponding to the largest ratio as the segmentation result.

5. The method for automatic compilation and management of digital archives based on big data analysis according to claim 1, characterized in that, The method of using a conditional random field model containing Chinese character stroke features and pinyin initial letter features to identify out-of-vocabulary words in the segmentation results includes: expanding the character position markers to six mutually exclusive states, and combining the dependency relationship between the current Chinese character and the characters in the preceding and following windows during feature serial scanning to output the identification sequence; wherein, the Chinese character stroke features are normalized stroke count feature values ​​obtained by dividing the absolute number of strokes of the Chinese character to be identified by the global average stroke constant value; The pinyin initial letter feature is obtained by extracting the initial letter of the pinyin of the Chinese character to be identified and adjacent Chinese characters, statistically distributing them in the double-array Trie tree, and calculating the semantic category smoothing probability using the Laplace smoothing algorithm.

6. The method for automatic digital archives compilation and management based on big data analysis according to claim 1, characterized in that, The step of calculating the weighted TF-IDF value based on the inverse document frequency by replacing the archive category distribution entropy, and feeding the distribution entropy back to the dual-array Trie tree to adjust the term frequency weights, includes: obtaining the conditional probability that documents containing the target term belong to a certain category under all categories, and calculating the archive category distribution entropy of the term according to the Shannon entropy formula; Determine whether the entropy of the file category distribution is lower than a set threshold. If so, update the word frequency weight of the term in the double-array Trie tree to: the original weight multiplied by an amplification factor, where the amplification factor is the sum of 1 and the reciprocal of the entropy of the file category distribution.

7. The method for automatic digital archives compilation and management based on big data analysis according to claim 1, characterized in that, The process of using a genetic algorithm to extract test questions from clusters and generate test papers includes determining crossover and mutation probabilities based on the number of iterations and the mean fitness of the clusters. Monitor the fitness variance of the population over generations. When the fitness variance decreases and tends to be homogeneous, linearly increase the probability of two-point crossover. The mutation probability is calculated using the following formula: the mutation probability is equal to the reciprocal of the square root of the current number of iterations in the genetic algorithm, multiplied by the negative exponent of the arithmetic mean of the fitness of all members in the cluster to which the mutated individual belongs, with the natural constant as the base.

8. The method for automatic compilation and management of digital archives based on big data analysis according to claim 7, characterized in that, The step of introducing the distribution entropy as a perturbation factor to perform test paper recombination when the genetic algorithm iterates to a stable state includes: when the fitness variance of the population for three consecutive generations is lower than the set stagnation threshold, it is determined that the iterative process has reached a stable state, and the chromosomes of the current population are traversed. The corresponding distribution entropy value is multiplied by the randomly generated standard normal distribution coefficient as a perturbation factor and added to the weighted TF-IDF feature corresponding to the gene locus. The clustering algorithm is called again to perform clustering and the questions are selected proportionally.

9. The method for automatic compilation and management of digital archives based on big data analysis according to claim 1, characterized in that, The primary key is composed of keyword hash values ​​extracted based on weighted TF-IDF values, comprehensive values ​​of test paper archive category distribution entropy, and generation timestamp, including: sorting the generated test paper text in descending order of weighted TF-IDF values, and extracting a preset number of keywords from the top to form a string; The hash encryption engine is invoked to generate a digest hash code for the string; the timestamp of the instant the test paper generation instruction is issued is obtained, and the comprehensive value of the file category distribution entropy of the test paper containing test questions is calculated; The digest hash code and the distribution entropy combined value are concatenated to form a unique identifier sequence, which is stored in a two-dimensional relational database as the primary key of the version index.

10. A digital archive automatic file generation management system based on big data analysis, characterized in that: include: A processor and a memory, wherein the memory stores computer program instructions that, when executed by the processor, implement the automatic digital archive compilation and management method based on big data analysis according to any one of claims 1-9.

Images