Model training data preprocessing method and device for school management system

Abstract

The invention provides a model training data preprocessing method and device for a school management system, and the method comprises the steps: obtaining to-be-processed original data, and constructing a training data set; dividing data in the training data set according to a preset dimension to obtain at least two data subsets with different dimensions; respectively processing the data in each data subset to obtain a sub-vector corresponding to each data subset; and splicing the sub-vectors to obtain a spliced result vector. Data preprocessing is carried out on original data in the training data set to remove redundant data in a database, and the problems that the data dimension is too high and training is over-fitted are solved; therefore, by processing the collected original data, redundant data in the database is removed, the model training efficiency is improved, and overfitting in model training is avoided.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a model training data preprocessing method and device for a school management system. Background Art

[0002] In the process of vulnerability detection in the education management system, artificial intelligence detection systems are usually used to improve the detection efficiency and accuracy of vulnerability detection using artificial intelligence recognition systems. When building an artificial intelligence detection system, it is necessary to collect a large amount of target data and build a huge database. Due to the large amount of data, different samples may contain similar information, so information redundancy may exist.

[0003] In the existing technology, when processing data, the method of embedding the content and response header information after parsing the HTML of the response content is easy to cause problems of too high dimension and overfitting of training. In addition, since all the content is parsed, the desired content information cannot be accurately learned.

[0004] In view of this, a model training data preprocessing method and device for a school management system are provided, in order to preprocess the original data in the training data set to eliminate redundant data in the database and solve the problems of too high data dimension and overfitting of training. Summary of the invention

[0005] The present invention provides a model training data preprocessing method and device for a school management system, in order to perform data preprocessing on the original data in the training data set, so as to eliminate redundant data in the database and solve the problems of too high data dimension and overfitting in training; thereby, by processing the collected original data, eliminating redundant data in the database, improving the model training efficiency and avoiding overfitting in model training.

[0006] The present invention provides a model training data preprocessing method for a school management system, the method comprising: Obtain the raw data to be processed and construct a training data set; Dividing the data in the training data set according to a preset dimension to obtain data subsets of at least two different dimensions; Process the data in each data subset respectively to obtain the sub-vector corresponding to each data subset; The sub-vectors are concatenated to obtain a concatenated result vector.

[0007] In some embodiments, the preset dimensions include content dimensions and structure dimensions.

[0008] In some embodiments, when the preset dimension is a content dimension, processing the data in the data subset specifically includes: The data in the data subset are processed in turn by removing special symbols, segmenting the content, removing high- and low-frequency words, converting into vectors, and enhancing the data.

[0009] In some embodiments, the data in the data subset is processed to remove special symbols, specifically including: Use regular expressions to remove special characters that cannot represent page content; Remove link information from the data, and remove whitespace and punctuation that does not aid understanding.

[0010] In some embodiments, data enhancement processing is performed on the data in the data subset, specifically including: The sub-vector obtained after vector transformation is input into the CTGAN network for data enhancement.

[0011] In some embodiments, when the preset dimension is a structural dimension, processing the data in the data subset specifically includes: The data in the data subset are processed in turn by removing duplicate structures, vector conversion and data enhancement.

[0012] The present invention also provides a model training data preprocessing device for a school management system, the device comprising: A data acquisition unit, used to acquire raw data to be processed and construct a training data set; A data partitioning unit, configured to partition the data in the training data set according to a preset dimension to obtain data subsets of at least two different dimensions; A vector conversion unit, used for processing the data in each data subset respectively to obtain a sub-vector corresponding to each data subset; The vector concatenation unit is used to concatenate the sub-vectors to obtain a concatenated result vector.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described above when executing the program.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the method described above when executed by a processor.

[0015] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method described above is implemented.

[0016] The model training data preprocessing method and device provided by the present invention for the school management system obtains the original data to be processed and constructs a training data set; divides the data in the training data set according to a preset dimension to obtain at least two data subsets of different dimensions; processes the data in each data subset respectively to obtain a sub-vector corresponding to each data subset; and splices each sub-vector to obtain a spliced ​​result vector. The method and device provided by the present invention use two information feature engineering processes, namely, the content and structure of the data, to obtain more data information to a great extent and improve the accuracy of subsequent models; at the same time, the data enhancement technology of CTGAN is used to enhance the data, reducing the workload of staff in collecting data; and when processing the data, low-frequency words and high-frequency words, as well as some words segmented out by the historical experience of the branch, are removed, so that the data information can be obtained more accurately and the accuracy of subsequent model training can be improved. In this way, the method and device provided by the present invention perform data preprocessing on the original data in the training data set to eliminate redundant data in the database, thereby solving the problems of too high data dimension and overfitting in training; thereby, by processing the collected original data and eliminating redundant data in the database, the model training efficiency is improved and overfitting in model training is avoided. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is one of the flow charts of the model training data preprocessing method for school management system provided by the present invention; Figure 2 This is the second flow chart of the model training data preprocessing method for a school management system provided by the present invention; Figure 3 It is a structural block diagram of a model training data preprocessing device for a school management system provided by the present invention; Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] In order to solve the problems existing in the prior art, especially the problems of large data dimension, easy overfitting and inability to accurately learn the desired content information in the data preprocessing process of the artificial intelligence model during the automated scanning of the ideological and political education and management system in colleges and universities, the present invention provides a data preprocessing method, which can accurately learn data information according to a fine-grained data preprocessing scheme. At the same time, the scheme also creatively proposes to use the information of the web page structure for training, which greatly improves the model accuracy. In view of the problems that vulnerability data is difficult to obtain and setting up a target range is time-consuming and labor-intensive, the method provided by the present invention uses reinforcement learning to perform data enhancement, thereby increasing the amount of training data and improving the accuracy of subsequent training models to a certain extent.

[0021] In a specific embodiment, Figure 1 As shown, the present invention provides a model training data preprocessing method for a school management system, comprising the following steps: S110: Obtaining raw data to be processed and constructing a training data set; specifically, building a shooting range, crawling all paths of the shooting range using a crawler, and obtaining a data set; S120: Divide the data in the training data set according to a preset dimension to obtain data subsets of at least two different dimensions; S130: Process the data in each data subset respectively to obtain a sub-vector corresponding to each data subset; S140: concatenate the sub-vectors to obtain a concatenated result vector.

[0022] In step S120, the preset dimensions include content dimension and structure dimension; in a specific usage scenario, the crawled data set (the structure of the html response data of the http request) is split using XPAHT into two parts: content and structure.

[0023] In some embodiments, when the preset dimension is a content dimension, processing the data in the data subset specifically includes: The data in the data subset are processed in turn by removing special symbols, segmenting the content, removing high- and low-frequency words, converting into vectors, and enhancing the data.

[0024] The data in the data subset is processed to remove special symbols, specifically including: Use regular expressions to remove special characters that cannot represent page content; Remove link information from the data, and remove whitespace and punctuation that does not aid understanding.

[0025] Perform data enhancement on the data subset, including: The sub-vector obtained after vector transformation is input into the CTGAN network for data enhancement.

[0026] In a specific usage scenario, such as Figure 2 As shown, obtaining the content part of the response data and processing the content part specifically include the following steps: 1. Use regular expressions to remove special characters that cannot represent page content, remove link information in the data, remove blanks, and punctuation marks that are not helpful for understanding; 2. Use a word segmenter to segment the information processed in step 1, and based on experience, retain some words needed for the current scenario when segmenting; 3. Count the frequency of the words that have been cut, and remove words with a frequency greater than 3000 and words with a frequency less than 100. Low-frequency word data actually has a certain amount of information, but when low-frequency words are put into the model for operation, they often maintain their random initial state, adding noise to the model; many high-frequency words are stop words with no information, and the stop words are processed directly by comparing them with the stop word list (depending on the actual task, many words do not need to appear in this task, and these words can also be added to the stop word list for processing, and the stop word list is made based on experience accumulation, and the data after word cutting is compared with the stop word list. If any words in the stop word list are found, they are deleted; 4. After comparative tests, it is found that the N-Gram model is the best for vector conversion of the data processed in step 3. A vector will be obtained for each page. N-Gram is an algorithm based on a statistical language model. Its basic idea is to perform a sliding window operation of size N on the content of the text according to bytes, forming a byte segment sequence of length N. In this embodiment, the final value of N is 4. Each byte segment is called a gram. The occurrence frequency of all grams is counted and filtered according to a pre-set threshold to form a key gram list, that is, the vector feature space of the text. Each gram in the list is a feature vector dimension. 5. The CTGAN algorithm is used to enhance the vector data obtained in step 4. GAN includes a discriminator network and a generator network, where the discriminator network is a convolutional neural network and the generator network is an inverse CNN. GAN obtains a random input and outputs it by upsampling the input data using transposed convolution. CTGAN designs a conditional generator that is trained through sampling to handle unbalanced discrete columns, which greatly ensures the consistency of the data distribution between the enhanced data and the original data.

[0027] In some embodiments, when the preset dimension is a structural dimension, processing the data in the data subset specifically includes: The data in the data subset are processed in turn by removing duplicate structures, vector conversion and data enhancement.

[0028] Still taking the above specific usage scenario as an example, obtaining the structure data of the response data and performing data processing specifically includes the following steps: 6. Remove duplicate parts in the response structure; 7. Vectorize the page structure: First, count all the tag types in the page, then encode different tags so that each tag has a unique code, and then map the tag data to convert the page into a vector representation. Since the encoding of each tag varies in size and is not in the same dimension, it affects the final model training effect. Therefore, normalize the transcoded data of each page so that the data is in the same dimension, improving the accuracy of the final training model. 8. Perform data enhancement on the data obtained in step 7 to obtain a large amount of training data; 9. Concatenate the data obtained in steps 5 and 8 to form a new vector for model training.

[0029] In the above specific implementation, the model training data preprocessing method for the school management system provided by the present invention obtains the original data to be processed and constructs a training data set; divides the data in the training data set according to a preset dimension to obtain at least two data subsets of different dimensions; processes the data in each data subset respectively to obtain the sub-vectors corresponding to each data subset; and splices each of the sub-vectors to obtain the spliced ​​result vector. The method provided by the present invention uses two information feature engineering processes, namely the content and structure of the data, to obtain more data information to a great extent and improve the accuracy of subsequent models; at the same time, the data enhancement technology of CTGAN is used to enhance the data, reducing the workload of staff in collecting data; and when processing the data, low-frequency words and high-frequency words, as well as some words segmented out by the historical experience of the branch, are removed, so that the data information can be obtained more accurately and the accuracy of subsequent model training can be improved. In this way, the method provided by the present invention performs data preprocessing on the original data in the training data set to eliminate redundant data in the database, thereby solving the problems of too high data dimension and overfitting in training; thereby, by processing the collected original data and eliminating redundant data in the database, the model training efficiency is improved and overfitting in model training is avoided.

[0030] In addition to the above method, the present invention also provides a model training data preprocessing device for a school management system, such as Figure 3 As shown, the device comprises: The data acquisition unit 310 is used to acquire the raw data to be processed and construct a training data set; A data partitioning unit 320 is used to partition the data in the training data set according to a preset dimension to obtain data subsets of at least two different dimensions; A vector conversion unit 330 is used to process the data in each data subset respectively to obtain a sub-vector corresponding to each data subset; The vector concatenation unit 340 is used to concatenate the sub-vectors to obtain a concatenated result vector.

[0031] In some embodiments, the preset dimensions include content dimensions and structure dimensions.

[0032] In some embodiments, when the preset dimension is a content dimension, processing the data in the data subset specifically includes: The data in the data subset are processed in turn by removing special symbols, segmenting the content, removing high- and low-frequency words, converting into vectors, and enhancing the data.

[0033] In some embodiments, the data in the data subset is processed to remove special symbols, specifically including: Use regular expressions to remove special characters that cannot represent page content; Remove link information from the data, and remove whitespace and punctuation that does not aid understanding.

[0034] In some embodiments, data enhancement processing is performed on the data in the data subset, specifically including: The sub-vector obtained after vector transformation is input into the CTGAN network for data enhancement.

[0035] In some embodiments, when the preset dimension is a structural dimension, processing the data in the data subset specifically includes: The data in the data subset are processed in turn by removing duplicate structures, vector conversion and data enhancement.

[0036] In the above specific implementation, the model training data preprocessing device for the school management system provided by the present invention obtains the original data to be processed and constructs a training data set; divides the data in the training data set according to a preset dimension to obtain at least two data subsets of different dimensions; processes the data in each data subset respectively to obtain the sub-vectors corresponding to each data subset; and splices each sub-vector to obtain the spliced ​​result vector. The device provided by the present invention uses two information feature engineering processes, namely the content and structure of the data, to obtain more data information to a great extent and improve the accuracy of subsequent models; at the same time, the data enhancement technology of CTGAN is used to enhance the data, reducing the workload of staff in collecting data; and when processing the data, low-frequency words and high-frequency words, as well as some words segmented out by the historical experience of the branch, are removed, so that the data information can be obtained more accurately and the accuracy of subsequent model training can be improved. In this way, the device provided by the present invention performs data preprocessing on the original data in the training data set to eliminate redundant data in the database, thereby solving the problems of too high data dimension and overfitting in training; thereby, by processing the collected original data and eliminating redundant data in the database, the model training efficiency is improved and overfitting in model training is avoided.

[0037] Figure 4 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 4 As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communication interface 420 and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute the above method.

[0038] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0039] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can perform the above method.

[0040] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0041] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0042] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A model training data preprocessing method for a school management system, characterized in that: The method comprises: Obtain the raw data to be processed and construct a training data set; Dividing the data in the training data set according to a preset dimension to obtain data subsets of at least two different dimensions; Process the data in each data subset respectively to obtain the sub-vector corresponding to each data subset; The sub-vectors are concatenated to obtain a concatenated result vector.

2. The model training data preprocessing method for school management system according to claim 1 is characterized in that: The preset dimensions include content dimension and structure dimension.

3. The model training data preprocessing method for school management system according to claim 2 is characterized in that: In the case where the preset dimension is a content dimension, processing the data in the data subset specifically includes: The data in the data subset are processed in turn by removing special symbols, segmenting the content, removing high- and low-frequency words, converting into vectors, and enhancing the data.

4. The model training data preprocessing method for school management system according to claim 3 is characterized in that: The data in the data subset is processed to remove special symbols, including: Use regular expressions to remove special characters that cannot represent page content; Remove link information from the data, and remove whitespace and punctuation that does not aid understanding.

5. The model training data preprocessing method for school management system according to claim 3 is characterized in that: Perform data enhancement on the data subset, including: The sub-vector obtained after vector transformation is input into the CTGAN network for data enhancement.

6. The model training data preprocessing method for school management system according to claim 2, characterized in that: In the case where the preset dimension is a structural dimension, processing the data in the data subset specifically includes: The data in the data subset are processed in turn by removing duplicate structures, vector conversion and data enhancement.

7. A model training data preprocessing device for a school management system, characterized in that: The device comprises: A data acquisition unit, used to acquire raw data to be processed and construct a training data set; A data partitioning unit, configured to partition the data in the training data set according to a preset dimension to obtain data subsets of at least two different dimensions; A vector conversion unit, used for processing the data in each data subset respectively to obtain a sub-vector corresponding to each data subset; The vector concatenation unit is used to concatenate the sub-vectors to obtain a concatenated result vector.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images