A personalized service-oriented intelligent online evaluation system design method

Abstract

The application discloses a kind of intelligent online evaluation system design methods for personalized service, comprising the following steps: (1) the domain-specific language created using different DSL design methods;(2) design the sandbox based on Docker and Linux command;(3) feature extraction and model building are carried out using deep learning model, Circle Loss is used as loss function, and deep learning model is trained to solve the multi-label classification problem of algorithm label;(4) build online evaluation module, integrate web, server and evaluation terminal;The application simultaneously considers security and intelligence.

Description

Technical Field

[0001] This invention relates to the field of software development technology, and more specifically to a design method for an intelligent online evaluation system oriented towards personalized services. Background Technology

[0002] To support various programming competitions, a robust online judge system is essential. This system needs to automatically evaluate the correctness of solutions in real time, ensuring that solutions do not exceed space and time limitations. Online judges (OJs) provide automated evaluation capabilities, significantly reducing the burden of manual judging and making the teaching process more efficient. Therefore, they are widely used in various computer competitions and educational settings.

[0003] However, most existing online judge (OJ) platforms lack scalability in evaluation, failing to support evaluation of communication-based problems and unable to handle various question types that may emerge in the future. Furthermore, they often suffer from complex operation and chaotic permission management. Even with some open-source OJs, it is difficult to perform secondary development on their foundations. Summary of the Invention

[0004] Purpose of the invention: The purpose of this invention is to provide a design method for an intelligent online assessment system oriented towards personalized services. It is a general assessment method based on JSON, FreeMarker, and JetBrains MPS-created DSL, which solves the problems of insufficient support for multiple question types and limited scalability of existing online assessment systems.

[0005] Technical Solution: The present invention provides a design method for an intelligent online evaluation system oriented towards personalized services, comprising the following steps:

[0006] (1) Domain-specific languages ​​created using different DSL design methodologies;

[0007] (2) Design a sandbox based on Docker and Linux commands;

[0008] (3) Deep learning models are used for feature extraction and model building, and Circle Loss is used as the loss function to train deep learning models to perform multi-label classification problems of algorithm labels;

[0009] (4) Construct an online evaluation module that integrates the web page, server and evaluation terminal.

[0010] Furthermore, in step (1), different DSL design methods include JSON, FreeMarker, and JetbrainsMPS.

[0011] Furthermore, step (1) includes the following steps:

[0012] (11) The test questions and time and space constraints are described using JSON, and parsed using the C++ nlohmann::json library during evaluation;

[0013] (12) Generate scripts for the test data, describe them using DSL-FreeMarker, use the <#list> syntax to describe a large number of regular scripts, and use Java to generate the corresponding scripts during evaluation;

[0014] (13) Use JetBrains’ meta-language programming system MPS to create the Strategy language, carry out conceptual design, editor design and generator design, and implement evaluation of traditional problem types, construction problem types, interactive problem types and communication problem types.

[0015] In the conceptual design, the JetBrains MPS Concept is used to describe the attributes of a concept and determine its inheritance relationship. In the editor design, a vertical layout is used for the syntax design of each concept. In the generator design, each concept is mapped to code snippets and Java classes, with specific code parameters described using the concept's attributes, combined with the LOOP macro, COPY_SRC macro, and the Collection language.

[0016] (14) After the DSL design is completed, the conversion from DSL to JAVA code to executable file is carried out.

[0017] Furthermore, step (2) includes the following steps:

[0018] (21) Use Docker to build an image and run it in the image;

[0019] (22) Use Linux commands to impose time and space constraints;

[0020] (23) Use seccomp to restrict system calls;

[0021] (24) Run the code using the nobody user.

[0022] Furthermore, step (3) includes the following steps:

[0023] (31) Use the Codeforces website to obtain open-source website datasets;

[0024] (32) For the multi-label classification problem, BERT is used to extract the title description features and a deep CNN network is used to extract the code features;

[0025] (33) Use Circle Loss as the loss function for the model; specifically as follows: Single-label Circle Loss

[0026] ;

[0027] in Represents the number of label categories. Represents the target category. This represents the score of the i-th category. The above formula is an approximation of the max function, which essentially aims to make the scores of other categories as small as possible compared to the target category. Generalizing to: for each target category... The scores of the target classes should be as high as possible. That is, the following losses are incurred:

[0028] ;

[0029] in A set representing the target categories. A set representing non-target categories.

[0030] If we include an empty class (0) in the prediction process and output all labels greater than the empty class as the answer, then the loss function should be changed to:

[0031] ;

[0032] in The score represents the empty class 0.

[0033] Furthermore, step (4) includes the following steps:

[0034] (41) Web page development based on Vue and Vant component library: using Vue’s responsiveness, component reuse, route management and Vant component library, the user module, question module and competition module of the web page are obtained;

[0035] (42) A Node.js-based server uses app.get() or app.post() to handle various web page requests, makes requests to the database and evaluation end as needed, and returns the corresponding results;

[0036] (43) The evaluation machine based on C++ and Java receives the code path, test question configuration path, evaluation type and code type through command line parameters; it uses the nlohmann::json library to parse the configuration file config.json and determines the evaluation behavior according to the parameters.

[0037] The present invention discloses a design system for an intelligent online evaluation system oriented towards personalized services, comprising:

[0038] DSL Design Module: Used to create domain-specific languages ​​using different DSL design methodologies;

[0039] Sandbox module: Used to design sandboxes based on Docker and Linux commands;

[0040] Feature building module: used for feature extraction and model building of deep learning models, using Circle Loss as the loss function to train deep learning models to solve multi-label classification problems with algorithmic labels;

[0041] Integration Module: Used to build online evaluation modules, integrating the web page, server, and evaluation platform.

[0042] An electronic device according to the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is loaded onto the processor, it implements a design method for an intelligent online evaluation system for personalized services according to any one of the claims.

[0043] The present invention provides a storage medium storing a computer program, which, when executed by a processor, implements any of the above-described intelligent online evaluation system design methods for personalized services.

[0044] Beneficial Effects: Compared with existing technologies, this invention has the following significant advantages: It uses different DSL design methods to address different needs, exhibiting strong scalability and facilitating the description of evaluation methods by domain experts and the understanding of evaluation methods by evaluation machines. For the characteristic of online evaluation systems requiring frequent execution of unknown programs, a sandbox design based on Docker and Linux commands is proposed to enhance system security. Deep learning models such as BERT and CNN are used for feature extraction and model building, and Circle Loss is used as the loss function to train deep learning models for multi-label classification problems, achieving good results. An online evaluation system is constructed, integrating the web page, server, and evaluation terminal, ensuring the effective application of the above modules. This invention simultaneously considers system security and intelligence, thus possessing high application value. Attached Figure Description

[0045] Figure 1 This is the system architecture diagram of the present invention.

[0046] Figure 2 This is a flowchart of the data generation process for this invention;

[0047] Figure 3 This is a schematic diagram illustrating the command generation using FreeMarker in this invention;

[0048] Figure 4 A diagram illustrating the inclusion and inheritance relationships of the DSL concept designed for this invention;

[0049] Figure 5 This is a schematic diagram illustrating the design and operation of a DSL using Jetbrains MPS in this invention;

[0050] Figure 6 This is a schematic diagram of the code feature extraction model using deep CNN in this invention;

[0051] Figure 7 This is a schematic diagram of the question description and code submission page of the online evaluation system of this invention. Detailed Implementation

[0052] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0053] like Figure 1 As shown, this embodiment of the invention provides a design and implementation method for an intelligent online assessment system oriented towards personalized services, comprising four steps: design of a general assessment method for the online assessment system, sandbox design based on Docker and Linux commands, and design of an intelligent label prediction algorithm. The system mainly consists of a web page, a server, and an assessment terminal. The Node.js server acts as a "middle layer," communicating with the web page, database, and assessment terminal. The assessment terminal parses a general assessment description file designed using DSL design tools such as JetBrains MPS, and the code is fed into the label prediction algorithm for predicting test item labels. During assessment, the assessment terminal also needs to run the code within Docker.

[0054] S1. Design of a general evaluation method for online evaluation systems

[0055] S11 uses JSON directly to describe information such as question descriptions and time / space constraints, which is already best practice. During evaluation, the C++ nlohmann::json library is used for parsing, allowing JSON to be managed like an STL container.

[0056] S12 as Figure 2 As shown, in the data generation process, the question setter uploads a generator and scripts to construct data and obtain the input for the test points. The generator commands resemble a large number of commands like `gen 1 2>1.txt`. To facilitate the maintenance of a large number of regular scripts, the existing DSL—FreeMarker—is used for description. The `<#list>` syntax can be used to describe and generate a large number of commands. During evaluation, since FreeMarker is a Java library, Java can be used to generate the corresponding commands. Figure 3This is a diagram illustrating the commands generated using FreeMarker; the comments section shows the generated commands.

[0057] S13 uses JetBrains' meta-language programming system MPS to create the Strategy language for evaluation methods, enabling conceptual design, editor design, and generator design. It can evaluate traditional, construction, interactive, and communication-related questions.

[0058] In conceptual design, the Concept class from JetBrains MPS can be used to describe a concept and determine its inheritance relationships. In the specific implementation steps, this may include the definition of a file (File), a pipe (for interactive tasks), and the execution of a program (user program, inspector). The former inherits from the abstract concept IO, and the latter inherits from the abstract concept Executable. Figure 4 This refers to the inheritance relationship of concepts when designing a DSL for test questions using JetBrains.

[0059] In editor design, taking the top-level concept "Strategy" as an example, its various steps can be arranged in a vertical layout, which will determine the syntax of the concept. The code is as follows.

[0060] Node cell layout:

[0061] [-

[0062] Strategy { name}

[0063] ( / % Steps % / )

[0064] / empty cell: <default>

[0065] -]

[0066] In generator design, the top-level concept `Strategy` is first mapped to a Java class, and the remaining concepts are mapped to code snippets. Taking a file (the `File` concept) as an example, the `LOOP` macro and the `COPY_SRC` macro are used to traverse all steps (the `Step` concept), and the `Collection` language is used to set the circular list.

[0067] iteration sequence: (genContext, node)->sequence <node<> > {

[0068] node.Steps.where({it => it.isInstanceOf(File);});

[0069] }

[0070] It also sets the mapping rule Reduce_File, which maps the attributes of the File concept to the corresponding code snippets.

[0071] The mapping of other concepts is described in a similar way. For example, User and Checker represent a program execution, which is mapped to Java using ProcessBuilder.

[0072] Once the generator is complete, you can attempt to generate Java code. If the program reports no errors, you can have Strategy implement IMainClass as the main class of the Java project and try running it in MPS. Figure 5 As shown.

[0073] S2, Sandbox Design Based on Docker and Linux Commands

[0074] S21Docker uses sandbox technology. When you want to run a program, you can use Docker to build an image and run the program within that image.

[0075] In Linux, `setitimer` can limit a process's time usage; exceeding the limit will trigger a timeout. The `setrlimit` function can limit a process's memory usage; memory allocation beyond the limit will fail. Linux commands can be used to apply time and space constraints.

[0076] S23 can use seccomp to restrict system calls. By pre-setting and enabling filtering rules, no additional context switching is required. Experiments show that the program runtime remains almost unchanged, reducing a significant amount of overhead compared to ptrace.

[0077] S24 uses the nobody user to run code. Even if the program bypasses sandbox rules, greater harm can be avoided through Linux access control. This can be achieved in C++ using setuid and setgid.

[0078] S3, Label Prediction Intelligent Algorithm

[0079] S31 used a Python web crawler to scrape basic information and partial AC (Accepted) code for 3697 code problems from the Codeforces website. A self-built dataset was used for model training. The final dataset was divided into training, validation, and test sets in a 7:1:2 ratio, containing 2587, 370, and 740 problems respectively. For the problem descriptions, only the main text was used, ignoring information such as sample inputs and outputs that are difficult to use for label prediction. For the AC code, all comments and some punctuation were removed, and word frequencies were calculated.

[0080] S32 This problem can be divided into three tasks: first, predicting the difficulty label, which is equivalent to a regression problem; second, predicting the algorithm label, which is a multi-label problem; and third, predicting the question type, which is equivalent to a three-class classification problem.

[0081] For the difficulty prediction problem, S33 uses the mean squared error (MSELoss) as the evaluation metric. For the multi-label prediction problem, the Jaccard score is used, which is the ratio of the intersection to the union of the predicted and true labels; a value closer to 1 is better. For the three-class classification problem, accuracy (ACC), precision, recall, and F1 score are used for evaluation. The optimizer for the model is Adam, with a learning rate of 5e-4. The PyTorch randomization seed is 42, and the number of epochs is 20. The loss function is the sum of MSELoss (difficulty prediction), Circle Loss (algorithm label prediction), and CrossEntropyLoss (item type prediction).

[0082] For multi-label classification problems, S34 uses BERT to extract features from the title description and a deep CNN network to extract features from the code. The extracted features are then fed into a fully connected layer for further classification. Figure 6 It is the model structure of a deep CNN network.

[0083] S35 addresses multi-label classification problems by using Circle Loss as the model's loss function. Specifically, it uses Circle Loss for single-label classification.

[0084] ;

[0085] in Represents the number of label categories. Represents the target category. This represents the score of the i-th category. The above formula is an approximation of the max function, which essentially aims to make the scores of other categories as small as possible compared to the target category. Generalizing to: for each target category... The scores of the target classes should be as high as possible. That is, the following losses are incurred:

[0086] ;

[0087] in A set representing the target categories. A set representing non-target categories.

[0088] If we include an empty class (0) in the prediction process and output all labels greater than the empty class as the answer, then the loss function should be changed to:

[0089] ;

[0090] in The score represents the empty class 0.

[0091] S36 Baseline Model Design: BERT is directly used to extract features from unprocessed code submissions, and CrossEntropyLoss is used instead of Circle Loss for multi-label classification. The results of comparative experiments using this invention's model and the baseline model are shown in Tables 1 and 2.

[0092] Table 1. Comparison of the model used in this invention with the baseline model on difficulty prediction and multi-label prediction problems.

[0093] ;

[0094] Table 2. Comparison of the model used in this invention with the baseline model on the question type prediction problem.

[0095] ;

[0096] S4. Design of Online Evaluation System

[0097] S41 is a web-based development platform using Vue and the Vant component library. It leverages Vue's responsiveness, component reuse, route management, and the convenient Vant component library to implement user modules, question modules, competition modules, and more on the web page. Figure 7 This is the question description and code submission page for the online assessment system.

[0098] S42 is a server-side application developed based on Node.js. It uses app.get() or app.post() to handle various web page requests, makes requests to the database and evaluation end as needed, and returns the corresponding results.

[0099] S43 is an evaluation machine developed based on C++ and Java. It receives code path, test configuration path, evaluation type, and code type via command-line arguments. It uses the nlohmann::json library to parse the configuration file config.json and determines the evaluation behavior based on the parameters.

[0100] This invention also provides a design system for an intelligent online evaluation system oriented towards personalized services, comprising:

[0101] DSL Design Module: Used to create domain-specific languages ​​using different DSL design methodologies;

[0102] Sandbox module: Used to design sandboxes based on Docker and Linux commands;

[0103] Feature building module: used for feature extraction and model building of deep learning models, using Circle Loss as the loss function to train deep learning models to solve multi-label classification problems with algorithmic labels;

[0104] Integration Module: Used to build online evaluation modules, integrating the web page, server, and evaluation platform.

[0105] This invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is loaded onto the processor, it implements a soil moisture fusion method based on XGB-SHAP-RFE feature optimization and SHAP interpretable analysis as described in any one of the claims.

[0106] This invention also provides a storage medium storing a computer program, which, when executed by a processor, implements any of the methods described above for fusing soil moisture based on XGB-SHAP-RFE feature optimization and SHAP interpretable analysis.< / default>

Claims

1. A design method for an intelligent online evaluation system oriented towards personalized services, characterized in that, Includes the following steps: (1) Domain-specific languages ​​created using different DSL design methodologies; (2) Design a sandbox based on Docker and Linux commands; (3) Use a deep learning model for feature extraction and model building, and use Circle Loss as the loss function to train the deep learning model to perform multi-label classification of algorithmic labels; step (3) includes the following steps: (31) Use the Codeforces website to obtain open-source website datasets; (32) For the multi-label classification problem, BERT is used to extract the title description features and a deep CNN network is used to extract the code features; (33) Use Circle Loss as the loss function for the model; specifically as follows: Single-label Circle Loss ； in, Represents the number of label categories. Represents the target category. Let represent the score of the i-th category; the above formula is an approximation of the max function, which essentially aims to make the scores of other categories as small as possible compared to the target class; if generalized to: each target class The scores of the target classes should be as high as possible. That is, the following losses are incurred: ； in A set representing the target categories. A set representing non-target categories; If we include an empty class (0) in the prediction process and output all labels greater than the empty class as the answer, then the loss function should be changed to: ； in The score representing the empty class 0; (4) Construct an online evaluation module that integrates the web page, server and evaluation terminal.

2. The design method of an intelligent online evaluation system for personalized services according to claim 1, characterized in that, In step (1), different DSL design methods include JSON, FreeMarker, and Jetbrains MPS.

3. The design method of an intelligent online evaluation system for personalized services according to claim 2, characterized in that, Step (1) includes the following steps: (11) The test questions and time and space constraints are described using JSON, and parsed using the C++ nlohmann::json library during evaluation; (12) Generate scripts for test data, describe them using DSL-FreeMarker, use the <#list> syntax to describe regular scripts, and use Java to generate the corresponding scripts during evaluation; (13) Use JetBrains’ meta-language programming system MPS to create the Strategy language, carry out conceptual design, editor design and generator design, and implement evaluation of traditional problem types, construction problem types, interactive problem types and communication problem types; In the concept design, the Concept in JetBrains MPS is used to describe the attributes of a concept and determine the inheritance relationship of the concept. In the editor design, a vertical layout is used for the syntax design of each concept. In the generator design, each concept is mapped to code snippets and Java classes. The specific code parameters are described using the attributes of the concept, combined with the LOOP macro, COPY_SRC macro, and Collection language. (14) After the DSL design is completed, the conversion from DSL to JAVA code to executable file is carried out.

4. The design method of an intelligent online evaluation system for personalized services according to claim 1, characterized in that, Step (2) includes the following steps: (21) Use Docker to build an image and run it in the image; (22) Use Linux commands to impose time and space constraints; (23) Use seccomp to restrict system calls; (24) Run the code using the nobody user.

5. The design method of an intelligent online evaluation system for personalized services according to claim 1, characterized in that, Step (4) includes the following steps: (41) Web page development based on Vue and Vant component library: using Vue’s responsiveness, component reuse, route management and Vant component library, the user module, question module and competition module of the web page are obtained; (42) A Node.js-based server uses app.get() or app.post() to handle various web page requests, makes requests to the database and evaluation end as needed, and returns the corresponding results; (43) The evaluation machine based on C++ and Java receives the code path, test question configuration path, evaluation type and code type through command line parameters; it uses the nlohmann::json library to parse the configuration file config.json and determines the evaluation behavior according to the parameters.

6. A design system for an intelligent online evaluation system oriented towards personalized services, characterized in that, The method described in any one of claims 1-5 is used to implement the method, comprising: DSL Design Module: Used to create domain-specific languages ​​using different DSL design methodologies; Sandbox module: Used to design sandboxes based on Docker and Linux commands; Feature building module: Used for feature extraction and model building using deep learning models, and uses Circle Loss as the loss function to train deep learning models to solve multi-label classification problems with algorithmic labels; Integration Module: Used to build online evaluation modules, integrating the web page, server, and evaluation platform.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the computer program is loaded into the processor, it implements the design method of an intelligent online evaluation system for personalized services according to any one of claims 1-5.

8. A storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the design method of an intelligent online evaluation system for personalized services according to any one of claims 1-5.

Images