Intelligent evaluation method and system for multi-target balance capability

Abstract

The invention relates to the technical field of educational evaluation and neural network crossing, in particular to an intelligent evaluation method and system for multi-objective balance ability, and aims at a complex problem to be solved through fine design, multi-objective constituent factors are researched for the complex problem to be solved, and the multi-objective balance ability is evaluated. A characterization method of multiple objectives under a formalized framework is established; then, a multi-target balance capability evaluation theoretical framework which is oriented to multiple targets and driven by the formalized framework is constructed by utilizing a representation method of the multiple targets under the formalized framework, a preset typical strategy and the collected process data; and finally, constructing an automatic evaluation model based on the process data by utilizing a multi-view feature extraction method, the process data and a multi-target balance capability evaluation theoretical framework. According to the invention, the accuracy and effectiveness of multi-target balance capability evaluation in complex problem solving can be improved.

Description

Technical Field

[0001] The present application relates to the field of educational assessment and cross-technology of neural networks, and in particular to an intelligent assessment method and system for multi-objective balancing ability. Background Art

[0002] Complex Problem Solving (CPS) refers to the ability of an individual to solve problems by applying professional knowledge and high-order thinking skills when facing a dynamic and complex task environment. With the development of society and technology, CPS ability is increasingly regarded as an important skill in the 21st century. It is not only valued in the field of education, but also considered a key ability in the workplace. In order to effectively evaluate the CPS ability of individuals, researchers have developed various assessment tools and techniques. In order to evaluate this ability, researchers usually use complex systems to simulate real-world multi-objective problem-solving situations. Such systems usually contain a large number of related variables that change over time, and there may be conflicts between the goals of the system. Existing complex system assessment tools and methods mainly include traditional micro-worlds (such as the town of Lohausen and the garment factory) and multi-complex systems (such as MicroDYN and genetic laboratories).

[0003] (1) Traditional Microcosm

[0004] Technical principle: Traditional microworlds are computer models that simulate real-world problem-solving situations, usually involving multiple interrelated variables. They are designed to evaluate individuals' performance in solving complex problems, especially how to balance different goals in a dynamic environment.

[0005] Representative examples: The town of Lohausen and the garment factory are typical representatives of the traditional microworld. These systems simulate the operation of a town or a factory, and participants need to make a series of decisions to maintain the stable operation of the town or the factory.

[0006] (2) Multi-complex systems

[0007] Technical principle: Multiple Complex Systems (MCSs) are another form of complex systems. They usually contain more variables and more complex interactions, and are designed to simulate problem-solving situations that are closer to the real world.

[0008] Representative examples: MicroDYN, MicroFIN and genetic laboratories are typical representatives of multi-complex systems. These systems involve dynamically changing environments, and participants need to balance multiple goals under changing conditions, such as economic benefits and social responsibility.

[0009] (3) Automated Scoring Method

[0010] Technical principle: Existing automated scoring methods usually rely on procedural data, which records the actions and decisions of participants in the problem-solving process. Scoring methods usually include theory-driven top-down feature extraction methods and data-driven bottom-up feature extraction methods.

[0011] Representative examples: PISA, ATC21S and other international assessment projects have adopted a theory-driven top-down feature extraction method. In addition, researchers have also tried to use data mining techniques, such as natural language analysis, dimensionality reduction algorithms and network analysis, to extract meaningful behavioral patterns from process data.

[0012] Specifically, existing technologies cannot effectively characterize the multi-objective nature of complex systems, resulting in a lack of comparability in the results of evaluation tools; and it is difficult to decouple the multi-objective balance capability from the measurement results of existing tools, and the indicator extraction and analysis methods of a single optimal strategy are not applicable to multi-objective scenarios; in addition, the scoring method based on result performance is difficult to comprehensively evaluate the multi-objective balance capability, and lacks an effective process data analysis method.

[0013] Multi-objective balancing ability is a key dimension of complex problem-solving ability. It refers to the ability of an individual to effectively identify, evaluate and coordinate multiple conflicting goals to achieve the overall optimal goal. In current research results, there is a lack of evaluation tools with formal representation of multi-objective properties, a lack of evaluation theoretical framework for multi-objective balancing ability, and a lack of automated scoring methods for multi-objective task process data. These limitations limit the effectiveness and practicality of multi-objective balancing ability evaluation in the field of complex problem solving. Therefore, the present invention aims to solve these problems and provide an intelligent evaluation method and system for multi-objective balancing ability. Summary of the invention

[0014] In view of this, the purpose of the present invention is to provide an intelligent evaluation method and system for multi-objective balancing capabilities to solve the above-mentioned problems existing in the prior art.

[0015] According to a first aspect of an embodiment of the present invention, an intelligent evaluation method for multi-objective balancing capability is provided, comprising:

[0016] Get carefully crafted complex problems to solve;

[0017] Aiming at the complex problems to be solved, the factors of multi-objectives are studied, and a representation method of multi-objectives is established in a formal framework;

[0018] By using the representation method of multi-objective in a formal framework, the preset typical strategies, and the collected process data, a theoretical framework for multi-objective balance ability evaluation driven by a formal framework is constructed.

[0019] An automated evaluation model based on process data is constructed by utilizing multi-view feature extraction methods, process data and a multi-objective balance capability evaluation theoretical framework.

[0020] Furthermore, for the complex problem to be solved, the multi-objective components are studied, and a representation method of the multi-objective in a formal framework is established, including:

[0021] For the complex problem to be solved, a preset linear structural equation formalization method is used to characterize the complex problem, and a formal explanation of multi-objective and the type of multi-objective are obtained;

[0022] Among them, the types of multi-objectives include: independent multi-objectives, collaborative multi-objectives, conflicting multi-objectives, and restriction-induced multi-objectives.

[0023] Furthermore, the method of characterizing multi-objective in a formal framework, the preset typical strategies, and the collected process data are used to construct a multi-objective oriented, formal framework-driven multi-objective balance ability evaluation theoretical framework, including:

[0024] By using the representation method of multi-objective in a formal framework and the preset typical strategies, a theoretical framework for testing multi-objective balance ability is constructed through correlation analysis.

[0025] By utilizing the behavioral patterns mined from the collected process data and distinguishing typical strategies, the association between strategies, behaviors, multi-objective types and capability dimensions is realized, the multi-objective balance capability evaluation framework of the test is verified and improved, and a multi-objective, formal framework-driven multi-objective balance capability evaluation theoretical framework is constructed.

[0026] Furthermore, the construction process of the typical strategy includes:

[0027] Using a preset reinforcement learning technique, the complex problem to be solved is modeled based on a preset linear structural equation formalization method as a benchmark for identifying individual strategies;

[0028] Based on the preset linear structural equation formalization method, the main elements of reinforcement learning technology and relevant data of the application process are obtained;

[0029] By using the main elements of reinforcement learning technology and relevant data of the application process, different policy networks can be trained using different reward functions to simulate different multi-objective balance strategies;

[0030] Based on different reward functions, the corresponding reinforcement learning agent is trained and the standard operation sequence of the corresponding strategy is output.

[0031] Furthermore, the preset multi-view feature extraction method, process data and multi-objective balance ability evaluation theoretical framework are used to construct an automated evaluation model based on process data, including:

[0032] Characterizing the collected process data using a preset view feature extraction method to obtain a first processing result;

[0033] Using the first processing result, collaborative modeling of consistency and complementarity among multi-view data is performed;

[0034] By utilizing the processing results of collaborative modeling of consistency and complementarity among multi-view data, the operation sequences generated by the reinforcement learning agent are assigned high scores and added to the training data. By measuring the differences in the deep representation of multi-view data, explainable information is provided for the scoring results.

[0035] Furthermore, the method of using a preset view feature extraction method to characterize the collected process data to obtain a first processing result includes:

[0036] The collected process data are characterized by using operation sequence-oriented feature extraction, state sequence-oriented feature extraction, and balance strategy-oriented feature extraction methods to obtain a first processing result.

[0037] Furthermore, the using the first processing result to perform collaborative modeling of consistency and complementarity among multi-view data includes:

[0038] Using the first processing result, according to the expert annotation information in the multi-objective balance ability evaluation theoretical framework, the support degree of each view data feature on the measurement target in different measurement dimensions is calculated;

[0039] Based on the results of the support of each view data feature for the measurement target in different measurement dimensions, expert fine-grained annotation information is introduced into the representation layer of the scoring model of multi-view features of process data, and collaborative modeling of consistency and complementarity between multi-view data is achieved by constructing supervision tasks.

[0040] According to a second aspect of an embodiment of the present invention, an intelligent evaluation system for multi-objective balancing capability is provided, comprising:

[0041] The acquisition module is used to acquire complex problems to be solved with fine design;

[0042] The first processing module is used to study the multi-objective components for the complex problem to be solved and establish a representation method of the multi-objective under a formal framework;

[0043] The second processing module is used to construct a multi-objective balance ability evaluation theoretical framework driven by a formal framework and oriented to multi-objective by using the representation method of multi-objective under a formal framework, the preset typical strategies, and the collected process data;

[0044] The third processing module is used to construct an automated evaluation model based on process data by utilizing a preset multi-view feature extraction method, process data, and a multi-objective balance capability evaluation theoretical framework.

[0045] The technical solution provided by the embodiments of the present invention may have the following beneficial effects:

[0046] It is understandable that the technical solution provided by the present invention, through the careful design of the complex problem to be solved, studies the multi-objective constituent factors for the complex problem to be solved, and establishes a characterization method for multi-objective under a formal framework; then, using the characterization method for multi-objective under a formal framework, the preset typical strategy, and the collected process data, a theoretical framework for multi-objective balance ability evaluation driven by a formal framework is constructed; finally, an automated evaluation model based on process data is constructed using the preset multi-view feature extraction method, process data, and the theoretical framework for multi-objective balance ability evaluation. It is understandable that the technical solution provided by the present invention helps to solve the problems of the lack of multi-objective formal representation of the existing complex problem solving evaluation tools, the lack of a theoretical framework for the evaluation of multi-objective balance ability, and the lack of an automated scoring method for multi-objective task process data, so as to improve the accuracy and effectiveness of the multi-objective balance ability evaluation in complex problem solving.

[0047] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0049] Figure 1 It is a flowchart of an intelligent evaluation method for multi-objective balancing ability;

[0050] Figure 2 It is a schematic diagram of typical strategies for multi-objective balance identification based on intelligent evaluation methods for multi-objective balance capabilities;

[0051] Figure 3 It is an intelligent evaluation method for multi-objective balance ability. Schematic diagram of the theoretical framework of multi-objective balance ability evaluation;

[0052] Figure 4It is a schematic diagram of the association between multi-view features and capability dimensions of an intelligent evaluation method for multi-objective balancing capabilities;

[0053] Figure 5 It is a schematic diagram of the scoring model framework based on multi-view features of process data for intelligent evaluation methods of multi-objective balance capabilities;

[0054] Figure 6 It is a schematic diagram of the composition flow of the intelligent evaluation system for multi-objective balancing ability. DETAILED DESCRIPTION

[0055] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.

[0056] Embodiment 1

[0057] See also Figure 1 , Figure 1 The figure is a flow chart of an intelligent evaluation method for multi-objective balance ability, the method comprising:

[0058] S1. Obtain a well-designed complex problem to be solved;

[0059] S2. For the complex problems to be solved, study the factors that constitute multi-objectives and establish a representation method for multi-objectives in a formal framework;

[0060] S3. Using the representation method of multi-objective in a formal framework, the preset typical strategies, and the collected process data, a theoretical framework for multi-objective balance ability evaluation driven by a formal framework is constructed;

[0061] S4. Utilize the preset multi-view feature extraction method, process data and multi-objective balance ability evaluation theoretical framework to build an automated evaluation model based on process data.

[0062] It is understandable that the technical solution provided by the present invention studies the multi-objective components for the complex problems to be solved through carefully designed complex problems to be solved, and establishes a characterization method for multi-objective under a formal framework; then, a multi-objective, formal framework-driven multi-objective balance ability evaluation theoretical framework is constructed using the characterization method for multi-objective under a formal framework, preset typical strategies, and collected process data; finally, an automated evaluation model based on process data is constructed using the preset multi-view feature extraction method, process data, and multi-objective balance ability evaluation theoretical framework. It is understandable that the technical solution provided by the present invention helps to solve the problems of the existing complex problem solving evaluation tools lacking a formal representation of multi-objective, lacking a theoretical framework for evaluating multi-objective balance ability, and lacking an automated scoring method for multi-objective task process data, so as to improve the accuracy and effectiveness of multi-objective balance ability evaluation in complex problem solving.

[0063] In one embodiment, as described in steps S1-S2, a complex problem to be solved is carefully designed, multi-objective components are studied for the complex problem to be solved, and a representation method of multi-objective in a formal framework is established, including:

[0064] In specific implementation, the carefully designed complex problem to be solved means extracting key element indicators based on real task scenarios and establishing a complex problem that conforms to multi-objective formal representation.

[0065] For the complex problem to be solved, a preset linear structural equation formalization method is used to characterize the complex problem, and a formal explanation of multi-objective and the type of multi-objective are obtained;

[0066] Among them, the types of multi-objectives include: independent multi-objectives, collaborative multi-objectives, conflicting multi-objectives, and restriction-induced multi-objectives.

[0067] In practice, formalized complex systems are mainly based on linear structural equations or finite state

[0068] Based on the two classic frameworks of motivation, the multi-objective of complex systems can be preliminarily divided into independent multi-objective, collaborative multi-objective and conflicting multi-objective. Considering that linear structural equations are more widely used in existing evaluation tools in the field of CPS, this application intends to give priority to the formalization method of linear structural equations to characterize complex systems and further realize the formal explanation of multi-objective.

[0069] Complex problem solving based on linear structural equation formalism includes the following:

[0070] Input variable (independent variable): X = {x1, x2, ..., x m}, xi ∈R + ,

[0071] Output variable (dependent variable): Y = {y1, y2, ..., y n},y j =f j (X j ), X j ∈P(X),

[0072] Operation space: {(x, b→d)|x∈X}

[0073] Initial state: z0=(y1=c 01 , y2=c 02 , ..., y n =c 0n ),

[0074] Task objective: z = {(y1, y2, ..., y n )|y1~C1,y2~C2,...,y n ~C n}.

[0075] Problem solvers interact with complex systems by manipulating input variables. Complex systems contain multiple output variables, each of which may be associated with one or more input variables. Problem solvers change the system output variables when performing operations, and the changes in these variables can be observed. The goals of complex systems are described by these output variables. For example, y~C represents adjusting a certain output variable to a specific real number space. A complex problem generally requires the subject to adjust the complex system from the initial state to the state space that meets the task goal through a series of operations.

[0076] When the number of system variables n>1, the complex problem can be considered to have multi-objective characteristics. Combined with the preliminary division results of multi-objective characteristics, the multi-objective can be formally defined. In the linear structural equation, the relationship between the system variables and the operation is represented by the slope The direction of the slope and the direction of change required to reach the target state from the current state determine how the subject operates. Therefore, by defining the function Indicates the direction of change of system variables when state c reaches target C.

[0077] Result-oriented independent multi-objective: If two objectives y i ~C i and j ~C j satisfy These two objectives are independent multiple objectives.

[0078] Result-oriented collaborative multi-objective: If two objectives y i ~C i and j ~C j satisfy

[0079] and Then these two multi-objectives are coordinated multi-objectives.

[0080] Result-oriented conflicting multiple goals: If two goals y i ~C i and j ~C j satisfy

[0081] and Then these two multi-objectives are conflicting multi-objectives.

[0082] Therefore, the types of multi-objectives are divided into independent multi-objectives, collaborative multi-objectives, conflicting multi-objectives, and constraint-induced multi-objectives.

[0083] In one embodiment, as described in step S3, the method of characterizing multi-objective in a formal framework, the preset typical strategy, and the collected process data are used to construct a multi-objective oriented, formal framework-driven multi-objective balance ability evaluation theoretical framework, including:

[0084] By using the representation method of multi-objective in a formal framework and the preset typical strategies, a theoretical framework for testing multi-objective balance ability is constructed through correlation analysis.

[0085] By mining behavioral patterns and distinguishing typical strategies from the collected process data, the association between strategies, behaviors, multi-objective types and capability dimensions is realized, the multi-objective balance capability evaluation framework of the test is verified and improved, and a multi-objective, formal framework-driven multi-objective balance capability evaluation theoretical framework is constructed.

[0086] In specific implementation, individuals may use different ability dimensions, associate different execution strategies, and form different behavior patterns in the process of solving different types of multi-objective problems. Only by deeply exploring these internal connections and representing them in a formal framework can we support the implementation of a transferable automated scoring method. To this end, this application uses the representation method of multi-objectiveness in a formal framework and preset typical strategies to construct a theoretical framework for testing multi-objective balance ability through correlation analysis.

[0087] Specifically, the theory-driven approach can provide some capability dimension information and balance strategy information for the multi-objective balance capability assessment framework. In addition, in-depth research on multi-objective balance strategies from a data-driven perspective can further improve the assessment framework.

[0088] In the specific implementation, this study intends to first introduce reinforcement learning technology, model the problem-solving strategy based on a formal framework, as a benchmark for identifying individual strategies. The core of reinforcement learning technology is to build an intelligent agent with a policy network to achieve the function of making the next decision based on the environment and the current state of the system. The parameters in the network are trained by obtaining the largest possible reward from the environment after the intelligent agent makes a series of decisions.

[0089] Specifically, based on the preset linear structural equation formalization method, the main elements of reinforcement learning technology and relevant data of the application process are obtained, including:

[0090] In the specific implementation, under the formal framework of the aforementioned linear structural equation, the main elements and application process of reinforcement learning technology can be described as follows:

[0091] Environment: determines the operation space that the agent can perform in the current state and how the output variables change after the operation is performed. It can be represented in the form of a graph as follows:

[0092] G=<V,E> , V = X ∪ Y,

[0093] Status:s t =(y 1t ,y 2t , ..., y nt ), Operation space: X t ={x:b→d|x∈X};

[0094] Policy network: π(a t |s t , G), a t ∈X t , the policy network can select the next operation from the currently available operation space according to the current state and environment.

[0095] Reward function: R(s t ) Feedback a value to the policy network based on the current state, and the policy network is trained by maximizing the cumulative reward.

[0096] Obviously, different reward functions can train different policy networks to simulate different multi-objective balance strategies. Based on the interviews and thinking aloud in the theory-driven research phase, this application will initially select several high-frequency expression strategies for reward function design. The following two examples are given to illustrate the design perspective of the reward function:

[0097] Comprehensively consider the multi-objective long-term benefit strategy: In this case, when S t Any output variable yjt does not satisfy the target C j When R(s t )=0; when S t When the target is met, R(S t ) should be set to a larger value as feedback, and the number of operations t can be considered and the feedback value can be associated with it, that is:

[0098]

[0099] A strategy to achieve goals one by one: In this case, when S t-1 The output variable yjt-1 does not meet the corresponding target and S t In yjt, C j When R(S t ) should give a certain positive feedback, the feedback value can be based on the full

[0100] The number and extent of the goals to be met are set, namely:

[0101] R(s t )=count(y jt-1 ! ~C j and y jt ~C j ) (2)

[0102] Based on the designed reward function, the corresponding reinforcement learning agent is trained and applied to the multi-scenario evaluation tool developed by the company to output the standard operation sequence of the corresponding strategy.

[0103] In the specific implementation, the collected process data is used to mine behavioral patterns, distinguish typical strategies, realize the association between strategies, behaviors and multi-objective types and capability dimensions, verify and improve the test multi-objective balance capability evaluation framework, and construct a multi-objective, formal framework-driven multi-objective balance capability evaluation theoretical framework, including:

[0104] Based on the research and development of the multi-objective balance ability assessment framework, this application intends to recruit 100 college students for trial testing, collect process data on multi-objective complex problem solving in multiple scenarios, and identify typical multi-objective balance strategies. Figure 2As shown in the figure, the operation sequence generated by each subject in the test will be compared with the operation sequence output by the reinforcement learning agent. If there is an operation sequence with a high similarity, the subject will be divided into the corresponding strategy group; if the operation sequence in the comparison standard is not similar enough to the subject, expert annotation will be introduced to decide whether to add a new strategy to the theoretical framework, design a new reward function, and train a new reinforcement learning agent to obtain a new standard operation sequence. In this stage of research, how to design a suitable operation sequence similarity measurement is a key issue. In the specific implementation, the measurement function will be designed and verified based on expert experience, system characteristics, and subject operation data.

[0105] Based on the results of theory-driven and data-driven research, the dimensions and typical strategies of multi-objective balance capabilities are basically clear, and it is necessary to further explore and verify the associations between various elements in the multi-objective balance capability evaluation framework. On the one hand, it is planned to group the objectives into pairs according to the multi-objective settings of different categories in the multi-objective balance capability evaluation framework, and further decompose the behavior patterns generated by the reinforcement learning agent to form the association between multi-objective categories and behavior patterns; on the other hand, it is planned to hire 3 experts or scholars in the CPS field to evaluate the typicality of the behavior patterns, and select behavior patterns with high consistency to add to the evaluation framework; establish external criterion for each capability dimension, and establish the connection between behavior patterns and capability dimensions through correlation analysis. Finally, this study will construct the following Figure 3 The multi-objective balance ability assessment framework shown.

[0106] Furthermore, the automated scoring model is the core of the integrated multi-objective balance ability assessment tool, which can provide instant evaluation feedback based on the subject's operation data, thereby supporting the popularization and application of the assessment tool. In order to model the automated scoring process, considering the limitations of previous studies that were mostly based on single-objective modeling, this study intends to design a multi-view feature extraction method based on the process data collected in the preliminary study, combined with the multi-objective balance ability assessment system, and construct a multi-objective balance ability automated scoring model based on process data. The specific research plan includes the following stages:

[0107] S41. Characterizing the collected process data using a preset view feature extraction method to obtain a first processing result;

[0108] S42. Using the first processing result, perform collaborative modeling of consistency and complementarity between multi-view data;

[0109] S43. Using the results of collaborative modeling of consistency and complementarity among multi-view data, the operation sequences generated by the reinforcement learning agent are assigned high scores and added to the training data. By measuring the differences in the deep representation of multi-view data, explainable information is provided for the scoring results.

[0110] In the specific implementation, the multi-objective balance ability is automatically scored based on the process data collected by the system, which first requires the process data to be fully characterized. In the specific implementation, the multi-view data feature extraction method is explored through the following three strategies.

[0111] 1. Feature extraction for operation sequence

[0112] This application intends to introduce the key node idea in the shortest path measurement in the field of graph theory to realize the extraction of key operations, and then integrate multiple information to model the criticality evaluation function of each step of operation. Specifically, the formal framework is represented as a graph, that is, G =<V,E> ,V=X∪Y, Considering the topological structure of the graph, the degree of the nodes in the graph, and other information to evaluate the ability of each operation to adjust the system variables; considering the gap between the current state and the target state, comprehensively considering the restrictions of the operation and system variables and the prior target priority information, further distinguish the criticality of the operations in the graph structure, and model the criticality assessment function H(zs t , G), to achieve dynamic calculation of operation criticality. Based on this, whether each operation in the operation sequence selects the more critical operation in the current state can be quantified and summarized as a numerical feature, denoted as X a .

[0113] 2. Feature extraction for state sequences

[0114] This application intends to implement a state consistency measurement method under multiple objectives and extract features of the process in which the state sequence approaches the goal. Specifically, the state sequence S = [s0, s1, s2, ..., s T ],s t =(y 1t ,y 2t , ..., y nt ), s0=z0 performs two mapping operations. On the one hand, through the difference operation s t -s t-1 The output variable change process characterization [Δs t ]=[(Δy 1t , Δy 2t , ..., Δy nt )]; On the other hand, by difference operation zs t-1 Get the change process representation of the gap between the current state and the target [Δs t ]=[(Δy 1t , Δy 2t , ..., Δy nt )]; On the other hand, by difference operation zs t-1 Get the change process representation of the gap between the current state and the target [Δ minz st-1 ]=[(min C1-y 1，t-1 , minC2-y 2，t-1 , ..., min C n -y n，t-1 )] and [Δ maxz s t-1 ]=[(max C1-y 1，t-1 , max C2-y 2，t-1 , ..., max C n -y n，t-1 )] Combined with system parameters such as target priority, a scoring function I(Δs t , Δ min z s t-1 , Δ max z s t-1 ), evaluate whether the current operation is closer to the overall goal and summarize it into numerical features, recorded as X b .

[0115] 3. Feature extraction for balancing strategies

[0116] This application intends to use the behavioral pattern identified in Study 3 as the standard, compare the subject's operation sequence with it, and then evaluate the subject's balance strategy a t =π(S t -1) Realize quantitative characterization. Specifically, with multi-objective y i ~C i and j ~C j Take Φ as an example, and its associated behavior pattern in the evaluation system is denoted as Φ ij =[φ1,φ2,...,φz ij ],Z ij represents the number of associated behavior patterns between two targets in the evaluation framework, and the operation sequence A = [a1, a2, ..., a T ], a t =(x i , b→d) satisfies x i ∈X i uX j The operation is performed to extract the new sequence A ij Based on the research experience of similarity measurement of operation sequences in strategy identification, the similarity measurement function J(A ij , Φ ij ), the output similarity value set is summarized according to the multi-objective type to form the numerical feature from the perspective of the multi-objective balance strategy, which is recorded as Xc.

[0117] In specific implementation, as described in step S42, using the first processing result, collaborative modeling of consistency and complementarity among multi-view data includes:

[0118] See also Figure 4 ,It is planned to first explore the support of each view data features for the ,measurement target in different measurement dimensions based on the expert annotation ,information in the evaluation framework research.

[0119] Specifically, suppose that the criterion scores of all subjects in the pre-study on a certain combination of ability dimensions are Q = [q1, q2, ..., q N ], the multiple scenarios in the evaluation tool are denoted as [M1, M2, ..., M O ], then the view features of these subjects in these scenes are recorded as formula (3), and the corresponding expert annotations are recorded as formula (4).

[0120]

[0121] By calculating the correlation between view features and capability dimensions through formula (5), the association between multiple view features and capability dimensions can be measured as the support of the view features for the measurement of the specified capability dimension. When the support is lower than a certain threshold, the view information should be ignored when constructing the scoring model for the specified measurement dimension to ensure the quality of the scoring model at the input level.

[0122]

[0123] Based on the above support research, the multi-view data modeling task has to solve the modeling problem of consistency and complementarity among multiple view data, and the consistency and complementarity of multi-view data features under each capability dimension may be different. Figure 5 As shown in Figure 1, the study preliminarily constructed a multi-view data-driven scoring model framework, intending to introduce expert fine-grained annotation information in the representation layer of the model, and achieve collaborative modeling of consistency and complementarity between multi-view data by constructing supervision tasks. Specifically, the extracted multi-view input data X a ,X b ,X c , mapped by neural network to and The supervision information of the model includes the expert's score on a certain ability dimension of the subject and whether the features of different views under this dimension tend to be consistent. For example, in a certain ability dimension, view a and view b are consistent, and both are complementary to view c. If we assume that in a certain high-dimensional space, and should be more similar to Different. w(a, b) represents the distance measurement function between a and b in the data representation space. By minimizing formula (6), the scoring model can be equipped with the ability to represent the consistency and complementarity of multimodal data.

[0124]

[0125] In the specific implementation, as described in step S43, the processing results of the collaborative modeling of consistency and complementarity between multi-view data are used to assign high scores to the operation sequences generated by the reinforcement learning agent and add them to the training data, and by measuring the differences in the deep representation of multi-view data, explainable information is provided for the scoring results, including:

[0126] Through expert fine-grained annotation, the consistency and complementarity of multi-view data representation are modeled. Then, the support of multi-view features for capability dimension measurement can be integrated to obtain the overall representation of process data. With the expert scores of each capability dimension as supervision information, multi-dimensional capability scoring based on process data can be achieved, as shown in formulas (7) and (8).

[0127]

[0128] However, due to its black-box nature, the end-to-end neural network model cannot directly associate the original features with the score. It is impossible to know the difference between high scorers and low scorers in the original features, and it is impossible to give a meaningful explanation for the scoring results. To solve this problem, this study intends to give high scores to the operation sequences generated by the reinforcement learning agent and add them to the training data. By measuring the differences in the deep representation of multi-view data, it provides explainable information for the scoring results. Specifically, after the scoring model is trained, the standard behavior sequence in the typical strategy research is extracted and output to the scoring model to obtain the deep representation of its multi-view features. and And store, where V represents the number of reinforcement learning agents constructed in the above study, that is, the number of typical strategies identified. When scoring the process data of any subject, the corresponding multi-view feature deep representation in its model is measured for similarity with the V stored representations, and the subject's strategy, execution of key operations, and approach to multiple goals are quantified. The effective and stable interpretation strategy verified on the pre-experimental data is applied, and finally a multi-dimensional and interpretable multi-goal balance ability scoring model is realized.

[0129] In the specific implementation, reinforcement learning technology is used, combined with expert experience and annotation information, to mine behavioral patterns and distinguish typical strategies from process data, realize the association between strategies, behaviors and multi-objective types and capability dimensions, and realize the verification and improvement of the multi-objective balance capability evaluation framework. Furthermore, by extracting multi-view behavioral features, the automatic evaluation of multi-objective balance capability is realized. Based on the analysis of process data based on typical pattern recognition, operation criticality assessment, etc., combined with expert annotation information, the support strength of each view data feature for the measurement target in different measurement dimensions is predefined, and the consistency and complementarity of multi-view features when facing multi-objective tasks are modeled, realizing the process data-driven scoring model construction method.

[0130] See also Figure 6 , Figure 6 The figure is a schematic diagram of the composition flow of an intelligent evaluation system for multi-objective balance ability, and the system includes:

[0131] An acquisition module 10, used to acquire a complex problem to be solved;

[0132] The first processing module 20 is used to study the multi-objective components for the complex problem to be solved and establish a representation method of the multi-objective under a formal framework;

[0133] The second processing module 30 is used to construct a multi-objective balance capability evaluation theoretical framework driven by a formal framework and oriented to multi-objective by using a representation method of multi-objective under a formal framework, a preset typical strategy, and collected process data;

[0134] The third processing module 40 is used to construct an automatic evaluation model based on process data by using a preset multi-view feature extraction method, process data and a multi-objective balance capability evaluation theoretical framework.

[0135] It can be understood that the same or similar parts of the above embodiments can be referenced to each other, and the contents not described in detail in some embodiments can refer to the same or similar contents in other embodiments.

[0136] It should be noted that, in the description of the present invention, the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of the present invention, unless otherwise specified, the meaning of "plurality" refers to at least two.

[0137] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention belong.

[0138] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0139] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0140] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0141] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0142] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0143] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. An intelligent evaluation method for multi-objective balancing ability, characterized by: The method comprises: Get carefully designed complex problems to solve; Aiming at the complex problems to be solved, the factors of multi-objectives are studied, and a representation method of multi-objectives is established in a formal framework; By using the representation method of multi-objective in a formal framework, the preset typical strategies, and the collected process data, a theoretical framework for multi-objective balance ability evaluation driven by a formal framework is constructed. An automated evaluation model based on process data is constructed by utilizing multi-view feature extraction methods, process data and a multi-objective balance capability evaluation theoretical framework.

2. The method according to claim 1, characterized in that In view of the complex problems to be solved, the factors constituting multi-objectives are studied, and a representation method of multi-objectives in a formal framework is established, including: For the complex problem to be solved, a preset linear structural equation formalization method is used to characterize the complex problem, and a formal explanation of multi-objective and the type of multi-objective are obtained; Among them, the types of multi-objectives include: independent multi-objectives, collaborative multi-objectives, conflicting multi-objectives, and restriction-induced multi-objectives.

3. The method according to claim 1, characterized in that The method of characterizing multi-objective in a formal framework, the preset typical strategies, and the collected process data are used to construct a multi-objective oriented, formal framework-driven multi-objective balance ability evaluation theoretical framework, including: By using the representation method of multi-objective in a formal framework and the preset typical strategies, a theoretical framework for testing multi-objective balance ability is constructed through correlation analysis. By mining behavioral patterns and distinguishing typical strategies from the collected process data, the association between strategies, behaviors, multi-objective types and capability dimensions is realized, the multi-objective balance capability evaluation framework of the test is verified and improved, and a multi-objective, formal framework-driven multi-objective balance capability evaluation theoretical framework is constructed.

4. The method according to claim 3, characterized in that The typical strategy construction process includes: Using a preset reinforcement learning technique, the complex problem to be solved is modeled based on a preset linear structural equation formalization method as a benchmark for identifying individual strategies; Based on the preset linear structural equation formalization method, the main elements of reinforcement learning technology and relevant data of the application process are obtained; By using the main elements of reinforcement learning technology and relevant data of the application process, different policy networks can be trained using different reward functions to simulate different multi-objective balance strategies; Based on different reward functions, the corresponding reinforcement learning agent is trained and the standard operation sequence of the corresponding strategy is output.

5. The method according to claim 1, characterized in that: The method uses the preset multi-view feature extraction method, process data and multi-objective balance ability evaluation theoretical framework to build an automatic evaluation model based on process data, including: Characterizing the collected process data using a preset multi-view feature extraction method to obtain a first processing result; Using the first processing result, collaborative modeling of consistency and complementarity among multi-view data is performed; By utilizing the processing results of collaborative modeling of consistency and complementarity among multi-view data, the operation sequences generated by the reinforcement learning agent are assigned high scores and added to the training data. By measuring the differences in the deep representation of multi-view data, explainable information is provided for the scoring results.

6. The method according to claim 5, characterized in that The method of using a preset view feature extraction method to characterize the collected process data to obtain a first processing result includes: The collected process data are characterized by using operation sequence-oriented feature extraction, state sequence-oriented feature extraction, and balance strategy-oriented feature extraction methods to obtain a first processing result.

7. The method according to claim 5, characterized in that The using the first processing result to perform collaborative modeling of consistency and complementarity among multi-view data includes: Using the first processing result, according to the expert annotation information in the multi-objective balance ability evaluation theoretical framework, the support degree of each view data feature on the measurement target in different measurement dimensions is calculated; Based on the results of the support degree of each view data feature for the measurement target in different measurement dimensions, expert fine-grained annotation information is introduced into the representation layer of the scoring model of the multi-view features of the collected process data, and collaborative modeling of consistency and complementarity between multi-view data is achieved by constructing supervision tasks.

8. An intelligent evaluation system for multi-objective balance ability, characterized by: The system comprises: The acquisition module is used to acquire complex problems to be solved with fine design; The first processing module is used to study the multi-objective components for the complex problem to be solved and establish a representation method of the multi-objective under a formal framework; The second processing module is used to construct a multi-objective balance ability evaluation theoretical framework driven by a formal framework and oriented to multi-objective by using the representation method of multi-objective under a formal framework, the preset typical strategies, and the collected process data; The third processing module is used to construct an automated evaluation model based on process data by utilizing a preset multi-view feature extraction method, process data, and a multi-objective balance capability evaluation theoretical framework.

Images