Multi-modal man-machine interaction view and component expression method and system

Abstract

According to the multi-mode man-machine interaction view and component expression method and system, the design state, the development state and the operation state are integrated, and the problems of scene linkage integration, multi-model association collaborative scheduling and the like of an aerospace equipment MOM system are solved by means of graphical interaction and the like. The method comprises construction of a knowledge graph and demand modeling in a design state, visual design and code generation in a development state, and fault diagnosis in an operation state. Compared with the prior art, knowledge can be deeply mined, codes can be automatically generated, system stability is guaranteed, development efficiency and reliability are improved, and the manufacturing and development requirements of aerospace equipment are met.

Description

Technical Field

[0001] The present invention relates to the fields of industrial software and artificial intelligence, and in particular to a multimodal human-computer interaction view and component expression method and system. Background Art

[0002] Manufacturing operations management (MOM) systems play a vital role in the manufacturing process of aerospace equipment. With the rapid development of aerospace technology, the functional requirements for MOM systems are increasing. Traditional single-modal interaction methods and component expression methods can no longer meet complex business needs.

[0003] At present, the MOM system is facing the problem of functional coupling dependency conflict between scene linkage integration and efficient deployment. The stable collaborative scheduling technology of model parsing, compilation, simulation, debugging and multi-model association also needs to be solved urgently. The component model parsing, compilation and code automatic generation technology under heterogeneous framework also need further research and improvement. Summary of the Invention

[0004] The main purpose of this invention is to overcome the defect that the single modal interaction mode and component expression method in the existing technology can no longer meet complex business needs. A multimodal human-computer interaction view and component expression method and system are proposed, which integrates the three modes of design, development and operation. Through graphical interaction, visual design and model-driven operation, the efficient development, flexible configuration and stable operation of the aerospace equipment MOM system are realized.

[0005] The present invention adopts the following technical solutions:

[0006] A multimodal human-computer interaction view and component expression method, comprising:

[0007] The S1 design state model assembly module uses semantic analysis and knowledge graph technology to deeply mine the business knowledge of the aerospace equipment MOM system, build a domain knowledge graph, and perform visual transformation. It generates a visual design view that includes demand modeling and component association views, and uses visual views to display the association relationships between components, realizing the transformation from business knowledge to component expression;

[0008] The S2 development model parsing module converts the business rules in the domain knowledge graph into executable logical expressions through rule reasoning algorithms, and uses constraint solving algorithms to ensure that the business logic conforms to the technical indicators defined in the domain knowledge graph. It operates on the components corresponding to the business logic in the interactive view of the visual orchestration tool. It automatically generates code based on the visual design view verified by rule reasoning and constraint solving, realizes the generation from interactive view to code components, completes functional assembly and logic orchestration, and ensures that the developed business logic accurately maps to business knowledge.

[0009] The S3 operational model driving module uses the component association rules defined in S1 and the business logic constraints verified by S2 to ensure the operational stability of the aerospace equipment MOM system through a fault diagnosis algorithm; it extends the visual design view of S1 and the code component mapping relationship of S2 to the operational stage, dynamically displays the system operational status view through a visual operation and maintenance interface, supports model visual debugging based on the design state knowledge graph, and ensures the reliability and maintainability of the aerospace equipment MOM system operation.

[0010] 2. A multimodal human-computer interaction view and component expression method according to claim 1, characterized in that S1 specifically comprises:

[0011] S101 performs semantic analysis on business documents containing business knowledge in the aerospace equipment MOM system, extracts key business entities and their relationships, and constructs a domain knowledge graph;

[0012] S102 maps the domain knowledge graph into a software requirement model and generates a visual design view containing functional modules, data flows, and interface definitions using the SysML language;

[0013] S103 dynamically displays the dependencies between components in a visual view, supports expressing business coupling strength through node connection line weights, and adjusts component configurations through interactive operations, realizing the transformation from business knowledge to component expression.

[0014] S2 specifically includes:

[0015] S201 defines a set of business rules. Each business rule includes trigger conditions and execution actions. The legitimacy of user operations on business logic components is verified in real time through rule inference algorithms.

[0016] S202 builds a multi-dimensional constraint set, including system performance constraints, resource allocation constraints, and business logic constraints. A constraint solving algorithm is used to find a parameter combination that satisfies all constraints, ensuring that the orchestration of business logic complies with the constraints of the aerospace equipment MOM system.

[0017] S203 is based on the visual orchestration results verified by rule reasoning and constraint solving. Users can intuitively perform final confirmation operations on business logic components. After confirmation, the basic architecture code is automatically generated through templated code generation technology, including data interface definition, service call logic and exception handling framework, realizing the generation from interactive view to code components.

[0018] The implementation of the S201 rule reasoning algorithm is as follows:

[0019] Rule definition, first define the business rule set R = {r1, r2, ..., r m}, each rule rk Represented as a logical expression Condition k →Action k , Condition k Action is the trigger condition. k To execute an action, k = 1, 2, ..., m; business rules are not only used to verify user operations, but also serve as the basic logic for function assembly;

[0020] Status acquisition: In the interactive view of the visual orchestration tool, the user operation status S is obtained in real time. The user operation status S includes the user operation status data and the business parameters entered by the user in the visual orchestration. Based on the user operation status S, it is determined whether the business rules are triggered, thereby determining the execution of the corresponding function.

[0021] Rule matching and execution: For each business rule, the real-time user operation status S is brought into the rule trigger condition Condition k In the trigger condition, k Whether it is established, dynamically determine which business rules will be triggered, and if it is established, execute the action k ;

[0022] Through rule reasoning, different functions are assembled in order according to business rules to achieve logical orchestration;

[0023] Business rules can be modified and expanded dynamically. When business needs change, only the rule definition needs to be adjusted without making large-scale modifications to the entire aerospace equipment MOM system.

[0024] The implementation of the constraint solving algorithm in S202 is as follows:

[0025] Definition of constraints, clarifying the constraint set C={c1,c2,…,c i}, each constraint c i It can be expressed as a function, f i (x1,x2,…,x p )≤b i , f i (x1,x2,…,x p ) represents the linear expression of the i-th constraint, where x j is the system parameter, b i is the constraint boundary, p represents the number of system parameters, and j is the index of the system parameter;

[0026] Conflict detection and resolution: Business rules are transformed into specific constraints in practical applications. When there are contradictions between these constraints, rule conflicts will occur. Linear programming algorithms are used to solve the parameters that meet all the constraints. The general form of linear programming is:

[0027] Objective function:

[0028] The objective function means finding a set of system parameter values ​​that minimizes the objective function value Z while satisfying all constraints.

[0029] Constraints: f i (x1,x2,…,x p )≤b i ,i=1,2,…l;x j ≥0, j=1,2,…,p

[0030] Right now:

[0031] where a ij Represents the constraint coefficient, which is used to define the linear relationship of the constraint;

[0032] By introducing the slack variable s i ≥0(i=1,2,…,l), converting inequality constraints into equality constraints to facilitate iterative calculation of linear programming:

[0033] a i1 x1+a i2 x2+…+a ip x p +s i =b i ;

[0034] Iterative optimization, for the decision variable x j The test number is Determine whether the variable can optimize the objective function, where Represents the base variable B i The coefficients in the objective function, the basis matrix B, is composed of a set of linearly independent column vectors selected from the coefficient matrix corresponding to the standard form equality constraint. The variables corresponding to these column vectors are called basis variables. i Represents the i-th basis variable, if all σ j ≥0, then the current decision variable x j This is the optimal solution;

[0035] Otherwise, repeat the above steps to iteratively determine the basis variables. j ≥0, the iteration ends and the parameters corresponding to the basic variables are determined, which is the optimal solution that satisfies all constraints. The optimal value of the objective function is

[0036] S203 specifically includes:

[0037] The visual orchestration tool displays the business logic architecture in the form of an interactive view, allowing users to add, delete, and modify business logic components directly in the interactive view. Based on the confirmed visual design results, the tool automatically generates some basic code according to specific code specifications, achieving efficient conversion from interactive view to component expression.

[0038] The visual orchestration tool has a component library that contains pre-defined functional modules. Through dragging and dropping and configuration operations, different functional modules are assembled together to complete complex logical orchestration.

[0039] S3 specifically includes:

[0040] S301 builds a fault knowledge base containing historical fault cases, characteristic data, and solutions, and uses Bayesian fault reasoning algorithms to conduct real-time monitoring, prediction, and fault diagnosis of aerospace equipment MOM systems;

[0041] S302 displays the operating status view of the aerospace equipment MOM system through a visual operation and maintenance interface, highlights faulty components and associates them with impact path analysis reports, realizes model visual debugging and cross-domain configurable deployment, and accurately locates and expresses faulty components to ensure the reliability and maintainability of system operations.

[0042] The implementation of the S301 Bayesian fault inference algorithm is as follows:

[0043] Fault model construction, analyzing the possible fault types that may occur during the low-code system development process, and defining these fault types as fault nodes F j (j=1,2,…,M); According to the previous development experience and historical data, the prior probability P(F j );

[0044] Fault feature extraction: When an aerospace equipment MOM system has an abnormality, relevant fault feature information is collected; fault feature information includes error information in the system log, user operation records, and component operation status; feature information is combined into a fault feature vector

[0045] Conditional probability determination, analysis of the correlation between each fault type and fault characteristics, and determination of conditional probability through statistical analysis of historical fault data

[0046] Bayesian inference calculation, using the Bayesian inference formula Calculate the observed fault feature vector In the case of each faulty node F j Posterior probability of occurrence M is the total number of nodes in the fault tree; At the faulty node F j When the fault characteristic vector The conditional probability of

[0047] Determine the fault source and compare the posterior probability of each fault node The fault node with the highest posterior probability is the most likely fault source.

[0048] A multimodal human-computer interaction view and component expression system, including

[0049] The design state model assembly module uses semantic analysis and knowledge graph technology to deeply mine the business knowledge of the aerospace equipment MOM system, construct a domain knowledge graph, and perform visual transformation. It generates a visual design view that includes demand modeling and component association views, and uses visual views to display the association relationship between components, realizing the transformation from business knowledge to component expression.

[0050] The development model parsing module uses rule-based reasoning algorithms to convert business rules in the domain knowledge graph into executable logical expressions. It also utilizes constraint-solving algorithms to ensure that the business logic complies with the technical specifications defined in the domain knowledge graph. It then operates on components corresponding to the business logic in the interactive view of the visual orchestration tool. It automatically generates code based on the visual design view verified by rule-based reasoning and constraint-solving, achieving the transition from interactive view to code components. This completes functional assembly and logical orchestration, ensuring that the developed business logic accurately maps to the business knowledge.

[0051] The operation state model driving module uses the component association rules defined by the design state model assembly module and the business logic constraints verified by the development state model parsing module to ensure the operational stability of the aerospace equipment MOM system through a fault diagnosis algorithm; extends the visual design view of the design state model assembly module and the code component mapping relationship of the development state model parsing module to the operation stage, dynamically displays the system operation status view through a visual operation and maintenance interface, supports model visual debugging based on the design state knowledge graph, and ensures the reliability and maintainability of the operation of the aerospace equipment MOM system.

[0052] From the above description of the present invention, it can be seen that compared with the prior art, the present invention has the following beneficial effects:

[0053] (1) Through semantic analysis and knowledge graph construction, business knowledge is deeply mined and visualized, enabling system developers to understand business requirements more comprehensively and deeply, avoiding the deviation in understanding requirements caused by the shallow processing of business knowledge in traditional design methods. The visual display of the knowledge graph directly presents the relationship between components, which helps to quickly build the system architecture, improve design efficiency and quality, and reduce the extension of development cycle and increase in costs caused by misunderstanding of requirements.

[0054] (2) Visual design based on rule reasoning and constraint solving, combined with code generation algorithms, enables automatic generation from interactive views to code components. This not only solves the problem of lack of rule constraints and automated code generation in traditional development processes, but also greatly improves development efficiency and reduces the workload and error rate of manual code writing. Non-professional developers can also participate in system development, further improving development flexibility and efficiency, accelerating system iteration speed, and meeting the needs of the rapid development of aerospace equipment manufacturing.

[0055] (3) Using fault diagnosis and reasoning algorithms, the system is monitored, predicted, and diagnosed in real time. Visual model debugging and cross-domain configurable deployment are achieved through a visual interface view. This makes system operation more reliable, enabling the timely detection and resolution of potential problems. Compared with traditional operation methods, it can more effectively ensure the stable operation of the system. The visual interface displays the system operation status and fault information, making it easier for operation and maintenance personnel to manage and maintain the system, reducing system maintenance costs and improving the availability and safety of aerospace equipment.

[0056] (4) The method of the present invention can effectively solve the functional coupling dependency conflict problem faced by the aerospace equipment MOM system, such as scene linkage integration and efficient deployment, as well as the difficult problems of model parsing, compilation, simulation, debugging and stable collaborative scheduling technology associated with multiple models. Through multimodal interaction and component expression, it can better adapt to complex and changing business needs, improve the flexibility and scalability of the system, enable the system to quickly respond to various changes in the aerospace equipment manufacturing process, and enhance the competitiveness of the enterprise. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 This is a system architecture diagram of the method of the present invention.

[0058] Figure 2 A flow chart for transforming design-state business knowledge into component expressions.

[0059] Figure 3 Generate a flowchart for developing business logic to code.

[0060] Figure 4 Design-development-operation flow chart for the entire life cycle of the system.

[0061] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments. DETAILED DESCRIPTION

[0062] To make the objectives, technical solutions, and advantages of the present invention more apparent, the present invention will be further described in detail below with reference to the accompanying drawings. It is apparent that the embodiments described are only some, not all, of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort are intended to fall within the scope of protection of the present invention.

[0063] The MOM (Manufacturing Operations Management) system for aerospace equipment is an integrated production management platform developed to address the complex manufacturing needs of this sector. Its core functions include modules such as production planning and scheduling, quality control, equipment maintenance, and resource optimization. By monitoring production processes in real time and integrating multi-source data, the system enables transparent management of the entire life cycle of aerospace component assembly and manufacturing, ensuring quality traceability and process compliance for key products such as high-precision structural parts and engine components. Compared to traditional MES systems, MOM adopts a modular architecture that can flexibly adapt to the high-variety, small-batch production characteristics of aerospace equipment. It optimizes production scheduling efficiency through data analysis and intelligent algorithms, reducing the risk of delivery delays due to process complexity.

[0064] See also Figure 1 The present invention provides a multimodal human-computer interaction view and component expression method, which is based on the implementation of the aerospace equipment MOM system and includes:

[0065] The S1 design state model assembly module uses semantic analysis and knowledge graph technology to deeply mine the business knowledge of the aerospace equipment MOM system, build a domain knowledge graph, and perform visual transformation. It generates a visual design view that includes demand modeling and component association views, and uses visual views to display the association relationship between components, realizing the transformation from business knowledge to component expression. This step specifically includes the following:

[0066] S101 performs semantic analysis on business documents containing business knowledge in the aerospace equipment MOM system, extracting key business entities and their relationships. This system then constructs a domain knowledge graph, graphically displaying the relationships between business knowledge and visually presenting the business architecture. Specifically, it collects real-world scenario information and demand models from the aerospace equipment MOM system, including business process documents, user requirement specifications, and historical production data, to fully understand the system's business context and requirements. Using semantic analysis tools, it then performs text analysis on this collected information, extracting key semantic information and concepts, and converting unstructured text into structured data.

[0067] S102 maps the domain knowledge graph into a software requirements model, generating a visual design view using the SysML language that includes functional modules, data flows, and interface definitions. This step uses the IBM Rational professional modeling tool for software requirements modeling, transforming business knowledge into the functional requirements and architectural design of the software system, creating a visual design model that provides a clear blueprint for subsequent development.

[0068] S103 dynamically displays the dependencies between components in a visual view, supports expressing business coupling strength through node connection line weights, and adjusts component configurations through interactive operations, realizing the transformation from business knowledge to component expression.

[0069] The S2 development state model parsing module converts the business rules in the domain knowledge graph into executable logical expressions through rule reasoning algorithms, and uses constraint solving algorithms to ensure that the business logic complies with the technical indicators defined in the domain knowledge graph, and operates on the components corresponding to the business logic in the interactive view of the visual orchestration tool; based on the visual design view verified by rule reasoning and constraint solving, it automatically generates code to realize the generation from interactive view to code components, completes functional assembly and logical orchestration, and ensures that the developed business logic accurately maps the business knowledge.

[0070] In this step, based on the visual design model, the development state model parsing module is used to start the rule-based reasoning and constraint solving algorithm, and the pre-defined rule base and constraints are loaded into the development environment.

[0071] See also Figure 2 , S2 specifically includes:

[0072] S201 defines a set of business rules. Each business rule includes trigger conditions and execution actions. The legitimacy of user operations on business logic components is verified in real time through rule reasoning algorithms.

[0073] The rule inference algorithm judges the legality and correctness of operations in real time based on user operations and preset rules, and checks whether the business logic complies with business rules and whether the data flow is correct. The details are as follows:

[0074] Rule definition, first define the business rule set R = {r1, r2, ..., r m}, each rule r k Represented as a logical expression Condition k →Action k , Condition k Action is the trigger condition. k To execute an action, k = 1, 2, …, m; business rules are not only used to verify user operations, but also serve as the basic logic for function assembly.

[0075] State acquisition: In the interactive view of the visual orchestration tool, the user operation state S is obtained in real time. The user operation state S includes user operation state data (such as the coordinates and timestamps of interactive behaviors such as clicks, connections, and dragging) and business parameters entered by the user for visual orchestration (such as sensor sampling frequency and control thresholds). Based on the user operation state S, the business rules are retrieved to determine whether the business rules are triggered, thereby determining the execution of the corresponding functions.

[0076] Rule matching and execution: For each business rule, the real-time user operation status S is brought into the rule trigger condition Condition k In the trigger condition, k Whether it is established, dynamically determine which business rules will be triggered, and if it is established, execute the action k .

[0077] Through rule reasoning, different functions are assembled in order according to business rules to achieve logical orchestration.

[0078] In the present invention, business rules can be dynamically modified and expanded. When business requirements change, only the rule definitions need to be adjusted without the need for large-scale modifications to the entire aerospace equipment MOM system.

[0079] S202 constructs a multi-dimensional set of constraints, including system performance constraints, resource allocation constraints, and business logic constraints. A constraint-solving algorithm is used to find a parameter combination that satisfies all constraints, ensuring that the business logic orchestration complies with the constraints of the aerospace equipment MOM system. Constraint-solving algorithms include, but are not limited to, linear programming, nonlinear programming, integer programming, or heuristic algorithms. The following uses a linear programming algorithm as an example for illustration.

[0080] The implementation of the constraint solving algorithm is as follows:

[0081] Definition of constraints, clarifying the constraint set C={c1,c2,…,c i}, each constraint c i It can be expressed as a function, f i (x1,x2,…,x p )≤b i , f i (x1,x2,…,x p ) represents the linear expression of the i-th constraint, where x j is the system parameter, b i is the constraint boundary, i is the index of the constraint, which is used to distinguish different constraints, p represents the number of system parameters, and j is the index of the system parameter, ranging from 1 to p.

[0082] Conflict detection and resolution: Business rules are transformed into specific constraints in practical applications. These constraints may come from aspects such as system performance requirements, resource allocation restrictions, and business logic specifications. When there is a contradiction between these constraints, a rule conflict occurs. For example, in some cases, two constraints may not be satisfied at the same time, resulting in a rule conflict. A linear programming algorithm is used to solve the parameters that satisfy all the constraints. The general form of linear programming is:

[0083] Objective function:

[0084] The objective function means finding a set of system parameter values ​​that minimizes the objective function value Z while satisfying all constraints.

[0085] Constraints: f i (x1,x2,…,x p )≤b i ,i=1,2,…l;x j ≥0, j=1,2,…,p

[0086] Right now:

[0087] where a ij Represents the constraint coefficient, which is used to define the linear relationship of the constraint;

[0088] By introducing the slack variable s i ≥0(i=1,2,…,l), converting inequality constraints into equality constraints to facilitate iterative calculation of linear programming:

[0089] a i1 x1+a i2 x2+…+a ip x p +s i =b i ;

[0090] Iterative optimization, for the decision variable x j The test number is Determine whether the variable can optimize the objective function, where Represents the base variable B i The coefficients in the objective function, the basis matrix B, is composed of a set of linearly independent column vectors selected from the coefficient matrix corresponding to the standard form equality constraint. The variables corresponding to these column vectors are called basis variables. i Represents the i-th basis variable, if all σ j ≥0, then the current decision variable x j This is the optimal solution;

[0091] Otherwise, repeat the above steps to iteratively determine the basis variables. j ≥0, the iteration ends and the parameters corresponding to the basic variables are determined, which is the optimal solution that satisfies all constraints. The optimal value of the objective function is

[0092] Constraint-solving algorithms are used to determine the optimal parameter solution that satisfies all constraints. Component logic is then orchestrated to ensure the desired business logic component orchestration and assembly process is implemented, meeting the integrity and accuracy requirements of the business process. Constraint-solving algorithms address issues such as inadequate system performance and resource utilization during functional assembly and logic orchestration. By defining constraints and solving for optimal parameters, resource allocation can be optimized while satisfying multiple performance and resource constraints, achieving an optimal system design.

[0093] S203 is based on the visual orchestration results verified by rule reasoning and constraint solving. Users can intuitively perform final confirmation operations on business logic components. After confirmation, the basic architecture code is automatically generated through templated code generation technology, including data interface definition, service call logic and exception handling framework, realizing the generation from interactive view to code components.

[0094] The visual orchestration tool displays the business logic architecture in the form of an interactive view. Users can directly add, delete, and modify business logic components in the interactive view. Based on the confirmed visual design results, some basic code is automatically generated according to specific code specifications to achieve efficient conversion from interactive view to component expression.

[0095] The visual orchestration tool has a component library that contains pre-defined functional modules (data panels, logic blocks, interface stubs, etc.). Users can assemble different functional modules together through simple drag and drop and configuration operations to complete complex logical orchestration.

[0096] In this step, after the business logic is orchestrated and verified, code generation algorithms are combined with template engine technology based on the visual design and business logic to automatically generate some code. Developers review and optimize the generated code before performing system integration and testing to ensure system quality and stability.

[0097] The S3 operational model driving module uses the component association rules defined in S1 and the business logic constraints verified by S2 to ensure the operational stability of the aerospace equipment MOM system through a fault diagnosis algorithm; it extends the visual design view of S1 and the code component mapping relationship of S2 to the operational stage, dynamically displays the system operational status view through a visual operation and maintenance interface, supports model visual debugging based on the design state knowledge graph, and ensures the reliability and maintainability of the aerospace equipment MOM system operation.

[0098] After the system is put into operation, the system operation data is collected in real time through sensors, data collectors and other equipment to build an interactive visual operation and maintenance interface, and the system is monitored in real time using fault diagnosis and reasoning algorithms.

[0099] This step specifically includes:

[0100] S301 builds a fault knowledge base containing historical fault cases, characteristic data, and solutions. It uses the Bayesian fault reasoning algorithm to conduct real-time monitoring, prediction, and fault diagnosis for the aerospace equipment MOM system. The Bayesian fault reasoning algorithm is implemented as follows:

[0101] Fault model construction, analyzing the possible fault types that may occur during the low-code system development process, and defining these fault types as fault nodes F j 9j=1,2,…,M); Based on previous development experience and historical data, determine the prior probability P(F j );

[0102] Fault feature extraction: When an aerospace equipment MOM system has an abnormality, relevant fault feature information is collected; fault feature information includes error information in the system log, user operation records, and component operation status; feature information is combined into a fault feature vector

[0103] Conditional probability determination, analysis of the correlation between each fault type and fault characteristics, and determination of conditional probability through statistical analysis of historical fault data

[0104] Bayesian inference calculation, using the Bayesian inference formula Calculate the observed fault feature vector In the case of each faulty node F j Posterior probability of occurrence M is the total number of nodes in the fault tree; At the faulty node F j When the fault characteristic vector The conditional probability of

[0105] Determine the fault source and compare the posterior probability of each fault node The fault node with the highest posterior probability is the most likely fault source. The faulty component corresponding to the fault source is highlighted in the visual operation and maintenance interface and linked to the impact path analysis report.

[0106] S302 displays the operating status view of the aerospace equipment MOM system through a visual operation and maintenance interface, highlights faulty components and associates them with impact path analysis reports, realizes model visual debugging and cross-domain configurable deployment, and accurately locates and expresses faulty components to ensure the reliability and maintainability of system operations.

[0107] The visual design view in step S1 provides the original model reference for operational status monitoring in step S3. Step S3 relies on the design model in step S1 and the code components in step S2, forming a complete "model-driven - code execution - real-time monitoring" process. If a component encounters a problem in a new scenario, step S3 can be used to identify and modify it.

[0108] In this step, when a system anomaly occurs, the fault diagnosis and reasoning algorithm, based on monitoring data and a fault knowledge base, quickly diagnoses the cause and location of the fault and displays the fault information through a visual operation and maintenance interface. When maintenance personnel adjust model parameters and configure data from the unified management center using the visual operation and maintenance interface, the system responds in real time and feeds the results back to the interface, enabling fault repair and system optimization based on the fault information.

[0109] Based on this, the present invention also proposes a multimodal human-computer interaction view and component expression system, including:

[0110] The design state model assembly module uses semantic analysis and knowledge graph technology to deeply mine the business knowledge of the aerospace equipment MOM system, construct a domain knowledge graph, and perform visual transformation. It generates a visual design view that includes demand modeling and component association views, and uses visual views to display the association relationship between components, realizing the transformation from business knowledge to component expression.

[0111] The development model parsing module uses rule-based reasoning algorithms to convert business rules in the domain knowledge graph into executable logical expressions. It also utilizes constraint-solving algorithms to ensure that the business logic complies with the technical specifications defined in the domain knowledge graph. It then operates on components corresponding to the business logic in the interactive view of the visual orchestration tool. It automatically generates code based on the visual design view verified by rule-based reasoning and constraint-solving, achieving the transition from interactive view to code components. This completes functional assembly and logical orchestration, ensuring that the developed business logic accurately maps to the business knowledge.

[0112] The operation state model driving module uses the component association rules defined by the design state model assembly module and the business logic constraints verified by the development state model parsing module to ensure the operational stability of the aerospace equipment MOM system through a fault diagnosis algorithm; extends the visual design view of the design state model assembly module and the code component mapping relationship of the development state model parsing module to the operation stage, dynamically displays the system operation status view through a visual operation and maintenance interface, supports model visual debugging based on the design state knowledge graph, and ensures the reliability and maintainability of the operation of the aerospace equipment MOM system.

[0113] It should be noted that although several modules or units of the device for action execution are mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be concretized in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.

[0114] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.

[0115] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein.

[0116] The above is only a specific implementation of the present invention, but the design concept of the present invention is not limited to this. Any non-substantial changes to the present invention using this concept shall be deemed as an infringement of the protection scope of the present invention.

Claims

1. A multimodal human-computer interaction view and component expression method, characterized in that: include: The S1 design state model assembly module uses semantic analysis and knowledge graph technology to deeply mine the business knowledge of the aerospace equipment MOM system, build a domain knowledge graph, and perform visual transformation. It generates a visual design view that includes demand modeling and component association views, and uses visual views to display the association relationships between components, realizing the transformation from business knowledge to component expression; S2 develops a state model parsing module that converts the business rules in the domain knowledge graph into executable logical expressions through a rule-based reasoning algorithm, and uses a constraint solving algorithm to ensure that the business logic complies with the technical indicators defined in the domain knowledge graph. It also operates on the components corresponding to the business logic in the interactive view of the visual orchestration tool. Automatically generate code based on visual design views verified by rule reasoning and constraint solving, achieving the generation of code components from interactive views, completing functional assembly and logic arrangement, and ensuring that the developed business logic accurately maps to business knowledge; The S3 operational model driving module uses the component association rules defined in S1 and the business logic constraints verified by S2 to ensure the operational stability of the aerospace equipment MOM system through a fault diagnosis algorithm; it extends the visual design view of S1 and the code component mapping relationship of S2 to the operational stage, dynamically displays the system operational status view through a visual operation and maintenance interface, supports model visual debugging based on the design state knowledge graph, and ensures the reliability and maintainability of the aerospace equipment MOM system operation.

2. A multimodal human-computer interaction view and component expression method according to claim 1, characterized in that: Said S1 specifically includes: S101 performs semantic analysis on business documents containing business knowledge in the aerospace equipment MOM system, extracts key business entities and their relationships, and constructs a domain knowledge graph; S102 maps the domain knowledge graph into a software requirement model, and generates a visual design view including functional modules, data flows, and interface definitions using the SysML language; S103 dynamically displays the dependency relationships between components in the visualization view, supports expressing business coupling strength through node connection line weights, and adjusts component configuration through interactive operations to achieve transformation from business knowledge to component expression.

3. A multimodal human-computer interaction view and component expression method according to claim 1, characterized in that: Said S2 specifically includes: S201 defines a set of business rules. Each business rule includes trigger conditions and execution actions. The legitimacy of user operations on business logic components is verified in real time through rule inference algorithms. S202 constructs a multi-dimensional constraint set, including system performance constraints, resource allocation constraints, and business logic constraints, and uses a constraint solving algorithm to solve a parameter combination that satisfies all constraints, ensuring that the orchestration of the business logic complies with the constraints of the aerospace equipment MOM system; S203 is based on the visual orchestration results verified by rule reasoning and constraint solving. Users can intuitively perform final confirmation operations on business logic components. After confirmation, the basic architecture code is automatically generated through templated code generation technology, including data interface definition, service call logic and exception handling framework, realizing the generation from interactive view to code components.

4. A multimodal human-computer interaction view and component expression method according to claim 3, characterized in that: The implementation of the rule reasoning algorithm in S201 is as follows: Rule definition, first define the business rule set R = {r1, r2, ..., r m }, each rule r k Represented as a logical expression Condition k →Action k , Condition k Action is the trigger condition. k To execute an action, k = 1, 2, ..., m; the business rules are not only used to verify user operations, but also serve as the basic logic for function assembly; Status acquisition: In the interactive view of the visual orchestration tool, the user operation status S is acquired in real time. The user operation status S includes the user operation status data and the business parameters input by the user for visual orchestration. Based on the user operation status S, it is determined whether the business rule is triggered, thereby determining the execution of the corresponding function. Rule matching and execution: For each business rule, the real-time user operation status S is brought into the rule trigger condition Condition k In the trigger condition, k Whether it is established, dynamically determine which business rules will be triggered, and if it is established, execute the action k ; Through rule reasoning, different functions are assembled in order according to business rules to achieve logical orchestration; The business rules can be modified and expanded dynamically. When business requirements change, only the rule definitions need to be adjusted without making large-scale modifications to the entire aerospace equipment MOM system.

5. A multimodal human-computer interaction view and component expression method according to claim 3, characterized in that: The implementation of the constraint solving algorithm in S202 is as follows: Definition of constraints, clarifying the constraint set C={c1,c2,…,c i }, each constraint c i It can be expressed as a function, f i (x1,x2,…,x p )≤b i ,fi i (x1,x2,…,x p ) represents the linear expression of the i-th constraint, where x j is the system parameter, b i is the constraint boundary, p represents the number of system parameters, and j is the index of the system parameter; Conflict detection and resolution: Business rules are transformed into specific constraints in practical applications. When there are contradictions between these constraints, rule conflicts will occur. Linear programming algorithms are used to solve the parameters that meet all the constraints. The general form of linear programming is: Objective function: The objective function means finding a set of system parameter values ​​that minimizes the objective function value Z while satisfying all constraints. Constraints: f i (x1,x2,…,x p )≤b i ,i=1,2,…l;x j ≥0, j=1,2,…,p Right now: where a ij Represents the constraint coefficient, which is used to define the linear relationship of the constraint; By introducing the slack variable s i ≥0(i=1,2,…,l), converting inequality constraints into equality constraints to facilitate iterative calculation of linear programming: a i1 x1+a i2 x2+…+a ip x p +s i =b i ; Iterative optimization, for the decision variable x j The test number is Determine whether the variable can optimize the objective function, where Represents the base variable B i The coefficients in the objective function, the basis matrix B, is composed of a set of linearly independent column vectors selected from the coefficient matrix corresponding to the standard form equality constraint. The variables corresponding to these column vectors are called basis variables. i Represents the i-th basis variable, if all σ j ≥0, then the current decision variable x j This is the optimal solution; Otherwise, repeat the above steps to iteratively determine the basis variables. j ≥0, the iteration ends and the parameters corresponding to the basic variables are determined, which is the optimal solution that satisfies all constraints. The optimal value of the objective function is 6. A multimodal human-computer interaction view and component expression method according to claim 3, characterized in that: The S203 specifically includes: The visual orchestration tool displays the business logic architecture in the form of an interactive view, allowing users to add, delete, and modify business logic components directly in the interactive view. Based on the confirmed visual design results, some basic code is automatically generated according to specific code specifications, achieving efficient conversion from the interactive view to the component expression. The visual arrangement tool is provided with a component library, which contains predefined functional modules. Different functional modules are assembled together through dragging and configuration operations to complete complex logical arrangement.

7. A multimodal human-computer interaction view and component expression method according to claim 1, characterized in that: Said S3 specifically includes: S301 builds a fault knowledge base containing historical fault cases, characteristic data, and solutions, and uses a Bayesian fault reasoning algorithm to perform real-time monitoring, prediction, and fault diagnosis on the aerospace equipment MOM system; S302 displays the operating status view of the aerospace equipment MOM system through a visual operation and maintenance interface, highlights faulty components and associates them with impact path analysis reports, realizes model visual debugging and cross-domain configurable deployment, and accurately locates and expresses faulty components to ensure the reliability and maintainability of system operations.

8. A multimodal human-computer interaction view and component expression method according to claim 3, characterized in that: The implementation of the Bayesian fault inference algorithm in S301 is as follows: Fault model construction, analyzing the possible fault types that may occur during the low-code system development process, and defining these fault types as fault nodes F j (j=1,2,…,M); According to the previous development experience and historical data, the prior probability P(F j ); Fault feature extraction: when the aerospace equipment MOM system is abnormal, relevant fault feature information is collected; the fault feature information includes error information in the system log, user operation records, and component operation status; the feature information is combined into a fault feature vector Conditional probability determination, analysis of the correlation between each fault type and fault characteristics, and determination of conditional probability through statistical analysis of historical fault data Bayesian inference calculation, using the Bayesian inference formula Calculate the observed fault feature vector In the case of each faulty node F j Posterior probability of occurrence M is the total number of nodes in the fault tree; At the faulty node F j When the fault characteristic vector The conditional probability of Determine the fault source and compare the posterior probability of each fault node The fault node with the highest posterior probability is the most likely fault source.

9. A multimodal human-computer interaction view and component expression system, characterized in that: include The design state model assembly module uses semantic analysis and knowledge graph technology to deeply mine the business knowledge of the aerospace equipment MOM system, construct a domain knowledge graph, and perform visual transformation. It generates a visual design view that includes demand modeling and component association views, and uses visual views to display the association relationship between components, realizing the transformation from business knowledge to component expression. Develop a state model parsing module that converts the business rules in the domain knowledge graph into executable logical expressions through rule-based reasoning algorithms, and uses constraint solving algorithms to ensure that the business logic complies with the technical indicators defined in the domain knowledge graph. Operate the components corresponding to the business logic in the interactive view of the visual orchestration tool; Automatically generate code based on visual design views verified by rule reasoning and constraint solving, achieving the generation of code components from interactive views, completing functional assembly and logic arrangement, and ensuring that the developed business logic accurately maps to business knowledge; The operation state model driving module uses the component association rules defined by the design state model assembly module and the business logic constraints verified by the development state model parsing module to ensure the operational stability of the aerospace equipment MOM system through a fault diagnosis algorithm; extends the visual design view of the design state model assembly module and the code component mapping relationship of the development state model parsing module to the operation stage, dynamically displays the system operation status view through a visual operation and maintenance interface, supports model visual debugging based on the design state knowledge graph, and ensures the reliability and maintainability of the operation of the aerospace equipment MOM system.

Images