Virtual collaborative modeling method and system for aero-engine

Abstract

The invention relates to the technical field of aero-engines, and particularly provides an aero-engine virtual collaborative modeling method and system, and the method comprises the steps: decomposing an aero-engine dismounting process into an operation task, and expressing the operation task as a JSON object; performing universal expression on the disassembly and assembly sequence of the JSON object, and constructing a task-driven process model based on a double-sequence directed graph; identifying a gesture corresponding to the hand feature data by using a KNN algorithm, and caching the identified gesture to a gesture sequence; describing the basic behavior based on the gesture sequence by utilizing a finite-state machine, and interacting the basic behavior with the gesture parameter of the dismounting object by utilizing a collision box to obtain an operation process of operating the object by a single hand; a behavioral model of a tool and a behavioral model of a single-hand or multi-hand operation virtual part are realized through a finite-state machine and a spatial transformation process, and an operation process of an actual maintenance scene is simulated through a visual feedback interaction mechanism. The disassembly and assembly process is described as a universal disassembly and assembly process model.

Description

Technical Field

[0001] The present invention belongs to the technical field of aero-engines, and in particular relates to a method and system for virtual collaborative modeling of an aero-engine. Background Art

[0002] Twin simulation model is an important technology in the industrial field. At present, the disassembly and assembly process model of the virtual disassembly and assembly system is mainly based on the process model of graph theory, such as AND / OR graph model, directed graph model and Petri net model. Among them, the directed graph consists of nodes and directed edges, which can intuitively represent the serial and parallel constraints of the disassembly and assembly nodes. The configurability of the system needs to be achieved with the flexibility of the disassembly and assembly process sequence, which means that it can be visualized as a corresponding directed graph structure according to different operating procedures.

[0003] The visualization method of the relevant process model is mainly driven by XML data. On the one hand, the transmission and parsing of XML files requires a large amount of time. On the other hand, the existing multi-person collaborative system involves a small amount of tasks. In a network environment, when dealing with multi-terminal and large-volume aircraft engine disassembly and assembly scenarios, a more lightweight and efficient method should be selected to improve deployment efficiency. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the prior art, the present invention provides an aero-engine virtual collaborative modeling method and system to solve the above-mentioned technical problems.

[0005] In a first aspect, the present invention provides an aircraft engine virtual collaborative modeling method, comprising: Decomposing the aircraft engine disassembly and assembly process into operation tasks, and expressing the operation tasks as JSON objects; The assembly and disassembly sequence of the JSON object is generally expressed, and a task-driven process model based on a double-sequence directed graph is constructed; Collect hand feature data through the Leap Motion device, use the KNN algorithm to identify the gestures corresponding to the hand feature data, and cache the identified gestures into a gesture sequence; A finite state machine is used to describe basic behaviors based on gesture sequences, and a collision box is used to interact the basic behaviors with gesture parameters of the disassembly and assembly objects to obtain the operation process of operating the objects with one hand; The behavioral models of tools and the behavioral models of single-handed and multi-handed operation of virtual parts are realized through finite state machines and space transformation processes, and the operation process of actual maintenance scenarios is simulated through the interactive mechanism of visual feedback.

[0006] In an optional implementation, the aircraft engine disassembly and assembly process is decomposed into operation tasks, and the operation tasks are expressed as JSON objects, including: Define a task object, and set basic properties, operation mode, runtime state identifier, sequence properties and operation prompt properties for the task object; the basic properties include: task sequence number, operation object name and operation object peer object relationship; the operation mode includes: free movement behavior and geometric motion behavior with adjustable parameters, wherein the free movement behavior includes single-handed grasping movement or multi-hand collaborative control posture transformation, the properties of the free movement behavior include operation mode, target position and gesture, and the properties of the geometric motion behavior include operation mode, axis vector, tool type, tool axial offset and target value; the runtime state identifier includes: collision body type, trigger addition status and completion status; the sequence properties include: preceding task list and subsequent task list, and the task sequence has a corresponding relationship with the fault type; the operation prompt properties include: task prompt title and specific information; In JSON, the parameter name and parameter value of a single task are combined and expressed as a key-value pair, each operation task is expressed as an object, and the collection of all objects is expressed as a list.

[0007] In an optional implementation, the assembly and disassembly sequence of the JSON object is universally expressed to construct a task-driven process model based on a dual-sequence directed graph, including: Define the total task list, unfinished task list, executable task list and completed task list; Parse the JSON object, generate the predecessor task set and the successor task set based on the basic attributes and sequence attributes of each JSON object, and build a task-driven process model based on a bi-sequence directed graph based on the predecessor task set and the successor task set; In the task-driven process model, the number of tasks that can be executed at the same time is often greater than or equal to 1; The task-driven process model is configured to read the parameterized file through the ReadAllText method encapsulated by the File class provided by C#, parse the JSON parameterized file and deserialize it into the Tasks class through the FromJson method encapsulated by the JsonUtility class provided by Unity3D, and retain it in the memory to complete the task deployment.

[0008] In an optional implementation, the driving method of the task-driven process model includes: Multi-terminal network requests and parses JSON data files, and initializes the total task set, unfinished task set, executable task set, and completed task set; Add element 0 to the completed task set, use this as a condition to traverse all tasks, activate all start tasks, and add the JSON parameter-specified behavior model encapsulation code to all activated tasks; When any i-th task is completed, the unfinished task set, executable task set, and completed task set are updated; Get the subsequent task set corresponding to the completed i-th task, traverse the subsequent task set, get the predecessor task set of the j-th task in the subsequent task set, and determine whether the j-th task in the subsequent task set can be activated according to the activation condition formula; Activate the activatable tasks, assign corresponding behavior models to encapsulate code components, and provide the correct interaction method; The subsequent tasks of the completed tasks are activated in a loop until the completed task set is the same as the total task set, indicating that all tasks have been completed.

[0009] In an optional implementation, hand feature data is collected by a Leap Motion device, a KNN algorithm is used to identify gestures corresponding to the hand feature data, and the identified gestures are cached in a gesture sequence, including: Collect hand feature data through the Leap Motion device, wherein the hand feature data includes the distance from the fingertips of five fingers to the palm and the fingertip direction vectors of the thumb, index finger and little finger; The hand feature data is saved as a csv format file; The KNN algorithm is used to calculate the Euclidean distance between the hand feature data and the standard feature data of multiple gestures, and the calculation results are sorted in ascending order; the multiple gestures include: grab, pinch, release, forward, backward and interface; The first K gestures are scored, and the gesture with the highest score is the gesture corresponding to the hand feature data; Cache the most recently recognized gesture at the end of the gesture sequence.

[0010] In an optional implementation, a finite state machine is used to describe basic behaviors based on gesture sequences, and a collision box is used to interact the basic behaviors with gesture parameters of the disassembly and assembly object to obtain an operation process of operating the object with one hand, including: The state change process of the basic behaviors of picking up, releasing and transferring objects with one hand is defined as a quaternion by using a finite state machine. Generate a state set of the operation object based on the quadruple of the hand that collides with the operation object; A trigger condition set and a finite jump trigger event set are constructed, and a collision box is used to detect dynamic gestures based on the state set, the trigger condition set and the finite jump trigger event set.

[0011] In an optional implementation, the behavior model of the tool and the behavior model of the single-handed and multi-handed operation of the virtual parts are realized through a finite state machine and a space transformation process, and the operation process of the actual maintenance scene is simulated through an interactive mechanism of visual feedback, including: Solve the in-position posture according to the tool axial offset in the parameterized file and other built-in offset values ​​of the tool, interpolate the in-position posture frame by frame, and describe the tool usage process through a finite state machine; Capture hand position via external controller or Leap Motion device; The position of the disassembled and assembled objects in the process of multi-hand collaborative handling is solved by the position of the hands; Based on the posture, a finite state machine and the spatial geometric transformation of the disassembled object are used to describe the behavior of the disassembled object.

[0012] In a second aspect, the present invention provides an aircraft engine virtual collaborative modeling system, comprising: An object expression module, used to decompose the aircraft engine disassembly and assembly process into operation tasks, and express the operation tasks as JSON objects; A driver configuration module is used to express the assembly and disassembly sequence of the JSON object in a universal manner and to construct a task-driven process model based on a dual-sequence directed graph; The gesture recognition module is used to collect hand feature data through the Leap Motion device, identify the gestures corresponding to the hand feature data using the KNN algorithm, and cache the identified gestures into a gesture sequence; An operation description module, used to describe basic behaviors based on gesture sequences using a finite state machine, and to interact the basic behaviors with gesture parameters of the disassembly and assembly objects using a collision box to obtain an operation process of operating the object with one hand; The scenario description module is used to implement the behavior model of the tool and the behavior model of single-handed and multi-handed operation of virtual parts through a finite state machine and space transformation process, and simulate the operation process of the actual maintenance scenario through an interactive mechanism of visual feedback.

[0013] In a third aspect, a terminal is provided, including: processor, memory, wherein: The memory is used to store computer programs. The processor is used to call and run the computer program from the memory, so that the terminal executes the above-mentioned terminal method.

[0014] According to a fourth aspect, a computer storage medium is provided, wherein the computer-readable storage medium stores instructions, and when the instructions are executed on a computer, the computer executes the methods described in the above aspects.

[0015] The beneficial effect of the present invention is that the aero-engine virtual collaborative modeling method and system provided by the present invention uses JSON as a data carrier, in which the parameter name and parameter value of a single task are combined and expressed as a key-value pair, each task is expressed as an object, and the collection of all task objects is expressed as a list, so as to realize the specific behavior driving of the task and the driving of the task sequence at the data level. The present invention universally expresses the disassembly and assembly sequence, extracts the key information of the disassembly and assembly process, and describes it as a universal disassembly and assembly process model that is easy to understand and use, thereby improving the configurability of the system.

[0016] In addition, in terms of collaborative disassembly and assembly by multiple people, the present invention improves the immersion of the system through a more natural bare-hand interaction method, and designs a collaborative disassembly and assembly behavior model based on a finite state machine and a spatial transformation process, namely, a behavior model of tools and a behavior model of single-handed and multi-handed operation of virtual parts, which can achieve synchronization and coordination of collaborative disassembly and assembly by multiple people and ensure the correctness and consistency of the disassembly and assembly process.

[0017] In addition, the invention has a reliable design principle, a simple structure and a very broad application prospect. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 is a schematic flow chart of a method according to an embodiment of the present invention.

[0020] Figure 2 The figure is a schematic diagram of a task-driven process model based on a dual-sequence directed graph of a method according to an embodiment of the present invention.

[0021] Figure 3 It is a task-driven flow chart of a method according to an embodiment of the present invention.

[0022] Figure 4 Schematic diagram of gesture types of a method according to an embodiment of the present invention.

[0023] Figure 5 Schematic diagram of hand features of a method according to an embodiment of the present invention.

[0024] Figure 6 It is a schematic diagram of basic behavior state transition of a single-handed operation object according to a method of an embodiment of the present invention.

[0025] Figure 7 It is a schematic diagram of tool parameters of a method according to an embodiment of the present invention.

[0026] Figure 8 It is a schematic diagram of the object rotation process of the method according to an embodiment of the present invention.

[0027] Fig. 9 The figure is a state transition diagram of an object rotation process of a method according to an embodiment of the present invention.

[0028] Fig.10 The figure is a schematic diagram of an object translation process of a method according to an embodiment of the present invention.

[0029] Fig.11 The following is a state transition diagram of an object translation process of a method according to an embodiment of the present invention.

[0030] Fig.12 It is a multi-hand collaboration state transition diagram of a method according to an embodiment of the present invention.

[0031] Fig.13 It is a schematic diagram of multi-hand transport according to a method of an embodiment of the present invention.

[0032] Fig.14 is a schematic block diagram of a system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0033] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0034] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0035] The aircraft engine virtual collaborative modeling method provided by the embodiment of the present invention is executed by a computer device, and accordingly, the aircraft engine virtual collaborative modeling system runs in the computer device.

[0036] Figure 1 is a schematic flow chart of a method according to an embodiment of the present invention. Figure 1 The execution subject may be an aircraft engine virtual collaborative modeling system. According to different requirements, the order of the steps in the flow chart may be changed, and some may be omitted.

[0037] like Figure 1 As shown, the method includes: Step 110, decomposing the aircraft engine disassembly and assembly process into operation tasks, and expressing the operation tasks as JSON objects; Step 120, generalizing the assembly and disassembly sequence of the JSON object and constructing a task-driven process model based on a dual-sequence directed graph; Step 130, collecting hand feature data through the Leap Motion device, using the KNN algorithm to identify the gesture corresponding to the hand feature data, and caching the identified gesture into a gesture sequence; Step 140, using a finite state machine to describe basic behaviors based on gesture sequences, and using a collision box to interact the basic behaviors with gesture parameters of the disassembly and assembly object to obtain an operation process of operating the object with one hand; Step 150, implement the behavior model of the tool and the behavior model of single-handed and multi-handed operation of virtual parts through the finite state machine and space transformation process, and simulate the operation process of the actual maintenance scene through the interactive mechanism of visual feedback.

[0038] To facilitate understanding of the present invention, the following further describes the method for virtual collaborative modeling of an aircraft engine provided by the present invention based on the principle of the method for virtual collaborative modeling of an aircraft engine of the present invention and the process of virtual collaborative modeling of an aircraft engine in an embodiment.

[0039] Specifically, the aero-engine virtual collaborative modeling method includes: S1. Decompose the aircraft engine disassembly and assembly process into operation tasks, and express the operation tasks as JSON objects.

[0040] Each step of VR collaborative disassembly and assembly for aircraft engines is regarded as a task, and the disassembly and assembly task sequences for different faults are also different. The definition of a single task needs to refer to the actual disassembly and assembly maintenance operations of aircraft engines. The complete operation of disassembling or installing a complex component can be regarded as a combination of several common simple movements, which can be abstractly summarized into five types of movements: translation, rotation, revolution, single-handed picking, and multi-handed carrying. The real disassembly and assembly operations are mapped to the program, and a task object is defined from multiple angles: From the task ontology, it should have three basic attributes: task sequence number, operation object name and the relationship between the operation object and the same-level objects.

[0041] From the perspective of operational behavior, the operation mode of the object should be constrained, and its operation mode can be divided into two categories: free movement behavior and geometric motion behavior with adjustable parameters. Free movement behavior refers to single-handed grasping and movement or multi-hand collaborative control posture transformation, which should have three attributes: operation mode, target position and gesture. Geometric motion behavior should have five attributes: operation mode, axis vector, tool type, tool axial offset and target value.

[0042] From a runtime perspective, there should be three runtime status identifiers: collision body type, trigger addition status, and completion status.

[0043] From the perspective of task coherence, it should have two sequence attributes: the list of preceding tasks and the list of succeeding tasks.

[0044] From the perspective of task communication, there should be two operation prompt attributes: task prompt title and specific information.

[0045] In summary, the operational elements are summarized from multiple perspectives as shown in Table 1: Table 1 Operation elements and execution effects

[0046] The present invention uses JSON as a data carrier. In JSON, the parameter name and parameter value of a single task are combined and expressed as a key-value pair, each task is expressed as an object, and the collection of all task objects is expressed as a list, so as to realize the specific behavior driving of the task and the driving of the task sequence at the data level. In the JSON parameterized file, an object is expressed by combining "{" and "}", a list is expressed by combining "[" and "]", and specific parameters are expressed in the form of key-value pairs. The set of all tasks "Tasks" is used as a key, and its value is a list composed of all task objects "Task". Each task object "Task" in the list is described by 17 parameters. Therefore, the JSON parameterized file is read and parsed into 19 disassembly sequence parameters, and their names and interpretations are shown in Table 2.

[0047] Table 2 Disassembly sequence parameter names and definitions

[0048] S2. The assembly and disassembly sequence of the JSON object is expressed in a universal manner, and a task-driven process model based on a double-sequence directed graph is constructed.

[0049] Data-driven refers to reading and parsing external parameterized data files according to certain rules to achieve the purpose of driving specific system behaviors. The system structure based on data-driven can reduce a large amount of duplicate code and effectively reduce the coupling between driving data and system behavior. Reducing the coupling between the operating parameters of the disassembly and assembly tasks and the system behavior in the form of data-driven, encapsulating the operating task sequence into a parameterized file, and driving the system behavior with the parameterized file is the prerequisite for realizing the configurability of the system.

[0050] The task driving includes three steps: activating the task, triggering the task behavior and retrieving the subsequent tasks, which involve the properties of Tasks, TaskID, TriggerBeAdd, ColliderType, PreviousTaskIDs and NextTaskIDs in the JSON configuration file. The system needs to define the total task list, unfinished task list, executable task list and completed task list, which can be expressed as mathematical sets , , , , the elements in the set are all task numbers. Suppose a task number is , the number of tasks is ,but It can be expressed as:

[0051] , , , The relationship is as follows:

[0052] Similarly, each task contains a predefined list of previous tasks (PreviousTaskIDs) and a list of next tasks (NextTaskIDs), both of which can also be expressed as sets. The previous task set is used to determine whether the task meets the activation conditions. The set of predecessor tasks of the step is ,but The cardinality is ,by Indicates the task Corresponding The number of completed tasks in The step activation condition can be expressed as:

[0053] Record The set of subsequent tasks of the step is , the subsequent task set is used to point to the Based on the set of preceding tasks and the set of following tasks, a disassembly and assembly process model based on a double-sequence directed graph is designed. The nodes of the directed graph are , and Together, Indicates the task , To point to a node After the node, For backtracking The number of completed nodes is used to determine the task Whether it can be activated.

[0054] The collaborative disassembly and assembly of multiple people requires the coordination of tasks, which not only requires the execution order of tasks to be constrained, but also the number of executable tasks at the same time is often greater than or equal to 1, that is, concurrent coordination of tasks. Figure 2 In the example, initially there are two tasks activated, Task 1 and Task 2. After Task 1 is completed, Tasks 3 and 4 will be activated. Task 5 does not meet the activation conditions and is not activated yet. At this time, the activated tasks are 3, 4, and 2. After Task 2 is completed, Tasks 5 and 6 will be activated. At this time, there are a total of 4 tasks available for execution in the scene, Tasks 3, 4, 5, and 6.

[0055] The ReadAllText method encapsulated by the File class provided by C# can read parameterized files, and the FromJson method encapsulated by the JsonUtility class provided by Unity3D can parse the JSON parameterized file and deserialize it into the Tasks class, which is retained in the memory to complete the task deployment. The key mathematical expressions involved in the task sequence parsing and driving process are shown in Table 3.

[0056] Table 3 Key mathematical expressions of task-driven processes

[0057] like Figure 3 As shown in the figure, according to the designed task-driven method, the system task-driven process is designed. The system task-driven process mainly has the following five steps: Step 1: Multiple terminals connect to the Internet to request JSON data files and parse and initialize them , , , .

[0058] Step 2: Add element 0 to , use this as a condition to traverse all tasks, activate all start tasks, and add the JSON parameters to all activated tasks to specify the behavior model encapsulation code.

[0059] Step 3: Any task Execution completed, update , , .

[0060] Step 4: Get completed tasks Corresponding , traverse , get The set of predecessor tasks is determined according to the activation condition formula Whether it can be activated.

[0061] Step 5: Activate the activatable tasks, assign corresponding behavior models to encapsulate code components, and provide the correct interaction method. Step 6: Repeat steps ③, ④ and ⑤ until == , indicating that all tasks have been completed.

[0062] S3. Collect hand feature data through the Leap Motion device, use the KNN algorithm to identify the gesture corresponding to the hand feature data, and cache the identified gesture to the gesture sequence.

[0063] S4. Using a finite state machine to describe basic behaviors based on gesture sequences, and using a collision box to interact the basic behaviors with gesture parameters of the disassembly and assembly object to obtain an operation process of operating the object with one hand.

[0064] like Figure 4 As shown, the present invention sets 6 gestures, including 3 working gestures and 3 auxiliary gestures, namely grab, pinch, release, forward, backward and call out interface. The names, definitions and functions of the gestures are shown in Table 4: Table 4 Gesture names, definitions and functions

[0065] Please refer to Figure 5 According to the characteristics of the gesture, there are 8 hand features that need to be collected, namely the distance from the fingertips to the palm of the hand and the fingertip direction vectors of the thumb, index finger and little finger. The collected hand feature data is saved as a csv file and stored in the system to provide a data set for the gesture recognition link.

[0066] The KNN (K-Nearst Neighbor) classification algorithm is used for gesture recognition. The system performs gesture recognition in real time during operation. The classification process is as follows: Step 1: Real-time sampling and feature extraction.

[0067] Step 2: Use the Euclidean distance formula to calculate the Euclidean distance between the data sample and the real-time sampling feature. The Euclidean distance between the real-time sampling feature and each sample data. To adopt features in real time, For the The characteristics of the sample data.

[0068]

[0069] Step 3: Sort the calculation results in ascending order.

[0070] Step 4: Take the first K samples for scoring, and the gesture with the highest score is the gesture recognition result.

[0071] Step 5: Output gesture recognition results.

[0072] The disassembly and assembly object refers to the aircraft engine unit that is disassembled or assembled by the virtual avatar, such as a virtual part or mechanism; the operation object includes the disassembly and assembly object and the tools. In the process of performing the VR disassembly and assembly task of the aircraft engine, many trainees need to frequently pick up, release and pass the tools and disassembly and assembly objects.

[0073] Currently, in most VR disassembly and assembly studies, the behaviors of picking up, delivering, and passing are triggered by static gestures or handle triggers. Static gesture interaction refers to the process of interacting with tools or disassembly and assembly objects, which only determines whether it is a specified gesture without going through the process of gesture changes. This will lead to unnatural picking behaviors and picking effects.

[0074] In the actual tool transfer process, the picker's palm has to go through a change process from an open state to a grasping state. Picking, releasing and transferring are usually for one-handed objects. One-handed objects refer to objects that can be operated or interacted with using one hand, including one-handed disassembly objects and tools. The picking, releasing and transferring processes of one-handed objects are the same. Therefore, a basic behavior model of one-handed objects based on dynamic gestures is designed based on a finite state machine to realize the picking, releasing and transferring of one-handed objects.

[0075] The picking and releasing process must detect gesture changes within the collision box. The picking process of tools and disassembly objects differs in gesture requirements. The gestures that can pick up tools are: pinching and grabbing, while the gesture of picking up a disassembly object is specified by the Gesture parameter of the disassembly object in the parametric model. The picking process of one-handed objects is specifically implemented as follows: determine the gesture of each frame. If the gesture of the previous frame is a non-grasping gesture and the gesture of the current frame is a grasping gesture, the one-handed object will be picked up; the releasing process of one-handed objects is specifically implemented as follows: in the grasping state, if the gesture of the previous frame is a non-releasing gesture and the gesture of the current frame is a releasing gesture, then add a gravity component to simulate the process of releasing and dropping. In order to allow students to perceive the state of the operating object, visual feedback with different color highlights is designed to reflect the operation process of picking up, transferring and releasing.

[0076] The state change process of the basic behavior model of picking up, releasing and passing objects with one hand is modeled and described by a finite state machine (FSM). FSM is a mathematical model used to describe the discrete input and output of a system, and is used to describe the state sequence and response actions of an object during its life cycle. is defined as a four-tuple: , , , }.

[0077] by Indicates the first hand that collides with the operation object. Represents the second hand that collides with the operand. is a state set, and its specific meaning is shown in Table 5, where It is the initial state and also the final state.

[0078] Table 5 Status of one-handed operation objects

[0079] It is a set of trigger conditions for state transition. Only one trigger condition can be activated at the same time. The specific meanings are shown in Table 6: Table 6 State transition triggering conditions for basic behaviors of one-handed operation objects

[0080] It is a finite jump trigger event set, and its specific meaning is shown in Table 7: Table 7 Trigger events for state transition of one-handed operation objects

[0081] is the state transition rule, . The state transition diagram is as follows Figure 6 shown.

[0082] The basic behavior state transition of one-handed operation objects is shown in Table 8: Table 8 Basic behavior state transition table of one-handed operation objects

[0083] S5. The behavior model of the tool and the behavior model of single-handed and multi-handed operation of virtual parts are realized through the finite state machine and space transformation process, and the operation process of the actual maintenance scene is simulated through the interactive mechanism of visual feedback.

[0084] S501, tool modeling.

[0085] When using tools for assembly and disassembly in a virtual environment, the lack of physical attachment points in the real environment will cause the tools to float unsteadily, affecting the operating experience. Therefore, it is necessary to solve the natural posture and adjust it.

[0086] The tool usage behavior model includes the posture solution and usage process of the tool in place. The tool in place posture refers to the natural initial usage posture calculated according to the posture of the tool after the tool collides with the collision body of the bolt. The tool in place posture solution process is specifically implemented as follows: first, the in-place posture needs to be solved according to the tool axial offset and other built-in offset values ​​of the tool in the parameterized file, and then the in-place posture is interpolated frame by frame. The tool can only be used after interpolating to the target posture frame by frame, and the tool usage process is the same. Therefore, it is necessary to solve the in-place posture first, and then implement the tool usage process through the finite state machine.

[0087] Taking the adjustable wrench and hexagonal bolt as examples, a geometric model for adjusting the wrench posture is constructed. Figure 7 As shown, the object coordinate system of the tool is determined in the virtual object modeling stage. is the set tool origin, the positive direction of the z-axis is the working direction, The positive direction of the axis is along the center axis of the tool toward the tool head, determined by the left-hand rule. Positive axis direction. The working contact point of the tool, Relative to exist The displacement on the axis is ,exist The displacement on the axis is ,exist The displacement on the axis is . is the origin of the part set in the modeling stage, and the contact point between the part and the tool is . Virtual Parts The specific position of the point is affected by the "Bias" and "Axis" properties set in the configuration file dynamically loaded at runtime. The "Bias" property is the Point relative to The offset of the point, i.e. value; the "Axis" attribute is the direction vector of the point offset, that is, the vector . That is, in the world coordinate system, it can be expressed as: .

[0088] When the tool is transformed from the free displacement state to the working position state, it is necessary to determine the starting posture and interpolate it to the position of the position state frame by frame. Solving the position of the position in position can be divided into two steps: calculating the coordinates of the position state and the quaternion of the position state.

[0089] Calculate the coordinates of the in-position state. Set point With vector The plane is defined as , using the formula to calculate the vector In plane The projection vector on , after normalization, we get the vector .

[0090]

[0091] After normalization, we get the vector , in position The coordinates of a point can be expressed as:

[0092] To calculate the quaternion in position, the reference coordinate system must be calibrated first, and then the conversion quaternion is obtained by the Euler angle difference between each frame attitude and the reference attitude, and the quaternion in position is calculated. First, determine Point is the origin of the local coordinate system. The square of the axis is the vector , that is, vector the opposite direction; The positive direction of the axis is the vector ; The positive direction of the axis is determined by the left-hand rule.

[0093] After the reference is determined, the Euler angle difference between the current attitude and the reference attitude can be obtained. , , Respectively around , , The angle of rotation of the axis, calculation Axis angle offset, get the calibrated Euler angle difference , , .

[0094]

[0095] Substitute the three Euler angle differences into the formula to get the converted quaternion , and then with the current attitude quaternion The in-position quaternion is obtained by the formula .

[0096]

[0097]

[0098] S502: Object modeling.

[0099] The behavior model of all disassembly objects is described by the finite state machine and the spatial geometric transformation of the disassembly object. can be defined as a four-tuple: , , , }. It is the state collection of the disassembly object; It is a set of trigger conditions for state transition. Only one trigger condition can be activated at the same time. It is a finite jump trigger event set in the basic behavior model, and this event set is included if the highlight changes; is the state transition rule, .

[0100] (1) Rotation. The parametric file defines two key parameters required for the assembly object to rotate around any axis: the rotation axis and the operation target value. Figure 8 As shown, the rotation axis is represented as a vector , set the tool around the part The rotation angle of each frame is , then the rotation angle of the disassembly object is also .

[0101] The rotational motion of each frame can be represented by quaternion as: ; Substitute the formula and multiply it by the current quaternion of the part to calculate the quaternion of the disassembled object after rotation. .

[0102] The meaning of the rotation process state is shown in Table 9: Table 9 Status of the rotation process

[0103] The meaning of the trigger conditions is shown in Table 10: Table 10 Triggering conditions for changes in the rotation process

[0104] The state transition of the rotation process is as follows Fig. 9 Shown: .

[0105] (2) Translation. The translation operation refers to the translation of a part along a given direction for a certain distance. The schematic diagram of the translation process of the disassembly object is as follows: Fig.10 As shown in the figure, the parametric file defines two key parameters required for the translation process of the disassembly object: translation direction and translation distance, where the translation direction is a given direction, expressed as , the position change of the hand within a frame is represented as a vector , exist The projection in the direction is ,but is the translation distance, and the current position of the part is , then the position of the part after one frame .

[0106] The meaning of the translation process status is shown in Table 11: Table 11 Status of the translation process

[0107] The meaning of the trigger conditions is shown in Table 12: Table 12 Trigger conditions for translation process

[0108] The state transfer process of the translation process is as follows Fig.11 shown.

[0109] (3) Multi-hand collaboration. Multi-hand collaborative operation of large objects is common in the process of transporting. The behavior modeling of collaborative transport of large objects in multi-person collaboration is carried out to simulate the effect of multiple hands jointly controlling the parts to be repaired. This method designs four multi-hand collaborative transport behavior models: two-hand collaboration, three-hand collaboration, four-hand collaboration, and multi-hand collaboration. Different numbers of hands correspond to different states.

[0110] The meaning of the multi-hand collaborative handling status of large objects is shown in Table 13: Table 13 Status of the multi-hand collaborative process of disassembling and assembling objects

[0111] The trigger conditions are shown in Table 14: Table 14 Trigger conditions for multi-hand collaboration process

[0112] The state transition process of multi-hand collaboration is as follows Fig.12 shown.

[0113] Solving the posture of the disassembled and assembled objects during multi-hand collaborative handling by using the hand positions is generally divided into two steps: Step 1: Solve the object position. The solution process of this step is consistent, and it is solved by calculating the center of mass of the spatial positions of multiple hands. In actual handling, the movement of a rigid object is only related to the external contact point. In virtual space, the movement of a virtual object is independent of its original coordinate origin and is related to the external hand position. Therefore, the hand position can be used to calculate and update the object's coordinate origin, replacing the object's original coordinate origin. The weight ω of each hand is 1. Suppose there are n hands for collaborative handling. The spatial position of the i-th hand in the absolute coordinate system is expressed as , then the formula can be used to solve the updated coordinate origin of the object location.

[0114]

[0115] Step 2: Solve the object posture. The solution process of this step is different. Redefine the new coordinate origin The rotation posture is updated through projection and rotation transformation steps rotation posture.

[0116] The following is an example of multi-hand collaborative handling process. Fig.13 : For multi-hand collaborative handling with n (n>4) hands participating in the interaction, the more collaborative hands there are, the more complicated the posture quaternion calculation method becomes. Collaborative handling in the real world is different from collaborative handling in virtual scenes. In the real world, the movement of the hand will drive the movement of the object. Collaborative handling has a force feedback effect, and the hand moves due to the feedback of the object. However, in virtual scenes, the position input source of the hand can only come from external handles, Leap Motion and other devices, which makes it difficult to simulate the feedback of the object on the hand.

[0117] Therefore, when the number of hands involved in the interaction increases, it is difficult to simulate the precise interaction of multiple hands with 6 degrees of freedom. For multi-hand collaborative handling when the number of hands involved in the handling is greater than 4, this paper uses the idea of ​​two-hand collaborative handling to simplify the multi-hand collaboration. The multi-hand collaborative handling posture solution process is as follows: Fig.13 As shown, the positions of the hands involved in the interaction on both sides are calculated using the formula , the object pose is solved through three steps: Initialization stage: The hand position set is divided into two sides through specific division conditions, and the midpoint position of each side is solved by the formula and ,pass and The coordinate operation of point to The unit direction vector and midpoint The coordinates of Sure Node y-axis positive, plane Depend on The y-axis in the local coordinate system is Nodes are defined together. Randomly select a direction vector in the plane The positive x-axis of the node can be determined by the right-hand rule. The positive z-axis of the node, after this step Coordinate and posture initialization is completed.

[0118] Update phase: In the update phase, the formula is used to calculate and update the object position in the current frame during the movement process . Continuously update while your hands are moving . Let the current frame be , the previous frame is , then R is the rotation matrix between the two frames, and the quaternion ΔQ corresponding to R can be obtained. The quaternion is Q, and we can calculate it by substituting it into the formula The quaternion of the current frame .

[0119] Repeat phase: Repeat the update phase until the current step is completed.

[0120] After simplifying the idea of ​​carrying with both hands, we get two coordinate points in space. and , similar to the collaborative carrying of the object by both hands, the object has 5 degrees of freedom under joint control, but 1 degree of rotational freedom is missing.

[0121] In some embodiments, the aircraft engine virtual collaborative modeling system may include a plurality of functional modules composed of computer program segments. The computer programs of the various program segments in the aircraft engine virtual collaborative modeling system may be stored in a memory of a computer device and executed by at least one processor to perform (see Figure 1 Description) Functions of virtual collaborative modeling of aircraft engines.

[0122] In this embodiment, the aircraft engine virtual collaborative modeling system can be divided into multiple functional modules according to the functions it performs, such as Fig.14As shown. The functional modules of the system 1400 may include: an object expression module 1410, a drive configuration module 1420, a gesture recognition module 1430, an operation description module 1440, and a scene description module 1450. The module referred to in the present invention refers to a series of computer program segments that can be executed by at least one processor and can complete fixed functions, which are stored in a memory. In this embodiment, the functions of each module will be described in detail in subsequent embodiments.

[0123] An object expression module, used to decompose the aircraft engine disassembly and assembly process into operation tasks, and express the operation tasks as JSON objects; A driver configuration module is used to express the assembly and disassembly sequence of the JSON object in a universal manner and to construct a task-driven process model based on a dual-sequence directed graph; The gesture recognition module is used to collect hand feature data through the Leap Motion device, identify the gestures corresponding to the hand feature data using the KNN algorithm, and cache the identified gestures into a gesture sequence; An operation description module, used to describe basic behaviors based on gesture sequences using a finite state machine, and to interact the basic behaviors with gesture parameters of the disassembly and assembly objects using a collision box to obtain an operation process of operating the object with one hand; The scenario description module is used to implement the behavior model of the tool and the behavior model of single-handed and multi-handed operation of virtual parts through a finite state machine and space transformation process, and simulate the operation process of the actual maintenance scenario through an interactive mechanism of visual feedback.

[0124] Although the present invention has been described in detail by referring to the accompanying drawings and in combination with the preferred embodiments, the present invention is not limited thereto. Without departing from the spirit and essence of the present invention, a person skilled in the art may make various equivalent modifications or substitutions to the embodiments of the present invention, and these modifications or substitutions shall be within the scope of the present invention. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, and they shall be within the scope of protection of the present invention.

Claims

1. A virtual collaborative modeling method for an aircraft engine, characterized in that: include: Decomposing the aircraft engine disassembly and assembly process into operation tasks, and expressing the operation tasks as JSON objects; The assembly and disassembly sequence of the JSON object is generally expressed, and a task-driven process model based on a double-sequence directed graph is constructed; Collect hand feature data through the Leap Motion device, use the KNN algorithm to identify the gestures corresponding to the hand feature data, and cache the identified gestures into a gesture sequence; A finite state machine is used to describe basic behaviors based on gesture sequences, and a collision box is used to interact the basic behaviors with gesture parameters of the disassembly and assembly objects to obtain the operation process of operating the objects with one hand; The behavioral models of tools and the behavioral models of single-handed and multi-handed operation of virtual parts are realized through finite state machines and space transformation processes, and the operation process of actual maintenance scenarios is simulated through the interactive mechanism of visual feedback.

2. The method according to claim 1, characterized in that: The aircraft engine disassembly and assembly process is decomposed into operation tasks, and the operation tasks are expressed as JSON objects, including: Define a task object, and set basic properties, operation mode, runtime state identifier, sequence properties and operation prompt properties for the task object; the basic properties include: task sequence number, operation object name and operation object peer object relationship; the operation mode includes: free movement behavior and geometric motion behavior with adjustable parameters, wherein the free movement behavior includes single-handed grasping movement or multi-hand collaborative control posture transformation, the properties of the free movement behavior include operation mode, target position and gesture, and the properties of the geometric motion behavior include operation mode, axis vector, tool type, tool axial offset and target value; the runtime state identifier includes: collision body type, trigger addition status and completion status; the sequence properties include: preceding task list and subsequent task list, and the task sequence has a corresponding relationship with the fault type; the operation prompt properties include: task prompt title and specific information; In JSON, the parameter name and parameter value of a single task are combined and expressed as a key-value pair, each operation task is expressed as an object, and the collection of all objects is expressed as a list.

3. The method according to claim 2, characterized in that The assembly and disassembly sequence of the JSON object is generally expressed, and a task-driven process model based on a double-sequence directed graph is constructed, including: Define the total task list, unfinished task list, executable task list and completed task list; Parse the JSON object, generate the predecessor task set and the successor task set based on the basic attributes and sequence attributes of each JSON object, and build a task-driven process model based on a bi-sequence directed graph based on the predecessor task set and the successor task set; In the task-driven process model, the number of tasks that can be executed at the same time is often greater than or equal to 1; The task-driven process model is configured to read the parameterized file through the ReadAllText method encapsulated by the File class provided by C#, parse the JSON parameterized file and deserialize it into the Tasks class through the FromJson method encapsulated by the JsonUtility class provided by Unity3D, and retain it in the memory to complete the task deployment.

4. The method according to claim 3, characterized in that The driving method of the task-driven process model includes: Multi-terminal network requests and parses JSON data files, and initializes the total task set, unfinished task set, executable task set, and completed task set; Add element 0 to the completed task set, use this as a condition to traverse all tasks, activate all start tasks, and add the JSON parameter-specified behavior model encapsulation code to all activated tasks; When any i-th task is completed, the unfinished task set, executable task set, and completed task set are updated; Get the subsequent task set corresponding to the completed i-th task, traverse the subsequent task set, get the predecessor task set of the j-th task in the subsequent task set, and determine whether the j-th task in the subsequent task set can be activated according to the activation condition formula; Activate the activatable tasks, assign corresponding behavior models to encapsulate code components, and provide the correct interaction method; The subsequent tasks of the completed tasks are activated in a loop until the completed task set is the same as the total task set, indicating that all tasks have been completed.

5. The method according to claim 1, characterized in that The hand feature data is collected through the Leap Motion device, the KNN algorithm is used to identify the gestures corresponding to the hand feature data, and the identified gestures are cached in the gesture sequence, including: Collect hand feature data through the Leap Motion device, wherein the hand feature data includes the distance from the fingertips of five fingers to the palm and the fingertip direction vectors of the thumb, index finger and little finger; The hand feature data is saved as a csv format file; The KNN algorithm is used to calculate the Euclidean distance between the hand feature data and the standard feature data of multiple gestures, and the calculation results are sorted in ascending order; the multiple gestures include: grab, pinch, release, forward, backward and interface; The first K gestures are scored, and the gesture with the highest score is the gesture corresponding to the hand feature data; Cache the most recently recognized gesture at the end of the gesture sequence.

6. The method according to claim 5, characterized in that The basic behavior is described based on the gesture sequence using a finite state machine, and the collision box is used to interact the basic behavior with the gesture parameters of the disassembly and assembly object to obtain the operation process of the single-handed operation object, including: The state change process of the basic behaviors of picking up, releasing and transferring objects with one hand is defined as a quaternion by using a finite state machine. Generate a state set of the operation object based on the quadruple of the hand that collides with the operation object; A trigger condition set and a finite jump trigger event set are constructed, and a collision box is used to detect dynamic gestures based on the state set, the trigger condition set and the finite jump trigger event set.

7. The method according to claim 1, characterized in that The behavior model of the tool and the behavior model of single-handed and multi-handed operation of virtual parts are realized through the finite state machine and space transformation process, and the operation process of the actual maintenance scene is simulated through the interactive mechanism of visual feedback, including: Solve the in-position posture according to the tool axial offset in the parameterized file and other built-in offset values ​​of the tool, interpolate the in-position posture frame by frame, and describe the tool usage process through a finite state machine; Capture hand position via external controller or Leap Motion device; The position of the disassembled and assembled objects in the process of multi-hand collaborative handling is solved by the position of the hands; Based on the posture, a finite state machine and the spatial geometric transformation of the disassembled object are used to describe the behavior of the disassembled object.

8. An aircraft engine virtual collaborative modeling system, characterized in that: include: An object expression module, used to decompose the aircraft engine disassembly and assembly process into operation tasks, and express the operation tasks as JSON objects; A driver configuration module is used to express the assembly and disassembly sequence of the JSON object in a universal manner and to construct a task-driven process model based on a dual-sequence directed graph; The gesture recognition module is used to collect hand feature data through the Leap Motion device, identify the gestures corresponding to the hand feature data using the KNN algorithm, and cache the identified gestures into a gesture sequence; An operation description module, used to describe basic behaviors based on gesture sequences using a finite state machine, and to interact the basic behaviors with gesture parameters of the disassembly and assembly objects using a collision box to obtain an operation process of operating the object with one hand; The scenario description module is used to implement the behavior model of the tool and the behavior model of single-handed and multi-handed operation of virtual parts through a finite state machine and space transformation process, and simulate the operation process of the actual maintenance scenario through an interactive mechanism of visual feedback.

9. The system according to claim 8, characterized in that The object expression module is used to: Define a task object, and set basic attributes, operation mode, runtime state identifier, sequence attributes, and operation prompt attributes for the task object; The basic attributes include: task sequence number, operation object name and operation object peer object relationship; the operation mode includes: free movement behavior and geometric motion behavior with adjustable parameters, wherein the free movement behavior includes single-handed grasping movement or multi-hand collaborative control posture transformation, the attributes of the free movement behavior include operation mode, target position and gesture, and the attributes of the geometric motion behavior include operation mode, axis vector, tool type, tool axial offset and target value; the runtime state identifier includes: collision body type, trigger addition state and completion state; the sequence attribute includes: preceding task list and subsequent task list, and the task sequence has a corresponding relationship with the fault type; the operation prompt attribute includes: task prompt title and specific information; In JSON, the parameter name and parameter value of a single task are combined and expressed as a key-value pair, each operation task is expressed as an object, and the collection of all objects is expressed as a list.

10. The system according to claim 9, characterized in that The driver configuration module is used for: Define the total task list, unfinished task list, executable task list and completed task list; Parse the JSON object, generate the predecessor task set and the successor task set based on the basic attributes and sequence attributes of each JSON object, and build a task-driven process model based on a bi-sequence directed graph based on the predecessor task set and the successor task set; In the task-driven process model, the number of tasks that can be executed at the same time is often greater than or equal to 1; The task-driven process model is configured to read the parameterized file through the ReadAllText method encapsulated by the File class provided by C#, parse the JSON parameterized file and deserialize it into the Tasks class through the FromJson method encapsulated by the JsonUtility class provided by Unity3D, and retain it in the memory to complete the task deployment.

Images