Combined hoisting operation virtual simulation training system and working method thereof

By designing a virtual simulation training system for joint lifting operations, using distributed networks and semi-physical simulation modules, a six-degree of freedom joint lifting model is simulated, which solves the training problems of single lifting and joint lifting operations in the existing technology, and achieves low cost and high efficiency The training effect meets the needs of navigation training institutions.

CN119007527APending Publication Date: 2024-11-22COSCO SHIPPING +1

Patent Information

Application Number
CN202411212821.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-30
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

It is difficult for the existing technology to conduct virtual simulation training for single-lift and joint-lift operations at the same time, resulting in unsatisfactory training results, high equipment costs and high hardware requirements, making it difficult to meet the needs of navigation universities and crew training institutions.

Method used

A virtual simulation training system for joint lifting operations was designed, including the teacher side, the student side, the semi-physical simulation module and the virtual simulation device. Through distributed network communication connection, it simulates a six-degree of freedom joint lifting model, obtains operation instructions, and performs security. Detect and simulate data generation, calculate quantitative deviations, perform training corrections, and provide immersive operation experience and low-cost training solutions.

Benefits of technology

It has achieved efficient training for single and joint lifting operations, reduced training costs, improved training efficiency, provided good teaching results, and met the needs of navigation training institutions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119007527A_ABST
    Figure CN119007527A_ABST
Patent Text Reader

Abstract

The invention provides a combined hoisting operation virtual simulation training system and a working method thereof. The system comprises a teacher terminal, a student terminal, a semi-physical simulation module and a virtual simulator. The teacher end is used for setting a training task and recording a training result, the student end is used for executing the training task and recording an operation instruction, and the training task comprises combined hoisting operation; the semi-physical simulation module is used for acquiring an interaction instruction of the student end, converting the interaction instruction into an operation instruction and sending the operation instruction to the teacher end; and the virtual simulator is used for simulating a linkage hoisting operation system to construct a six-degree-of-freedom linkage hoisting model, acquiring the operation instruction, generating simulation data of linkage hoisting operation according to the instruction parameters and safety detection, calculating quantitative deviation of the simulation data according to the training task, and correcting the linkage hoisting operation based on the quantitative deviation. According to the invention, the training efficiency of single-lifting operation and combined-lifting operation training is improved, and a good teaching training effect is achieved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of virtual simulation, and in particular to a virtual simulation training system for a crane operation and a working method thereof. Background Art

[0002] Joint lifting and single lifting operations are important operations on heavy lift vessels. Crane operators and on-site operation commanders must master the core experience such as the working principle, application method and operation precautions of lifting equipment. Therefore, it is necessary to train the heavy lift vessel staff to master the core experience. In the past, due to the complex structure of joint lifting and single lifting operations, the expensive equipment and dangerous operation, most training institutions had a limited number of real machines, and practical courses were mostly conducted through videos or simple models and physical models to understand the operation of real machines, lacking a more intuitive and direct operation experience and lacking a sense of reality.

[0003] In the existing technology, with the rapid development of virtual reality technology, three-dimensional simulation technology and semi-physical simulation control technology, the virtual training system of linked and single-lift operations has received more and more attention. Virtual reality technology can create realistic three-dimensional scenes and provide immersive operation experience. Three-dimensional simulation technology can accurately simulate the operation details of linked and single-lift operations. Semi-physical simulation control technology combines virtual and physical equipment to enable trainees to operate in an environment that is close to the real thing. Compared with traditional physical operation training, virtual reality and three-dimensional simulation technology have the advantages of low hardware requirements, high flexibility and relatively low cost. They do not require special training venues and can greatly reduce training costs and improve training efficiency.

[0004] However, the difference between single-sling and coupled-sling operation methods limits the implementation cost and technology. It is difficult for general maritime colleges or crew training institutions to introduce single-sling and coupled-sling operation equipment on a large scale at the same time, and it is difficult to meet the growing training needs. As a result, it is difficult for the training simulation environment to have training for both coupled and single-sling operations, and it is impossible to achieve ideal teaching and training results.

[0005] Therefore, there is an urgent need for a virtual simulation training system for joint crane operations to fill the gaps in existing virtual simulation training for joint crane operations. Summary of the invention

[0006] In view of the deficiencies in the prior art, the present invention provides a virtual simulation training system for joint crane operation and a working method thereof.

[0007] The first aspect of the present invention discloses a virtual simulation training system for joint crane operation, the system comprising:

[0008] Instructor side, student side, semi-physical simulation module and virtual simulator;

[0009] The instructor terminal is used to set training tasks and record training results, and the trainee terminal is used to execute the training tasks and record operation instructions. The training tasks include joint crane operations;

[0010] The semi-physical simulation module is used to obtain the interactive instructions of the trainee end, convert the interactive instructions into operation instructions and send them to the instructor end;

[0011] The virtual simulator is used to simulate the joint lifting operation system to construct a six-degree-of-freedom joint lifting model, obtain the operation instructions, generate simulation data of the joint lifting operation according to the instruction parameters and safety detection, calculate the quantitative deviation of the simulation data according to the training task, and correct the joint lifting operation based on the quantitative deviation.

[0012] In an optional embodiment, the instructor end is configured as a server, the trainee end is configured as a slave, the server and the slave are connected using a distributed network communication, the semi-physical simulation module is connected to the slave, the virtual simulator is connected to the server, and the server and the slave virtually display the training process of the linkage model through a display device;

[0013] The semi-physical simulation module includes operable physical hardware of the lifting task, collects hardware interaction instructions according to the operation process of the trainee user on the operable physical hardware, and converts the interaction instructions into operation instructions;

[0014] The virtual simulator calculates actual motion parameters of the crane in the training task according to the operation instructions, and calculates target motion parameters of the crane according to the training task, and simulates the target motion trajectory and actual motion trajectory of the crane on the joint crane model based on the actual motion parameters and the target motion parameters, respectively, and compares the motion trajectories to perform auxiliary training correction and record training scores.

[0015] In an optional embodiment, the instructor terminal includes a role management module, a training task configuration module, a training subject management module, a training scoring module, a training scenario generation module, a training environment simulation module, a training operation recording module, a training auxiliary module and a training synchronization module;

[0016] The role management module is used for authority management of instructor roles and trainee roles, and the trainee roles include operators and commanders;

[0017] The training task configuration module is used to set training tasks, training subjects and task parameter configurations. The training tasks include single-lift coordination and joint-lift coordination. The training subjects include ferry lifting, large oil and gas lifting, RTG lifting, wind blade lifting, tower lifting, nacelle lifting, hub lifting, high-speed rail lifting, transformer lifting and long steel pipe lifting. The task parameters include path planning of crane target motion trajectory;

[0018] The training subject management module is used to select, add, modify, delete and save the training tasks;

[0019] The training scoring module is used to set scoring rules, and to perform training scoring on the corresponding trainees for the operation instructions fed back by the semi-physical simulation module based on the scoring rules;

[0020] The training scene generation module is used to match the training task to generate virtual scene parameters, and realize the virtual reality task scene based on the virtual scene parameters, wherein the virtual scene parameters include ship parameters, crane parameters, cargo parameters and cargo loading plan;

[0021] The training environment simulation module is used to simulate weather conditions with different visibility in different training scenarios;

[0022] The training operation record module is used to save and record the trainee's operation data during the training process for the training scoring module to score, and for the training auxiliary module to review the training, the operation data includes operation time, operation instructions, crane path movement process, cargo path movement process, crane collision event, cargo collision event, cargo tilt event, crane hook head height and training completion degree;

[0023] The training synchronization module is used to synchronize the operation process of the trainee end and virtually display it on the display device through the virtual simulator.

[0024] In an optional embodiment, the trainee terminal includes a pre-training inspection module, a safety detection prompt module, a training emergency module, a training auxiliary module and a training display module;

[0025] The pre-training inspection module is used for routine inspection of trainees before starting a simulation task;

[0026] The safety detection reminder module is used for trainees to pay attention to the precautions before starting the simulation operation before the training task;

[0027] The training emergency module is used to temporarily terminate the training task according to an emergency event during the training task;

[0028] The training assistance module is used to assist trainees in correcting the actual operation of training tasks and training review, including path navigation, deviation prompts, collision prompts, command gestures and command guidance functions;

[0029] The training display module is used to virtually display the operation process of the trainee end on a display device through the virtual simulator.

[0030] In an optional embodiment, the virtual simulator performs motion modeling of the linked crane model in six degrees of freedom according to the lifting motion, rotation motion and luffing motion of the first crane and the second crane; based on the crane base as the root node, the hierarchical relationship of the base unit, rotation unit, boom unit and hook head unit are marked as child nodes in sequence, and the structural modeling of the linked crane model in six degrees of freedom is constructed based on the association mechanism of the motion state between the root node and the child nodes.

[0031] In an optional embodiment, the virtual simulator initializes the posture parameters of the first crane and the second crane, obtains the motion control parameters between the root node and the child node and between the child node and the child node in the first crane and / or the second crane, and defines the geometric relationship between the root node and the child node and between the child node and the child node by using joints and connecting rods;

[0032] Based on the joints and the connecting rods, the kinematic solution of the six degrees of freedom is converted into the kinematic solution of the mechanical arm unit in the single degree of freedom, wherein the mechanical arm unit in the single degree of freedom respectively represents the slewing structure, the boom structure and the hook structure of the first crane and the second crane; according to the motion control parameters, variable parameters describing the association relationship between the joints and the connecting rods are generated, and the variable parameters include the joint angle θ i , connecting rod length a i , connecting rod offset d i and the twist angle α i ; Calculate the motion data reflecting the end effector posture information of the robot unit according to the variable parameters; According to the joint angle θ i Confirm the rotation angle of the corresponding joint of the robot arm unit, according to the connecting rod length a i Confirm that the distance between the corresponding joints of the robot arm unit on the same joint axis is based on the link offset d i Confirm the distance between the corresponding joints of the robot unit along the previous joint axis, according to the torsion angle α i Confirm the angle between two adjacent joint axes of the robotic arm unit; construct a homogeneous transformation matrix of the robotic arm unit in a single-degree-of-freedom joint coordinate system according to the variable parameters; sequentially use the homogeneous transformation matrix to transform the joint coordinate system of the robotic arm units corresponding to the base to the end effector, and output the target transformation matrix of the joint suspension model, wherein the target transformation matrix includes the posture information of the end effector and the motion data of the motion process of the robotic arm unit in each degree of freedom;

[0033] The virtual simulator collects position information of cargo and obstacles, and sets a safety distance threshold between the linkage model and the cargo and obstacles according to the posture information of the end effector and the motion data of the robotic arm unit; updates the virtual simulation motion state of the linkage model frame by frame according to the posture information and motion data, calculates simulation data that does not meet the safety distance threshold using spatial position parameter constraint check and / or collision check based on the bounding box, removes simulation data that does not meet the safety distance threshold, and obtains simulation data that passes the safety test.

[0034] In an optional embodiment, the virtual simulator collects target task data to generate a reference path of the linkage model, and generates a simulation path of the linkage model according to the simulation data; uses polynomial fitting to decompose the reference path and the simulation path into sub-paths according to the number of motion frames, and calculates the sub-path coordinates of the reference path and the simulation path respectively; calculates the quantitative error and determination coefficient of the simulation path and the reference path on the sub-path coordinates; evaluates the goodness of fit of the simulation path according to the quantitative error and determination coefficient, corrects the simulation path based on the goodness of fit and returns a control instruction for path correction.

[0035] The second aspect of the present invention discloses a working method of a virtual simulation training system for a joint crane operation, the method comprising:

[0036] Initialize the student side, and the teacher side loads the training task configuration to the student side;

[0037] Trainees perform training tasks for joint crane operations, and convert interactive instructions of the semi-physical simulation module into operation instructions;

[0038] The simulated joint lifting operation system constructs a six-degree-of-freedom joint lifting model, obtains the operation instruction, generates simulation data of the joint lifting operation according to the instruction parameters and safety detection, calculates the quantitative deviation of the simulation data according to the training task, and corrects the joint lifting operation based on the quantitative deviation.

[0039] In an optional embodiment, before initializing the student terminal, the process further includes:

[0040] Before starting training, a monitoring server is established to confirm that the corresponding devices of each role in the system are in the power-on state, the instructor role end is set as the collaborative server host, and the operator role end and the commander role end are set as slaves of the server host. The instructor role end collects slave information through the monitoring server for data synchronization and confirms that the trainee end has entered the server;

[0041] Import and set training task related information on the instructor interface before training, the related information includes trainee information, training task information and training score information;

[0042] The instructor sets the training subjects, training scenarios and training role configurations of the training tasks through the management interface. The training subjects include ferry lifting, large oil and gas lifting, RTG lifting, wind blade lifting, tower lifting, nacelle lifting, hub lifting, high-speed rail lifting, transformer lifting and long steel pipe lifting. The training scenarios include weather conditions with different visibility, wave amplitudes and wind speeds. The training role configuration includes the allocation of roles corresponding to the training tasks to trainees and the semi-physical simulation modules corresponding to the roles.

[0043] In an optional embodiment, the simulated joint lifting operation system constructs a six-degree-of-freedom joint lifting model, obtains the operation instruction, generates simulation data of the joint lifting operation according to the instruction parameters and safety detection, calculates the quantitative deviation of the simulation data according to the training task, and corrects the joint lifting operation based on the quantitative deviation, including:

[0044] According to the lifting motion, slewing motion and luffing motion of the first crane and the second crane, the cranes are subjected to motion modeling of the linkage model in six degrees of freedom, the crane base is used as the root node, the hierarchical relationship of the base unit, the slewing unit, the boom unit and the hook unit are marked as child nodes in sequence, and the structural modeling of the linkage model in six degrees of freedom is constructed based on the association mechanism of the motion state between the root node and the child nodes;

[0045] Initializing the posture parameters of the first crane and the second crane, and obtaining the motion control parameters between the root node and the child node and between the child node and the child node in the first crane and / or the second crane;

[0046] Joints and connecting rods are used to define the geometric relationship between the root node and the child node and between the child node and the child node. Based on the joints and connecting rods, the kinematic solution of the six degrees of freedom is converted into the kinematic solution of the mechanical arm unit in the single degree of freedom. The mechanical arm unit in the single degree of freedom represents the slewing structure, the boom structure and the hook structure of the first crane and the second crane respectively. According to the motion control parameters, variable parameters describing the association relationship between the joints and the connecting rods are generated. The variable parameters include the joint angle θ i , connecting rod length a i , connecting rod offset d i and the twist angle α i , calculating motion data reflecting the position and posture information of the end effector of the robotic arm unit according to the variable parameters;

[0047] According to the joint angle θ i Confirm the rotation angle of the corresponding joint of the robot arm unit, according to the connecting rod length a i Confirm that the distance between the corresponding joints of the robot arm unit on the same joint axis is based on the link offset d iConfirm the distance between the corresponding joints of the robot unit along the previous joint axis, according to the torsion angle α i Determine the angle between two adjacent joint axes of the robotic arm unit, and construct a homogeneous transformation matrix of the robotic arm unit in a single-degree-of-freedom joint coordinate system according to the variable parameters;

[0048] The manipulator units corresponding to the base and the end effector use the homogeneous transformation matrix to transform the joint coordinate system in sequence, and output the target transformation matrix of the joint suspension model, wherein the target transformation matrix includes the posture information of the end effector and the motion data of the motion process of the manipulator unit in each degree of freedom;

[0049] Collecting the position information of the cargo and the obstacle, setting the safety distance threshold between the linkage model and the cargo and the obstacle according to the position information of the end effector and the motion data of the manipulator unit, updating the virtual simulation motion state of the linkage model frame by frame according to the position information and the motion data, calculating the simulation data that does not meet the safety distance threshold by using the spatial position parameter constraint check and / or the collision check based on the bounding box, eliminating the simulation data that does not meet the safety distance threshold, and obtaining the simulation data that passes the safety detection;

[0050] Collect target task data to generate a reference path of the linkage model, generate a simulation path of the linkage model according to the simulation data, use polynomial fitting to decompose the reference path and the simulation path into sub-paths according to the number of motion frames, calculate the sub-path coordinates of the reference path and the simulation path respectively, calculate the quantitative error and determination coefficient between the simulation path and the reference path on the sub-path coordinates; evaluate the goodness of fit of the simulation path according to the quantitative error and the determination coefficient, correct the simulation path based on the goodness of fit and return a control instruction for path correction.

[0051] The third aspect of the present invention discloses a virtual simulation training device for joint crane operation, comprising:

[0052] at least one processor, and

[0053] a memory communicatively connected to the at least one processor; wherein,

[0054] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the working method of the virtual simulation training system for the joint lifting operation as described in any one of the second aspects of the present invention.

[0055] The fourth aspect of the present invention discloses a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the working method of the virtual simulation training system for the joint lifting operation as described in any one of the second aspect of the present invention.

[0056] Compared with the prior art, the present invention has the following advantages:

[0057] (1) The present invention constructs a training system for single crane operation and coupled crane operation through virtual simulation technology, providing crane operation and command operation experience for different roles in different operation scenarios, with good scene immersion and low training cost, thus improving the training efficiency of single crane operation and coupled crane operation training. It is specifically used for the teaching and training of cranes for navigation students and crew members, and achieves good teaching and training effects.

[0058] (2) The present invention provides trainees with a real operating environment through a semi-physical simulation module, which can complete the entire loading and unloading process of single-lifting operations and linked-lifting operations. It has the advantages of low hardware requirements, high flexibility, and relatively low cost. It does not require a special training venue, and can significantly reduce training costs and improve training efficiency.

[0059] (3) The present invention uses a virtual simulator to perform three-dimensional simulation of single-lift operations and linked-lift operations, can visualize the students' operations in a virtual display, can intuitively simulate actual operation scenes, evaluate the students' operations through scoring, and assist in correcting the students' training errors through visualized quantitative deviations, so that single-lift operations and linked-lift operations can achieve ideal teaching and training effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0061] Figure 1 A schematic diagram of a virtual simulation training system for joint crane operation according to the present invention;

[0062] Figure 2 The present invention is a flowchart of the working method of the virtual simulation training system for the joint crane operation. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0064] Obviously, the drawings described below are only some examples or embodiments of the present application. For ordinary technicians in this field, the present application can also be applied to other similar scenarios based on these drawings without creative work. In addition, it can also be understood that although the efforts made in this development process may be complicated and lengthy, for ordinary technicians in this field related to the content disclosed in this application, some changes in design, manufacturing or production based on the technical content disclosed in this application are just conventional technical means, and should not be understood as insufficient content disclosed in this application.

[0065] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0066] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantitative limitation, and may represent the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0067] Example 1

[0068] See also Figure 1 The embodiment of the present invention discloses a virtual simulation training system for joint crane operation, including:

[0069] Instructor side, student side, semi-physical simulation module and virtual simulator;

[0070] The instructor terminal is used to set training tasks and record training results, and the trainee terminal is used to execute the training tasks and record operation instructions. The training tasks include joint crane operations;

[0071] The semi-physical simulation module is used to obtain the interactive instructions of the trainee end, convert the interactive instructions into operation instructions and send them to the instructor end;

[0072] The virtual simulator is used to simulate the joint lifting operation system to construct a six-degree-of-freedom joint lifting model, obtain the operation instructions, generate simulation data of the joint lifting operation according to the instruction parameters and safety detection, calculate the quantitative deviation of the simulation data according to the training task, and correct the joint lifting operation based on the quantitative deviation.

[0073] In an optional embodiment, the instructor end is configured as a server, the trainee end is configured as a slave, the server and the slave are connected using a distributed network communication, the semi-physical simulation module is connected to the slave, the virtual simulator is connected to the server, and the server and the slave virtually display the training process of the linkage model through a display device;

[0074] The semi-physical simulation module includes operable physical hardware of the lifting task, collects hardware interaction instructions according to the operation process of the trainee user on the operable physical hardware, and converts the interaction instructions into operation instructions;

[0075] The virtual simulator calculates actual motion parameters of the crane in the training task according to the operation instructions, and calculates target motion parameters of the crane according to the training task, and simulates the target motion trajectory and actual motion trajectory of the crane on the joint crane model based on the actual motion parameters and the target motion parameters, respectively, and compares the motion trajectories to perform auxiliary training correction and record training scores.

[0076] Specifically, the process of the semi-physical simulation module and the virtual simulator jointly completing the virtual simulation of the joint lifting operation is as follows: select the main crane; select the movement form of the auxiliary crane: rotation or amplitude variation, and lock other control operations; operate the main crane to make the main crane rotate or vary its amplitude; the simulator solves the motion of the double crane mechanism, calculates the rotation angle or amplitude variation range of the auxiliary crane, and drives the auxiliary crane to move according to the calculated motion parameters; if the main crane or the auxiliary crane is performing amplitude variation motion, calculates the span and the height of the top of the boom, and calculates the extension or contraction distance of the sling according to the height of the hook head; according to the position of the main crane and the auxiliary crane (rotation angle, span), the position change of the cargo is calculated through the mechanism motion solution; the simulator performs three-dimensional simulation of the movement of the double crane, the hook head, and the cargo; during the simulation process, calculates the safe distance between the crane, the cargo, and the obstacle, and performs collision checks. After completing the operation of a crane, the user switches the main crane and the auxiliary crane, changes the control mode and the movement mode of the auxiliary crane, and performs the next operation; continuously adjusts and controls the movement of the dual cranes, and finally realizes the simulation of the loading process from the lifting point to the placement point; generates control instructions for the dual cranes, including: the main crane, the main crane operation movement and parameters, the auxiliary crane movement and parameters, the movement parameters of the dual crane slings, and the position and posture of the cargo at key locations.

[0077] Furthermore, the real-time motion solution of the simulator operation to the three-dimensional simulation model is established: the 6-DOF mechanism model of the double-hoisting system is established, the 6-DOF mechanism is transformed into a single-DOF mechanism, the forward motion solution of the mechanism (DH algorithm), collision check (spatial position parameter constraint check, collision check based on bounding box), the output of loading operation instructions and motion data, and three-dimensional motion simulation; after the hoisting path is generated, if the program is opened in training mode, the motion trajectory of the crane, hook head and cargo can be seen. The motion trajectory is displayed in the operation scene in a three-dimensional visualization way, as an auxiliary display for the crane operator and the commander, to realize the hoisting operation training function, and can be used for auxiliary display during the operation process, comparing the theoretical crane path with the actual operation results of the trainees in the virtual operating system, and providing quantitative deviation feedback, correction suggestions and other auxiliary operation suggestions. According to the requirements of the crane operation simulation task, the crane operation process events and event trigger conditions can be associated. It can drive the operation simulation task process and trigger simulation events in the task process.

[0078] Specifically, when performing multi-person collaborative tasks, one machine acts as a server and the other machines act as clients. Each client only communicates with the server and does not directly send information to other clients. The server can limit the client to meet the bandwidth (network speed) requirements for sending and receiving data in order to communicate. This training system consists of a training-end server and a training-end slave. The server and the slave are added to a local area network or a distributed network, in which the hardware of the semi-physical simulation module is connected to the slave, and the server is connected to multiple displays for displaying the operator's screen; during training, the operator inputs operation interactions to the slave through semi-physical hardware operations, and the slave parses and processes the operation input of the semi-physical hardware and converts it into operation instruction information and sends it to the server. The server performs simulation screen rendering and operation calculations according to the received operation instructions, and outputs the screen to the operator's display. Relying on the semi-physical simulation simulator system, a collaborative module with the "Ship Crane Operation Virtual Training System" is developed to import the theoretical lifting path of the training task in the intelligent simulation planning into the auxiliary display device of the virtual simulation training system, and compare the theoretical crane path with the actual operation results of the trainees in the virtual operating system, and give auxiliary operation suggestions such as quantitative deviation feedback and correction suggestions.

[0079] Furthermore, in addition to being able to simulate a single crane operation during the simulation execution, the three roles of crane operator 1, crane operator 2, and on-site commander can also be simulated through cluster distribution. For this purpose, a multi-person collaborative network processing framework is designed to implement the run time support framework (RTI) required for distributed simulation. For example, when the crane is operating (such as crane 1), the operation information of the crane's rotation, boom pitch, and hook head landing is sent to the server. The server performs unified calculations and then sends the information about the crane's (such as crane 1) movement to other clients. In this way, all data information is taken over by the server, which also means that only the instance version running on the server is the correct version, and the information on it is the basis for the client to run the operation scenario.

[0080] It should be noted that the software related to the virtual simulation training system has general functions and instructor and student functions designed specifically for teaching, including generation of task scenarios, hoisting operation demonstrations, operation environment simulation (day, night, cloudy, rainy days), crane beam swing, engine starting / braking, cargo collision, cargo being affected by wind, ship surface fluctuations and other simulated situations, and also supports automatic demonstration of hoisting operations, operation training (path navigation, deviation prompts, collision prompts, crane operation command gestures and command guidance), and hoisting operation records; the system also has semi-physical hardware operations, including connected screen displays, operating levers, switches, buttons, instruments, etc., as well as other operable objects related to hoisting tasks on physical simulation simulators.

[0081] As an example, taking the training operation of realizing three-person collaboration as an example, two semi-physical simulation hardware can be used to perform two-person collaborative operation, and the commander can participate in the command and collaborative training from a third perspective; record the data during the operation, including but not limited to the operation time, operation instructions, path operation status, collision status, operation completion status, cargo tilt, wire rope tilt, hook head height, etc.; by formulating scoring standards, score according to the training status of the crane operator and the commander;

[0082] Furthermore, for the crane simulation in the training process, a data association is established between the operating console of the semi-physical simulation module and the virtual simulator. The student user interactively selects the operating instructions of the main crane and the auxiliary crane in the semi-physical simulation module to complete the control of the two cranes. The system calculates the actual motion state of the crane operated by the student user in the model according to the feedback of the operating instructions through the virtual simulator, and calculates the motion trajectory of the cargo. The motion state of the crane and the motion trajectory of the cargo are displayed virtually throughout the entire operation process, and then compared through the virtual display of the target motion state and the target motion trajectory in the training task, and the student user's operation process is continuously adjusted until the control of the two cranes is completed.

[0083] In an optional embodiment, the instructor terminal includes a role management module, a training task configuration module, a training subject management module, a training scoring module, a training scenario generation module, a training environment simulation module, a training operation recording module, a training auxiliary module and a training synchronization module;

[0084] The role management module is used for authority management of instructor roles and trainee roles, and the trainee roles include operators and commanders;

[0085] The training task configuration module is used to set training tasks, training subjects and task parameter configurations. The training tasks include single-lift coordination and joint-lift coordination. The training subjects include ferry lifting, large oil and gas lifting, RTG lifting, wind blade lifting, tower lifting, nacelle lifting, hub lifting, high-speed rail lifting, transformer lifting and long steel pipe lifting. The task parameters include path planning of crane target motion trajectory;

[0086] The training subject management module is used to select, add, modify, delete and save the training tasks;

[0087] The training scoring module is used to set scoring rules, and to perform training scoring on the corresponding trainees for the operation instructions fed back by the semi-physical simulation module based on the scoring rules;

[0088] The training scene generation module is used to match the training task to generate virtual scene parameters, and realize the virtual reality task scene based on the virtual scene parameters, wherein the virtual scene parameters include ship parameters, crane parameters, cargo parameters and cargo loading plan;

[0089] The training environment simulation module is used to simulate weather conditions with different visibility in different training scenarios;

[0090] The training operation record module is used to save and record the trainee's operation data during the training process for the training scoring module to score, and for the training auxiliary module to review the training, the operation data includes operation time, operation instructions, crane path movement process, cargo path movement process, crane collision event, cargo collision event, cargo tilt event, crane hook head height and training completion degree;

[0091] The training synchronization module is used to synchronize the operation process of the trainee end and virtually display it on the display device through the virtual simulator.

[0092] It should be noted that for instructor users, the system software provides functions such as login, personnel information management (student management, instructor management), hoisting subject management, score setting, hoisting subject operation (subject selection, task setting, role allocation, mode selection, start operation), etc. Instructors can view the camera screen in the simulator, the operator's virtual operation perspective screen and the commander screen through the screen display, and query the results and replay the operation at the same time.

[0093] Furthermore, the login function includes three different account types: administrator, instructor, and student. The administrator's account is a system-built-in account that can manage instructors, students, and subject content. The instructor's account is responsible for managing students, subject content, and scoring student operations. The student account is responsible for recording student operations and scoring data. The instructor actively logs into the training system, and the student is ready to perform training or assessment operations in the simulator. The instructor logs into the system by entering a user name and password, and the student account is generated in the system by the instructor, without the need to log in.

[0094] Furthermore, after logging in, the administrator can manage students and teachers, see the list of students and teachers, query, modify, create, and delete student accounts and information, and set the password required for teachers to log in to the system. For the management of lifting subjects, all subject contents in the current system can be previewed, and operations including searching and deleting subjects can be performed. When creating a subject, the planning end path and scene settings used by the subject will be set, and after creation, they will be used as the selected content when the subject is running.

[0095] Furthermore, for the task scene generation function: according to the task configuration parameters and path data, a virtual task scene is generated, and a virtual simulation operation scene is presented by building a three-dimensional model. The PBR material pipeline is used for production during the scene construction process, and photo-realistic texture sampling and dynamic global illumination system are used for high-definition real-time rendering to restore a realistic and credible virtual terminal operation scene. At the same time, the overall scene is optimized and adjusted to ensure smooth and stable operation during real-time operation and interaction. The on-site climate and environmental conditions are simulated through dynamic lighting and particle systems to achieve the natural environment visual effects of sunny days, heavy rains, and day and night changes.

[0096] Furthermore, with regard to the screen display function: the coach can observe the lifting task process from a bird's eye view or from the perspectives of various roles participating in the task. After the task starts, the coach can simultaneously see the webcam screen, the management software interface and the virtual scene operation screen, where the webcam screen and the virtual scene operation screen specifically include: the camera screen in the crane 1 simulator, the camera screen in the crane 2 simulator, the crane 1 operator's virtual operation perspective screen, the crane 2 operator's virtual operation perspective screen, and the on-site commander's virtual operation perspective screen.

[0097] In an optional embodiment, the trainee terminal includes a pre-training inspection module, a safety detection prompt module, a training emergency module, a training auxiliary module and a training display module;

[0098] The pre-training inspection module is used for routine inspection of trainees before starting a simulation task;

[0099] The safety detection reminder module is used for trainees to pay attention to the precautions before starting the simulation operation before the training task;

[0100] The training emergency module is used to temporarily terminate the training task according to an emergency event during the training task;

[0101] The training assistance module is used to assist trainees in correcting the actual operation of training tasks and training review, including path navigation, deviation prompts, collision prompts, command gestures and command guidance functions;

[0102] The training display module is used to virtually display the operation process of the trainee end on a display device through the virtual simulator.

[0103] It should be noted that before the trainees operate, the crane operation screen and the on-site commander screen will play the operation guide (operation process, precautions, start and stop operations) in a loop. The crane operator (operator) uses a semi-physical hardware platform to operate during the operation. The crane operator can perform single-lifting tasks or joint-lifting tasks. When performing joint-lifting operations, the coordinated operation of operator 1 and operator 2 can be achieved through the distributed network data architecture.

[0104] Furthermore, the on-site commander at the trainee end uses the handle to move around on the virtual deck or dock and observe the operation of the crane in the virtual scene. When the crane operator moves the hook head to a binding position (the hook head is lower than the lifting point, with a distance of less than 0.4 meters), the commander can perform binding and unbinding actions. During a single-lift operation, the commander can work together with the crane operator to rotate the cargo to bring it to the expected angle.

[0105] Furthermore, the software system records the operation process data into the performance database, giving the operation score and time statistics. The recorded results can be queried through the performance query interface, and the operation playback can play back the trainee's operation at that time in the form of video. The playback video content is composed of the operation perspective of each role. Specifically, the crane operator uses the physical camera in the semi-physical simulator to shoot the actual operation screen as the operation record video, and the commander uses the virtual scene screen of the virtual crane and cargo in the virtual scene as the record video.

[0106] In an optional embodiment, the virtual simulator performs motion modeling of the linked crane model in six degrees of freedom according to the lifting motion, rotation motion and luffing motion of the first crane and the second crane; based on the crane base as the root node, the hierarchical relationship of the base unit, rotation unit, boom unit and hook head unit are marked as child nodes in sequence, and the structural modeling of the linked crane model in six degrees of freedom is constructed based on the association mechanism of the motion state between the root node and the child nodes.

[0107] Specifically, when the heavy-lift ship crane is working, it mainly realizes the lifting operation through three actions. The motion processes of the three actions can be simplified in the form of mechanisms, which can include the motion processes of the lifting mechanism, the slewing mechanism and the luffing mechanism. The lifting mechanism can include structures such as a drive motor, a sling, a hook and a pulley block. The lifting mechanism is mainly used to lift and lower the goods in the motion modeling of virtual simulation. The slewing mechanism can include a slewing support structure and a slewing drive structure. The slewing mechanism is mainly used to slew the fixed turntable and slew the turntable to rotate in the motion modeling of virtual simulation. The luffing mechanism may include a single-boom luffing structure and a combined-boom luffing structure. The luffing mechanism is mainly manifested in the motion modeling of virtual simulation as being used to realize the change of the amplitude of the joint lifting operation by rotating the hinge shaft on the fuselage. It is easy to know that the motion process of six degrees of freedom is jointly completed by the main and auxiliary cranes in the joint lifting model. Therefore, it is necessary to simulate the simultaneous motion process of the main crane and the auxiliary crane in the motion modeling. The motion process can be described by the spatial position coordinates and posture information at the previous time point and the spatial coordinates and posture information at the next time point to describe the motion state of the double crane joint lifting operation process in unit time.

[0108] Furthermore, through the structural analysis of the double cranes, based on the mechanism models and joints of the crane base, slewing, boom and hook head, the size, structure and hierarchical relationship of the three-dimensional model are organized according to the structural relationship of the actual motion mechanism. The base unit is taken as the root node of the crane, the slewing unit is taken as its child node, the boom unit and the hook head unit are taken as child nodes of the slewing unit, and their respective positions and angles can be solved respectively, so as to abstractly simplify the structural expression of the linked crane model, and cooperate with the above-mentioned motion modeling to complete the static and dynamic expressions of the linked crane model, providing a model basis for the subsequent virtual simulation of the linked crane operation through operating instructions.

[0109] In an optional embodiment, the virtual simulator initializes the posture parameters of the first crane and the second crane, and obtains the motion control parameters between the root node and the child node and between the child node and the child node in the first crane and / or the second crane;

[0110] Specifically, a single crane can be simplified into the motion process of three robotic arms through the root node and child nodes, where the root node-child node represents the motion state of the robotic arm from the base unit to the slewing unit, where the root node and child node can be regarded as the joints of the robotic arm, and the robotic arm between the joints can be regarded as the connecting rod. Such simplification facilitates the subsequent kinematic solution of the coupled crane model. The command parameters of the crane include control commands for the main crane and the auxiliary crane. The control operations corresponding to the control commands include the slewing control and boom control of the main crane and the auxiliary crane. The kinematic solution of the control commands on the coupled crane model can calculate the crane's slewing angle, boom range, span, boom height, hook head height, sling telescopic length, and cargo position change state, respectively, and then complete the cargo displacement process from the lifting point to the placement point under the coupled crane model, and perform dynamic virtual simulation of the coupled crane operation through the posture parameters and motion control parameters in the command parameters.

[0111] Joints and connecting rods are used to define the geometric relationship between the root node and the child node and between the child node and the child node. Based on the joints and connecting rods, the kinematic solution of the six degrees of freedom is converted into the kinematic solution of the mechanical arm unit in the single degree of freedom. The mechanical arm unit in the single degree of freedom represents the slewing structure, the boom structure and the hook structure of the first crane and the second crane respectively. According to the motion control parameters, variable parameters describing the association relationship between the joints and the connecting rods are generated, and the variable parameters include the joint angle θ i , connecting rod length a i , connecting rod offset d i and the twist angle α i ; Calculate the motion data reflecting the end effector posture information of the robot unit according to the variable parameters; According to the joint angle θ i Confirm the rotation angle of the corresponding joint of the robot arm unit, according to the connecting rod length a i Confirm that the distance between the corresponding joints of the robot arm unit on the same joint axis is based on the link offset d i Confirm the distance between the corresponding joints of the robot unit along the previous joint axis, according to the torsion angle α i Confirm the angle between two adjacent joint axes of the robotic arm unit; construct a homogeneous transformation matrix of the robotic arm unit in a single-degree-of-freedom joint coordinate system according to the variable parameters; sequentially use the homogeneous transformation matrix to transform the joint coordinate system of the robotic arm units corresponding to the base to the end effector, and output the target transformation matrix of the joint suspension model, wherein the target transformation matrix includes the posture information of the end effector and the motion data of the motion process of the robotic arm unit in each degree of freedom;

[0112] Specifically, in this embodiment, the position and angle of the six degrees of freedom mechanism of the joint crane operation can be calculated by using the Denavit-Hartenberg algorithm. Since the DH algorithm is a mathematical method for solving the forward kinematics of a manipulator or robot mechanism, the position and posture of the end effector of the manipulator can be calculated by defining the geometric relationship between the joints and the connecting rods. The slewing mechanism, the boom mechanism and the hook head system in the mechanical structure of the crane can be abstracted as the form of a manipulator, and the relationship between the joints and the connecting rods in each manipulator can be described by four variable parameters. First, the joint axis is coincident with the Z axis in the coordinate system. For the rotary joint, the length direction of the connecting rod can be set to be along the X axis of the coordinate system, and the variable parameter can be set to the joint angle θ. i , connecting rod length a i , connecting rod offset d i and the twist angle α i , where the joint angle is expressed as the rotation angle around the joint axis (the turning angle of the connecting rod moving from the previous position to the next position along the Z-axis direction), usually expressed as a joint variable, the connecting rod length is expressed as the distance between two joints on the same joint axis (the distance the connecting rod is displaced on the Z-axis along the X-axis direction), the connecting rod offset represents the distance between two joints along the previous joint axis (the distance the connecting rod moves from the previous position to the next position along the Z-axis direction), usually expressed as a joint variable, and the torsion angle is expressed as the angle between two adjacent joint axes (the turning angle of the connecting rod moving from the previous joint axis position to the next joint axis position along the X-axis direction). The relative position and direction between the connecting rods can be described by the variable parameters, which can then be used to describe the virtual simulation motion process of the linkage model.

[0113] Furthermore, for each of the above links, variable parameters can be defined as a 4×4 homogeneous matrix. The transformation process of the homogeneous matrix in the joint coordinate system is used to express the motion state of the single-degree-of-freedom manipulator corresponding to the link in the virtual simulation process in the joint-hanging model. The homogeneous matrix of the current link can be expressed as:

[0114]

[0115] Among them, θ i Represents the joint angle of the current link, α i Indicates the torsion angle of the current connecting rod, a i Indicates the connecting rod length of the current connecting rod, d i represents the link offset of the current link, i represents the time node corresponding to the current link motion process (which can be expressed in frames), T iIt represents the current state of the connecting rod. In order to further calculate the posture (position and direction) of the end effector of the robot arm, it is necessary to accumulate the homogeneous transformation matrix of each connecting rod from the base to the end effector in turn to obtain the target transformation matrix T. Since the target transformation matrix is ​​obtained by calculating the connecting rod under six degrees of freedom in the joint lifting model, the target transformation matrix contains the position coordinates and angles of each joint and connecting rod of the joint lifting model in the static modeling process, as well as the motion process of each connecting rod under dynamic modeling. Therefore, the joint lifting model can be used to virtually simulate the process of the double crane joint lifting operation through the motion changes of each connecting rod at different positions and the movement process, which can effectively express the geometric relationship between each joint.

[0116] Furthermore, the program implementation steps for the forward solution of the three-dimensional simulation process of the crane mechanism can be expressed as:

[0117] (1) Initialization: call the UpdateKinematics() method to initialize the joint pose.

[0118] (2) Update kinematics and construct the UpdateKinematics() method; calculate the transformation matrix: call DHMatrix() to calculate T1 (rotating platform), T2 (boom), and T3 (hook); calculate the total transformation matrix T by cumulative multiplication; update joint poses; update the rotation of baseRotation; update the rotation and position of arm; and update the position of hook;

[0119] (3) Calculation of the transformation moment T of the rotating platform: Call DHMatrix(theta1,0,0,0) to calculate the DH transformation matrix T1 of the rotating platform; where theta1 represents the rotation angle of the rotating platform, a=0, d=0, alpha=0, because the platform only rotates without other displacement or tilt.

[0120] (4) Calculation of the boom transformation matrix T2: Call DHMatrix(theta2,0,armLength,0) to calculate the DH transformation matrix T2 of the boom; where theta2 represents the pitch angle of the boom, and armLength represents the length of the boom; d=0, alpha=0, assuming no tilt or other movement.

[0121] (5) Calculation of hook head transformation matrix T3: Call DHMatrix(0,hookLength,0,0) to calculate the DH transformation matrix T3 of the hook head, where hookLength represents the vertical hanging length of the hook head; theta=0, a=0, alpha=0 and the hook head is directly suspended at the end of the boom.

[0122] (5) Calculation of the total transformation matrix T: The total transformation matrix T = T1*T2*T3 is calculated by matrix multiplication to obtain the position of the hook head relative to the base.

[0123] (6) Update Transform: Update the rotation of baseRotation: Convert theta1 to degrees and apply, baseRotation.localRotation = Quaternion.Euler(0,theta1*Mathf.Rad2Deg,0).

[0124] (7) Update the rotation and position of arm: Rotation: Convert theta2 to degrees and apply. Position: Set to armLength on the local z-axis, indicating the extension of the arm. arm.localRotation = Quaternion.Euler(theta2*Mathf.Rad2Deg,0,0); arm.localPosition = new Vector3(0,0,armLength);

[0125] (8) Update the position of the hook: Extract the fourth column of T to update the position of the hook, hook.localPosition = T.GetColumn(3);

[0126] Furthermore, the calculation process of the DH transformation matrix in the forward solution of the three-dimensional simulation of the crane mechanism is specifically as follows:

[0127] (1) Input DH parameters, calculate and return the transformation matrix. Create DHMatrix(float theta, float d, float a, float alpha) method: Input four DH parameters: theta, d, a, alpha. Calculate cosine and sine values: float cosTheta = Mathf.Cos(theta); float sinTheta = Mathf.Sin(theta); float cosAlpha = Mathf.Cos(alpha); float sinAlpha = Mathf.Sin(alpha);

[0128] (2) Create and set the 4x4 transformation matrix: Matrix4x4 matrix = new Matrix4x4(); matrix.SetColumn(0, new Vector4(cosTheta, sinTheta, 0, 0)); matrix.SetColumn(1, new Vector4(-sinTheta*cosAlpha, cosTheta*cosAlpha, sinAlpha, 0)); matrix.SetColumn(2, new Vector4(sinTheta*sinAlpha, -cosTheta*sinAlpha, cosAlpha, 0)); matrix.SetColumn(3, new Vector4(a*cosTheta, a*sinTheta, d, 1)); Return the calculated transformation matrix.

[0129] (3) Update each frame and call UpdateKinematics() to update the joint status in each frame, so that the joint crane model is always in a visual state that dynamically displays the movement of the joint crane operation.

[0130] The virtual simulator collects position information of cargo and obstacles, and sets a safety distance threshold between the linkage model and the cargo and obstacles according to the posture information of the end effector and the motion data of the robotic arm unit; updates the virtual simulation motion state of the linkage model frame by frame according to the posture information and motion data, calculates simulation data that does not meet the safety distance threshold using spatial position parameter constraint check and / or collision check based on the bounding box, removes simulation data that does not meet the safety distance threshold, and obtains simulation data that passes the safety test.

[0131] Specifically, in order to speed up the detection speed and improve the efficiency, the geometric model of the crane can be reasonably and effectively simplified. In this embodiment, a joint crane model is used to represent the geometric relationship of the two cranes. The spatial position and posture of the crane at any time are solved by forward kinematics through the obtained instruction parameters, and are represented by a homogeneous matrix in the establishment and transformation of the three-dimensional coordinate system. Based on the position information of the reference cargo and obstacles, the spatial position relationship between the crane and the cargo and the obstacle is converted into the distance between spatial points, lines and surfaces through distance calculation for solution. At the same time, the shortest distance between the two cranes is calculated, and the influence of its dynamic changes (such as moving speed, rotation angle, etc.) on the distance is considered. According to the size, A predefined safety distance threshold is set based on factors such as working scope, safety regulations, and location information of cargo and obstacles. The calculated shortest distance is compared with the safety distance threshold through comparison and judgment. If the shortest distance is less than or equal to the safety distance threshold, it is judged that a collision may occur; otherwise, it is considered that no collision will occur in the current state, where the shortest distance includes the shortest distance between cranes and the shortest distance between cranes and cargo and obstacles. According to the above-mentioned spatial position parameter constraint check, the control steps with collision risks in the crane virtual simulation process corresponding to the control parameters can be identified, the simulation data corresponding to the control steps are deleted, and the simulation data without collision risks are retained to complete the simulation process of the crane joint operation.

[0132] Furthermore, the collision detection of the crane can also be performed by a bounding box. In this embodiment, an oriented bounding box (OBB) is taken as an example. The bounding box of the crane and the bounding boxes of the cargo and obstacles are respectively established according to the position information of the cargo and obstacles and the position information and motion data of the crane. The construction of the bounding box should be based on the geometric shape and size of the object to ensure that the object can be tightly and accurately surrounded. First, a preliminary screening is performed to use a relatively simple bounding box for preliminary collision detection. If the bounding boxes of the two objects do not intersect, the two objects are unlikely to collide, thereby avoiding more complex collision detection. If the bounding boxes of the two objects intersect after the preliminary screening, a more accurate collision detection is required, which involves direct calculation and analysis of the geometric shape of the object itself. The accurate data reflected by the joint crane model can support the direct calculation and analysis of the bounding box. Taking the oriented bounding box (OBB) as an example, the separating axis theorem (SAT) and other methods can be used to further determine whether the two objects intersect, thereby determining the possible collision risk in the virtual simulation process of the joint crane operation, and then improving the virtual simulation process of the joint crane operation.

[0133] Furthermore, the specific implementation of the crane collision check may include:

[0134] Collision check between crane boom and obstacles: crane boom and obstacles are represented by bounding boxes respectively; during the crane movement, the positions of the bounding boxes of the boom and obstacles are updated in real time; a collision detection algorithm is used to determine whether the boom bounding box intersects with the obstacle bounding box; if so, further precise collision detection is performed to determine the specific location and extent of the collision.

[0135] The collision check between the hoisted object and obstacles is performed, and a bounding box is also constructed for the hoisted object. During the hoisting process, the position and posture of the hoisted object are updated in real time, and its bounding box is also updated. A similar collision detection algorithm is used to determine whether the bounding box of the hoisted object intersects with the bounding box of the obstacle.

[0136] Collision check inside the crane: For the mechanical structures inside the crane (such as wire ropes, pulleys, etc.), collision check can also be performed using bounding boxes. By constructing bounding boxes for these components and updating their positions and postures in real time, internal collisions can be predicted and avoided.

[0137] In an optional embodiment, the virtual simulator collects target task data to generate a reference path of the linkage model, and generates a simulation path of the linkage model according to the simulation data; uses polynomial fitting to decompose the reference path and the simulation path into sub-paths according to the number of motion frames, and calculates the sub-path coordinates of the reference path and the simulation path respectively; calculates the quantitative error and determination coefficient of the simulation path and the reference path on the sub-path coordinates; evaluates the goodness of fit of the simulation path according to the quantitative error and determination coefficient, corrects the simulation path based on the goodness of fit and returns a control instruction for path correction.

[0138] Specifically, by comparing the recorded instruction parameter data with the loaded planning task standard data, the accuracy and deviation of the path execution are suggested, so as to achieve the effect of simulation correction, which is realized by data fitting. Data fitting is a mathematical method used to find a function or model that describes the trend and distribution of a set of data points as accurately as possible. It is widely used in scientific research, engineering, economics and other fields to analyze and predict data.

[0139] Furthermore, the crane mechanism movement determines that the actual movement data is not a linear movement. Therefore, in this embodiment, a polynomial fitting method is used to perform data fitting calculation, and the path is decomposed into a straight line or a polynomial curve. At the same time, after the fitting calculation, the fitting result is adjusted and corrected by introducing an error analysis method, wherein the polynomial fitting can be expressed as:

[0140] y=a n x n +a n-1 x n-1 +…+a 1x+a 0

[0141] The calculation process of error analysis includes:

[0142] The mean square error of path A and path B is calculated to quantify the degree of deviation between them. The calculation process of the mean square error can be expressed as:

[0143]

[0144] in, represents the coordinates of the i-th point of path A, Represents the coordinates of the i-th point on path B;

[0145] The maximum error between path A and path B is calculated to identify the maximum deviation point. The calculation process of the maximum error is expressed as:

[0146]

[0147] The coefficient of determination was calculated to assess the goodness of fit. The calculation process of the coefficient of determination is expressed as:

[0148]

[0149] in, Represents the average value of path A.

[0150] Furthermore, the program implementation steps of the forward solution of the three-dimensional simulation correction process of the crane mechanism can be expressed as:

[0151] 1. Initialize and input path data pathA and pathB; store the data of known paths and actual operation paths respectively through two lists pathA and pathB. These lists are composed of Vector3 objects, each of which represents a point on the path;

[0152] public List <vector3>pathA; / / known path A;

[0153] public List <vector3>pathB; / / Actual operation path B;

[0154] Determine the minimum number of points pointCount, and determine the number of points as follows: Make sure that the two paths have the same length for comparison, and use the Mathf.Min function to get the smaller length of the two lists, int pointCount = Mathf.Min (pathA.Count, pathB.Count);

[0155] 2. Error calculation, calculate the mean square error (MSE), declare an error accumulator parameter mse: set mse to 0, used to accumulate the square error of each point; float mse = 0f; traverse the path data: use a for loop to traverse each point, calculate the square error between path A and path B and accumulate;

[0156]

[0157] Calculate the average error: divide the accumulated error sum by the number of points to get the mean square error, mse / =pointCount;

[0158] Debug.Log("Mean Squared Error(MSE):"+mse);

[0159] Calculate the maximum error: Initialize the maximum error variable: set maxError to 0 to record the maximum error; float maxError = 0f; traverse the path data: use a for loop to calculate the error of each point and update maxError.

[0160]

[0161]

[0162] 3. Fitting and analysis

[0163] Extract the coordinate points of path B and store the x and y coordinates of path B into arrays xValues ​​and yValues;

[0164]

[0165] Perform polynomial fitting and use the Fit.Polynomial method of the MathNet.Numerics library to fit the path B data and obtain the polynomial coefficients.

[0166] var polynomial = Fit.Polynomial(xValues, yValues, 2); / / 2nd order polynomial fitting.

[0167] Calculate the mean of path A: Calculate the average of all y coordinates in path A;

[0168]

[0169] Initialize the sum variables: ssTot and ssRes are used to store the total sum of squares and the residual sum of squares respectively;

[0170] double ssTot = 0;

[0171] double ssRes = 0;

[0172] Calculate the fitted value, residual sum of squares and total sum of squares for each point through a loop;

[0173]

[0174] Calculate R2 and calculate the coefficient of determination;

[0175] double rSquared=1-(ssRes / ssTot);

[0176] Debug.Log("R^2:"+rSquared);

[0177] End the program.

[0178] The present invention constructs a training system for single-lift operation and coupled-lift operation training through virtual simulation technology, provides crane operation and command operation experience for different roles in different operation scenarios, has good scene immersion, low training cost, improves the training efficiency of single-lift operation and coupled-lift operation training, and is specially used for the teaching and training of cranes for navigation students and crew members, achieving good teaching and training effects; the semi-physical simulation module provides a real operating environment for trainees, can complete the entire loading and unloading process of single-lift operation and coupled-lift operation, has the advantages of low hardware requirements, high flexibility, and relatively low cost, does not require a special training venue, can greatly reduce training costs, and improve training efficiency; single-lift operation and coupled-lift operation are simulated in three dimensions through a virtual simulator, can visualize the trainee's operation in a virtual display, can intuitively simulate actual operation scenarios, evaluate the trainee's operation through scoring, and assist in correcting the trainee's training errors through visualized quantitative deviations, so that single-lift operation and coupled-lift operation achieve ideal teaching and training effects.

[0179] Example 2

[0180] like Figure 2 As shown, the second aspect of the present invention discloses a working method of a virtual simulation training system for joint crane operation:

[0181] 101. Initialize the student side, and the instructor side loads the training task configuration to the student side;

[0182] 102. Trainees perform training tasks for joint crane operations and convert interactive instructions of the semi-physical simulation module into operation instructions;

[0183] Specifically, the crane operator uses a semi-physical hardware platform to operate during the operation. After operation, the operator's perspective simulates the real cockpit perspective. At the same time, the commander's screen will be synchronized to the operator's operating interface so that the commander's gesture status can be viewed.

[0184] The specific operation process is as follows:

[0185] Power on the system equipment and check whether all screens are lit (Note: there can be no signal, as long as they are lit);

[0186] Turn on the management computer and start the management program;

[0187] Start the computers of crane 1, crane 2 and the command terminal. After startup, the simulation program will automatically run and connect to the management terminal.

[0188] The management end checks whether the command end, crane 1, and crane 2 are connected. After the connection is completed, configure the management end training parameters or import training data. After completion, click the "Start" button and the entire system enters the training scenario.

[0189] The crane operator enters the crane cabin and presses Figure 1 Power request signal button (Power requests ignal).

[0190] After the power request signal button is pressed, the "Ready for start" indicator light enters the flashing state, flashes 3 times (on for 1 second and off for 1 second) and then enters the steady state.

[0191] After the Ready for start indicator light turns on, press the Crane ON button, and the Ready for start indicator light turns off automatically. You can then start the crane hoisting operation.

[0192] During the hoisting process, under the premise of NMF crane, when the load span selection knob (Load steps with) on the right operating console is switched, the crane will adjust to the corresponding span and load:

[0193] Gear 1 means 38t, 4-28m;

[0194] Gear 2 means 50t, 4-28m;

[0195] Gear 3 refers to 70t, 4-24m;

[0196] Gear 4 means 120t, 4-14m;

[0197] Gear position 5 refers to automatic gear, 4-28m;

[0198] During the hoisting process, after selecting the tonnage gear, when the hoisting weight reaches 95% of the allowable hoisting weight tonnage, the yellow "Warning Overload" light on the left console lights up, and the crane operates normally; when the hoisting weight exceeds 105% of the allowable hoisting weight tonnage, the red "Overload" light on the left console lights up, and the crane overload alarms. At this time, the hook cannot be raised, the boom has no function, but the hook can be lowered.

[0199] According to the training configuration of the cargo lifting process, the two operators use the left operating console ( Figure 1 ) marked on the lever controls the boom pitch, slewing mechanism rotation, right console ( Figure 2 ) is used to control the extension and retraction of the rope to carry out lifting operations on the cargo.

[0200] Press the "Crane OFF" button to disable crane rope extension, boom luffing and slewing mechanism rotation. To operate the crane again, repeat steps 5-7.

[0201] When the lifting task is completed, close the system process:

[0202] Close the management program and the host.

[0203] Close the administrator program and the host.

[0204] Shut down the two crane control hosts.

[0205] Remove all power from the system.

[0206] Furthermore, the commander operates in the virtual training task: after the training task is started, the commander will be generated in the virtual task scene, and the commander has two different working planes to correspond to different working positions in the lifting process, specifically 1. The dock working plane. 2. The working plane on the ship. The working plane is switched by the handle buttons A and B.

[0207] The commander can control the commander's perspective in the scene through a separate handle. The left handle joystick moves forward, backward, left and right, and the right handle joystick moves the perspective roaming.

[0208] When the operator moves the hook head to the binding position (the hook head is lower than the lifting point and the distance is less than 0.4 meters), the commander presses the Y button on the right handle to perform binding and unbinding actions.

[0209] When the task is a single-lift operation, the rotation of the cargo during the lifting process is performed by the commander with the assistance of the crane operator. The handle is rotated to the desired angle position.

[0210] When the cargo is moved to the designated lifting position, a commander will confirm and end the task.

[0211] Furthermore, the coach's operation in the virtual lifting task: When the virtual lifting task is started, the coach can observe the lifting task process from a bird's eye view or from the perspective of each role participating in the task. Specifically including: 1. Crane 1 simulator operation perspective; 2. Crane 2 simulator operation perspective; 3. Commander operation perspective; 4. God's eye view. The God's eye view is a completely free three-dimensional control perspective. You can observe and roam anywhere in the field through the keyboard W / S / A / D keys and the mouse. The coach can override and end the virtual training task in the event of a major physical task. If training is to be conducted again, the coach can select a new subject in the subject management page, configure the environment and roles, and then start the training again. If the training is all over, the coach clicks to exit the system and close the software.

[0212] 103. Simulate the joint lifting operation system to construct a six-degree-of-freedom joint lifting model, obtain the operation instruction, generate simulation data of the joint lifting operation according to the instruction parameters and safety detection, calculate the quantitative deviation of the simulation data according to the training task, and correct the joint lifting operation based on the quantitative deviation.

[0213] In an optional embodiment, before initializing the student terminal, the process further includes:

[0214] Before starting training, a monitoring server is established to confirm that the corresponding devices of each role in the system are in the power-on state, the instructor role end is set as the collaborative server host, and the operator role end and the commander role end are set as slaves of the server host. The instructor role end collects slave information through the monitoring server for data synchronization and confirms that the trainee end has entered the server;

[0215] Import and set training task related information on the instructor interface before training, the related information includes trainee information, training task information and training score information;

[0216] The instructor sets the training subjects, training scenarios and training role configurations of the training tasks through the management interface. The training subjects include ferry lifting, large oil and gas lifting, RTG lifting, wind blade lifting, tower lifting, nacelle lifting, hub lifting, high-speed rail lifting, transformer lifting and long steel pipe lifting. The training scenarios include weather conditions with different visibility, wave amplitudes and wind speeds. The training role configuration includes the allocation of roles corresponding to the training tasks to trainees and the semi-physical simulation modules corresponding to the roles.

[0217] Specifically, the coach starts the training task through the management page, the system starts network synchronization, and the coach PC starts to distribute the relevant task configuration information of this training. The coach PC, crane simulator and commander PC dynamically load the scene resources according to the task configuration distributed by the coach PC. At the same time, the coach PC monitors the scene loading and creation process in real time. When starting the training task, there are two situations: 1. Single crane operation task. When the training task started is a single crane operation task, the system will start the training task in the way of 1 crane simulator and 1 on-site commander assigned. The coach PC, the crane simulator assigned the task, and the on-site commander will join the training scene for collaborative operation, and the crane simulator that is not assigned will join the training scene as an observer. 2. Joint crane operation task. When the training task started is a joint crane operation task, the system will start the training task in the way of 2 crane simulators and 1 on-site commander assigned. The coach PC, the 2 crane simulators assigned the task, and the on-site commander will join the training scene for collaborative operation.

[0218] In an optional embodiment, the simulated joint lifting operation system constructs a six-degree-of-freedom joint lifting model, obtains the operation instruction, generates simulation data of the joint lifting operation according to the instruction parameters and safety detection, calculates the quantitative deviation of the simulation data according to the training task, and corrects the joint lifting operation based on the quantitative deviation, including:

[0219] According to the lifting motion, slewing motion and luffing motion of the first crane and the second crane, the cranes are subjected to motion modeling of the linkage model in six degrees of freedom, the crane base is used as the root node, the hierarchical relationship of the base unit, the slewing unit, the boom unit and the hook unit are marked as child nodes in sequence, and the structural modeling of the linkage model in six degrees of freedom is constructed based on the association mechanism of the motion state between the root node and the child nodes;

[0220] Specifically, when the heavy-lift ship crane is working, it mainly realizes the lifting operation through three actions. The motion processes of the three actions can be simplified in the form of mechanisms, which can include the motion processes of the lifting mechanism, the slewing mechanism and the luffing mechanism. The lifting mechanism can include structures such as a drive motor, a sling, a hook and a pulley block. The lifting mechanism is mainly used to lift and lower the goods in the motion modeling of virtual simulation. The slewing mechanism can include a slewing support structure and a slewing drive structure. The slewing mechanism is mainly used to slew the fixed turntable and slew the turntable to rotate in the motion modeling of virtual simulation. The luffing mechanism may include a single-boom luffing structure and a combined-boom luffing structure. The luffing mechanism is mainly manifested in the motion modeling of virtual simulation as being used to realize the change of the amplitude of the joint lifting operation by rotating the hinge shaft on the fuselage. It is easy to know that the motion process of six degrees of freedom is jointly completed by the main and auxiliary cranes in the joint lifting model. Therefore, it is necessary to simulate the simultaneous motion process of the main crane and the auxiliary crane in the motion modeling. The motion process can be described by the spatial position coordinates and posture information at the previous time point and the spatial coordinates and posture information at the next time point to describe the motion state of the double crane joint lifting operation process in unit time.

[0221] Furthermore, through the structural analysis of the double cranes, based on the mechanism models and joints of the crane base, slewing, boom and hook head, the size, structure and hierarchical relationship of the three-dimensional model are organized according to the structural relationship of the actual motion mechanism. The base unit is taken as the root node of the crane, the slewing unit is taken as its child node, the boom unit and the hook head unit are taken as child nodes of the slewing unit, and their respective positions and angles can be solved respectively, so as to abstractly simplify the structural expression of the linked crane model, and cooperate with the above-mentioned motion modeling to complete the static and dynamic expressions of the linked crane model, providing a model basis for the subsequent virtual simulation of the linked crane operation through operating instructions.

[0222] Initializing the posture parameters of the first crane and the second crane, and obtaining the motion control parameters between the root node and the child node and between the child node and the child node in the first crane and / or the second crane;

[0223] Specifically, a single crane can be simplified into the motion process of three robotic arms through the root node and child nodes, where the root node-child node represents the motion state of the robotic arm from the base unit to the slewing unit, where the root node and child node can be regarded as the joints of the robotic arm, and the robotic arm between the joints can be regarded as the connecting rod. Such simplification facilitates the subsequent kinematic solution of the coupled crane model. The command parameters of the crane include control commands for the main crane and the auxiliary crane. The control operations corresponding to the control commands include the slewing control and boom control of the main crane and the auxiliary crane. The kinematic solution of the control commands on the coupled crane model can calculate the crane's slewing angle, boom range, span, boom height, hook head height, sling telescopic length, and cargo position change state, respectively, and then complete the cargo displacement process from the lifting point to the placement point under the coupled crane model, and perform dynamic virtual simulation of the coupled crane operation through the posture parameters and motion control parameters in the command parameters.

[0224] Joints and connecting rods are used to define the geometric relationship between the root node and the child node and between the child node and the child node. Based on the joints and connecting rods, the kinematic solution of the six degrees of freedom is converted into the kinematic solution of the mechanical arm unit in the single degree of freedom. The mechanical arm unit in the single degree of freedom represents the slewing structure, the boom structure and the hook structure of the first crane and the second crane respectively. According to the motion control parameters, variable parameters describing the association relationship between the joints and the connecting rods are generated. The variable parameters include the joint angle θ i , connecting rod length a i , connecting rod offset d i and the twist angle α i , calculate the motion data reflecting the position information of the end effector of the robot unit according to the variable parameters; according to the joint angle θ i Confirm the rotation angle of the corresponding joint of the robot arm unit, according to the connecting rod length a i Confirm that the distance between the corresponding joints of the robot arm unit on the same joint axis is based on the link offset d i Confirm the distance between the corresponding joints of the robot unit along the previous joint axis, according to the torsion angle α i Confirm the angle between two adjacent joint axes of the robotic arm unit, and construct a homogeneous transformation matrix of the robotic arm unit in a single-degree-of-freedom joint coordinate system according to the variable parameters; sequentially use the homogeneous transformation matrix to transform the joint coordinate system of the robotic arm units corresponding to the base to the end effector, and output the target transformation matrix of the joint suspension model, wherein the target transformation matrix includes the posture information of the end effector and the motion data of the motion process of the robotic arm unit in each degree of freedom;

[0225] Specifically, in this embodiment, the position and angle of the six degrees of freedom mechanism of the joint crane operation can be calculated by using the Denavit-Hartenberg algorithm. Since the DH algorithm is a mathematical method for solving the forward kinematics of a manipulator or robot mechanism, the position and posture of the end effector of the manipulator can be calculated by defining the geometric relationship between the joints and the connecting rods. The slewing mechanism, the boom mechanism and the hook head system in the mechanical structure of the crane can be abstracted as the form of a manipulator, and the relationship between the joints and the connecting rods in each manipulator can be described by four variable parameters. First, the joint axis is coincident with the Z axis in the coordinate system. For the rotary joint, the length direction of the connecting rod can be set to be along the X axis of the coordinate system, and the variable parameter can be set to the joint angle θ. i , connecting rod length a i , connecting rod offset d i and the twist angle α i , where the joint angle is expressed as the rotation angle around the joint axis (the turning angle of the connecting rod moving from the previous position to the next position along the Z-axis direction), usually expressed as a joint variable, the connecting rod length is expressed as the distance between two joints on the same joint axis (the distance the connecting rod is displaced on the Z-axis along the X-axis direction), the connecting rod offset represents the distance between two joints along the previous joint axis (the distance the connecting rod moves from the previous position to the next position along the Z-axis direction), usually expressed as a joint variable, and the torsion angle is expressed as the angle between two adjacent joint axes (the turning angle of the connecting rod moving from the previous joint axis position to the next joint axis position along the X-axis direction). The relative position and direction between the connecting rods can be described by the variable parameters, which can then be used to describe the virtual simulation motion process of the linkage model.

[0226] Furthermore, for each of the above links, variable parameters can be defined as a 4×4 homogeneous matrix. The transformation process of the homogeneous matrix in the joint coordinate system is used to express the motion state of the single-degree-of-freedom manipulator corresponding to the link in the virtual simulation process in the joint-hanging model. The homogeneous matrix of the current link can be expressed as:

[0227]

[0228] Among them, θ i Represents the joint angle of the current link, α i Indicates the torsion angle of the current connecting rod, a i Indicates the connecting rod length of the current connecting rod, d i represents the link offset of the current link, i represents the time node corresponding to the current link motion process (which can be expressed in frames), T i It represents the current state of the connecting rod. In order to further calculate the posture (position and direction) of the end effector of the robot arm, it is necessary to accumulate the homogeneous transformation matrix of each connecting rod from the base to the end effector in turn to obtain the target transformation matrix T. Since the target transformation matrix is ​​obtained by calculating the connecting rod under six degrees of freedom in the joint lifting model, the target transformation matrix contains the position coordinates and angles of each joint and connecting rod of the joint lifting model in the static modeling process, as well as the motion process of each connecting rod under dynamic modeling. Therefore, the joint lifting model can be used to virtually simulate the process of the double crane joint lifting operation through the motion changes of each connecting rod at different positions and the movement process, which can effectively express the geometric relationship between each joint.

[0229] Collecting the position information of the cargo and the obstacle, setting the safety distance threshold between the linkage model and the cargo and the obstacle according to the position information of the end effector and the motion data of the manipulator unit, updating the virtual simulation motion state of the linkage model frame by frame according to the position information and the motion data, calculating the simulation data that does not meet the safety distance threshold by using the spatial position parameter constraint check and / or the collision check based on the bounding box, eliminating the simulation data that does not meet the safety distance threshold, and obtaining the simulation data that passes the safety detection;

[0230] Specifically, in order to speed up the detection speed and improve the efficiency, the geometric model of the crane can be reasonably and effectively simplified. In this embodiment, a joint crane model is used to represent the geometric relationship of the two cranes. The spatial position and posture of the crane at any time are solved by forward kinematics through the obtained instruction parameters, and are represented by a homogeneous matrix in the establishment and transformation of the three-dimensional coordinate system. Based on the position information of the reference cargo and obstacles, the spatial position relationship between the crane and the cargo and the obstacle is converted into the distance between spatial points, lines and surfaces through distance calculation for solution. At the same time, the shortest distance between the two cranes is calculated, and the influence of its dynamic changes (such as moving speed, rotation angle, etc.) on the distance is considered. According to the size, A predefined safety distance threshold is set based on factors such as working scope, safety regulations, and location information of cargo and obstacles. The calculated shortest distance is compared with the safety distance threshold through comparison and judgment. If the shortest distance is less than or equal to the safety distance threshold, it is judged that a collision may occur; otherwise, it is considered that no collision will occur in the current state, where the shortest distance includes the shortest distance between cranes and the shortest distance between cranes and cargo and obstacles. According to the above-mentioned spatial position parameter constraint check, the control steps with collision risks in the crane virtual simulation process corresponding to the control parameters can be identified, the simulation data corresponding to the control steps are deleted, and the simulation data without collision risks are retained to complete the simulation process of the crane joint operation.

[0231] Furthermore, the collision detection of the crane can also be performed by a bounding box. In this embodiment, an oriented bounding box (OBB) is taken as an example. The bounding box of the crane and the bounding boxes of the cargo and obstacles are respectively established according to the position information of the cargo and obstacles and the position information and motion data of the crane. The construction of the bounding box should be based on the geometric shape and size of the object to ensure that the object can be tightly and accurately surrounded. First, a preliminary screening is performed to use a relatively simple bounding box for preliminary collision detection. If the bounding boxes of the two objects do not intersect, the two objects are unlikely to collide, thereby avoiding more complex collision detection. If the bounding boxes of the two objects intersect after the preliminary screening, a more accurate collision detection is required, which involves direct calculation and analysis of the geometric shape of the object itself. The accurate data reflected by the joint crane model can support the direct calculation and analysis of the bounding box. Taking the oriented bounding box (OBB) as an example, the separating axis theorem (SAT) and other methods can be used to further determine whether the two objects intersect, thereby determining the possible collision risk in the virtual simulation process of the joint crane operation, and then improving the virtual simulation process of the joint crane operation.

[0232] Furthermore, the specific implementation of the crane collision check may include:

[0233] Collision check between crane boom and obstacles: crane boom and obstacles are represented by bounding boxes respectively; during the crane movement, the positions of the bounding boxes of the boom and obstacles are updated in real time; a collision detection algorithm is used to determine whether the boom bounding box intersects with the obstacle bounding box; if so, further precise collision detection is performed to determine the specific location and extent of the collision.

[0234] The collision check between the hoisted object and obstacles is performed, and a bounding box is also constructed for the hoisted object. During the hoisting process, the position and posture of the hoisted object are updated in real time, and its bounding box is also updated. A similar collision detection algorithm is used to determine whether the bounding box of the hoisted object intersects with the bounding box of the obstacle.

[0235] Collision check inside the crane: For the mechanical structures inside the crane (such as wire ropes, pulleys, etc.), collision check can also be performed using bounding boxes. By constructing bounding boxes for these components and updating their positions and postures in real time, internal collisions can be predicted and avoided.

[0236] Collect target task data to generate a reference path of the linkage model, generate a simulation path of the linkage model according to the simulation data, use polynomial fitting to decompose the reference path and the simulation path into sub-paths according to the number of motion frames, calculate the sub-path coordinates of the reference path and the simulation path respectively, calculate the quantitative error and determination coefficient between the simulation path and the reference path on the sub-path coordinates; evaluate the goodness of fit of the simulation path according to the quantitative error and the determination coefficient, correct the simulation path based on the goodness of fit and return a control instruction for path correction.

[0237] Specifically, by comparing the recorded instruction parameter data with the loaded planning task standard data, the accuracy and deviation of the path execution are suggested, so as to achieve the effect of simulation correction, which is realized by data fitting. Data fitting is a mathematical method used to find a function or model that describes the trend and distribution of a set of data points as accurately as possible. It is widely used in scientific research, engineering, economics and other fields to analyze and predict data.

[0238] Furthermore, the crane mechanism movement determines that the actual movement data is not a linear movement. Therefore, in this embodiment, a polynomial fitting method is used to perform data fitting calculation, and the path is decomposed into a straight line or a polynomial curve. At the same time, after the fitting calculation, the fitting result is adjusted and corrected by introducing an error analysis method, wherein the polynomial fitting can be expressed as:

[0239] y=a n x n +a n-1 x n-1 +…+a 1 x+a 0

[0240] The calculation process of error analysis includes:

[0241] The mean square error of path A and path B is calculated to quantify the degree of deviation between them. The calculation process of the mean square error can be expressed as:

[0242]

[0243] in, represents the coordinates of the i-th point of path A, Represents the coordinates of the i-th point on path B;

[0244] The maximum error between path A and path B is calculated to identify the maximum deviation point. The calculation process of the maximum error is expressed as:

[0245]

[0246] The coefficient of determination was calculated to assess the goodness of fit. The calculation process of the coefficient of determination is expressed as:

[0247]

[0248] in, Represents the average value of path A.

[0249] Example 3

[0250] The third aspect of the present invention discloses a virtual simulation training device for joint crane operation, comprising:

[0251] at least one processor, and

[0252] a memory communicatively connected to the at least one processor; wherein,

[0253] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the working method of the virtual simulation training system for the joint lifting operation as described in any one of the second aspects of the present invention.

[0254] The computer device may be a terminal, and the computer device includes a processor, a memory, a network interface, a display screen, and an input device connected through a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the working method of the virtual simulation training system for the joint crane operation is realized. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covering the display screen, or a key, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.

[0255] Example 4

[0256] The fourth aspect of the present invention discloses a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the working method of the virtual simulation training system for the joint lifting operation as described in any one of the second aspect of the present invention.

[0257] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing related hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the working methods of the virtual simulation training system for the joint lifting operation as described above. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0258] Alternatively, if the above modules of the present invention are implemented in the form of software function modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product that contributes to the relevant technology. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, terminal, or network device, etc.) to execute all or part of the methods of each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, RAM, ROM, magnetic disks or optical disks.

[0259] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A virtual simulation training system for joint crane operation, characterized in that: The system comprises: a teacher terminal, a student terminal, a semi-physical simulation module and a virtual simulator; The instructor terminal is used to set training tasks and record training results, and the trainee terminal is used to execute the training tasks and record operation instructions. The training tasks include joint crane operations; The semi-physical simulation module is used to obtain the interactive instructions of the trainee end, convert the interactive instructions into operation instructions and send them to the instructor end; The virtual simulator is used to simulate the joint lifting operation system to construct a six-degree-of-freedom joint lifting model, obtain the operation instructions, generate simulation data of the joint lifting operation according to the instruction parameters and safety detection, calculate the quantitative deviation of the simulation data according to the training task, and correct the joint lifting operation based on the quantitative deviation.

2. The virtual simulation training system for joint crane operation according to claim 1 is characterized in that: The instructor end is configured as a server, the trainee end is configured as a slave, the server and the slave are connected by a distributed network communication, the semi-physical simulation module is connected to the slave, the virtual simulator is connected to the server, and the server and the slave virtually display the training process of the linkage model through a display device; The semi-physical simulation module includes operable physical hardware of the lifting task, collects hardware interaction instructions according to the operation process of the trainee user on the operable physical hardware, and converts the interaction instructions into operation instructions; The virtual simulator calculates actual motion parameters of the crane in the training task according to the operation instructions, and calculates target motion parameters of the crane according to the training task, and simulates the target motion trajectory and actual motion trajectory of the crane on the joint crane model based on the actual motion parameters and the target motion parameters, respectively, and compares the motion trajectories to perform auxiliary training correction and record training scores.

3. The virtual simulation training system for joint crane operation according to claim 1 is characterized in that: The instructor terminal includes a role management module, a training task configuration module, a training subject management module, a training scoring module, a training scenario generation module, a training environment simulation module, a training operation recording module, a training auxiliary module and a training synchronization module; The role management module is used for authority management of instructor roles and trainee roles, and the trainee roles include operators and commanders; The training task configuration module is used to set training tasks, training subjects and task parameter configurations. The training tasks include single-lift coordination and joint-lift coordination. The training subjects include ferry lifting, large oil and gas lifting, RTG lifting, wind blade lifting, tower lifting, nacelle lifting, hub lifting, high-speed rail lifting, transformer lifting and long steel pipe lifting. The task parameters include path planning of crane target motion trajectory; The training subject management module is used to select, add, modify, delete and save the training tasks; The training scoring module is used to set scoring rules, and to perform training scoring on the corresponding trainees for the operation instructions fed back by the semi-physical simulation module based on the scoring rules; The training scene generation module is used to match the training task to generate virtual scene parameters, and realize the virtual reality task scene based on the virtual scene parameters, wherein the virtual scene parameters include ship parameters, crane parameters, cargo parameters and cargo loading plan; The training environment simulation module is used to simulate weather conditions with different visibility in different training scenarios; The training operation record module is used to save and record the trainee's operation data during the training process for the training scoring module to score, and for the training auxiliary module to review the training, the operation data includes operation time, operation instructions, crane path movement process, cargo path movement process, crane collision event, cargo collision event, cargo tilt event, crane hook head height and training completion degree; The training synchronization module is used to synchronize the operation process of the trainee end and virtually display it on the display device through the virtual simulator.

4. The virtual simulation training system for joint crane operation according to claim 1 is characterized in that: The trainee terminal includes a pre-training inspection module, a safety detection prompt module, a training emergency module, a training auxiliary module and a training display module; The pre-training inspection module is used for routine inspection of trainees before starting a simulation task; The safety detection reminder module is used for trainees to pay attention to the precautions before starting the simulation operation before the training task; The training emergency module is used to temporarily terminate the training task according to an emergency event during the training task; The training assistance module is used to assist trainees in correcting the actual operation of training tasks and training review, including path navigation, deviation prompts, collision prompts, command gestures and command guidance functions; The training display module is used to virtually display the operation process of the trainee end on a display device through the virtual simulator.

5. The virtual simulation training system for joint crane operation according to claim 1 is characterized in that: The virtual simulator performs motion modeling of the linked crane model in six degrees of freedom according to the lifting motion, rotation motion and luffing motion of the first crane and the second crane; based on the crane base as the root node, the hierarchical relationship of the base unit, the rotation unit, the boom unit and the hook head unit are marked as child nodes in sequence, and the structural modeling of the linked crane model in six degrees of freedom is constructed based on the association mechanism of the motion state between the root node and the child nodes.

6. The virtual simulation training system for joint crane operation according to claim 1 is characterized in that: The virtual simulator initializes the posture parameters of the first crane and the second crane, obtains the motion control parameters between the root node and the child node and the child node in the first crane and / or the second crane, and defines the geometric relationship between the root node and the child node and the child node by using joints and connecting rods; based on the joints and connecting rods, the kinematic solution of the six-degree-of-freedom is converted into the kinematic solution of the mechanical arm unit in the single degree of freedom, and the mechanical arm unit in the single degree of freedom represents the slewing structure, the boom structure and the hook structure of the first crane and the second crane respectively; according to the motion control parameters, variable parameters describing the association relationship between the joints and the connecting rods are generated, and the variable parameters include the joint angle θ i , connecting rod length a i , connecting rod offset d i and the twist angle α i ; Calculate the motion data reflecting the end effector posture information of the robot unit according to the variable parameters; According to the joint angle θ i Confirm the rotation angle of the corresponding joint of the robot arm unit, according to the connecting rod length a i Confirm that the distance between the corresponding joints of the robot arm unit on the same joint axis is based on the link offset d i Confirm the distance between the corresponding joints of the robot unit along the previous joint axis, according to the torsion angle α i Determine the angle between two adjacent joint axes of the robotic arm unit; Constructing a homogeneous transformation matrix of the manipulator unit in a single-degree-of-freedom joint coordinate system according to the variable parameters; The manipulator units corresponding to the base and the end effector use the homogeneous transformation matrix to transform the joint coordinate system in sequence, and output the target transformation matrix of the joint suspension model, wherein the target transformation matrix includes the posture information of the end effector and the motion data of the motion process of the manipulator unit in each degree of freedom; The virtual simulator collects position information of cargo and obstacles, and sets a safety distance threshold between the linkage model and the cargo and obstacles according to the posture information of the end effector and the motion data of the robotic arm unit; updates the virtual simulation motion state of the linkage model frame by frame according to the posture information and motion data, calculates simulation data that does not meet the safety distance threshold using spatial position parameter constraint check and / or collision check based on the bounding box, removes simulation data that does not meet the safety distance threshold, and obtains simulation data that passes the safety test.

7. The virtual simulation training system for joint crane operation according to claim 1 is characterized in that: The virtual simulator collects target task data to generate a reference path of the linkage model, and generates a simulation path of the linkage model according to the simulation data; uses polynomial fitting to decompose the reference path and the simulation path into sub-paths according to the number of motion frames, and calculates the sub-path coordinates of the reference path and the simulation path respectively; Calculating the quantitative error and the coefficient of determination between the simulation path and the reference path on the sub-path coordinates; The goodness of fit of the simulation path is evaluated according to the quantitative error and the determination coefficient, the simulation path is corrected based on the goodness of fit, and a control instruction for path correction is returned.

8. A working method of a virtual simulation training system for joint crane operation, characterized in that: The method comprises: Initialize the student side, and the teacher side loads the training task configuration to the student side; Trainees perform training tasks for joint crane operations, and convert interactive instructions of the semi-physical simulation module into operation instructions; The simulated joint lifting operation system constructs a six-degree-of-freedom joint lifting model, obtains the operation instruction, generates simulation data of the joint lifting operation according to the instruction parameters and safety detection, calculates the quantitative deviation of the simulation data according to the training task, and corrects the joint lifting operation based on the quantitative deviation.

9. The working method of the virtual simulation training system for joint crane operation according to claim 8 is characterized in that: Before the student terminal is initialized, the method further includes: Before starting training, a monitoring server is established to confirm that the corresponding devices of each role in the system are in the power-on state, the instructor role end is set as the collaborative server host, and the operator role end and the commander role end are set as slaves of the server host. The instructor role end collects slave information through the monitoring server for data synchronization and confirms that the trainee end has entered the server; Import and set training task related information on the instructor interface before training, the related information includes trainee information, training task information and training score information; The instructor sets the training subjects, training scenarios and training role configurations of the training tasks through the management interface. The training subjects include ferry lifting, large oil and gas lifting, RTG lifting, wind blade lifting, tower lifting, nacelle lifting, hub lifting, high-speed rail lifting, transformer lifting and long steel pipe lifting. The training scenarios include weather conditions with different visibility, wave amplitudes and wind speeds. The training role configuration includes the allocation of roles corresponding to the training tasks to trainees and the semi-physical simulation modules corresponding to the roles.

10. The working method of the virtual simulation training system for joint crane operation according to claim 9 is characterized in that: The simulated joint lifting operation system constructs a six-degree-of-freedom joint lifting model, obtains the operation instruction, generates simulation data of the joint lifting operation according to the instruction parameters and safety detection, calculates the quantitative deviation of the simulation data according to the training task, and corrects the joint lifting operation based on the quantitative deviation, including: According to the lifting motion, slewing motion and luffing motion of the first crane and the second crane, the cranes are subjected to motion modeling of the linkage model in six degrees of freedom, the crane base is used as the root node, the hierarchical relationship of the base unit, the slewing unit, the boom unit and the hook unit are marked as child nodes in sequence, and the structural modeling of the linkage model in six degrees of freedom is constructed based on the association mechanism of the motion state between the root node and the child nodes; Initializing the posture parameters of the first crane and the second crane, and obtaining the motion control parameters between the root node and the child node and between the child node and the child node in the first crane and / or the second crane; Joints and connecting rods are used to define the geometric relationship between the root node and the child node and between the child node and the child node. Based on the joints and connecting rods, the kinematic solution of the six degrees of freedom is converted into the kinematic solution of the mechanical arm unit in the single degree of freedom. The mechanical arm unit in the single degree of freedom represents the slewing structure, the boom structure and the hook structure of the first crane and the second crane respectively. According to the motion control parameters, variable parameters describing the association relationship between the joints and the connecting rods are generated. The variable parameters include the joint angle θ i , connecting rod length a i , connecting rod offset d i and the twist angle α i , calculating motion data reflecting the position and posture information of the end effector of the robotic arm unit according to the variable parameters; According to the joint angle θ i Confirm the rotation angle of the corresponding joint of the robot arm unit, according to the connecting rod length a i Confirm that the distance between the corresponding joints of the robot arm unit on the same joint axis is based on the link offset d i Confirm the distance between the corresponding joints of the robot unit along the previous joint axis, according to the torsion angle α i Determine the angle between two adjacent joint axes of the robotic arm unit, and construct a homogeneous transformation matrix of the robotic arm unit in a single-degree-of-freedom joint coordinate system according to the variable parameters; The manipulator units corresponding to the base and the end effector use the homogeneous transformation matrix to transform the joint coordinate system in sequence, and output the target transformation matrix of the joint suspension model, wherein the target transformation matrix includes the posture information of the end effector and the motion data of the motion process of the manipulator unit in each degree of freedom; Collecting the position information of the cargo and the obstacle, setting the safety distance threshold between the linkage model and the cargo and the obstacle according to the position information of the end effector and the motion data of the manipulator unit, updating the virtual simulation motion state of the linkage model frame by frame according to the position information and the motion data, calculating the simulation data that does not meet the safety distance threshold by using the spatial position parameter constraint check and / or the collision check based on the bounding box, eliminating the simulation data that does not meet the safety distance threshold, and obtaining the simulation data that passes the safety detection; Collect target task data to generate a reference path of the linkage model, generate a simulation path of the linkage model according to the simulation data, use polynomial fitting to decompose the reference path and the simulation path into sub-paths according to the number of motion frames, calculate the sub-path coordinates of the reference path and the simulation path respectively, calculate the quantitative error and determination coefficient of the simulation path and the reference path on the sub-path coordinates, evaluate the goodness of fit of the simulation path according to the quantitative error and determination coefficient, correct the simulation path based on the goodness of fit and return a control instruction for path correction.

Citation Information

Patent Citations

  • Double mobile crane cooperative hoisting simulation method based on kinematics and dynamics

    CN103064297A

  • Hoisting simulation method based on crane vehicle-mounted control system

    CN106325100A

  • Virtual training system of ship deck crane and operating method of virtual training system

    CN107025819A

  • Double-crane hoisting operation simulation modeling method

    CN110032817A

  • Hoisting simulation construction method and system based on BIM

    CN112069698A

Cited By

  • Multi-tower crane cooperative driving training system based on virtual reality technology

    CN119992915A

  • Scene reproduction and augmented reality simulation training method for floating installation

    CN120047050A

  • Mining directional drilling track simulation learning system

    CN121563737A

  • BIM (Building Information Modeling)-based ethylene device hoisting operation control system and method

    CN122021336A

  • Pumped storage power station semi-physical simulation operation and maintenance training system, method and medium

    CN122176985A