Satellite digital intelligent assembling method

Abstract

The invention discloses a satellite digital intelligent assembly method, and belongs to the technical field of digital twin virtual assembly simulation. The problems that a traditional satellite assembly process scheme is long in iteration period and high in cost are solved. Comprising the following steps: establishing a gesture matching relationship between a virtual palm and a real palm, and defining a man-machine interaction gesture; the method comprises the following steps: establishing all satellite component models based on a digital twinning technology, configuring a Transform script component, a Mesh Filter script component and a Mesh Render script component to obtain a satellite component grid rendering model, and constructing a collision body; designing and assembling a control panel and configuring an interactive button; configuring an intelligent auxiliary assembly logic script for the satellite component grid rendering model to prompt an installation position, and constructing an interference detection early warning model to output an early warning interference result; establishing a common assembly tool model to obtain an assembly reachable result; and obtaining the problems existing in the satellite design and assembly process scheme according to the early warning interference result and the assembly reachable result. According to the method, the actual satellite assembling efficiency and quality can be improved.

Description

Technical Field

[0001] The invention relates to a satellite digital intelligent assembly method, and belongs to the technical field of digital twin virtual assembly simulation. Background Art

[0002] The traditional satellite assembly process has a significant bottleneck: it requires that all satellite parts must be designed and produced before assembly can begin. This linear workflow leads to multiple problems. First, if defects or errors are found in the design of the entire satellite structure during the assembly phase, the problem must be fed back to the design phase, and then redesigned and manufactured. This process is not only time-consuming but also inefficient because it involves repeated communication and coordination among multiple departments and stages. In addition, this long-cycle feedback iteration mechanism severely limits the flexibility and responsiveness of the project. In a rapidly changing technological environment and market demand, this lack of agility in the process may cause the project to miss critical windows. In order to improve the efficiency of satellite assembly and the ability to respond to the market, a more flexible and efficient workflow and a virtual-real design verification method are urgently needed to reduce the time delay between design, production and assembly, and allow potential design problems to be discovered and resolved at an early stage.

[0003] At present, the application of digital twin technology in the field of intelligent manufacturing is mainly focused on the optimization of batch production lines. By creating digital copies of physical entities, the production process can be monitored, predicted and optimized. However, for products that require personalization and high customization, such as high-value products such as satellites, the application of digital twin technology is relatively limited. This limitation not only affects the efficiency of product development, but also leads to an extension of the development cycle. The characteristics of the satellite industry are that its project process is long, the investment cost is high, and it has extremely high requirements for the quality and efficiency of the design, manufacturing, and operation and maintenance processes. Although there have been some digitalization works, such as model-based system engineering (MBSE) research, there are still problems such as low degree of digitalization and weak information interaction capabilities between systems throughout the life cycle of satellite engineering.

[0004] At present, digital twin technology is mainly concentrated in the field of computer simulation, using computer computing power to obtain the required data, but it lacks in human-computer interaction and simulation visualization. In the existing application framework, people are not given a dominant position, but digital models are used as the core of simulation. This limits the application scope of digital twin technology to a certain extent, especially in those engineering projects that require direct human participation. How to effectively combine human operators with digital models has become a challenge that needs to be solved urgently. Summary of the invention

[0005] In view of the problems of long iteration cycle and high cost of traditional satellite assembly process solutions, the present invention provides a satellite digital intelligent assembly method.

[0006] A satellite digital intelligent assembly method of the present invention comprises:

[0007] Construct a virtual palm model and establish a gesture matching relationship between the virtual palm and the real palm; define the human-computer interaction gestures of the real palm based on mixed reality technology;

[0008] Based on the digital twin technology, all satellite component models are built and imported into Unity3D. Transform script component, Mesh Filter script component and Mesh Renderer script component are configured for each satellite component model to obtain the satellite component mesh rendering model; and a collision body is constructed for each satellite component mesh rendering model;

[0009] An assembly control panel is designed so that when the real palm drives the virtual palm to move through human-computer interaction gestures, the assembly control panel is correspondingly configured at the corresponding position of the virtual palm; the assembly control panel is configured with an interactive button, and the interactive button recognizes the click behavior of the virtual palm on the interactive button based on the contact state, and triggers the corresponding satellite component mesh rendering model assembly function;

[0010] An intelligent auxiliary assembly logic script is configured for the satellite component mesh rendering model, and the corresponding installation position is prompted through the intelligent auxiliary assembly logic script, and the satellite component mesh rendering models are fixed to each other;

[0011] Construct an interference detection and early warning model to detect collision interference or situations where the distance is less than the preset component distance threshold during the assembly process of the satellite component mesh rendering model, and output early warning interference results;

[0012] A common assembly tool model is established to detect the reachability of the satellite component mesh rendering model in a small space and the reachability of the common assembly tool model to the target position to obtain the assembly reachability result;

[0013] According to the early warning interference results and assembly achievable results, the problems existing in the design and assembly process of the current target satellite are obtained.

[0014] According to the satellite digital intelligent assembly method of the present invention, the virtual palm is positioned and constructed according to 26 joint points, including five joint points set for each finger and one joint point set at the wrist;

[0015] The gesture matching relationship between the virtual palm and the real palm is established based on the OpenPose algorithm;

[0016] Human-computer interaction gestures include finger posture, knuckle contact relationship, and hand orientation;

[0017] The definition of finger gestures includes:

[0018] The flexion of the metacarpophalangeal joint was defined as flexion; the bending of the distal phalanx of the finger was defined as flexion; the opening and closing between two adjacent fingers was defined as abduction;

[0019] Each finger posture is set through Any, Open and Close, where Any is used to set the finger posture that does not affect gesture recognition, Open means setting the degree of extension / opening of the finger, and Close is used to set the degree of bending / closing of the finger;

[0020] The finger joint contact relationship is determined by calculating the distance between the joint points, and the joint points whose distance between the joint points is less than the finger joint distance threshold are determined as the finger joint contact relationship;

[0021] Hand orientations include facing the face, away from the face, palms up, and palms down.

[0022] According to the satellite digital intelligent assembly method of the present invention, the Transform script component is used to determine the position, rotation and scale of the satellite component mesh rendering model in the assembly scene;

[0023] The Mesh Filter script component is used to reference mesh resources;

[0024] The Mesh Renderer script component is used in conjunction with the Mesh Filter script component to render the mesh referenced by the MeshFilter script component;

[0025] Colliders are constructed using mesh assets.

[0026] According to the satellite digital intelligent assembly method of the present invention, after the assembly control panel recognizes the human-computer interaction gesture given by the real palm of the left hand, it generates an assembly control panel on the right side of the corresponding virtual palm and can move with the virtual palm.

[0027] According to the satellite digital intelligent assembly method of the present invention, the human-computer interaction gestures for generating the assembly control panel are set as follows:

[0028] Finger posture settings: flexion and bending of the five fingers are set to Open, and abduction is set to Any;

[0029] The finger joint contact relationship is set to no contact relationship;

[0030] For the tracking axis of the hand orientation, select the palm as the main axis extension, and for the hand orientation, select facing the face.

[0031] According to the satellite digital intelligent assembly method of the present invention, the human-computer interaction gesture of clicking the interactive button is defined as:

[0032] Finger posture settings: The flexion, bending and abduction of the four fingers except the index finger are all set to Close state; the flexion and bending of the index finger are all set to Open state, and the abduction is set to Any;

[0033] The finger joint contact relationship is set to no contact relationship;

[0034] For the tracking axis of the hand orientation, select the palm as the main axis extension, and select the hand orientation away from the face.

[0035] According to the satellite digital intelligent assembly method of the present invention, the design of the assembly control panel includes building a control panel model, writing a panel generation logic script, and designing interactive buttons;

[0036] The control panel model is set to configure several button blocks on the bottom plate;

[0037] The panel generation logic script is used to generate the assembly control panel at the right target position of the corresponding virtual palm after recognizing the human-computer interaction gesture for generating the assembly control panel, positioning based on the thumb metacarpal joint, and making the assembly control panel move with the virtual palm;

[0038] Designing interactive buttons includes adding Box Collider script components to button blocks and constructing button collision bodies;

[0039] The method for realizing the click function of the interactive button is to calculate whether the top bone joint of the index finger is in contact with the interactive button after receiving the human-computer interaction gesture of clicking the interactive button. If so, the interactive button is highlighted and the corresponding assembly function is triggered.

[0040] According to the satellite digital intelligent assembly method of the present invention, each satellite component grid rendering model corresponds to a configuration installation position model;

[0041] The intelligent auxiliary assembly logic script is used to display the installation position model at the corresponding position when the satellite component mesh rendering model is picked up during the assembly process, and to make the installation position model disappear after the installation is completed;

[0042] The method of fixing satellite component mesh rendering models to each other by intelligent auxiliary assembly logic script is as follows:

[0043] Create a calibrated empty object as the sub-object to be assembled, and set the pose coordinates of the calibrated empty object to the pose coordinates of the corresponding satellite component mesh rendering model installation position; when the satellite component mesh rendering model is picked up, calculate the position deviation and pose deviation between the satellite component mesh rendering model and the calibrated empty object. When the position deviation and pose deviation are both less than the set deviation threshold, release the satellite component mesh rendering model to complete the installation of the satellite component mesh rendering model;

[0044] The position deviation calculation formula is:

[0045] Set the position coordinates of the calibration empty object Satellite component mesh rendering model location coordinates Among them, x0 and x1 are the X-axis coordinates in the spatial position system, y0 and y1 are the Y-axis coordinates in the spatial position system, and z0 and z1 are the Z-axis coordinates in the spatial position system. The calculated position deviation d is:

[0046]

[0047] Set the quaternion of the calibration empty object to The attitude quaternion of the satellite component mesh rendering model is Deviation Quaternion

[0048]

[0049]

[0050]

[0051]

[0052] Where q * is the conjugate of q.

[0053] According to the satellite digital intelligent assembly method of the present invention, the interference detection warning model is used to detect the distance state during the movement process of the satellite component grid rendering model installation. When the collision body interference or the distance is less than the preset component distance threshold occurs during the assembly process of the satellite component grid rendering model, the relevant collision body flashes red as a warning interference; the display of the warning interference is realized by triggering the warning condition, and the warning condition is that the collision body interferes or the distance is less than the preset component distance threshold;

[0054] The satellite component mesh rendering model is configured with a rigid body script component, and the rigid body script component enables a real collision effect to be presented when a collision occurs between the satellite component mesh rendering models.

[0055] According to the satellite digital intelligent assembly method of the present invention, the commonly used assembly tool model is used to support the inspection function of the installation tool after the satellite component grid rendering model is installed and fixed; the inspection function includes detecting whether the satellite component grid rendering model reaches the specified fixed position and whether the installation tool interferes with other models after imitating the assembly action, and analyzing the accessibility of the installation tool to the target position and whether the operating space is sufficient.

[0056] The beneficial effects of the present invention are as follows: the method of the present invention carries out assembly process simulation verification based on the digital twin satellite model, which can quickly iterate the assembly process plan before satellite production and manufacturing, thereby improving the efficiency and quality of actual satellite assembly.

[0057] The method of the present invention carries out assembly process simulation verification based on the digital twin satellite model, can interactively operate the three-dimensional digital model of the product, and use the product, tooling tools, virtual palms, etc. to simulate the assembly process of the virtual satellite in the simulation environment. By analyzing the rationality and feasibility of the assembly process, the defects or errors in the whole satellite structure design can be discovered in advance, and the deficiencies in the original process plan can be continuously discovered and timely improved, and finally an optimized product assembly process plan can be obtained. The method of the present invention is used for satellite digital assembly, which can greatly shorten the feedback correction cycle of the satellite assembly process plan and the satellite design, reduce the time and cost of design iteration, and avoid expensive rework and material waste.

[0058] The implementation of the method of the present invention can complete the complete virtual assembly process of the satellite without referring to the drawings, which can greatly improve the workers' familiarity and experience with the assembly process and greatly improve the assembly efficiency and quality of the satellite sample.

[0059] Figure 1 This is a schematic diagram of the joints of the two hands; in the figure, Tip is the fingertip, Distal is the distal end, Intermediate is the middle, Proximal is the proximal end, Palm is the palm, Metacarpal is the metacarpal bone, Wrist is the wrist, Thumb is the thumb, Little is the little finger, Ring is the ring finger, Middle is the middle finger, and Index is the index finger;

[0060] Figure 2 This is a schematic diagram of the matching effect of the virtual and real palm models;

[0061] Figure 3 It is a schematic diagram of finger posture;

[0062] Figure 4 It is a schematic diagram of the thumb and four fingers in a flexed posture;

[0063] Figure 5 It is a schematic diagram of the thumb and four fingers in a bent posture;

[0064] Figure 6 It is a schematic diagram of the thumb and four fingers in the abducted posture;

[0065] Figure 7 This is a diagram of the Okay gesture under the joint contact relationship;

[0066] Figure 8 This is a schematic diagram with four fingers as the main axis when the tracking category is finger;

[0067] Fig. 9 This is a schematic diagram extending with the palm as the main axis when the tracking category is palm;

[0068] Fig.10 It is a schematic diagram extending with the thumb as the main axis when the tracking category is thumb;

[0069] Fig.11 It is a schematic diagram of the hand orientation facing the face;

[0070] Fig.12 It is a diagram of the direction of the hands facing away from the face;

[0071] Fig.13 It is a schematic diagram of the hand orientation with the palm facing upward;

[0072] Fig.14 It is a diagram of the hand orientation with the palm facing downward;

[0073] Fig.15 It is the Mesh preview effect picture;

[0074] Fig.16 This is a comparison chart of the model with and without Mesh Collider added;

[0075] Fig.17 Schematic diagram for visualizing finger posture;

[0076] Fig.18 It is a visual diagram of hand orientation;

[0077] Fig.19 This is a successful diagram of gesture interaction application;

[0078] Fig. 20 It is a visualization diagram of finger postures in the definition of the assembly control panel;

[0079] Fig.21 It is a visualization diagram of the hand orientation in the assembly control panel definition;

[0080] Fig. 22 It is a visualization diagram of the click gesture of the interactive button;

[0081] Fig.23 It is a visualization diagram of the hand orientation for the click gesture of the interactive button;

[0082] Fig.24 It is a schematic diagram of the control panel model;

[0083] Fig.25 It is a schematic diagram of the generated effect of assembling the control panel;

[0084] Fig.26 This is a diagram of the actual effect of running the Box Collider script component;

[0085] Fig. 27 It is a schematic diagram of the satellite component model and the corresponding installation location model;

[0086] Fig.28 It is the effect diagram of the intelligent auxiliary assembly logic script running;

[0087] Fig.29 This is the actual effect diagram of running the rigid body Rigidbody script component;

[0088] Fig.30 It is a schematic diagram of a simple cylindrical tool in a commonly used assembly tool model;

[0089] Fig.31 Schematic diagram of using cylindrical tools to check tool accessibility in a narrow space and spatial redundancy of manual assembly. DETAILED DESCRIPTION

[0090] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0091] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0092] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, but they are not intended to limit the present invention.

[0093] Specific implementation method 1. Combination Figure 1 As shown, the present invention provides a satellite digital intelligent assembly method, comprising:

[0094] Construct a virtual palm model and establish a gesture matching relationship between the virtual palm and the real palm; define the human-computer interaction gestures of the real palm based on mixed reality technology;

[0095] Based on the digital twin technology, all satellite component models are built and imported into Unity3D. Transform script component, Mesh Filter script component and Mesh Renderer script component are configured for each satellite component model to obtain the satellite component mesh rendering model; and a collision body is constructed for each satellite component mesh rendering model;

[0096] An assembly control panel is designed so that when the real palm drives the virtual palm to move through human-computer interaction gestures, the assembly control panel is correspondingly configured at the corresponding position of the virtual palm; the assembly control panel is configured with an interactive button, and the interactive button recognizes the click behavior of the virtual palm on the interactive button based on the contact state, and triggers the corresponding satellite component mesh rendering model assembly function;

[0097] An intelligent auxiliary assembly logic script is configured for the satellite component mesh rendering model, and the corresponding installation position is prompted through the intelligent auxiliary assembly logic script, and the satellite component mesh rendering models are fixed to each other;

[0098] Construct an interference detection and early warning model to detect collision interference or situations where the distance is less than the preset component distance threshold during the assembly process of the satellite component mesh rendering model, and output early warning interference results;

[0099] A common assembly tool model is established to detect the reachability of the satellite component mesh rendering model in a small space and the reachability of the common assembly tool model to the target position to obtain the assembly reachability result;

[0100] According to the early warning interference results and assembly achievable results, the problems existing in the design and assembly process of the current target satellite are obtained.

[0101] In this embodiment, the virtual palm is positioned and constructed according to 26 joint points, including five joint points set for each finger and one joint point set at the wrist;

[0102] The gesture matching relationship between the virtual palm and the real palm is established based on the OpenPose algorithm;

[0103] Human-computer interaction gestures include finger posture, knuckle contact relationship, and hand orientation;

[0104] The definition of finger gestures includes:

[0105] The flexion of the metacarpophalangeal joint was defined as flexion; the bending of the distal phalanx of the finger was defined as flexion; the opening and closing between two adjacent fingers was defined as abduction;

[0106] Each finger posture is set through Any, Open and Close, where Any is used to set the finger posture that does not affect gesture recognition, Open means setting the degree of extension / opening of the finger, and Close is used to set the degree of bending / closing of the finger;

[0107] The finger joint contact relationship is determined by calculating the distance between the joint points, and the joint points whose distance between the joint points is less than the finger joint distance threshold are determined as the finger joint contact relationship;

[0108] Hand orientations include facing the face, away from the face, palms up, and palms down.

[0109] In this implementation, according to the OpenXR standard, a virtual palm model is defined through 26 hand joints, such as Figure 1 As shown in Figure 2, a high-precision three-dimensional model of the palm is constructed based on the positioning of 26 joint points.

[0110] Establish the matching relationship between virtual and real palms based on the OpenPose algorithm:

[0111] OpenPose is an advanced multi-task human key point detection algorithm that can accurately identify and locate 2D and 3D human joints, facial expressions, hand postures and other rich information. OpenPose adopts a fusion method of deep learning and separable graph models. It regards the human body as a graph structure composed of multiple parts, each part represents a joint or feature point, uses convolutional neural network CNN to extract features from the input image, and then predicts the position of each part through graph optimization technology.

[0112] The architecture of OpenPose is divided into two main parts: image flow and graph optimization. First, the image flow module uses a pre-trained convolutional neural network to perform multi-scale analysis on the input image to generate a series of feature maps. These feature maps capture the hand features of different scales and positions. Next, these feature maps predict the location of key points through Heatmap Regression and Part Affinity Fields (PAF). Heatmap Regression is used to directly predict the heat map corresponding to each joint, while PAF depicts the relationship between joints by connecting the probability of adjacent joints to form a hand posture map.

[0113] The core mathematical principles of this algorithm include the following three aspects:

[0114] Keypoint detection: OpenPose processes image data through convolutional neural networks. Assuming the input image is I, it outputs a set of keypoints ({P} = {p_1, p_2, ..., p_n}), where n is the number of keypoints and p_i = (x_i, y_i) represents the pixel coordinates of the i-th keypoint. The output of CNN can be expressed as:

[0115] P = f(I; θ),

[0116] Where f is the function of CNN and θ is the parameter of the grid.

[0117] Pose Estimation: To estimate 3D pose, OpenPose usually combines regression and classification methods. First, a deep learning network is used to predict the depth d of each key point. i , and then calculate the 3D coordinates through camera parameters (such as focal length, optical center, etc.):

[0118] Q i =K·[d i , p ix , p iy ] T ,

[0119] Where K is the camera intrinsic parameter matrix, p ix and p iy are the horizontal and vertical coordinates of the key points on the image.

[0120] Loss function: During training, OpenPose usually uses a multi-task loss function, including key point detection loss L kp , body part segmentation loss L seg and 3D pose loss L 3D wait:

[0121] L=L kp +L seg +L 3D ;

[0122] Based on the OpenPose algorithm, a matching relationship between the virtual palm model and the real palm model in the real space can be established. Figure 2 shown.

[0123] Definition of human-computer interaction gestures based on MR technology:

[0124] MR (mixed reality) technology is a technology that integrates virtual information with the real world. Among them, interacting with virtual models through gesture recognition is an important way of interaction. In MR technology, gestures can be defined by three parts: finger posture Shapes, finger joint contact relationship Bones, and hand orientation Transform:

[0125] Gesture = Shapes + Bones + Transform;

[0126] Step 1: Define finger gesture Shapes:

[0127] First, the finger postures are divided into three types: flexion, curvature, and abduction. Figure 3 shown.

[0128] Finger gestures support three states: Any, Open and Close:

[0129] Any: indicates that the finger can be in any state and will not affect gesture recognition;

[0130] Open: used to set the extension / opening degree of the fingers;

[0131] Close: Used to set the bending / closing degree of the finger.

[0132] Figure 4 , Figure 5 and Figure 6 The diagrams are schematic diagrams showing the states (Open or Close) of the thumb and four fingers in flexion, bending and abduction. Each diagram provides a specific Open and Close range.

[0133] Figure 6 In the middle, abduction is the expansion between the current finger and the next finger starting from the thumb, that is, the expansion between the thumb and the index finger, the index finger and the middle finger, the middle finger and the ring finger, and the ring finger and the little finger. There is no separate abduction setting for the little finger.

[0134] Step 2: Define joint contact relationships Bones:

[0135] It mainly calculates the distance between joint points, and triggers the contact relationship when the distance is less than a certain threshold. Figure 7 As shown, when the distance between the thumb tip (Thumb_tip) and the index finger tip (Index_tip) is less than the set threshold, the contact relationship is triggered, and the "Okay" gesture can be implemented.

[0136] Step 3: Define the hand orientation Transform:

[0137] First, define the tracking axis as the reference axis for gesture tracking;

[0138] When the tracking category is finger, four fingers are used as the main axis to extend, such as Figure 8 As shown; when the tracking category is palm, the palm is extended as the main axis, such as Fig. 9 As shown; when the tracking category is thumb, the thumb is extended as the main axis, such as Fig.10 shown.

[0139] The hand orientation is defined by the relative orientation of the tracking axis and the tracking target: facing the face, such as Fig.11 As shown; away from the face as Fig.12 As shown; palms facing up Fig.13 As shown, palms facing down Fig.14 shown.

[0140] The following is the interactive attribute subsystem design of satellite components:

[0141] The Transform script component is used to determine the position, rotation and scale of the satellite component mesh rendering model in the assembly scene;

[0142] The Mesh Filter script component is used to reference mesh resources;

[0143] The Mesh Renderer script component is used in conjunction with the Mesh Filter script component to render the mesh referenced by the MeshFilter script component;

[0144] Colliders are constructed using mesh assets.

[0145] This implementation method first constructs and renders the digital twin satellite model mesh: first, the digital twin satellite model after interface conversion is imported into unity3d. At this time, the model automatically adds three script components: Transform, MeshFilter and Mesh Renderer.

[0146] The Transform script component determines the position, rotation, and scale of each GameObject in the scene. Each GameObject has a Transform. Table 1 introduces the Transform script properties.

[0147] Table 1 Transform script property description

[0148] property Function Position Transforms the position in x, y, and z coordinates. Rotation Transforms the rotation around the x-, y-, and z-axes, in degrees. Scale The scale of the transformation along the x-, y-, and z-axes. A value of "1" is the original size.

[0149] Mesh Filter is mainly a script component that references the mesh resource Mesh.

[0150] The Mesh Inspector displays information about how the data of a mesh asset is stored, but does not display the value of the data itself. For example, the Inspector shows that Unity stores the Position value of each vertex as a set of (three) Float32 values, but does not display the Position value of a specific vertex. Table 2 provides a detailed introduction to Mesh properties.

[0151] Table 2 Mesh properties detailed description

[0152]

[0153] In the Inspector section, you can preview the appearance of the grid and view the display effect of the grid data, such as Fig.15 Each view displays the following data about the mesh below the mesh preview: the mesh name, the number of vertices in the mesh, the type and number of mesh faces, and the names of any UV maps on the mesh.

[0154] The Mesh Renderer component is used in conjunction with the Mesh Filter component on the same model object; the Mesh Renderer component renders the mesh referenced by the Mesh Filter component. The most important property of the Mesh Renderer component is the Materials property, which lists all the materials used by this component. Table 3 gives a detailed introduction.

[0155] Table 3 Materials property introduction

[0156] property describe Size The number of elements in the material list. Element List of materials. Each element can be assigned a material.

[0157] Construction of collision bodies for digital twin satellite models: The three script components Transform, Mesh Filter, and Mesh Renderer are mainly responsible for rendering the model at the specified pose coordinates. In addition, to achieve the interactivity of the model, it is also necessary to build collision bodies based on Mesh grid resources. The script component involved is Mesh Collider.

[0158] Mesh Collider takes a mesh asset and builds its collider based on that mesh. Mesh Collider is more accurate than primitives that use complex meshes when performing collision detection. Mesh Colliders marked as Convex can collide with other Mesh Colliders. Their properties and functions are described below:

[0159] Convex: Checking this checkbox will make the Mesh Collider collide with other Mesh Colliders. Convex Mesh Colliders can have a maximum of 255 triangles.

[0160] Is Trigger: Checking this box will cause Unity to use this collider to trigger events, while the physics engine will ignore it.

[0161] Cooking: Enable or disable mesh cooking options that affect how the physics engine handles the mesh.

[0162] Options:

[0163] None: Disables all Cooking Options listed below.

[0164] Everything: Enables all the Cooking Options listed below.

[0165] Cook for Faster Simulation: Makes the physics engine cook the mesh for faster simulation. When this setting is enabled, this runs some extra steps to guarantee that the resulting mesh is optimal for runtime performance. This affects the performance of physics queries and contact generation. When this setting is disabled, the physics engine uses a faster cooking speed and generates results as quickly as possible. As a result, the cooked Mesh Collider may not be optimal.

[0166] Enable Mesh Cleaning: Enables the physics engine to clean up the mesh. When this setting is enabled, the cooking process attempts to eliminate degenerate triangles and other geometric artifacts from the mesh. This process produces a mesh that is more suitable for use in collision detection and tends to produce more accurate hit points.

[0167] Weld Colocated Vertices: Causes the physics engine to remove equal vertices in the mesh. When this setting is enabled, the physics engine will merge vertices with the same position. This is important for collision feedback that occurs at runtime.

[0168] Use Fast Midphase: Makes the physics engine use the fastest midphase acceleration structures and algorithms available for your output platform. When this option is enabled, the physics engine will use a faster algorithm that does not require any R-Trees for spatial access. If you encounter midphase issues when running on certain platforms, you can still disable this option to use the slower legacy midphase algorithm instead.

[0169] Material: A reference to a physical material that determines how this collider interacts with other objects.

[0170] Mesh: Reference to the mesh to be used for collision.

[0171] Fig.16 This is a comparison chart of the effects of a model with and without Mesh Collider added.

[0172] Interaction method design:

[0173] After completing the construction of the collision body, the interaction interface between the palm model and the digital twin satellite model is realized. Based on this interaction interface, the interaction method can be designed to realize the interaction of specific gestures with the digital twin satellite model. The main script component used is Object Manipulator.

[0174] The key properties of Object Manipulator are described as follows:

[0175] ObjectManipulator Settings: Mainly used to set the allowed interaction types, such as move, rotate, scale, etc.; and set the type of interactor used, such as handle, hand, line of sight, etc.;

[0176] Manipulation Logic Types: Contains method logic for interaction types such as moving, rotating, and scaling.

[0177] The Object Manipulator script component mainly designs the interactive method of digital twin satellite component assembly based on the human-computer interaction system.

[0178] First, design the finger postures. The Flexion and Curl of the thumb and index finger are set to Close, that is, flexion and bending, and the Abduction is set to Any, that is, closed or open. The postures of the remaining fingers are set to Any, such as Fig.17 Then design the finger joint contact relationship Bones, so that the tip of the index finger contacts the tip of the thumb.

[0179] Finally, design the hand orientation Transform. The Track Axis is defined as Palm, which is extended from the palm as the main axis. The hand orientation is defined as AwayFrom Face, which is away from the face. Fig.18 shown.

[0180] Fig.19 This is a schematic diagram of an AR device using an algorithm to intelligently recognize a grabbing gesture and successfully interact with a satellite virtual component model using the gesture.

[0181] Assembly control panel subsystem design:

[0182] Considering the timeliness of the control panel and the convenience of user operation, an assembly control panel subsystem based on intelligent gesture recognition technology is designed. The system can intelligently perceive the needs of users. When the user makes a preset specific gesture with his left hand, the control panel is automatically generated on the right side of the palm, and can intelligently follow the movement of the palm to ensure that the control panel is always in a convenient position for the user to operate. The design of this intelligent control panel not only improves the convenience of operation, but also improves the naturalness and intuitiveness of user interaction, making the entire assembly process smoother and more efficient.

[0183] After recognizing the human-computer interaction gesture given by the real palm of the left hand, the assembly control panel generates an assembly control panel on the right side of the corresponding virtual palm and can move with the virtual palm.

[0184] Furthermore, the human-computer interaction gestures for generating the assembly control panel are set as follows:

[0185] Finger posture Shapes settings: Flexion and Curl of the five fingers are set to Open, that is, all five fingers are straight; Abduction is set to Any, that is, closed or open is acceptable, such as Fig. 20 As shown;

[0186] The finger joint contact relationship is set to no contact relationship;

[0187] For the Track Axis of the Transform, select the palm as the main axis extension, and select Towards Face as the hand orientation. Fig.21 shown.

[0188] The human-computer interaction gesture for clicking an interactive button is defined as:

[0189] Finger posture settings: The flexion, bending and abduction of the four fingers except the index finger are all set to Close state, that is, they are all closed and bent; the flexion and bending of the index finger are all set to Open state, that is, straight state, and the abduction is set to Any, that is, closed and open are both possible; Fig. 22 shown.

[0190] The finger joint contact relationship is set to no contact relationship;

[0191] For the tracking axis of the hand orientation, select the palm as the main axis extension, and select the hand orientation away from the face, such as Fig.23 shown.

[0192] After designing the control panel gesture, the corresponding hand control panel design is carried out. The design of the assembly control panel includes building the control panel model, writing the panel generation logic script and designing the interactive buttons.

[0193] The control panel model is set to configure several button blocks on the bottom plate; Fig.24 As shown;

[0194] The panel generation logic script is used to generate the assembly control panel at the right target position of the corresponding virtual palm after recognizing the human-computer interaction gesture for generating the assembly control panel, positioning based on the thumb metacarpal joint, and making the assembly control panel move with the virtual palm;

[0195] Designing interactive buttons includes adding Box Collider script components to button blocks and constructing button collision bodies;

[0196] The method for realizing the click function of the interactive button is to calculate whether the top bone joint of the index finger is in contact with the interactive button after receiving the human-computer interaction gesture of clicking the interactive button. If so, the interactive button is highlighted and the corresponding assembly function is triggered.

[0197] Generation logic script writing: The basic algorithm principle is to determine whether the hand control panel gesture is recognized. If so, a control panel is generated at a certain distance to the right of the thumb metacarpal joint Thumb_metacarpal, and the panel posture is updated in real time with the joint. This system only sets the left hand generation logic to avoid the wrong generation of the control panel when the right hand is operating and affecting the line of sight.

[0198] Fig.25 The AR device uses an algorithm to intelligently recognize the control panel gesture, and intelligently generates a control panel on the right side of the palm to run the actual effect diagram of the script;

[0199] Interactive button design: First, you need to add a Box Collider script component to the button model, that is, to build a collision body. The basic algorithm principle for identifying clicks is to first intelligently identify whether it is a button click gesture. If so, calculate whether the index finger's top bone joint Index_tip is in contact with the button. If so, the button is highlighted. In real time, it is determined whether the index finger's top bone joint Index_tip moves a certain distance toward the normal of the button plane. If so, it is determined to be a button click behavior. At the same time, some trigger events can be added for this behavior. The actual effect of running the Box Collider script component is as follows: Fig.26 As shown, the AR device can intelligently recognize the click gesture and successfully click to trigger the corresponding event.

[0200] Going a step further, the satellite parts installation subsystem design is carried out:

[0201] Intelligent assisted assembly design: In the process of assembling parts, it is designed as a drawing-free intelligent virtual assembly system. Therefore, before assembly, workers do not need to refer to drawings. They can clearly see the corresponding assembly positions of each part through the intelligent assisted assembly function. Intelligent assisted assembly design mainly includes the construction of the installation position prompt model and the writing of intelligent assisted assembly logic scripts.

[0202] Each satellite component mesh rendering model corresponds to the configuration installation location model; each component needs to build a corresponding installation location model. To reduce the workload, the components can be directly copied and the material can be changed to transparent green. The effect is as follows Fig. 27 shown.

[0203] The main function of the intelligent auxiliary assembly logic script is to display the corresponding prompt installation location model at the installation location of the component when the operator picks up the component, and the prompt installation location model disappears when the operator puts it down. The basic algorithm principle of the script is to broadcast an event to display the prompt model at the beginning of the picking action, and broadcast an event to hide the prompt display model at the end of the picking action. The actual effect of running the script is as follows Fig.28 As shown in the figure, the AR device intelligently recognizes the action of holding the model and displays the prompt model, and intelligently recognizes the action of putting it down and hides the prompt model.

[0204] The intelligent auxiliary assembly logic script is used to display the installation position model at the corresponding position when the satellite component mesh rendering model is picked up during the assembly process, and to make the installation position model disappear after the installation is completed; after the components are placed in the installation position, the components can be intelligently identified and automatically fixed on the installation plate, and become a whole, moving with the installation plate.

[0205] The method of fixing satellite component mesh rendering models to each other by intelligent auxiliary assembly logic script is as follows:

[0206] Create a calibrated empty object as the sub-object to be assembled, and set the pose coordinates of the calibrated empty object to the pose coordinates of the corresponding satellite component mesh rendering model installation position; when the satellite component mesh rendering model is picked up, calculate the position deviation and pose deviation between the satellite component mesh rendering model and the calibrated empty object. When the position deviation and pose deviation are both less than the set deviation threshold, release the satellite component mesh rendering model to complete the installation of the satellite component mesh rendering model;

[0207] The position deviation calculation formula is:

[0208] Set the position coordinates of the calibration empty object Satellite component mesh rendering model location coordinates Among them, x0 and x1 are the X-axis coordinates in the spatial position system, y0 and y1 are the Y-axis coordinates in the spatial position system, and z0 and z1 are the Z-axis coordinates in the spatial position system. The calculated position deviation d is:

[0209]

[0210] Set the quaternion of the calibration empty object to The attitude quaternion of the satellite component mesh rendering model is Deviation Quaternion

[0211]

[0212]

[0213]

[0214]

[0215] Where q * is the conjugate of q.

[0216] Write a script algorithm based on the above deviation formula, DanJis is the reference of the assembled parts, and Points is the reference of the calibrated empty object.

[0217] The actual effect of running the script is to hold the model, move it to the target position and adjust the posture. After letting go, the AR device intelligently identifies whether the calculated deviation meets the requirements. If so, the parts are automatically fixed to the mounting plate.

[0218] The interference detection warning model is used to detect the distance status during the moving process of the satellite component mesh rendering model installation. When the collision body interference or the distance is less than the preset component distance threshold occurs during the assembly process of the satellite component mesh rendering model, the relevant collision body flashes red as a warning interference; the display of the warning interference is realized by triggering the warning condition, and the warning condition is that the collision body interferes or the distance is less than the preset component distance threshold; the basic algorithm principle is to calculate the closest distance between the handheld component and the surface of the assembled component mesh collision body in real time, and the warning condition is triggered when it is less than a certain value;

[0219] The satellite component mesh rendering model is configured with a rigid body script component, which ensures that when collisions occur between satellite component mesh rendering models, the model will not be penetrated, but a real collision effect will be presented. Considering that no mechanical effect is needed, gravity is turned off, Is Kinematic is turned on, and only the kinematic relationship is considered.

[0220] In the process of virtual assembly, the interference situation of the assembly process can be known through the component interference warning, and the component assembly sequence can be adjusted by analyzing the interference situation, while the assembly path can be optimized. Finally, an optimal assembly sequence and assembly path can be found to obtain the best assembly solution.

[0221] The actual effect of running the Rigidbody script component is as follows Fig.29 As shown, when the AR device intelligently detects that the handheld component is too close to other components or has collided with other components, the relevant component will flash red once.

[0222] Tool interference warning and accessibility analysis design in narrow spaces:

[0223] The commonly used assembly tool model is used to support the inspection function of the installation tool after the satellite component mesh rendering model is installed and fixed; the inspection function includes detecting whether the satellite component mesh rendering model reaches the specified fixed position and whether the installation tool interferes with other models after imitating the assembly action, and analyzing the accessibility of the installation tool to the target position and whether the operating space is sufficient.

[0224] Model common assembly tools and add Transform, Mesh Filter, Mesh Renderer, Mesh Collider, ObjectManipulator, Rigidbody, and Collition Detection script components to them. Take the simple cylinder tool as an example. Fig.30 shown.

[0225] Analysis and design of tool accessibility in narrow spaces: For components in narrow spaces, after the components are automatically fixed on the mounting plate, hold the corresponding installation tool to inspect whether it can reach the specified fixed position, use the tool to imitate various assembly actions, observe whether there is interference with other components, and analyze the accessibility of the tool and whether there is enough space for operation. Fig.31 Schematic diagram of using cylindrical tools to verify tool accessibility in confined spaces and spatial redundancy of manual assembly.

[0226] Although the present invention is described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the present invention. It should therefore be understood that many modifications may be made to the exemplary embodiments and that other arrangements may be devised without departing from the spirit and scope of the present invention as defined by the appended claims. It should be understood that the various dependent claims and features described herein may be combined in a manner different from that described in the original claims. It should also be understood that the features described in conjunction with a single embodiment may be used in other described embodiments.

Claims

1. A satellite digital intelligent assembly method, characterized in that: include, Construct a virtual palm model and establish a gesture matching relationship between the virtual palm and the real palm; define the human-computer interaction gestures of the real palm based on mixed reality technology; Based on the digital twin technology, all satellite component models are built and imported into Unity3D. Transform script component, Mesh Filter script component and Mesh Renderer script component are configured for each satellite component model to obtain the satellite component mesh rendering model; and a collision body is constructed for each satellite component mesh rendering model; Design an assembly control panel so that when the real palm drives the virtual palm to move through human-computer interaction gestures, the assembly control panel is correspondingly configured at the corresponding position of the virtual palm; The assembly control panel is configured with an interactive button, which recognizes the click behavior of the virtual palm on the interactive button based on the contact state, and triggers the corresponding satellite component mesh rendering model assembly function; An intelligent auxiliary assembly logic script is configured for the satellite component mesh rendering model, and the corresponding installation position is prompted through the intelligent auxiliary assembly logic script, and the satellite component mesh rendering models are fixed to each other; Construct an interference detection and early warning model to detect collision interference or situations where the distance is less than the preset component distance threshold during the assembly process of the satellite component mesh rendering model, and output early warning interference results; A common assembly tool model is established to detect the reachability of the satellite component mesh rendering model in a small space and the reachability of the common assembly tool model to the target position to obtain the assembly reachability result; According to the early warning interference results and assembly achievable results, the problems existing in the design and assembly process of the current target satellite are obtained.

2. The satellite digital intelligent assembly method according to claim 1, characterized in that: The virtual palm is positioned and constructed according to 26 joint points, including five joint points set for each finger and one joint point set at the wrist; The gesture matching relationship between the virtual palm and the real palm is established based on the OpenPose algorithm; Human-computer interaction gestures include finger posture, knuckle contact relationship, and hand orientation; The definition of finger gestures includes: The flexion of the metacarpophalangeal joint was defined as flexion; the bending of the distal phalanx of the finger was defined as flexion; the opening and closing between two adjacent fingers was defined as abduction; Each finger posture is set through Any, Open and Close, where Any is used to set the finger posture that does not affect gesture recognition, Open means setting the degree of extension / opening of the finger, and Close is used to set the degree of bending / closing of the finger; The finger joint contact relationship is determined by calculating the distance between the joint points, and the joint points whose distance between the joint points is less than the finger joint distance threshold are determined as the finger joint contact relationship; Hand orientations include facing the face, away from the face, palms up, and palms down.

3. The satellite digital intelligent assembly method according to claim 2, characterized in that: The Transform script component is used to determine the position, rotation and scale of the satellite component mesh rendering model in the assembly scene; The Mesh Filter script component is used to reference mesh resources; The Mesh Renderer script component is used in conjunction with the Mesh Filter script component to render the mesh referenced by the Mesh Filter script component; Colliders are constructed using mesh assets.

4. The satellite digital intelligent assembly method according to claim 3, characterized in that: After recognizing the human-computer interaction gesture given by the real palm of the left hand, the assembly control panel generates an assembly control panel on the right side of the corresponding virtual palm and can move with the virtual palm.

5. The satellite digital intelligent assembly method according to claim 4, characterized in that: Generate the human-computer interaction gesture settings for the assembly control panel: Finger posture settings: flexion and bending of the five fingers are set to Open, and abduction is set to Any; The finger joint contact relationship is set to no contact relationship; For the tracking axis of the hand orientation, select the palm as the main axis extension, and for the hand orientation, select facing the face.

6. The satellite digital intelligent assembly method according to claim 5, characterized in that: The human-computer interaction gesture for clicking an interactive button is defined as: Finger posture settings: The flexion, bending and abduction of the four fingers except the index finger are all set to Close state; the flexion and bending of the index finger are all set to Open state, and the abduction is set to Any; The finger joint contact relationship is set to no contact relationship; For the tracking axis of the hand orientation, select the palm as the main axis extension, and select the hand orientation away from the face.

7. The satellite digital intelligent assembly method according to claim 6, characterized in that: The design of the assembly control panel includes building the control panel model, writing the panel generation logic script and designing the interactive buttons; The control panel model is set to configure several button blocks on the bottom plate; The panel generation logic script is used to generate the assembly control panel at the right target position of the corresponding virtual palm after recognizing the human-computer interaction gesture for generating the assembly control panel, positioning based on the thumb metacarpal joint, and making the assembly control panel move with the virtual palm; Designing interactive buttons includes adding Box Collider script components to button blocks and constructing button collision bodies; The method for realizing the click function of the interactive button is to calculate whether the top bone joint of the index finger is in contact with the interactive button after receiving the human-computer interaction gesture of clicking the interactive button. If so, the interactive button is highlighted and the corresponding assembly function is triggered.

8. The satellite digital intelligent assembly method according to claim 7, characterized in that: Each satellite component mesh rendering model corresponds to the configuration installation location model; The intelligent auxiliary assembly logic script is used to display the installation position model at the corresponding position when the satellite component mesh rendering model is picked up during the assembly process, and to make the installation position model disappear after the installation is completed; The method of fixing satellite component mesh rendering models to each other by intelligent auxiliary assembly logic script is as follows: Create a calibrated empty object as the sub-object to be assembled, and set the pose coordinates of the calibrated empty object to the pose coordinates of the corresponding satellite component mesh rendering model installation position; when the satellite component mesh rendering model is picked up, calculate the position deviation and pose deviation between the satellite component mesh rendering model and the calibrated empty object. When the position deviation and pose deviation are both less than the set deviation threshold, release the satellite component mesh rendering model to complete the installation of the satellite component mesh rendering model; The position deviation calculation formula is: Set the position coordinates of the calibration empty object Satellite component mesh rendering model location coordinates Among them, x0 and x1 are the X-axis coordinates in the spatial position system, y0 and y1 are the Y-axis coordinates in the spatial position system, and z0 and z1 are the Z-axis coordinates in the spatial position system. The calculated position deviation d is: Set the quaternion of the calibration empty object to The attitude quaternion of the satellite component mesh rendering model is Deviation Quaternion Where q * is the conjugate of q.

9. The satellite digital intelligent assembly method according to claim 8, characterized in that: The interference detection warning model is used to detect the distance status during the installation and movement of the satellite component mesh rendering model. When collision body interference occurs during the assembly of the satellite component mesh rendering model or the distance is less than the preset component distance threshold, the relevant collision body flashes red as a warning interference. The display of the early warning interference is realized by triggering the early warning condition, wherein the early warning condition is the interference of the collision body or the distance is less than the preset component distance threshold; The satellite component mesh rendering model is configured with a rigid body script component, and the rigid body script component enables a real collision effect to be presented when a collision occurs between the satellite component mesh rendering models.

10. The satellite digital intelligent assembly method according to claim 9, characterized in that: The commonly used assembly tool model is used to support the inspection function of the installation tool after the satellite component mesh rendering model is installed and fixed; the inspection function includes detecting whether the satellite component mesh rendering model reaches the specified fixed position and whether the installation tool interferes with other models after imitating the assembly action, and analyzing the accessibility of the installation tool to the target position and whether the operating space is sufficient.

Images