Satellite digital intelligent assembly method

Abstract

A method for digital intelligent assembly of satellites belongs to the field of digital twin virtual assembly simulation technology. The present invention addresses the problems of long iteration cycle and high cost of traditional satellite assembly process solutions. It includes establishing a gesture matching relationship between a virtual palm and a real palm, defining human-computer interaction gestures; building all satellite component models based on digital twin technology, configuring Transform script components, Mesh Filter script components and Mesh Renderer script components, obtaining satellite component mesh rendering models, and constructing collision bodies; designing an assembly control panel to configure interactive buttons; configuring an intelligent auxiliary assembly logic script for the satellite component mesh rendering model to prompt the installation position, constructing an interference detection warning model to output warning interference results; establishing a commonly used assembly tool model to obtain assembly reachable results; and obtaining problems existing in satellite design and assembly process solutions based on the warning interference results and assembly reachable results. The present invention can improve the efficiency and quality of actual satellite assembly.

Description

Technical Field

[0001] The present invention relates to a satellite digital intelligent assembly method, and belongs to the technical field of digital twin virtual assembly simulation. Background Art

[0002] The traditional satellite assembly process suffers from a significant bottleneck: it requires that all satellite components be designed and produced before assembly can begin. This linear workflow leads to multiple problems. First, if flaws or errors in the satellite's structural design are discovered during the assembly phase, the issues must be fed back to the design phase, requiring redesign and manufacturing. This process is not only time-consuming but also inefficient, as it involves repeated communication and coordination across multiple departments and stages. Furthermore, this long feedback loop severely limits project flexibility and responsiveness. In a rapidly changing technological environment and market demands, this lack of agility can cause projects to miss critical windows. To improve satellite assembly efficiency and market responsiveness, a more flexible and efficient workflow and a virtual-physical design verification approach are urgently needed to reduce time delays between design, production, and assembly and allow potential design issues to be identified and resolved at an early stage.

[0003] Currently, the application of digital twin technology in intelligent manufacturing is primarily focused on optimizing batch production lines. By creating digital replicas of physical entities, production processes can be monitored, predicted, and optimized. However, for products that require personalization and high customization, such as high-value products like satellites, the application of digital twin technology is relatively limited. This limitation not only affects product development efficiency but also leads to extended development cycles. The satellite industry is characterized by long project processes, high investment costs, and extremely high requirements for the quality and efficiency of design, manufacturing, and operation and maintenance processes. Although some digitalization work has been carried out, such as research on model-based systems engineering (MBSE), problems such as a low degree of digitalization and weak information exchange capabilities between systems still exist throughout the satellite engineering lifecycle.

[0004] Currently, digital twin technology primarily focuses on computer simulation, leveraging computing power to obtain the necessary data. However, it lacks human-computer interaction and simulation visualization. In existing application frameworks, humans are not given a central role, with digital models serving as the core of the simulation. This, to a certain extent, limits the application scope of digital twin technology, especially in engineering projects requiring direct human involvement. Effectively integrating human operators with digital models has become a pressing challenge. Summary of the Invention

[0005] Aiming at the problems of long iteration cycle and high cost in traditional satellite assembly process, the present invention provides a satellite digital intelligent assembly method.

[0006] A satellite digital intelligent assembly method of the present invention comprises:

[0007] Construct a virtual hand model and establish gesture matching between the virtual hand and the real hand; define the human-computer interaction gestures of the real hand based on mixed reality technology;

[0008] Based on digital twin technology, all satellite component models are built and imported into Unity3D. Transform script components, Mesh Filter script components, and Mesh Renderer script components are configured for each satellite component model to obtain the satellite component mesh rendering model; and collision bodies are constructed for each satellite component mesh rendering model;

[0009] An assembly control panel is designed so that when the real hand drives the virtual hand through human-computer interaction gestures, the assembly control panel is correspondingly configured at the corresponding position of the virtual hand; the assembly control panel is configured with interactive buttons, which recognize the click behavior of the virtual hand on the interactive button based on the contact state and trigger the assembly function of the corresponding satellite component mesh rendering model;

[0010] Configure intelligent auxiliary assembly logic scripts for satellite component mesh rendering models, prompt the corresponding installation positions through the intelligent auxiliary assembly logic scripts, and realize the fixation of satellite component mesh rendering models to each other;

[0011] Construct an interference detection and warning model to detect collision interference or situations where the distance is less than the preset component distance threshold during the assembly process of the satellite component mesh rendering model, and output warning interference results;

[0012] Establish a common assembly tool model to detect the reachability of satellite component mesh rendering models in confined spaces and the reachability of common assembly tool models to target locations, obtaining assembly reachability results.

[0013] According to the early warning interference results and assembly achievable results, the problems existing in the design and assembly process plan of the current target satellite are obtained.

[0014] According to the satellite digital intelligent assembly method of the present invention, the virtual palm is positioned and constructed based on 26 joint points, including five joint points set for each finger and one joint point set at the wrist;

[0015] The gesture matching relationship between the virtual palm and the real palm is established based on the OpenPose algorithm;

[0016] Human-computer interaction gestures include finger posture, knuckle contact relationship, and hand orientation;

[0017] The definition of finger gestures includes:

[0018] Flexion of the metacarpophalangeal joint was defined as flexion; bending of the distal phalanx of the finger was defined as flexion; and opening and closing between two adjacent fingers was defined as abduction.

[0019] Each finger posture is set through Any, Open and Close. Any is used to set the finger posture that does not affect gesture recognition. Open means setting the degree of extension / opening of the finger, and Close is used to set the degree of bending / closing of the finger.

[0020] The finger joint contact relationship is determined by calculating the distance between the joint points, and the joint points whose distance between the joint points is less than the finger joint distance threshold are determined as the finger joint contact relationship;

[0021] Hand orientations include facing toward the face, away from the face, palms up, and palms down.

[0022] According to the satellite digital intelligent assembly method of the present invention, the Transform script component is used to determine the position, rotation and scale of the satellite component mesh rendering model in the assembly scene;

[0023] The Mesh Filter script component is used to reference mesh resources;

[0024] The Mesh Renderer script component is used in conjunction with the Mesh Filter script component to render the mesh referenced by the MeshFilter script component;

[0025] Colliders are constructed using mesh assets.

[0026] According to the satellite digital intelligent assembly method of the present invention, after the assembly control panel recognizes the human-computer interaction gesture given by the real left palm, it generates an assembly control panel on the right side of the corresponding virtual palm and can move with the virtual palm.

[0027] According to the satellite digital intelligent assembly method of the present invention, the human-computer interaction gestures for generating the assembly control panel are set as follows:

[0028] Finger posture settings: Flexion and bending of all five fingers are set to Open, and abduction is set to Any;

[0029] The finger joint contact relationship is set to no contact relationship;

[0030] For the tracking axis of the hand orientation, select the palm as the main axis extension, and select the hand orientation to face the face.

[0031] According to the satellite digital intelligent assembly method of the present invention, the human-computer interaction gesture of clicking the interactive button is defined as:

[0032] Finger posture settings: Flexion, bend, and abduction of the four fingers except the index finger are all set to Close; Flexion and bend of the index finger are all set to Open, and abduction is set to Any;

[0033] The finger joint contact relationship is set to no contact relationship;

[0034] For the tracking axis of the hand's direction, select the palm as the main axis extension, and select the direction of the hand away from the face.

[0035] According to the satellite digital intelligent assembly method of the present invention, the design of the assembly control panel includes constructing a control panel model, writing a panel generation logic script, and designing interactive buttons;

[0036] The control panel model is configured to configure several button blocks on the bottom plate;

[0037] The panel generation logic script is used to identify the human-computer interaction gesture for generating the assembly control panel, locate it based on the thumb metacarpal joint, generate the assembly control panel at the target position on the right side of the corresponding virtual palm, and make the assembly control panel follow the movement of the virtual palm;

[0038] Designing interactive buttons involves adding Box Collider script components to button blocks and constructing button collision bodies.

[0039] The method for realizing the click function of the interactive button is to calculate whether the top bone joint of the index finger is in contact with the interactive button after receiving the human-computer interaction gesture of clicking the interactive button. If so, the interactive button is highlighted and the corresponding assembly function is triggered.

[0040] According to the satellite digital intelligent assembly method of the present invention, each satellite component grid rendering model corresponds to a configuration installation position model;

[0041] The intelligent auxiliary assembly logic script is used to display the installation position model at the corresponding position when the satellite component mesh rendering model is picked up during the assembly process, and to make the installation position model disappear after the installation is completed;

[0042] The method used by the intelligent auxiliary assembly logic script to fix the satellite component mesh rendering models to each other is as follows:

[0043] Create a calibrated empty object as the sub-object to be assembled, and set the pose coordinates of the calibrated empty object to the pose coordinates of the corresponding satellite component mesh rendering model installation position; when the satellite component mesh rendering model is picked up, calculate the position deviation and pose deviation between the satellite component mesh rendering model and the calibrated empty object. When the position deviation and pose deviation are both less than the set deviation threshold, release the satellite component mesh rendering model to complete the installation of the satellite component mesh rendering model;

[0044] The position deviation calculation formula is:

[0045] Set the position coordinates of the calibration empty object The position coordinates of the satellite component mesh rendering model Among them, x0 and x1 are the X-axis coordinates in the spatial position system, y0 and y1 are the Y-axis coordinates in the spatial position system, and z0 and z1 are the Z-axis coordinates in the spatial position system. The calculated position deviation d is:

[0046]

[0047] Set the attitude quaternion of the calibration empty object to The attitude quaternion of the satellite component mesh rendering model is Deviation Quaternion

[0048]

[0049]

[0050]

[0051]

[0052] Where q * is the conjugate of q.

[0053] According to the satellite digital intelligent assembly method of the present invention, the interference detection warning model is used to detect the distance status during the movement process of the satellite component grid rendering model installation. When collision body interference occurs during the assembly process of the satellite component grid rendering model or the distance is less than a preset component distance threshold, the relevant collision body flashes red as a warning interference; the display of the warning interference is achieved by triggering the warning condition, and the warning condition is that the collision body interferes or the distance is less than the preset component distance threshold;

[0054] The satellite component mesh rendering model is configured with a rigid body script component, and the rigid body script component enables a real collision effect to be presented when a collision occurs between the satellite component mesh rendering models.

[0055] According to the satellite digital intelligent assembly method of the present invention, a commonly used assembly tool model is used to support the inspection function of the installation tool after the satellite component grid rendering model is installed and fixed; the inspection function includes detecting whether the satellite component grid rendering model reaches the specified fixed position and whether the installation tool interferes with other models after imitating the assembly action, and analyzing the accessibility of the installation tool to the target position and whether the operating space is sufficient.

[0056] The beneficial effects of the present invention are as follows: the method of the present invention carries out assembly process simulation verification based on the digital twin satellite model, which can quickly iterate the assembly process plan before satellite production and manufacturing, thereby improving the efficiency and quality of actual satellite assembly.

[0057] The method of the present invention conducts assembly process simulation and verification based on a digital twin satellite model. It can interactively manipulate the three-dimensional digital model of the product, simulating the assembly process of a virtual satellite in a simulation environment using the product, tooling, and a virtual hand. By analyzing the rationality and feasibility of the assembly process, it can proactively identify defects or errors in the overall satellite structural design, continuously identify and promptly improve deficiencies in the original process plan, and ultimately obtain an optimized product assembly process plan. Using the method of the present invention for digital satellite assembly can significantly shorten the feedback and correction cycle for satellite assembly process plans and satellite designs, reducing the time and cost of design iterations while avoiding expensive rework and material waste.

[0058] The implementation of the method of the present invention can complete the complete virtual assembly process of the satellite without referring to drawings, which can greatly improve the workers' familiarity and experience with the assembly process and greatly improve the assembly efficiency and quality of the satellite prototype.

[0059] Figure 1 This is a diagram of the hand joints of two hands; in the figure, Tip is the fingertip, Distal is the distal end, Intermediate is the middle, Proximal is the proximal end, Palm is the palm, Metacarpal is the metacarpal bone, Wrist is the wrist, Thumb is the thumb, Little is the little finger, Ring is the ring finger, Middle is the middle finger, and Index is the index finger;

[0060] Figure 2 This is a schematic diagram of the matching effect of the virtual and real palm models;

[0061] Figure 3 It is a diagram of finger posture;

[0062] Figure 4 This is a schematic diagram of the thumb and four fingers in a flexed posture;

[0063] Figure 5 This is a schematic diagram of the thumb and four fingers in a bent posture;

[0064] Figure 6 This is a schematic diagram of the thumb and four fingers in the abducted position;

[0065] Figure 7 This is a diagram of the Okay gesture under the joint contact relationship;

[0066] Figure 8 This is a schematic diagram extending with four fingers as the main axis when the tracking category is fingers;

[0067] Figure 9 This is a schematic diagram extending with the palm as the main axis when the tracking category is palm;

[0068] Figure 10 This is a schematic diagram extending with the thumb as the main axis when the tracking category is thumb;

[0069] Figure 11 It is a diagram of the direction of the hand facing the face;

[0070] Figure 12 It is a diagram of the direction of the hands facing away from the face;

[0071] Figure 13 It is a diagram of the hand orientation with the palm facing upwards;

[0072] Figure 14 It is a diagram of the hand orientation with the palm facing downwards;

[0073] Figure 15 It is the Mesh preview effect picture;

[0074] Figure 16 This is a comparison chart of the model with and without Mesh Collider added;

[0075] Figure 17 A diagram showing the visualization of finger postures;

[0076] Figure 18 It is a visual diagram of hand orientation;

[0077] Figure 19 This is a successful diagram of gesture interaction application;

[0078] Figure 20 It is a visualization diagram of finger posture in the definition of assembly control panel;

[0079] Figure 21 It is a visualization diagram of the hand orientation in the definition of the assembly control panel;

[0080] Figure 22 It is a visual diagram of the click gesture of the interactive button;

[0081] Figure 23 It is a visualization diagram of the hand orientation for clicking gestures on interactive buttons;

[0082] Figure 24 This is a schematic diagram of the control panel model;

[0083] Figure 25 This is a schematic diagram of the generated effect of assembling the control panel;

[0084] Figure 26 This is a diagram showing the actual effect of running the Box Collider script component;

[0085] Figure 27 It is a schematic diagram of the satellite component model and the corresponding installation location model;

[0086] Figure 28 It is the effect diagram of the intelligent auxiliary assembly logic script running;

[0087] Figure 29 This is the actual effect diagram of running the rigid body Rigidbody script component;

[0088] Figure 30 It is a schematic diagram of a simple cylindrical tool in a commonly used assembly tool model;

[0089] Figure 31 Schematic diagram of using cylindrical tools to verify tool accessibility in confined spaces and spatial redundancy in manual assembly. DETAILED DESCRIPTION

[0090] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0091] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0092] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but they are not intended to limit the present invention.

[0093] Specific implementation method 1. Combination Figure 1 As shown, the present invention provides a satellite digital intelligent assembly method, comprising:

[0094] Construct a virtual hand model and establish gesture matching between the virtual hand and the real hand; define the human-computer interaction gestures of the real hand based on mixed reality technology;

[0095] Based on digital twin technology, all satellite component models are built and imported into Unity3D. Transform script components, Mesh Filter script components, and Mesh Renderer script components are configured for each satellite component model to obtain the satellite component mesh rendering model; and collision bodies are constructed for each satellite component mesh rendering model;

[0096] An assembly control panel is designed so that when the real hand drives the virtual hand through human-computer interaction gestures, the assembly control panel is correspondingly configured at the corresponding position of the virtual hand; the assembly control panel is configured with interactive buttons, which recognize the click behavior of the virtual hand on the interactive button based on the contact state and trigger the assembly function of the corresponding satellite component mesh rendering model;

[0097] Configure intelligent auxiliary assembly logic scripts for satellite component mesh rendering models, prompt the corresponding installation positions through the intelligent auxiliary assembly logic scripts, and realize the fixation of satellite component mesh rendering models to each other;

[0098] Construct an interference detection and warning model to detect collision interference or situations where the distance is less than the preset component distance threshold during the assembly process of the satellite component mesh rendering model, and output warning interference results;

[0099] Establish a common assembly tool model to detect the reachability of satellite component mesh rendering models in confined spaces and the reachability of common assembly tool models to target locations, obtaining assembly reachability results.

[0100] According to the early warning interference results and assembly achievable results, the problems existing in the design and assembly process plan of the current target satellite are obtained.

[0101] In this embodiment, the virtual palm is positioned and constructed based on 26 joint points, including five joint points set for each finger and one joint point set at the wrist;

[0102] The gesture matching relationship between the virtual palm and the real palm is established based on the OpenPose algorithm;

[0103] Human-computer interaction gestures include finger posture, knuckle contact relationship, and hand orientation;

[0104] The definition of finger gestures includes:

[0105] Flexion of the metacarpophalangeal joint was defined as flexion; bending of the distal phalanx of the finger was defined as flexion; and opening and closing between two adjacent fingers was defined as abduction.

[0106] Each finger posture is set through Any, Open and Close. Any is used to set the finger posture that does not affect gesture recognition. Open means setting the degree of extension / opening of the finger, and Close is used to set the degree of bending / closing of the finger.

[0107] The finger joint contact relationship is determined by calculating the distance between the joint points, and the joint points whose distance between the joint points is less than the finger joint distance threshold are determined as the finger joint contact relationship;

[0108] Hand orientations include facing toward the face, away from the face, palms up, and palms down.

[0109] This implementation defines a virtual palm model based on the OpenXR standard using 26 hand joints. Figure 1 As shown in Figure 2, a high-precision 3D model of the palm is constructed based on the positioning of 26 joints.

[0110] Establish matching relationship between virtual and real palms based on OpenPose algorithm:

[0111] OpenPose is an advanced multi-task human keypoint detection algorithm that accurately identifies and locates rich 2D and 3D information, including joints, facial expressions, and hand gestures. OpenPose utilizes a fusion of deep learning and separable graph models. By treating the human body as a graph composed of multiple parts, each representing a joint or feature point, it uses a convolutional neural network (CNN) to extract features from the input image and then predicts the position of each part using graph optimization techniques.

[0112] The architecture of OpenPose is divided into two main parts: image flow and graph optimization. First, the image flow module uses a pre-trained convolutional neural network to perform multi-scale analysis on the input image to generate a series of feature maps. These feature maps capture the characteristics of the hand at different scales and positions. Next, these feature maps are used to predict the positions of key points through two methods: Heatmap Regression and Part Affinity Fields (PAF). Heatmap Regression is used to directly predict the heat map corresponding to each joint, while PAF depicts the relationship between joints by connecting the probabilities of adjacent joints to form a hand pose graph.

[0113] The core mathematical principles of this algorithm include the following three aspects:

[0114] Keypoint detection: OpenPose processes image data through convolutional neural networks. Assuming the input image is I, it outputs a set of keypoints ({P} = {p_1, p_2, ..., p_n}), where n is the number of keypoints and p_i = (x_i, y_i) represents the pixel coordinates of the i-th keypoint. The output of CNN can be expressed as:

[0115] P=f(I;θ),

[0116] Where f is the function of CNN and θ is the parameter of the grid.

[0117] Pose Estimation: To estimate 3D pose, OpenPose usually combines regression and classification methods. First, a deep learning network is used to predict the depth d of each key point. i , and then calculate the 3D coordinates through camera parameters (such as focal length, optical center, etc.):

[0118] Q i =K·[d i , p ix , p iy ] T ,

[0119] Where K is the camera intrinsic parameter matrix, p ix and p iy are the horizontal and vertical coordinates of the key points on the image.

[0120] Loss function: During training, OpenPose usually uses a multi-task loss function, including key point detection loss L kp , body part segmentation loss L seg and 3D pose loss L 3D wait:

[0121] L=L kp +L seg +L 3D ;

[0122] Based on the OpenPose algorithm, a matching relationship between the virtual palm model and the real palm model in the real space can be established, such as Figure 2 shown.

[0123] Definition of human-computer interaction gestures based on MR technology:

[0124] Mixed reality (MR) technology integrates virtual information with the real world, and gesture recognition is an important way to interact with virtual models. In MR technology, gestures can be defined by three parts: finger posture shapes, finger joint contact relationships, and hand orientation transforms:

[0125] Gesture = Shapes + Bones + Transform;

[0126] Step 1: Define finger gesture Shapes:

[0127] First, the finger postures are divided into three types: flexion, curvature, and abduction. Figure 3 shown.

[0128] Finger gestures support three states: Any, Open, and Close:

[0129] Any: Indicates that the finger state can be in any state and will not affect gesture recognition;

[0130] Open: used to set the extension / opening degree of the fingers;

[0131] Close: Used to set the degree of bending / closing of the finger.

[0132] Figure 4 、 Figure 5 and Figure 6 Schematic diagrams showing the thumb and four fingers in flexion, bending, and abduction (Open or Close), each diagram providing specific Open and Close ranges.

[0133] Figure 6 In the middle, abduction is the spread between the current finger and the next finger in sequence, starting with the thumb. This includes the spread between the thumb and index finger, index finger and middle finger, middle finger and ring finger, and ring finger and pinky finger. There is no separate abduction setting for the pinky.

[0134] Step 2: Define joint contact relationships Bones:

[0135] It mainly calculates the distance between joint points, and triggers the contact relationship when the distance is less than a certain threshold. Figure 7 As shown, when the distance between the thumb tip (Thumb_tip) and the index finger tip (Index_tip) is less than the set threshold, the contact relationship is triggered, and the "Okay" gesture can be implemented.

[0136] Step 3: Define the hand orientation Transform:

[0137] First, define the tracking axis as the reference axis for gesture tracking;

[0138] When the tracking type is finger, four fingers are used as the main axis to extend, such as Figure 8 As shown; when the tracking category is palm, the palm is extended as the main axis, as shown Figure 9 As shown; when the tracking category is thumb, the thumb is extended as the main axis, as shown Figure 10 shown.

[0139] The hand orientation is defined by the relative orientation of the tracking axis and the tracking target: facing the face, such as Figure 11 As shown; away from the face as Figure 12 As shown; palms facing up as Figure 13 As shown, palms facing down Figure 14 shown.

[0140] The following is the design of the satellite component interactive attribute subsystem:

[0141] The Transform script component is used to determine the position, rotation, and scale of the satellite component mesh rendering model in the assembly scene;

[0142] The Mesh Filter script component is used to reference mesh resources;

[0143] The Mesh Renderer script component is used in conjunction with the Mesh Filter script component to render the mesh referenced by the MeshFilter script component;

[0144] Colliders are constructed using mesh assets.

[0145] This implementation method first constructs and renders the digital twin satellite model mesh: first, the digital twin satellite model after interface conversion is imported into Unity3D. At this time, three script components are automatically added to the model: Transform, MeshFilter, and Mesh Renderer.

[0146] The Transform script component determines the position, rotation, and scale of each GameObject in the scene. Each GameObject has a Transform. Table 1 describes the Transform script properties.

[0147] Table 1 Transform script property description

[0148] property Function Position Transforms the position in x, y, and z coordinates. Rotation The transformation's rotation around the x-, y-, and z-axes, in degrees. Scale The scale of the transformation along the x-, y-, and z-axes. A value of "1" is the original size.

[0149] Mesh Filter is mainly a script component that references the mesh resource Mesh.

[0150] The Mesh Inspector displays information about how the mesh asset's data is stored, rather than displaying the values ​​of the data itself. For example, the Inspector shows that Unity stores the Position value of each vertex as a set of (three) Float32 values, but it does not display the Position value of a specific vertex. Table 2 provides a detailed description of the Mesh properties.

[0151] Table 2 Mesh properties detailed description

[0152]

[0153] In the Inspector section, you can preview the appearance of the grid and view the display effect of the grid data, such as Figure 15 Each view displays the following data about the mesh below the mesh preview: the mesh name, the number of vertices in the mesh, the type and number of mesh faces, and the names of any UV maps on the mesh.

[0154] The Mesh Renderer component works in conjunction with the Mesh Filter component on the same model object; the Mesh Renderer component renders the mesh referenced by the Mesh Filter component. The most important property of the Mesh Renderer component is the Materials property, which lists all the materials used by the component. Table 3 provides a detailed description.

[0155] Table 3 Materials property introduction

[0156] property describe Size The number of elements in the material list. Element List of materials. Each element can be assigned a material.

[0157] Constructing colliders for the digital twin satellite model: The three script components, Transform, Mesh Filter, and Mesh Renderer, are primarily responsible for rendering the model at a specified pose. Furthermore, to achieve model interactivity, colliders must be constructed based on mesh resources. The script component involved is Mesh Collider.

[0158] A Mesh Collider takes a Mesh asset and constructs its collider based on that mesh. Mesh Colliders are more accurate than primitives that use complex meshes when performing collision detection. Mesh Colliders marked as Convex can collide with other Mesh Colliders. Their properties and functionality are described below:

[0159] Convex: Checking this box will make the Mesh Collider collide with other Mesh Colliders. A Convex Mesh Collider can have a maximum of 255 triangles.

[0160] Is Trigger: Checking this box will cause Unity to use this collider to trigger events, while the physics engine will ignore it.

[0161] Cooking: Enables or disables mesh cooking options that affect how the physics engine handles the mesh.

[0162] Options:

[0163] None: Disables all Cooking Options listed below.

[0164] Everything: Enables all Cooking Options listed below.

[0165] Cook for Faster Simulation: Causes the physics engine to cook meshes for faster simulations. When this setting is enabled, it performs some extra steps to ensure that the resulting mesh is optimal for runtime performance. This can affect the performance of physics queries and contact generation. When this setting is disabled, the physics engine uses a faster cooking speed and generates results as quickly as possible. As a result, the cooked Mesh Collider may not be optimal.

[0166] Enable Mesh Cleaning: Enables the physics engine to clean up the mesh. When this setting is enabled, the cooking process attempts to remove degenerate triangles and other geometric artifacts from the mesh. This process produces a mesh that is more suitable for use in collision detection and tends to produce more accurate hit points.

[0167] Weld Colocated Vertices: Causes the physics engine to remove equal vertices in the mesh. When this setting is enabled, the physics engine will merge vertices with the same position. This is important for collision feedback at runtime.

[0168] Use Fast Midphase: Causes the physics engine to use the fastest midphase acceleration structures and algorithms available for your output platform. When this option is enabled, the physics engine will use a faster algorithm that does not require any R-Trees for spatial access. If you encounter midphase issues when running on certain platforms, you can still disable this option to use the slower legacy midphase algorithm instead.

[0169] Material: A reference to a physical material that determines how this collider interacts with other objects.

[0170] Mesh: Reference to the mesh to be used for collision.

[0171] Figure 16 This is a comparison chart of the effects of a model with and without a Mesh Collider.

[0172] Interaction method design:

[0173] After constructing the collision body, we implemented an interactive interface between the hand model and the digital twin satellite model. This interface allows us to design interaction methods to enable specific gestures to interact with the digital twin satellite model. The main script component used is the Object Manipulator.

[0174] The key properties of Object Manipulator are described as follows:

[0175] ObjectManipulator Settings: Mainly used to set the allowed interaction types, such as move, rotate, scale, etc.; and set the type of interactor used, such as handle, hand, line of sight, etc.;

[0176] Manipulation Logic Types: Contains method logic for interaction types such as movement, rotation, and scaling.

[0177] The Object Manipulator script component mainly designs the interactive method of digital twin satellite component assembly based on the human-computer interaction system.

[0178] First, design the finger postures. The Flexion and Curl of the thumb and index finger are set to Close, that is, flexion and bending, and the Abduction is set to Any, that is, closed or open. The postures of the other fingers are set to Any, such as Figure 17 Then design the finger joint contact relationship Bones, where the tip of the index finger contacts the tip of the thumb.

[0179] Finally, design the hand orientation Transform. The Track Axis is defined as Palm, which is extended from the palm as the main axis. The hand orientation is defined as AwayFromFace, which is away from the face. Figure 18 shown.

[0180] Figure 19 This is a schematic diagram of an AR device intelligently recognizing a grabbing gesture through an algorithm and successfully interacting with a virtual satellite component model using this gesture.

[0181] Assembly control panel subsystem design:

[0182] Considering the timeliness of the control panel and user convenience, a new assembly control panel subsystem based on intelligent gesture recognition technology was designed. This system intelligently senses the user's needs and automatically generates a control panel on the right side of the palm when the user makes a specific, preset gesture with their left hand. The system intelligently follows the palm's movements, ensuring the control panel is always positioned for convenient operation. This intelligent control panel design not only improves operational convenience but also enhances the naturalness and intuitiveness of user interaction, making the entire assembly process smoother and more efficient.

[0183] After recognizing the human-computer interaction gesture given by the real palm of the left hand, the assembly control panel generates an assembly control panel on the right side of the corresponding virtual palm and can move with the virtual palm.

[0184] Furthermore, the human-computer interaction gestures for generating the assembly control panel are set as follows:

[0185] Finger posture Shapes settings: Flexion and Curl of the five fingers are set to Open, that is, all five fingers are straight; Abduction is set to Any, that is, closed or open, such as Figure 20 As shown;

[0186] The finger joint contact relationship is set to no contact relationship;

[0187] For the Track Axis of the Hand Towards Transform, select the Palm as the main axis extension, and select Towards Face as the hand direction. Figure 21 shown.

[0188] The human-computer interaction gesture for clicking an interactive button is defined as:

[0189] Finger posture settings: The flexion, bending and abduction of the four fingers except the index finger are all set to Close state, that is, they are all closed and bent; the flexion and bending of the index finger are all set to Open state, that is, straight state, and the abduction is set to Any, that is, closed and open; Figure 22 shown.

[0190] The finger joint contact relationship is set to no contact relationship;

[0191] For the hand tracking axis, select the palm as the main axis extension and the hand direction away from the face, such as Figure 23 shown.

[0192] After designing the control panel gesture, the corresponding hand control panel design is carried out. The design of the assembly control panel includes building the control panel model, writing the panel generation logic script and designing the interactive buttons.

[0193] The control panel model is set to configure several button blocks on the bottom plate; Figure 24 As shown;

[0194] The panel generation logic script is used to identify the human-computer interaction gesture for generating the assembly control panel, locate it based on the thumb metacarpal joint, generate the assembly control panel at the target position on the right side of the corresponding virtual palm, and make the assembly control panel follow the movement of the virtual palm;

[0195] Designing interactive buttons involves adding Box Collider script components to button blocks and constructing button collision bodies.

[0196] The method for realizing the click function of the interactive button is to calculate whether the top bone joint of the index finger is in contact with the interactive button after receiving the human-computer interaction gesture of clicking the interactive button. If so, the interactive button is highlighted and the corresponding assembly function is triggered.

[0197] Generation logic script writing: The basic algorithm principle is to determine whether the hand control panel gesture is recognized. If so, the control panel is generated a certain distance to the right of the thumb metacarpal joint Thumb_metacarpal, and the panel posture is updated in real time with this joint. This system only sets the left-hand generation logic to avoid erroneous generation of the control panel when the right hand is operating and affecting the line of sight.

[0198] Figure 25 The AR device uses an algorithm to intelligently recognize the control panel gesture, intelligently generate a control panel on the right side of the palm, and run the actual effect diagram of the script;

[0199] Interactive button design: First, you need to add a Box Collider script component to the button model, that is, to construct a collision body. The basic algorithm principle for identifying clicks is to first intelligently identify whether it is a button click gesture. If so, calculate whether the index finger's top bone joint Index_tip is in contact with the button. If so, the button is highlighted. In real time, it is determined whether the index finger's top bone joint Index_tip moves a certain distance toward the normal of the button plane. If so, it is determined to be a button click behavior. At the same time, some trigger events can be added for this behavior. The actual effect of running the Box Collider script component is as follows: Figure 26 As shown, the AR device can intelligently recognize the click gesture and successfully click to trigger the corresponding event.

[0200] Going a step further, the satellite parts installation subsystem design is carried out:

[0201] Intelligent Assisted Assembly Design: During the component assembly process, a drawing-free intelligent virtual assembly system is designed. Prior to assembly, workers no longer need to refer to drawings; instead, they can clearly see the corresponding assembly positions of each component through intelligent assisted assembly functions. Intelligent assisted assembly design primarily involves the construction of a model that indicates the installation position and the development of intelligent assisted assembly logic scripts.

[0202] Each satellite component grid rendering model corresponds to the configuration installation position model; each component needs to build a corresponding installation position model. To reduce the workload, the components can be directly copied and the material can be changed to transparent green. The effect is as follows Figure 27 shown.

[0203] The main function of the intelligent assisted assembly logic script is to display the corresponding installation location model at the installation location of the component when the operator picks up the component, and the installation location model disappears when the operator puts it down. The basic algorithm principle of this script is to broadcast an event to display the prompt model at the beginning of the picking action, and broadcast an event to hide the prompt display model at the end of the picking action. The actual effect of running the script is as follows Figure 28 The AR device intelligently recognizes the action of holding the model and displays the prompt model, and intelligently recognizes the action of putting it down and hides the prompt model.

[0204] The intelligent auxiliary assembly logic script is used to display the installation position model at the corresponding position when the satellite component mesh rendering model is picked up during the assembly process, and to make the installation position model disappear after the installation is completed; after the parts are placed in the installation position, the parts can be intelligently identified and automatically fixed on the installation plate, and become a whole, moving with the installation plate.

[0205] The method used by the intelligent auxiliary assembly logic script to fix the satellite component mesh rendering models to each other is as follows:

[0206] Create a calibrated empty object as the sub-object to be assembled, and set the pose coordinates of the calibrated empty object to the pose coordinates of the corresponding satellite component mesh rendering model installation position; when the satellite component mesh rendering model is picked up, calculate the position deviation and pose deviation between the satellite component mesh rendering model and the calibrated empty object. When the position deviation and pose deviation are both less than the set deviation threshold, release the satellite component mesh rendering model to complete the installation of the satellite component mesh rendering model;

[0207] The position deviation calculation formula is:

[0208] Set the position coordinates of the calibration empty object The position coordinates of the satellite component mesh rendering model Among them, x0 and x1 are the X-axis coordinates in the spatial position system, y0 and y1 are the Y-axis coordinates in the spatial position system, and z0 and z1 are the Z-axis coordinates in the spatial position system. The calculated position deviation d is:

[0209]

[0210] Set the attitude quaternion of the calibration empty object to The attitude quaternion of the satellite component mesh rendering model is Deviation Quaternion

[0211]

[0212]

[0213]

[0214]

[0215] Where q * is the conjugate of q.

[0216] Write a script algorithm based on the above deviation formula, DanJis is the reference of the assembly parts, and Points is the reference of the calibration empty object.

[0217] The actual effect of running the script is to hold the model, move it to the target position and adjust the posture. After letting go, the AR device intelligently identifies whether the calculated deviation meets the requirements. If so, the component is automatically fixed to the mounting plate.

[0218] The interference detection and warning model is used to detect the distance status during the installation and movement of the satellite component mesh rendering model. When collision body interference occurs during the assembly of the satellite component mesh rendering model or the distance is less than the preset component distance threshold, the relevant collision body will flash red as a warning interference. The warning interference display is achieved by triggering the warning condition, which is the collision body interference or the distance is less than the preset component distance threshold. The basic algorithm principle is to calculate the closest distance between the handheld component and the surface of the assembled component mesh collision body in real time, and trigger the warning condition when it is less than a certain value.

[0219] The satellite component mesh rendering model is configured with a Rigidbody script component. This ensures that collisions between satellite component mesh rendering models do not intersect the model, resulting in a realistic collision effect. Since no mechanical effects are required, gravity is disabled and Is Kinematic is enabled, allowing only kinematic relationships to be considered.

[0220] During the virtual assembly process, the interference situation of the assembly process can be known through the component interference warning. By analyzing the interference situation, the component assembly sequence can be adjusted and the assembly path can be optimized. Finally, an optimal assembly sequence and assembly path can be found to obtain the best assembly solution.

[0221] The actual effect of running the Rigidbody script component is as follows Figure 29 As shown, when the AR device intelligently detects that the handheld component is too close to or has collided with other components, the relevant component will flash red once.

[0222] Tool interference warning and accessibility analysis and design in narrow spaces:

[0223] The commonly used assembly tool model is used to support the inspection function of the installation tool after the satellite component mesh rendering model is installed and fixed; the inspection function includes detecting whether the satellite component mesh rendering model has reached the specified fixed position and whether the installation tool interferes with other models after imitating the assembly action, and analyzing the installation tool's accessibility to the target position and whether there is sufficient operating space.

[0224] Model common assembly tools and add Transform, Mesh Filter, Mesh Renderer, Mesh Collider, ObjectManipulator, Rigidbody, and Collition Detection script components to them. Take the simple cylinder tool as an example. Figure 30 shown.

[0225] Analysis and design of tool accessibility in confined spaces: For components in confined spaces, after the components are automatically fixed on the mounting plate, the corresponding installation tool is held to inspect whether it can reach the designated fixed position. The tool is used to simulate various assembly actions to observe whether there is interference with other components. The tool accessibility and whether there is enough space for operation are analyzed. Figure 31 Schematic diagram of using cylindrical tools to verify tool accessibility in confined spaces and spatial redundancy in manual assembly.

[0226] Although the present invention is described herein with reference to specific embodiments, it should be understood that these embodiments are merely illustrative of the principles and applications of the invention. It should be understood that many modifications may be made to the illustrative embodiments, and that other arrangements may be devised, without departing from the spirit and scope of the invention as defined by the appended claims. It should be understood that the various dependent claims and features described herein may be combined in ways other than those described in the original claims. It should also be understood that features described in conjunction with individual embodiments may be used in conjunction with other described embodiments.

Claims

1. A satellite digital intelligent assembly method, characterized in that: include, Construct a virtual hand model and establish a gesture matching relationship between the virtual hand and the real hand; define the human-computer interaction gestures of the real hand based on mixed reality technology; Based on digital twin technology, all satellite component models are built and imported into Unity3D. Transform script components, Mesh Filter script components, and Mesh Renderer script components are configured for each satellite component model to obtain the satellite component mesh rendering model; and collision bodies are constructed for each satellite component mesh rendering model; Design an assembly control panel so that when the real palm drives the virtual palm movement through human-computer interaction gestures, the assembly control panel is correspondingly configured at the corresponding position of the virtual palm; The assembly control panel is configured with an interactive button, which recognizes the click behavior of the virtual palm on the interactive button based on the contact state and triggers the assembly function of the corresponding satellite component mesh rendering model; Configure intelligent auxiliary assembly logic scripts for satellite component mesh rendering models, prompt the corresponding installation positions through the intelligent auxiliary assembly logic scripts, and realize the fixation of satellite component mesh rendering models to each other; Construct an interference detection and warning model to detect collision interference or situations where the distance is less than the preset component distance threshold during the assembly process of the satellite component mesh rendering model, and output warning interference results; Establish a common assembly tool model to detect the reachability of satellite component mesh rendering models in confined spaces and the reachability of common assembly tool models to target locations, obtaining assembly reachability results. According to the early warning interference results and assembly achievable results, the problems existing in the design and assembly process plan of the current target satellite are obtained.

2. The satellite digital intelligent assembly method according to claim 1, characterized in that: The virtual palm is positioned and constructed based on 26 joint points, including five joint points set for each finger and one joint point set at the wrist; The gesture matching relationship between the virtual palm and the real palm is established based on the OpenPose algorithm; Human-computer interaction gestures include finger posture, knuckle contact relationship, and hand orientation; The definition of finger gestures includes: Flexion of the metacarpophalangeal joint was defined as flexion; bending of the distal phalanx of the finger was defined as flexion; and opening and closing between two adjacent fingers was defined as abduction. Each finger posture is set through Any, Open and Close. Any is used to set the finger posture that does not affect gesture recognition. Open means setting the degree of extension / opening of the finger. Close is used to set the degree of bending / closing of the finger. The finger joint contact relationship is determined by calculating the distance between the joint points, and the joint points whose distance between the joint points is less than the finger joint distance threshold are determined as the finger joint contact relationship; Hand orientations include facing toward the face, away from the face, palms up, and palms down.

3. The satellite digital intelligent assembly method according to claim 2, characterized in that: The Transform script component is used to determine the position, rotation, and scale of the satellite component mesh rendering model in the assembly scene; The Mesh Filter script component is used to reference mesh resources; The Mesh Renderer script component is used in conjunction with the Mesh Filter script component to render the mesh referenced by the Mesh Filter script component; Colliders are constructed using mesh assets.

4. The satellite digital intelligent assembly method according to claim 3, characterized in that: After recognizing the human-computer interaction gesture given by the real palm of the left hand, the assembly control panel generates an assembly control panel on the right side of the corresponding virtual palm and can move with the virtual palm.

5. The satellite digital intelligent assembly method according to claim 4, characterized in that: Generate the human-computer interaction gesture settings for the assembly control panel: Finger posture settings: Flexion and bending of all five fingers are set to Open, and abduction is set to Any; The finger joint contact relationship is set to no contact relationship; For the tracking axis of the hand orientation, select the palm as the main axis extension, and select the hand orientation to face the face.

6. The satellite digital intelligent assembly method according to claim 5, characterized in that: The human-computer interaction gesture for clicking an interactive button is defined as: Finger posture settings: Flexion, bend, and abduction of the four fingers except the index finger are all set to Close; Flexion and bend of the index finger are all set to Open, and abduction is set to Any; The finger joint contact relationship is set to no contact relationship; For the tracking axis of the hand's direction, select the palm as the main axis extension, and select the direction of the hand away from the face.

7. The satellite digital intelligent assembly method according to claim 6, characterized in that: The design of the assembly control panel includes building the control panel model, writing the panel generation logic script and designing interactive buttons; The control panel model is configured to configure several button blocks on the bottom plate; The panel generation logic script is used to identify the human-computer interaction gesture for generating the assembly control panel, locate it based on the thumb metacarpal joint, generate the assembly control panel at the target position on the right side of the corresponding virtual palm, and make the assembly control panel follow the movement of the virtual palm; Designing interactive buttons involves adding Box Collider script components to button blocks and constructing button collision bodies. The method for realizing the click function of the interactive button is to calculate whether the top bone joint of the index finger is in contact with the interactive button after receiving the human-computer interaction gesture of clicking the interactive button. If so, the interactive button is highlighted and the corresponding assembly function is triggered.

8. The satellite digital intelligent assembly method according to claim 7, characterized in that: Each satellite component mesh rendering model corresponds to the configuration installation location model; The intelligent auxiliary assembly logic script is used to display the installation position model at the corresponding position when the satellite component mesh rendering model is picked up during the assembly process, and to make the installation position model disappear after the installation is completed; The method used by the intelligent auxiliary assembly logic script to fix the satellite component mesh rendering models to each other is as follows: Create a calibrated empty object as the sub-object to be assembled, and set the pose coordinates of the calibrated empty object to the pose coordinates of the corresponding satellite component mesh rendering model installation position; when the satellite component mesh rendering model is picked up, calculate the position deviation and pose deviation between the satellite component mesh rendering model and the calibrated empty object. When the position deviation and pose deviation are both less than the set deviation threshold, release the satellite component mesh rendering model to complete the installation of the satellite component mesh rendering model; The position deviation calculation formula is: Set the position coordinates of the calibration empty object The position coordinates of the satellite component mesh rendering model Among them, x0 and x1 are the X-axis coordinates in the spatial position system, y0 and y1 are the Y-axis coordinates in the spatial position system, and z0 and z1 are the Z-axis coordinates in the spatial position system. The calculated position deviation d is: Set the attitude quaternion of the calibration empty object to The attitude quaternion of the satellite component mesh rendering model is Deviation Quaternion Where q * is the conjugate of q.

9. The satellite digital intelligent assembly method according to claim 8, characterized in that: The interference detection warning model is used to detect the distance status during the installation and movement of the satellite component mesh rendering model. When collision body interference occurs during the assembly of the satellite component mesh rendering model or the distance is less than the preset component distance threshold, the relevant collision body will flash red as a warning interference; The display of the early warning interference is realized by triggering the early warning condition, which is the interference of the collision body or the distance being less than the preset component distance threshold; The satellite component mesh rendering model is configured with a rigid body script component, and the rigid body script component enables a real collision effect to be presented when a collision occurs between the satellite component mesh rendering models.

10. The satellite digital intelligent assembly method according to claim 9, characterized in that: The commonly used assembly tool model is used to support the inspection function of the installation tool after the satellite component mesh rendering model is installed and fixed; the inspection function includes detecting whether the satellite component mesh rendering model has reached the specified fixed position and whether the installation tool interferes with other models after imitating the assembly action, and analyzing the installation tool's accessibility to the target position and whether there is sufficient operating space.

Images