Low-code programming-based human-machine collaborative assembly method and system, and storage medium

Abstract

The present application provides a low-code programming-based human-machine collaborative assembly method and system, and a storage medium. The method comprises: using a target augmented reality (AR) device to collect a target scene containing a target object, to generate a three-dimensional segmentation model and display same; acquiring initial pose information of the three-dimensional segmentation model and gesture information of a user manipulating the three-dimensional segmentation model, generating mechanical arm control data on the basis of the initial pose information and the gesture information, and generating a simulation preview of a mechanical arm grabbing the target object; upon receiving confirmation information of the user for the simulation preview, the target AR device generating a control instruction on the basis of the mechanical arm control data, and sending the control instruction to the mechanical arm; and the mechanical arm receiving and executing the control instruction, so as to move the target object to a target position in a target pose. The present application utilizes AR technology to enable users to intuitively manipulate the three-dimensional segmentation model, thereby generating a control instruction for the mechanical arm, reducing the workload of coding, and effectively improving the programming efficiency of the mechanical arm.

Description

Human-computer collaborative assembly method, system, and storage medium based on low-code programming Technical Field

[0001] This application relates to the field of human-machine collaborative assembly, and in particular to a human-machine collaborative assembly method, system and storage medium based on low-code programming. Background Technology

[0002] As a prime example of flexible automation in the industrial sector, robotic arms have brought about a tremendous leap in productivity to the manufacturing industry. They can replace human operators in performing repetitive and precision-required tasks, greatly improving production efficiency and product quality.

[0003] Traditional offline programming methods for robotic arms allow engineers to plan and debug the arm's movements in a virtual simulation environment. This method does not interrupt the actual production process and can handle more complex motion trajectories. However, this type of offline programming heavily relies on accurate digital models, including digital representations of the robotic arm itself, the workpiece it manipulates, and the entire working environment. Building these digital models requires significant time and effort to ensure that the simulation results in the virtual environment accurately reflect the actual workspace. Furthermore, traditional offline programming methods still require writing a large amount of code to define the robotic arm's motion trajectory, control logic, and interaction with external sensors, and require fine-tuning numerous parameters to make the robotic arm work as expected, resulting in low programming efficiency.

[0004] Therefore, existing technologies have shortcomings and need to be improved and developed. Summary of the Invention

[0005] The technical problem to be solved by this application is to provide a human-machine collaborative assembly method, system and storage medium based on low-code programming, which addresses the above-mentioned deficiencies of the prior art and aims to solve the problem of low programming efficiency of robotic arms in the prior art.

[0006] The technical solution adopted by this application to solve the technical problem is as follows:

[0007] In a first aspect, embodiments of this application provide a human-computer collaborative assembly method based on low-code programming, the method comprising:

[0008] The target AR device is used to capture the target scene containing the target object, so as to generate a 3D segmentation model of the target object and display it on the device.

[0009] The initial pose information of the three-dimensional segmentation model and the gesture information of the user controlling the three-dimensional segmentation model are obtained. Based on the initial pose information and the gesture information, robotic arm control data is generated, and a simulation preview of the robotic arm grasping the target object is generated based on the robotic arm control data.

[0010] Upon receiving confirmation from the user regarding the simulation preview, the target AR device generates control commands based on the robotic arm control data and sends the control commands to the robotic arm.

[0011] The robotic arm receives and executes the control commands to move the target object to the target position in the target posture.

[0012] In one implementation, the step of using a target AR device to acquire a target scene containing a target object, generating a 3D segmentation model corresponding to the target object, and displaying it on the device includes:

[0013] The target AR device is used to collect the target scene containing the target object, and the RGB image of the target object, the camera pose when the target object was captured, and the initial point cloud of the target object are obtained.

[0014] The RGB image, the camera pose, and the initial point cloud are input into a trained 3D Gaussian sputtering model. The 3D Gaussian sputtering model is then used to process the data to obtain the initial 3D Gaussian distribution of the target object.

[0015] The RGB image is input into the All-Segmentation Model, and after processing by the All-Segmentation Model, a two-dimensional segmentation mask of the target object is obtained. The initial three-dimensional Gaussian distribution is then subjected to inverse rasterization using the two-dimensional segmentation mask to obtain the initial segmentation result of the target object.

[0016] The initial segmentation result is filtered using the KNN statistical filtering algorithm to obtain several three-dimensional Gaussian spheres. All three-dimensional Gaussian spheres are converted into point clouds to obtain the three-dimensional segmentation model corresponding to the target object, which is then displayed on this device.

[0017] In one implementation, the training steps of the three-dimensional Gaussian sputtering model include:

[0018] Construct an initial 3D Gaussian sputtering model;

[0019] Obtain a training dataset, which includes multiple training scenarios. Each training scenario consists of a training RGB image of the target training object, the training camera pose when the target training object is photographed, and the initial training point cloud of the target training object.

[0020] In each iteration of training, the training RGB image, the training camera pose, and the initial training point cloud in each training scene are input into the initial 3D Gaussian sputtering model, while the training RGB image in each training scene is input into the segmentation model.

[0021] The loss function is calculated based on the output of the initial three-dimensional Gaussian sputtering model and the output of the segmentation model, and the parameters of the initial three-dimensional Gaussian sputtering model are updated based on the loss function.

[0022] Repeated iterative training is performed until the loss function converges to a preset value, resulting in a trained three-dimensional Gaussian sputtering model.

[0023] In one embodiment, the gesture information includes target pose information, grasping strategy information, and path guidance information; the step of acquiring the initial pose information of the 3D segmentation model and the gesture information of the user manipulating the 3D segmentation model, and generating robotic arm control data based on the initial pose information and the gesture information, includes:

[0024] Obtain the initial pose information of the three-dimensional segmentation model;

[0025] Acquire the target pose information, grasping strategy information, and path guidance information of the user-controlled 3D segmentation model;

[0026] Based on the initial pose information and the target pose information, pose calculation is performed to transform the coordinates of the 3D segmentation model from the augmented reality coordinate system to the robotic arm coordinate system;

[0027] Based on the grasping strategy information and the path guidance information, robotic arm control data is generated in the robotic arm coordinate system. The robotic arm control data includes the motion angles of each joint of the robotic arm to reach the target pose, as well as the path planning for the robotic arm to grasp the target object.

[0028] In one implementation, generating robotic arm control data in the robotic arm coordinate system based on the grasping strategy information and the path guidance information includes:

[0029] The grasping strategy information is processed using a cyclic coordinate descent algorithm to obtain the motion angles of each joint of the robotic arm in the robotic arm coordinate system to reach the target pose.

[0030] Based on the path guidance information, path planning is performed to obtain the path planning for the robotic arm to grasp the target object in the robotic arm coordinate system.

[0031] In one embodiment, before the robotic arm executes the control command, it further includes:

[0032] Obtain the prerequisites corresponding to the control command and determine whether the robotic arm meets the prerequisites. The prerequisites are used to determine whether the robotic arm can execute the corresponding control command.

[0033] If the robotic arm meets the prerequisite, then it is determined that the robotic arm can execute the control command.

[0034] In one embodiment, the method further includes:

[0035] While the robotic arm executes the control command, the actual state of the robotic arm is monitored in real time, and actual state timing data is generated.

[0036] Based on the simulation preview of the robotic arm grasping the target object, simulation state timing data is generated;

[0037] Calculate the difference between the actual state timing data and the simulated state timing data within each time step. If the difference within each time step is within a preset threshold, then the actual state of the robotic arm is determined to be consistent with the simulated state.

[0038] If the difference at any time step exceeds a preset threshold, it is determined that the actual state of the robotic arm is inconsistent with the simulation state.

[0039] In one embodiment, after the robotic arm executes the control command, it further includes:

[0040] Obtain the postcondition corresponding to the control command, and determine whether the robotic arm meets the postcondition after executing the control command. The postcondition is used to determine whether the robotic arm has completed the control command and is in the specified target state.

[0041] If the robotic arm meets the postcondition, it sends a task success message to the target AR device.

[0042] Secondly, embodiments of this application also provide a human-machine collaborative assembly system based on low-code programming, comprising:

[0043] The model display module is used to acquire a target scene containing the target object using the target AR device, generate a 3D segmentation model of the target object, and display it on the device.

[0044] The simulation preview module is used to acquire the initial pose information of the three-dimensional segmentation model and the gesture information of the user controlling the three-dimensional segmentation model, generate robotic arm control data based on the initial pose information and the gesture information, and generate a simulation preview of the robotic arm grasping the target object based on the robotic arm control data.

[0045] The instruction sending module is used to, upon receiving confirmation information from the user regarding the simulation preview, generate control instructions based on the robotic arm control data and send the control instructions to the robotic arm.

[0046] The instruction execution module is used for the robotic arm to receive and execute the control instructions to move the target object to the target position in the target posture.

[0047] Thirdly, embodiments of this application also provide a computer-readable storage medium storing a low-code programming-based human-computer collaborative assembly program, which can be executed to implement the steps of the low-code programming-based human-computer collaborative assembly method as described above.

[0048] The beneficial effects of this application are as follows: This application utilizes a target AR device to collect a target scene containing the target object, generates a 3D segmentation model, and displays it. It acquires the initial pose information of the 3D segmentation model and the user's gesture information regarding the model. Based on the initial pose and gesture information, it generates robotic arm control data and generates a simulation preview of the robotic arm grasping the target object. Upon receiving confirmation from the user of the simulation preview, the target AR device generates control commands based on the robotic arm control data and sends these commands to the robotic arm. The robotic arm receives and executes the control commands to move the target object to the target position in the target posture. This application utilizes augmented reality technology to enable users to intuitively manipulate the 3D segmentation model, thereby generating control commands for the robotic arm, reducing the amount of coding work, and effectively improving the programming efficiency of the robotic arm. Attached Figure Description

[0049] Figure 1 is a flowchart of a preferred embodiment of the human-computer collaborative assembly method based on low-code programming in this application.

[0050] Figure 2 is a training diagram of the three-dimensional Gaussian sputtering model in this application.

[0051] Figure 3 is a schematic diagram of the three-dimensional segmentation model selected by the user in this application.

[0052] Figure 4 is a schematic diagram of the user adjusting the pose of the 3D segmentation model using gestures in this application.

[0053] Figure 5 is a schematic diagram of how a user adjusts the pose of a 3D segmentation model using gestures in this application.

[0054] Figure 6 is a schematic diagram of how a user adjusts the pose of a 3D segmentation model using gestures in this application.

[0055] Figure 7 is a schematic diagram of a user dragging a 3D segmentation model using gestures in this application.

[0056] Figure 8 is a schematic diagram of the overall architecture of this application.

[0057] Figure 9 is a functional principle block diagram of a preferred embodiment of the human-machine collaborative assembly system in this application. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of this application clearer and more explicit, the following detailed description of this application is provided with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only for explaining this application and are not intended to limit this application.

[0059] Robotic arms are the quintessential embodiment of flexible automation in the industrial field, significantly improving productivity when introduced into manufacturing processes. However, robotic arm programming, especially in human-robot collaborative assembly, still faces challenges such as time consumption and over-reliance on programmer skill levels. In traditional online programming, operators control the robotic arm by physically moving joysticks or control boxes. While this method requires lower programming skills and costs, it is inefficient for complex assembly tasks and difficult to modify easily. On the other hand, offline programming allows programmers to program robotic arms in a simulation environment without downtime, making it suitable for more complex tasks. However, this type of offline programming heavily relies on accurate digital models, including the robotic arm itself, the workpiece being manipulated, and a digital representation of the entire working environment. Building these digital models requires significant time and effort to ensure that the simulation results in the virtual environment accurately reflect the actual workspace. Furthermore, traditional offline programming methods still require writing substantial amounts of code to define the robotic arm's motion trajectory, control logic, and interactions with external sensors, and necessitate fine-tuning numerous parameters to ensure the robotic arm functions as intended, resulting in low programming efficiency.

[0060] To address the aforementioned deficiencies in existing technologies, this application provides a human-machine collaborative assembly method, system, and storage medium based on low-code programming. The method includes: using a target AR device to acquire a target scene containing a target object, generating and displaying a 3D segmented model; acquiring initial pose information of the 3D segmented model and user gesture information of manipulating the 3D segmented model, generating robotic arm control data based on the initial pose information and gesture information, and generating a simulation preview of the robotic arm grasping the target object based on the robotic arm control data; upon receiving confirmation from the user of the simulation preview, the target AR device generates control commands based on the robotic arm control data and sends the control commands to the robotic arm; the robotic arm receives and executes the control commands to move the target object to the target position in the target posture. This application utilizes augmented reality technology to enable users to intuitively manipulate the 3D segmented model, thereby generating control commands for the robotic arm, reducing the workload of coding and effectively improving the programming efficiency of the robotic arm.

[0061] Please refer to Figure 1. The human-computer collaborative assembly method based on low-code programming described in this embodiment of the application includes the following steps:

[0062] Step S100: Use the target AR device to collect the target scene containing the target object, so as to generate a three-dimensional segmentation model corresponding to the target object and display it on the device.

[0063] Specifically, the target object is a real-world object that requires manipulation using a robotic arm. For example, the robotic arm might grasp the target object and move it to a designated target location in a target pose. To illustrate, if a box needs to be moved using a robotic arm, the user wears a target AR device to collect data about the box and its surroundings, generating a 3D segmented model of the box, which is then displayed on the target AR device. Later, the user can use the target AR device to manipulate the 3D segmented model to generate control commands for the robotic arm in the real world to manipulate the target object.

[0064] In one implementation, the target AR device is AR glasses.

[0065] In one implementation, the target AR device is a Microsoft HoloLens 2.

[0066] In one implementation, the step of using a target AR device to acquire a target scene containing a target object, generating a 3D segmentation model corresponding to the target object, and displaying it on the local device includes:

[0067] The target AR device is used to collect the target scene containing the target object, and the RGB image of the target object, the camera pose when the target object was captured, and the initial point cloud of the target object are obtained.

[0068] The RGB image, the camera pose, and the initial point cloud are input into a trained 3D Gaussian sputtering model. The 3D Gaussian sputtering model is then used to process the data to obtain the initial 3D Gaussian distribution of the target object.

[0069] The RGB image is input into the All-Segmentation Model, and after processing by the All-Segmentation Model, a two-dimensional segmentation mask of the target object is obtained. The initial three-dimensional Gaussian distribution is then subjected to inverse rasterization using the two-dimensional segmentation mask to obtain the initial segmentation result of the target object.

[0070] The initial segmentation result is filtered using the KNN statistical filtering algorithm to obtain several three-dimensional Gaussian spheres. All three-dimensional Gaussian spheres are converted into point clouds to obtain the three-dimensional segmentation model corresponding to the target object, which is then displayed on this device.

[0071] Specifically, this application employs a combination of a 3D Gaussian Splatting model (3DGS) and a Segment Anything Model (SAM) to process data acquired by the target AR device. First, the target scene is acquired using a head-mounted AR device, with the target object at the center, resulting in an RGB image, camera pose, and a coarse initial point cloud. The RGB image, camera pose, and initial point cloud are then input into a trained 3D Gaussian Splatting model to obtain the initial 3D Gaussian distribution of the target object. That is, the 3D Gaussian sphere set G = {g1, g2, ..., g...} N} Where G represents the set of 3D Gaussian spheres, g1 is a 3D Gaussian sphere in the set of 3D Gaussian spheres, and N represents the number of 3D Gaussian spheres in the scene.

[0072] An RGB image is input into a segmentation model, which processes it to obtain a 2D segmentation mask for the target object. For example, a user wearing a target AR device can select the target object with a gesture. The AR device then acquires an RGB image of the camera pose, which, after processing by the segmentation model, yields a 2D segmentation mask. This 2D segmentation mask is then used to backtrack pixel-by-pixel to the initial 3D Gaussian distribution, i.e., a rasterized 3D Gaussian sphere, to obtain the initial segmentation result G. t However, relying solely on this segmentation result may result in omissions or redundant Gaussian balls. Therefore, it is necessary to use the KNN statistical filtering algorithm to calculate the segmentation result G. t In the middle, the nearest surrounding each Gaussian distribution The average distances of the Gaussian distributions are calculated. The mean μ and standard deviation σ of the average distances of all Gaussian distributions are then removed, and the new set G is obtained. t‘ It is a collection of 3D Gaussian spheres representing the segmented target object. The center positions of these Gaussian spheres are obtained and converted into point clouds to create a 3D segmentation model, which can then be displayed on the target AR device, coinciding with the target object.

[0073] In one implementation, the training steps of the three-dimensional Gaussian sputtering model include:

[0074] Construct an initial 3D Gaussian sputtering model;

[0075] Obtain a training dataset, which includes multiple training scenarios. Each training scenario consists of a training RGB image of the target training object, the training camera pose when the target training object is photographed, and the initial training point cloud of the target training object.

[0076] In each iteration of training, the training RGB image, the training camera pose, and the initial training point cloud in each training scene are input into the initial 3D Gaussian sputtering model, while the training RGB image in each training scene is input into the segmentation model.

[0077] The loss function is calculated based on the output of the initial three-dimensional Gaussian sputtering model and the output of the segmentation model, and the parameters of the initial three-dimensional Gaussian sputtering model are updated based on the loss function.

[0078] Repeated iterative training is performed until the loss function converges to a preset value, resulting in a trained three-dimensional Gaussian sputtering model.

[0079] Specifically, this application iteratively trains an initial 3D Gaussian sputtering model to obtain a 3D Gaussian sputtering model that conforms to the actual application scenario. During the training process, as shown in Figure 2, the training RGB image, training camera pose, and initial training point cloud are input into the initial 3D Gaussian sputtering model to obtain a set of 3D Gaussian spheres. An associated feature f is then attached to each 3D Gaussian sphere in the set. g ∈R. This feature is rasterized, and the loss function is calculated using the rasterized 2D rendering mask result and the 2D mask reference ground value output by the Segmentation All Model (SAM). During the calculation, using the training camera pose and parameters corresponding to each training RGB image I, the rasterized rendering result of each pixel p under the training camera pose is calculated, which can be expressed as: In the formula, N is the sequential set of Gaussian spheres covering the pixel, and f i For the i-th associated feature, α i Let α be the current scattering intensity of the Gaussian sphere. j It represents the scattering intensity of all Gaussian spheres preceding the current Gaussian sphere, arranged by pixel, where r represents the rendering identifier. Based on the rasterization rendering result of each pixel p, the rasterization features of the 2D mask rendering image can be obtained. The pixel-level sigmoid function σ is used to process it, resulting in the rendered 2D segmentation mask, which can be represented as follows:

[0080] The training RGB images are input into the segmentation model to obtain the two-dimensional segmentation mask reference ground truth, denoted as M = SAM(I,P), where I is the set of input RGB images. In either case, P is the coordinate on the RGB image converted from the spatial coordinates of the target object input by the human operator using gestures.

[0081] The loss function is defined as the two-dimensional segmentation mask rendering result P. MThe binary cross-entropy of the two-dimensional segmentation mask reference M extracted by the Segmentation of Everything Model (SAM) is expressed as:

[0082] In the formula, M I Let p represent the 2D segmentation mask of the target object in the current image I, where p represents each pixel, H represents the height of the current image I, W represents the width of the current image I, and M represents the height of the target object in the current image I. p P represents the 2D mask value (0 or 1) of that pixel. M,p P represents M The value at that pixel. Finally, the initial 3D Gaussian sputtering model is trained using the loss function until the loss function converges to the preset value, at which point the trained 3D Gaussian sputtering model is obtained.

[0083] Please refer to Figure 1. The human-computer collaborative assembly method based on low-code programming described in this embodiment of the application further includes the following steps:

[0084] Step S200: Obtain the initial pose information of the three-dimensional segmentation model and the gesture information of the user controlling the three-dimensional segmentation model; generate robotic arm control data based on the initial pose information and the gesture information; and generate a simulation preview of the robotic arm grasping the target object based on the robotic arm control data.

[0085] Specifically, after generating the 3D segmented model, users can wear a target AR device to manipulate the 3D segmented model, such as grasping target objects, adjusting the pose of target objects, and moving target objects. These user operations on the 3D segmented model will generate gesture information. This application will combine the initial pose information and gesture information of the 3D segmented model to generate robotic arm control data for controlling the robotic arm. At the same time, it can also generate a simulation preview of the robotic arm grasping the target object in the target AR device.

[0086] In this application, the 3D segmentation of the target object is achieved using a 3D Gaussian sputtering model and a segmentation model, resulting in a corresponding 3D segmentation model of the target object. The acquisition of user gesture information from these 3D segmentation models is also based on the coarse-grained human visual guidance module of this application. In the fine-grained robot autonomous perception module, pose calculation and path planning are performed, generating robotic arm control data. The digital twin simulation and control module of this application generates a simulation preview based on the robotic arm control data.

[0087] In one implementation, the gesture information includes target pose information, grasping strategy information, and path guidance information; the step of acquiring the initial pose information of the 3D segmentation model and the gesture information of the user manipulating the 3D segmentation model, and generating robotic arm control data based on the initial pose information and the gesture information, includes:

[0088] Obtain the initial pose information of the three-dimensional segmentation model;

[0089] Acquire the target pose information, grasping strategy information, and path guidance information of the user-controlled 3D segmentation model;

[0090] Based on the initial pose information and the target pose information, pose calculation is performed to transform the coordinates of the 3D segmentation model from the augmented reality coordinate system to the robotic arm coordinate system;

[0091] Based on the grasping strategy information and the path guidance information, robotic arm control data is generated in the robotic arm coordinate system. The robotic arm control data includes the motion angles of each joint of the robotic arm to reach the target pose, as well as the path planning for the robotic arm to grasp the target object.

[0092] Specifically, this application uses gesture input as the interaction method. When a user wears the target AR device, they can select an object by pinching their thumb and forefinger together. Users can select the 3D segmented model using either a single or double-handed pinch gesture. Figure 3 illustrates the operation of selecting a 3D model using a double-handed pinch gesture. After selecting the 3D segmented model, gripping point reference information is generated; this information belongs to the gripping strategy information. After selecting the 3D segmented model with a single hand, the user can rotate it using gestures to change the model's posture. Figures 4, 5, and 6 illustrate how the user adjusts the posture of the 3D segmented model using gestures. Additionally, the user can drag the 3D segmented model to the target location; this process generates path guidance information. Figure 7 illustrates the dragging effect.

[0093] Once the user's gestures interacting with the 3D segmented model are obtained, robotic arm control data can be generated by combining this information with the initial pose information of the 3D segmented model. The gesture information includes target pose information, grasping strategy information, and path guidance information. Initial pose information includes the initial coordinate system of the 3D object in the virtual augmented environment. Target pose information includes the target pose coordinate system of the 3D object in the virtual augmented environment. Grasping strategy information includes the grasping points for grasping the 3D segmented model. Path guidance information is the path the user takes to move the 3D segmented model to the target location.

[0094] In the process of pose calculation based on initial pose information and target pose information, the first step is to obtain the homogeneous transformation matrix preset by the target AR device. To set the coordinate system Ψ of the virtual robotic arm model in the target AR device base′ And the coordinate system Ψ of a real robotic arm base Alignment. Simultaneously, the target AR device can identify the initial coordinate system Ψ of the 3D segmentation model. org Homogeneous coordinate transformation matrix between augmented reality coordinate system and augmented reality coordinate system and the target pose coordinate system Ψ of the 3D segmentation model tar Homogeneous coordinate transformation moments between augmented reality and augmented reality coordinate systems Then use Calculate the homogeneous coordinate transformation matrix between the initial coordinate system of the 3D object and the coordinate system of the robotic arm, and use... Calculate the homogeneous coordinate transformation matrix between the target attitude coordinate system of the 3D object and the robot arm coordinate system.

[0095] After the pose calculation is completed, robotic arm control data is generated in the robotic arm coordinate system based on the grasping strategy information and the path guidance information.

[0096] In one implementation, generating robotic arm control data in the robotic arm coordinate system based on the grasping strategy information and the path guidance information includes:

[0097] The grasping strategy information is processed using a cyclic coordinate descent algorithm to obtain the motion angles of each joint of the robotic arm in the robotic arm coordinate system to reach the target pose.

[0098] Based on the path guidance information, path planning is performed to obtain the path planning for the robotic arm to grasp the target object in the robotic arm coordinate system.

[0099] Specifically, the Cyclic Coordinate Descent (CCDIK) algorithm based on inverse kinematics is used to process the user-provided gripping point information to calculate the joint motion angles required for the robotic arm to reach the target posture. Based on the path guidance information provided by the user, a path planning from the initial position to the target position is calculated. This application employs a pre-defined interpolation algorithm for path planning of the robotic arm. The input term of this interpolation algorithm is Ψ. org Ψ tar [P0,…,P] n-1 ] and Δt. Where, [P0,…,P n-1 [q0, ..., qt] represents n path points uniformly sampled on the reference path, and Δt represents the time interval between the previous frame and the current frame in the digital twin simulation and preview module, in seconds. The digital twin simulation and preview module in this application is used to generate a simulation preview based on the user's operations on the 3D segmentation model. The interpolation algorithm outputs the motion angles [q0, ..., qt] of each joint of the robotic arm at each path point. N ], and path planning data.

[0100] Please refer to Figure 1. The human-computer collaborative assembly method based on low-code programming described in this embodiment of the application further includes the following steps:

[0101] Step S300: After receiving confirmation information from the user regarding the simulation preview, the target AR device generates control commands based on the robotic arm control data and sends the control commands to the robotic arm.

[0102] Specifically, the digital twin simulation and control module includes a digital twin and preview module, as well as a physical execution module. The digital twin and preview module displays the robotic arm's movements in advance on the target AR device for user inspection. Users can confirm via voice or gesture control. Upon receiving confirmation, control commands are generated and sent to the robotic arm. This application utilizes digital twin technology to create a simulation preview of the robotic arm's operation on a target object, allowing for advance display of its motion trajectory and expected effects, enabling users to intuitively see the robotic arm's actual working state. In this way, users can identify and correct potential problems before actual operation, thereby improving work efficiency and safety.

[0103] Please refer to Figure 1. The human-computer collaborative assembly method based on low-code programming described in this embodiment of the application further includes the following steps:

[0104] Step S400: The robotic arm receives and executes the control command to move the target object to the target position in the target posture.

[0105] Specifically, after the robotic arm receives a control command, the physical execution module drives the robotic arm to execute the control command.

[0106] In one implementation, before the robotic arm executes the control command, it further includes:

[0107] Obtain the prerequisites corresponding to the control command and determine whether the robotic arm meets the prerequisites. The prerequisites are used to determine whether the robotic arm can execute the corresponding control command.

[0108] If the robotic arm meets the prerequisite, then it is determined that the robotic arm can execute the control command.

[0109] Specifically, before the robotic arm executes control commands, prerequisite conditions need to be determined. For example, if the control command is a grasping action, it is necessary to determine whether the robotic arm's gripper is open and not holding an object before grasping. Only when this prerequisite condition is met can the grasping action be executed. This application, through the determination of prerequisite conditions, ensures that the robotic arm is in a safe and executable state when executing control commands, avoiding the failure of control commands to be completed due to improper robotic arm status.

[0110] In one implementation, the method further includes:

[0111] While the robotic arm executes the control command, the actual state of the robotic arm is monitored in real time, and actual state timing data is generated.

[0112] Based on the simulation preview of the robotic arm grasping the target object, simulation state timing data is generated;

[0113] Calculate the difference between the actual state timing data and the simulated state timing data within each time step. If the difference within each time step is within a preset threshold, then the actual state of the robotic arm is determined to be consistent with the simulated state.

[0114] If the difference at any time step exceeds a preset threshold, it is determined that the actual state of the robotic arm is inconsistent with the simulation state.

[0115] Specifically, during the execution of control commands by the robotic arm, the target AR device continuously monitors the robotic arm's execution status. When the difference at any time step exceeds a preset threshold, it is considered that the actual state of the robotic arm is inconsistent with the simulation state, and a corresponding prompt mechanism is triggered. During the monitoring process, the target AR device and the robotic arm communicate using the TCP / IP communication protocol.

[0116] When executing control commands, the robotic arm operates based on device meta-language, relying on a series of smallest hardware units to perform actions, such as the opening and closing of grippers, data acquisition by force sensors, and joint movements of the robotic arm. In recent years, digital twin (DT) and augmented reality (AR) technologies have shown great promise in human-computer interaction applications. AR-based robotic arm programming blurs the lines between traditional online and offline programming, providing an immersive and natural interactive interface for human-machine collaborative assembly. Simultaneously, the integration of digital twins drives product assembly technology towards intelligent assembly with deep integration of virtual and real elements. However, current methods of programming robotic arms using AR technology primarily focus on low-level device meta-language execution, such as controlling basic robotic arm movements (translation and rotation) or gripper operations (opening and closing), resulting in poor accuracy and flexibility. This application utilizes a more advanced method of manipulating the 3D segmentation model of the target object through gestures to trigger the execution of the underlying device meta-language, achieving more natural and flexible control and making the robotic arm's execution more aligned with human thinking and habits.

[0117] Meanwhile, the robotic arm of this application is equipped with a force and torque sensor on the end effector, so that the robotic arm can automatically perform operations such as searching for the plane and searching for the aperture through force feedback when performing operations.

[0118] In one implementation, after the robotic arm executes the control command, it further includes:

[0119] Obtain the postcondition corresponding to the control command, and determine whether the robotic arm meets the postcondition after executing the control command. The postcondition is used to determine whether the robotic arm has completed the control command and is in the specified target state.

[0120] If the robotic arm meets the postcondition, it sends a task success message to the target AR device.

[0121] Specifically, after executing the instruction, a postcondition check is performed. For example, if the operation of grabbing and moving the target object is performed, the postcondition is that the gripper is released. This indicates that the task has been completed and the next operation can proceed.

[0122] Figure 8 shows the overall architecture diagram of this application. This application includes a coarse-grained human vision guidance module, a fine-grained robot autonomous perception module, and a digital twin simulation and control module. In the coarse-grained human vision guidance module, an AR-based spatial interaction interface enables 3D segmentation of the target object, allowing users to intuitively specify the target pose, grasping strategy, and reference path through gesture interaction. In the fine-grained robot autonomous perception module, pose calculation and path planning are performed. In the digital twin simulation and control module, a simulation preview is first generated using the digital twin simulation and preview module. After the user confirms the simulation preview, control commands are generated, and the physical execution module drives the robotic arm to execute. The simulation preview improves operational safety and drives the robotic arm to complete the assembly task. Prerequisite checks are performed before execution. During execution, execution is based on device meta-statements, with force feedback sensing and continuous digital twin monitoring. Post-execution checks are performed after execution. Through force feedback, pose calculation, and path planning, high-precision on-site adaptive motion execution can be achieved. This application employs a multi-granularity, low-code programming approach, significantly reducing the workload of programming while ensuring the robotic arm better understands human guidance, thus improving the efficiency and flexibility of human-robot collaborative assembly programming. It effectively addresses the limitations of existing online and offline programming methods in handling complex tasks, as well as the current limitation of AR programming focusing only on low-level device primitive-level control, achieving a better balance between intuitiveness, safety, and efficiency in robotic arm programming.

[0123] In one embodiment, as shown in FIG9, based on the above-described low-code programming-based human-machine collaborative assembly method, this application also provides a human-machine collaborative assembly system, including:

[0124] The model display module 100 is used to acquire a target scene containing the target object using the target AR device, generate a three-dimensional segmentation model of the target object, and display it on the device.

[0125] The simulation preview module 200 is used to acquire the initial pose information of the three-dimensional segmentation model and the gesture information of the user controlling the three-dimensional segmentation model, generate robotic arm control data based on the initial pose information and the gesture information, and generate a simulation preview of the robotic arm grasping the target object based on the robotic arm control data.

[0126] The instruction sending module 300 is used to, upon receiving confirmation information from the user regarding the simulation preview, generate control instructions based on the robotic arm control data and send the control instructions to the robotic arm.

[0127] The instruction execution module 400 is used for the robotic arm to receive and execute the control instructions to move the target object to the target position in the target posture.

[0128] It should be noted that the foregoing explanation of the human-machine collaborative assembly method embodiment based on low-code programming also applies to the human-machine collaborative assembly system of this embodiment, and will not be repeated here.

[0129] This application also provides a computer-readable storage medium storing a low-code programming-based human-computer collaborative assembly program. When the low-code programming-based human-computer collaborative assembly program is executed by a processor, it implements the steps of any low-code programming-based human-computer collaborative assembly method provided in this application.

[0130] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0131] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the above device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0132] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0133] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0134] In the embodiments provided in this application, it should be understood that the disclosed apparatus / terminal devices and methods can be implemented in other ways. For example, the apparatus / terminal device embodiments described above are merely illustrative. For instance, the division of modules or units described above is only a logical functional division, and in actual implementation, it can be divided in other ways. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0135] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions are not in essence a departure from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A human-computer collaborative assembly method based on low-code programming, characterized in that, The method includes: The target AR device is used to capture the target scene containing the target object, so as to generate a 3D segmentation model of the target object and display it on the device. The initial pose information of the three-dimensional segmentation model and the gesture information of the user controlling the three-dimensional segmentation model are obtained. Based on the initial pose information and the gesture information, robotic arm control data is generated, and a simulation preview of the robotic arm grasping the target object is generated based on the robotic arm control data. Upon receiving confirmation from the user regarding the simulation preview, the target AR device generates control commands based on the robotic arm control data and sends the control commands to the robotic arm. The robotic arm receives and executes the control commands to move the target object to the target position in the target posture.

2. The human-computer collaborative assembly method based on low-code programming according to claim 1, characterized in that, The step of using a target AR device to acquire a target scene containing the target object, generating a 3D segmentation model corresponding to the target object, and displaying it on the local device includes: The target AR device is used to collect the target scene containing the target object, and the RGB image of the target object, the camera pose when the target object was captured, and the initial point cloud of the target object are obtained. The RGB image, the camera pose, and the initial point cloud are input into a trained 3D Gaussian sputtering model. The 3D Gaussian sputtering model is then used to process the data to obtain the initial 3D Gaussian distribution of the target object. The RGB image is input into the All-Segmentation Model, and after processing by the All-Segmentation Model, a two-dimensional segmentation mask of the target object is obtained. The initial three-dimensional Gaussian distribution is then subjected to inverse rasterization using the two-dimensional segmentation mask to obtain the initial segmentation result of the target object. The initial segmentation result is filtered using the KNN statistical filtering algorithm to obtain several three-dimensional Gaussian spheres. All three-dimensional Gaussian spheres are converted into point clouds to obtain the three-dimensional segmentation model corresponding to the target object, which is then displayed on this device.

3. The human-computer collaborative assembly method based on low-code programming according to claim 2, characterized in that, The training steps for the three-dimensional Gaussian sputtering model include: Construct an initial 3D Gaussian sputtering model; Obtain a training dataset, which includes multiple training scenarios. Each training scenario consists of a training RGB image of the target training object, the training camera pose when the target training object is photographed, and the initial training point cloud of the target training object. In each iteration of training, the training RGB image, the training camera pose, and the initial training point cloud in each training scene are input into the initial 3D Gaussian sputtering model, while the training RGB image in each training scene is input into the segmentation model. The loss function is calculated based on the output of the initial three-dimensional Gaussian sputtering model and the output of the segmentation model, and the parameters of the initial three-dimensional Gaussian sputtering model are updated based on the loss function. Repeated iterative training is performed until the loss function converges to a preset value, resulting in a trained three-dimensional Gaussian sputtering model.

4. The human-computer collaborative assembly method based on low-code programming according to claim 1, characterized in that, The gesture information includes target pose information, grasping strategy information, and path guidance information; the process of acquiring the initial pose information of the 3D segmentation model and the user's gesture information in manipulating the 3D segmentation model, and generating robotic arm control data based on the initial pose information and the gesture information, includes: Obtain the initial pose information of the three-dimensional segmentation model; Acquire the target pose information, grasping strategy information, and path guidance information of the user-controlled 3D segmentation model; Based on the initial pose information and the target pose information, pose calculation is performed to transform the coordinates of the 3D segmentation model from the augmented reality coordinate system to the robotic arm coordinate system; Based on the grasping strategy information and the path guidance information, robotic arm control data is generated in the robotic arm coordinate system. The robotic arm control data includes the motion angles of each joint of the robotic arm to reach the target pose, as well as the path planning for the robotic arm to grasp the target object.

5. The human-computer collaborative assembly method based on low-code programming according to claim 4, characterized in that, The step of generating robotic arm control data in the robotic arm coordinate system based on the grasping strategy information and the path guidance information includes: The grasping strategy information is processed using a cyclic coordinate descent algorithm to obtain the motion angles of each joint of the robotic arm in the robotic arm coordinate system to reach the target pose. Based on the path guidance information, path planning is performed to obtain the path planning for the robotic arm to grasp the target object in the robotic arm coordinate system.

6. The human-computer collaborative assembly method based on low-code programming according to claim 1, characterized in that, Before the robotic arm executes the control command, it also includes: Obtain the prerequisites corresponding to the control command and determine whether the robotic arm meets the prerequisites. The prerequisites are used to determine whether the robotic arm can execute the corresponding control command. If the robotic arm meets the prerequisite, then it is determined that the robotic arm can execute the control command.

7. The human-computer collaborative assembly method based on low-code programming according to claim 1, characterized in that, The method further includes: While the robotic arm executes the control command, the actual state of the robotic arm is monitored in real time, and actual state timing data is generated. Based on the simulation preview of the robotic arm grasping the target object, simulation state timing data is generated; Calculate the difference between the actual state timing data and the simulated state timing data within each time step. If the difference within each time step is within a preset threshold, then the actual state of the robotic arm is determined to be consistent with the simulated state. If the difference at any time step exceeds a preset threshold, it is determined that the actual state of the robotic arm is inconsistent with the simulation state.

8. The human-computer collaborative assembly method based on low-code programming according to claim 1, characterized in that, After the robotic arm executes the control command, it also includes: Obtain the postcondition corresponding to the control command, and determine whether the robotic arm meets the postcondition after executing the control command. The postcondition is used to determine whether the robotic arm has completed the control command and is in the specified target state. If the robotic arm meets the postcondition, it sends a task success message to the target AR device.

9. A human-machine collaborative assembly system based on low-code programming, characterized in that, include: The model display module is used to acquire a target scene containing the target object using the target AR device, generate a 3D segmentation model of the target object, and display it on the device. The simulation preview module is used to acquire the initial pose information of the three-dimensional segmentation model and the gesture information of the user controlling the three-dimensional segmentation model, generate robotic arm control data based on the initial pose information and the gesture information, and generate a simulation preview of the robotic arm grasping the target object based on the robotic arm control data. The instruction sending module is used to, upon receiving confirmation information from the user regarding the simulation preview, have the target AR device generate control instructions based on the robotic arm control data and send the control instructions to the robotic arm. The instruction execution module is used for the robotic arm to receive and execute the control instructions to move the target object to the target position in the target posture.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a low-code programming-based human-computer collaborative assembly program, which can be executed to implement the steps of the low-code programming-based human-computer collaborative assembly method as described in any one of claims 1-8.

Images