Fastener assembling method and device and electronic equipment

By controlling the robot to pick up and align the fasteners, and continuously adjusting the assembly force using a predictive model, the problem of traditional robots' inability to accurately assemble fasteners of different specifications has been solved, achieving high-precision fastener assembly.

CN118977251APending Publication Date: 2024-11-19BEIJING GALBOT AI CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411388771.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-30
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

Traditional robots struggle to precisely adjust the force according to different fastener specifications, thus hindering the accurate assembly of fasteners.

Method used

By controlling the robot to pick up the target fastener and align it with the hole, the assembly status parameters are continuously acquired. The assembly force parameters are predicted using a predictive model, and the assembly force is adjusted to achieve precise assembly of the fastener.

Benefits of technology

This improved the robot's assembly accuracy for fasteners, avoided endless trial and error, ensured a continuous improvement in the fit between the fasteners and the holes, and successfully completed the assembly.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118977251A_ABST
    Figure CN118977251A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides a fastener assembly method and device and electronic equipment, and relates to the technical field of robotics.The method comprises the steps that a first robot is controlled to pick up a target fastener, the target fastener is aligned to a target hole site, and specification parameters of the target hole site are matched with specification parameters of the target fastener; in the assembling process, current assembling state parameters of the target fastener are continuously obtained; inputting the specification parameter of the target fastener and the current assembly state parameter into a prediction model to obtain a current assembly force parameter; and the first robot is controlled to assemble the target fastener into the target hole site according to the current assembly force parameter. According to the scheme, the assembling precision of the robot on the fastener can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of robotics, and in particular to a fastener assembly method, device and electronic equipment. Background Art

[0002] In some mechanical assembly scenarios, robots are often used to complete the assembly of fasteners, such as screws or bolts. However, fasteners come in a variety of specifications, and the force and force changes required to assemble fasteners of different specifications vary. It is difficult for traditional robots to accurately assemble fasteners. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a fastener assembly method, device and electronic equipment to improve the assembly accuracy of fasteners by robots. The specific technical solution is as follows:

[0004] In a first aspect, an embodiment of the present application provides a fastener assembly method, the method comprising:

[0005] Controlling the first robot to pick up a target fastener and align the target fastener with a target hole position, wherein specification parameters of the target hole position match specification parameters of the target fastener;

[0006] During the assembly process, continuously obtaining current assembly state parameters of the target fastener;

[0007] Inputting the specification parameters and current assembly state parameters of the target fastener into the prediction model to obtain current assembly force parameters;

[0008] The first robot is controlled to assemble the target fastener into the target hole according to the current assembly force parameter.

[0009] In some embodiments, the prediction model is trained using the following steps:

[0010] Acquire a real data set, wherein the real data set includes specification parameters, assembly state parameters, and assembly force parameters of a real fastener;

[0011] Inputting the real data set into a simulation model to obtain a simulation data set, wherein the simulation data set includes specification parameters, assembly state parameters, and assembly force parameters of the simulated fastener;

[0012] The prediction model is trained using the simulation data set, or the prediction model is trained using the simulation data set and the real data set.

[0013] In some embodiments, the step of obtaining a real data set includes:

[0014] Controlling the second robot to pick up a real fastener and align the real fastener with a real hole position, wherein specification parameters of the real hole position match specification parameters of the real fastener;

[0015] In the process of controlling the second robot to assemble the real fastener into the real hole, the assembly force parameters of the real fastener are continuously adjusted according to the activity freedom of the second robot, and the assembly state parameters of the real fastener are continuously obtained.

[0016] In some embodiments, the step of controlling the first robot to pick up the target fastener and aligning the target fastener with the target hole position includes:

[0017] Acquire images of target fasteners and target holes captured by an image acquisition device;

[0018] Recognize the image to obtain a first position of the target fastener and a second position of the target hole;

[0019] The first robot is controlled to pick up the target fastener at the first position, and align the target fastener with the target hole at the second position.

[0020] In some embodiments, the step of acquiring images of the target fastener and the target hole position acquired by the image acquisition device includes:

[0021] Acquire a first image of a target fastener and second images of a plurality of hole positions acquired by an image acquisition device;

[0022] Recognize the first image of the target fastener and the second image of the plurality of hole positions to obtain specification parameters of the target fastener and the specification parameters of the plurality of hole positions;

[0023] A target hole position whose specification parameters match the specification parameters of the target fastener is determined from the plurality of hole positions.

[0024] In some embodiments, the step of acquiring images of the target fastener and the target hole position acquired by the image acquisition device includes:

[0025] Acquire a first image of a plurality of fasteners and a second image of a target hole position acquired by an image acquisition device;

[0026] Recognize the first images of the plurality of fasteners and the second image of the target hole position to obtain specification parameters of the plurality of fasteners and the specification parameters of the target hole position;

[0027] A target fastener whose specification parameters match the specification parameters of the target hole position is determined from a plurality of fasteners.

[0028] In some embodiments, the step of acquiring first images of a plurality of fasteners and a second image of a target hole position acquired by an image acquisition device comprises:

[0029] Acquire a first image of a plurality of fasteners and a second image of a plurality of hole positions acquired by an image acquisition device;

[0030] Recognize the second images of the plurality of hole positions to obtain second positions of the plurality of hole positions;

[0031] A target hole position closest to the first robot is determined according to the second positions of the plurality of hole positions.

[0032] In some embodiments, the step of acquiring first images of a plurality of fasteners and a second image of a target hole position acquired by an image acquisition device comprises:

[0033] Acquire a first image of a plurality of fasteners and a second image of a plurality of hole positions acquired by an image acquisition device;

[0034] Recognize the second images of the plurality of hole positions to obtain second positions of the plurality of hole positions;

[0035] Connecting the plurality of hole positions according to the second positions of the plurality of hole positions to obtain an assembly path, wherein a starting point on the assembly path is a hole position closest to the first robot, and along the assembly path, a distance from a current hole position to a next hole position is less than a distance from the current hole position to other unreached hole positions;

[0036] Starting from the starting point on the assembly path, determining the current hole position on the assembly path as the target hole position;

[0037] When the target hole position is assembled, the next hole position on the assembly path is updated as the current hole position, and the step of determining the current hole position on the assembly path as the target hole position is re-executed.

[0038] In some embodiments, the step of controlling the first robot to pick up the target fastener and aligning the target fastener with the target hole position includes:

[0039] Controlling the first robot to pick up a target fastener and place the target fastener against a target hole;

[0040] The first robot is controlled to adjust the posture of the target fastener until it is detected that the target fastener moves a first distance toward the target hole position, or it is detected that a projection of the target fastener on the plane where the target hole position is located coincides with the target hole position.

[0041] In some embodiments, the current assembly state parameter includes a current contact position between the target fastener and the target hole position;

[0042] The step of inputting the specification parameters and current assembly state parameters of the target fastener into the prediction model to obtain the current assembly force parameters includes:

[0043] Continuously detecting a second distance between the current contact position and the last contact position; the last contact position is the last contact position input into the prediction model;

[0044] If the second distance is greater than the first distance threshold, the specification parameters of the target fastener and the current contact position are input into the prediction model to obtain the current assembly force parameters.

[0045] In some embodiments, the method further comprises:

[0046] If the second distance is less than or equal to the first distance threshold, the step of continuously detecting the second distance between the current contact position and the previous contact position is repeated.

[0047] In some embodiments, the step of controlling the first robot to assemble the target fastener into the target hole according to the current assembly force parameter includes:

[0048] Continuously detecting the resistance from the target hole; if the resistance is less than or equal to the resistance threshold, controlling the first robot to push the target fastener toward the target hole according to the current assembly force parameter; if the resistance is greater than the resistance threshold, controlling the first robot to stop pushing the target fastener toward the target hole according to the current assembly force parameter; or

[0049] Continuously detecting the resistance from the target hole; if the resistance is less than or equal to a resistance threshold, controlling the first robot to advance the target fastener toward the target hole according to the current assembly force parameter; if the resistance is greater than the resistance threshold, controlling the first robot to advance the target fastener toward the target hole by a third distance according to the current assembly force parameter; or

[0050] Continuously detecting a fourth distance of the target fastener advancing in the direction of the target hole position; if the fourth distance is less than the second distance threshold, controlling the first robot to advance the target fastener in the direction of the target hole position according to the current assembly force parameter; if the fourth distance is greater than or equal to the second distance threshold, controlling the first robot to stop advancing the target fastener in the direction of the target hole position according to the current assembly force parameter.

[0051] In some embodiments, the assembly force parameters include the magnitude of the assembly force, the law of change of the magnitude of the assembly force, the direction of the assembly force, and the law of change of the direction of the assembly force;

[0052] The assembly state parameters include the contact position between the fastener and the hole, the distance the fastener moves toward the hole, and the inclination angle of the fastener relative to the plane where the hole is located.

[0053] In a second aspect, an embodiment of the present application provides a fastener assembly device, the device comprising:

[0054] A first control module is used to control the first robot to pick up a target fastener and align the target fastener with a target hole position, wherein the specification parameters of the target hole position match the specification parameters of the target fastener;

[0055] An acquisition module, used for continuously acquiring current assembly state parameters of the target fastener during the assembly process;

[0056] A prediction module, used for inputting the specification parameters and current assembly state parameters of the target fastener into a prediction model to obtain current assembly force parameters;

[0057] The second control module is used to control the first robot to assemble the target fastener into the target hole according to the current assembly force parameter.

[0058] In a third aspect, an embodiment of the present application provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0059] Memory, used to store computer programs;

[0060] The processor is used to implement any of the above-mentioned method steps when executing the program stored in the memory.

[0061] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, any of the method steps described above is implemented.

[0062] The embodiment of the present application also provides a computer program product including instructions, which, when executed on a computer, enables the computer to execute any of the method steps described above.

[0063] Beneficial effects of the embodiments of the present application:

[0064] In the technical solution provided in the embodiment of the present application, the robot aligns the fasteners with matching specification parameters and the hole positions. Subsequently, the prediction model continuously uses the specification parameters and assembly state parameters of the fasteners to obtain the assembly force parameters of the fasteners, thereby enabling the robot to assemble the fasteners into the hole positions according to the assembly force parameters. In the embodiment of the present application, the prediction model assists the robot to continuously adjust the assembly force parameters. Each time the assembly force parameters are adjusted, the fit between the fasteners and the hole positions is improved. The assembly of the fasteners can be successfully completed without infinite trial and error, thereby improving the assembly accuracy of the robot for the fasteners.

[0065] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0067] Figure 1a This is the first schematic diagram of the assembly scenario;

[0068] Figure 1b This is the second schematic diagram of the assembly scene;

[0069] Figure 1c This is the third schematic diagram of the assembly scene;

[0070] Figure 2 A schematic diagram of a first process flow of a fastener assembly method provided in an embodiment of the present application;

[0071] Figure 3a A first schematic diagram of an assembly scenario provided in an embodiment of the present application;

[0072] Figure 3b for Figure 3a A schematic diagram for determining the target hole position in FIG.

[0073] Figure 4a A second schematic diagram of an assembly scenario provided in an embodiment of the present application;

[0074] Figure 4b for Figure 4a A first schematic diagram of determining the target hole position;

[0075] Figure 4c for Figure 4a A second schematic diagram of determining the target hole position;

[0076] Figure 4dfor Figure 4a A third schematic diagram for determining the target hole position;

[0077] Figure 5 A first schematic diagram of a screw aligning hole provided in an embodiment of the present application;

[0078] Figure 6 A second schematic diagram of the screw alignment hole provided in the embodiment of the present application;

[0079] Figure 7 A schematic diagram of a flow chart of a model training method provided in an embodiment of the present application;

[0080] Figure 8 A first schematic diagram of assembly force parameters during the assembly of a target fastener provided in an embodiment of the present application;

[0081] Fig. 9 A second schematic diagram of assembly force parameters during the assembly of a target fastener provided in an embodiment of the present application;

[0082] Fig.10 A schematic diagram showing the completion of the assembly of the target fastener provided in the embodiment of the present application;

[0083] Fig.11 A second schematic flow chart of the fastener assembly method provided in an embodiment of the present application;

[0084] Fig.12 A signaling interaction diagram of an assembly system provided in an embodiment of the present application;

[0085] Fig.13 A schematic diagram of a structure of a fastener assembly device provided in an embodiment of the present application;

[0086] Fig.14 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0087] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.

[0088] Fasteners can be screws, bolts, or pins. In some mechanical assembly scenarios, robots are often used to complete the assembly of fasteners to reduce labor costs, such as Figure 1a In the assembly scenario shown, the robot can assemble the pin 11 into the hole 12 to achieve the connection between the upper and lower components; Figure 1bIn the assembly scenario shown, the robot can assemble the pin 13 into the hole 14 to achieve the connection between the left and right devices; Figure 1c In the assembly scenario shown, the robot can assemble the screw 15 into the hole 16 .

[0089] At present, fasteners have various specifications. The assembly of fasteners of different specifications requires different forces and force changes. It is difficult for traditional robots to accurately assemble fasteners.

[0090] In order to improve the assembly accuracy of fasteners by robots, the present application embodiment provides a fastener assembly method, such as Figure 2 As shown, the method includes:

[0091] Step S201, controlling the first robot to pick up a target fastener and align the target fastener with a target hole position, wherein the specification parameters of the target hole position match the specification parameters of the target fastener;

[0092] Step S202, during the assembly process, continuously obtaining current assembly state parameters of the target fastener;

[0093] Step S203, inputting the specification parameters of the target fastener and the current assembly state parameters into the prediction model to obtain the current assembly force parameters;

[0094] Step S204, controlling the first robot to assemble the target fastener into the target hole according to the current assembly force parameter.

[0095] In the technical solution provided in the embodiment of the present application, the robot aligns the fasteners with matching specification parameters and the hole positions. Subsequently, the prediction model continuously uses the specification parameters and assembly state parameters of the fasteners to obtain the assembly force parameters of the fasteners, thereby enabling the robot to assemble the fasteners into the hole positions according to the assembly force parameters. In the embodiment of the present application, the prediction model assists the robot to continuously adjust the assembly force parameters. Each time the assembly force parameters are adjusted, the fit between the fasteners and the hole positions is improved. The assembly of the fasteners can be successfully completed without infinite trial and error, thereby improving the assembly accuracy of the robot for the fasteners.

[0096] The above-mentioned fastener assembly method can be applied to electronic devices with data processing functions such as servers, clusters, and personal computers. For ease of description, the following description is based on the electronic device as the execution subject, which does not serve as a limitation. The electronic device can be an independent physical machine or integrated into a robot, which is not limited.

[0097] In the above step S201, the first robot can be any robot that performs the fastener assembly task. The target hole position is any hole position that needs to be adapted to the fastener. The specification parameters of the hole position are adapted to the specification parameters of the fastener, and the specification parameters of the hole position and the specification parameters of the fastener may include but are not limited to the fastener size, cross-sectional shape, etc. When the fastener is a screw or a bolt, the hole position is a hole position provided with an internal thread, and the specification parameters of the hole position and the specification parameters of the fastener may also include thread density, thread spacing, length of the thread end, inclination angle of the thread, thread diameter, etc. The fastener is aligned with the hole position, indicating that the projection of the cross section of the fastener on the plane where the hole position is located coincides with the hole position or the deviation is less than a set threshold.

[0098] When performing assembly, the electronic device can obtain a first position of a target fastener and a second position of a target hole; control the end effector of the first robot to pick up the target fastener at the first position, move the end effector, and adjust the posture of the end effector to align the target fastener with the target hole at the second position.

[0099] Here, when the electronic device is an independent physical machine, the electronic device may send a control instruction to the first robot. The control instruction is used to instruct the first robot to pick up the fastener, and the control instruction may include the first position of the target fastener. The first robot controls its own end effector to pick up the target fastener according to the control instruction, and aligns the target fastener with the target hole at the second position. When the electronic device is integrated on the robot, the first robot may directly control its own end effector to pick up the target fastener at the first position, and align the target fastener with the target hole at the second position.

[0100] In the embodiments of the present application, the electronic device may pick up the target fastener in any of the following ways.

[0101] Mode a1, cooperate with the image acquisition device to pick up the target fastener and align with the target hole. The image acquisition device can be an independent physical machine, that is, the image acquisition device can be installed in the assembly scene to capture the image of the hole in the assembly scene (such as the first image) and the image of the fastener (such as the second image). The image acquisition device can also be integrated on the robot, that is, the image of the hole in the assembly scene (such as the first image) and the image of the fastener (such as the second image) are captured by the image acquisition device on the robot.

[0102] In the embodiment of the present application, the image acquisition device acquires images of the target fastener and the target hole. The electronic device acquires the images of the target fastener and the target hole acquired by the image acquisition device; recognizes the images of the target fastener and the target hole to obtain the first position of the target fastener and the second position of the target hole; controls the first robot to pick up the target fastener at the first position, and aligns the target fastener with the target hole at the second position. The target fastener and the target hole can be included in the same image or in different images, which is not limited.

[0103] Take the case where the target fastener is included in the first image and the target hole is included in the second image as an example. The electronic device acquires the first image of the target fastener and the second image of the target hole acquired by the image acquisition device; recognizes the first image of the target fastener and the second image of the target hole to obtain the first position of the target fastener and the second position of the target hole; and then controls the first robot to pick up the target fastener at the first position and align the target fastener with the target hole at the second position.

[0104] In the embodiment of the present application, the electronic device may determine the target fastener and the target hole position in any of the following ways.

[0105] In mode a11, the electronic device acquires a first image of a target fastener and second images of multiple hole positions acquired by an image acquisition device. The electronic device recognizes the first image of the target fastener and the second images of the multiple hole positions to obtain specification parameters of the target fastener and specification parameters of the multiple hole positions; and determines a target hole position whose specification parameters match the specification parameters of the target fastener from the multiple hole positions.

[0106] In mode a12, the electronic device acquires a first image of a plurality of fasteners and a second image of a target hole position acquired by an image acquisition device. The electronic device recognizes the first images of the plurality of fasteners and the second images of the target hole position to obtain specification parameters of the plurality of fasteners and specification parameters of the target hole position; and determines a target fastener whose specification parameters match the specification parameters of the target hole position from among the plurality of fasteners.

[0107] In an embodiment of the present application, the electronic device can obtain the second image of any hole position captured by the image acquisition device as the second image of the target hole position; the electronic device can also obtain the second images of multiple hole positions captured by the image acquisition device; and determine the second image of the target hole position based on the second images of the multiple hole positions.

[0108] For example, when the image acquisition device acquires the second images of multiple hole positions, the electronic device can identify the second images of the multiple hole positions to obtain the second positions of the multiple hole positions; according to the second positions of the multiple hole positions, the hole position closest to the first robot is determined as the target hole position. The electronic device preferentially controls the robot to assemble the nearest hole position, thereby improving assembly efficiency.

[0109] For example Figure 3a In the assembly scenario shown, robot A collects image 1 of hole 1 and image 2 of hole 2. Robot A recognizes image 1 and image 2 respectively to obtain the position of hole 1 and the position of hole 2. Based on the position of hole 1 and the position of hole 2, robot A can determine that the distance d1 between hole 1 and robot A is smaller than the distance d2 between hole 2 and robot A, and then control the end effector to pick up the fastener whose specification parameters match those of hole 1 and align it with hole 1, as shown in FIG. Figure 3b shown.

[0110] For example, when the image acquisition device acquires the second images of multiple hole positions, the electronic device identifies the second images of the multiple hole positions to obtain the second positions of the multiple hole positions; according to the second positions of the multiple hole positions, the multiple hole positions are connected to obtain an assembly path, the starting point on the assembly path is the hole position closest to the first robot, and along the assembly path, the distance from the current hole position to the next hole position is less than the distance from the current hole position to other unreached holes. Starting from the starting point on the assembly path, the electronic device determines the current hole position on the assembly path as the target hole position; then, the electronic device determines the target fastener whose specification parameters match the specification parameters of the target hole position; controls the first robot to pick up the target fastener and assembles the target fastener into the target hole position. When the target hole position is assembled, the electronic device updates the next hole position on the assembly path to the current hole position, and re-executes the step of determining the current hole position on the assembly path as the target hole position until the multiple hole positions on the assembly path are assembled. In the embodiment of the present application, the electronic device pre-plans the assembly path, and the robot can complete the assembly work of the hole position closest to the current position one by one according to the pre-planned assembly path.

[0111] For example Figure 4a In the assembly scene shown, robot A collects image 1 of hole 1, image 2 of hole 2, and image 3 of hole 3. Robot A recognizes image 1, image 2, and image 3 respectively to obtain the position of hole 1, hole 2, and hole 3. Based on the position of hole 1, hole 2, and hole 3, robot A connects multiple holes to obtain an assembly path, such as Figure 4aIn the assembly path of hole position 1→hole position 2→hole position 3, hole position 1 is closest to robot A. Robot A determines hole position 1 as the target hole position, controls the end effector to pick up the fastener whose specification parameters match those of hole position 1, and aligns it with hole position 1. Figure 4b After the fastener corresponding to hole position 1 is assembled, robot A determines hole position 2 as the target hole position, controls the end effector to pick up the fastener whose specification parameters match those of hole position 2, and aligns it with hole position 2, as shown in FIG. Figure 4c After the fastener corresponding to hole position 2 is assembled, robot A determines hole position 3 as the target hole position, controls the end effector to pick up the fastener whose specification parameters match those of hole position 3, and aligns it with hole position 3, as shown in FIG. Figure 4d shown.

[0112] In mode a13, the electronic device can obtain the first images of multiple fasteners and the second images of multiple hole positions captured by the image acquisition device, and display the first images of multiple fasteners and the second images of multiple hole positions on the human-computer interaction interface. The user selects the target fastener and the target hole position according to the image displayed on the human-computer interaction interface. The electronic device determines the target fastener and the target hole position selected by the user; then recognizes the first image of the target fastener and the second image of the target hole position to obtain the first position of the target fastener and the second position of the target hole position; controls the first robot to pick up the target fastener at the first position, and aligns the target fastener with the target hole position at the second position.

[0113] In the embodiments of the present application, the electronic device may also determine the target fastener and the target hole position in other ways, which are not limited to this.

[0114] After determining the target fastener and the target hole position, the electronic device controls the first robot to pick up the target fastener and align the target fastener with the target hole position.

[0115] Method a2: In accordance with user instructions, the target fastener is picked up and aligned with the target hole position.

[0116] In an embodiment of the present application, a server, a platform or other electronic device may send a user command to the electronic device, or the electronic device may provide a user interface (UI), and the user may input a user command to the electronic device through the UI. The user command may include a first position of the target fastener and a second position of the target hole. The user command may also include other information such as specification parameters and texture features of the target fastener, which is not limited. After receiving the user command, the electronic device may control the first robot to pick up the target fastener at the first position and align the target fastener with the target hole at the second position.

[0117] In the embodiment of the present application, the electronic device may also use other methods to control the first robot to pick up the target fastener.

[0118] For example, by combining the image acquisition device and the user command, the target fastener is picked up and the target hole is aligned. Specifically:

[0119] The electronic device receives a user command, and in the process of controlling the first robot to pick up the target fastener at the first position according to the user command, continuously acquires a first image of the target fastener acquired by the image acquisition device; based on the continuously acquired first image, adjusts the posture of the end effector of the first robot, and controls the first robot to continue to pick up the target fastener according to the adjusted posture, so as to correct the posture deviation of the end effector and achieve accurate picking up of the target fastener;

[0120] After picking up the target fastener, during the process of aligning with the target hole, the first image of the target fastener captured by the image acquisition device is continuously acquired; based on the continuously acquired first image, the posture of the end effector of the first robot is adjusted, and the first robot is controlled to continue to align the target fastener with the target hole according to the adjusted posture, so as to correct the posture deviation of the end effector and achieve accurate alignment with the target hole.

[0121] In some embodiments, in order to improve the accuracy of aligning the target fastener with the target hole, the electronic device may implement the above step S201 in any of the following ways.

[0122] Mode b1, using distance change to align the target hole position. Specifically, the electronic device controls the first robot to pick up the target fastener and place the target fastener against the target hole position; controls the first robot to adjust the posture of the target fastener until it is detected that the target fastener moves a first distance toward the target hole position.

[0123] In the embodiment of the present application, the electronic device controls the end effector of the first robot to pick up the target fastener, and then moves the end effector to place the target fastener against the target hole; the electronic device controls the first robot to adjust the posture of the end effector, thereby adjusting the posture of the target fastener. In the process of adjusting the posture of the target fastener, the electronic device can continuously detect the distance moved by the target fastener in the direction of the target hole. When it is detected that the target fastener moves a first distance in the direction of the target hole, it indicates that the alignment between the target fastener and the target hole is completed, and the electronic device stops controlling the first robot to adjust the posture of the end effector.

[0124] In the embodiment of the present application, a sensor (such as a first sensor) for detecting distance may be configured on the first robot or in the assembly scene. The first sensor may detect the contact position between the target fastener and the target hole position, and the electronic device determines the distance moved by the target fastener to the target hole position based on the contact position detected by the first sensor at two moments. In the embodiment of the present application, the movement of the target fastener to the target hole position may be understood as the movement of the target fastener in the direction of the target hole position.

[0125] For example, the robot controls the end effector to clamp the screw and insert the screw into the hole at an angle perpendicular to or nearly perpendicular to the plane where the hole is located. The first sensor detects the contact position between the screw and the hole as follows: Figure 5 The contact position a shown in (1) in the figure. At this time, the screw is located within the area where the hole is located, butts against the edge of the hole or some areas inside the hole, and is not aligned. The robot adjusts the posture of the end effector. As the posture of the end effector is adjusted, the posture of the screw clamped by the end effector is adjusted accordingly. When the first sensor detects the contact position between the screw and the hole as shown in Figure 5 When the contact position is b as shown in (2), the target fastener moves a small distance toward the target hole, that is, the distance between the contact position b and the contact position a, that is, the first distance, indicating that the alignment between the target fastener and the target hole is complete, and the robot A stops controlling and adjusting the posture of the end effector. Figure 5 Robots are omitted.

[0126] In the embodiment of the present application, a sensor for detecting distance (such as a second sensor) may be configured on the first robot or in the assembly scene. The second sensor may detect the distance between the target fastener and the target hole, and the electronic device determines the distance moved by the target fastener to the target hole based on the distance detected by the second sensor at two moments, such as calculating the difference in the distance detected at two moments as the distance moved by the target fastener to the target hole.

[0127] Method b2 uses fastener projection to align the target hole position.

[0128] In the embodiment of the present application, the electronic device controls the end effector of the first robot to pick up the target fastener, and then moves the end effector to place the target fastener against the target hole; controls the first robot to adjust the posture of the end effector, thereby adjusting the posture of the target fastener. In the process of adjusting the posture of the target fastener, the electronic device can continuously detect the projection of the target fastener on the plane where the target hole is located. When it is detected that the projection of the target fastener on the plane where the target hole is located coincides with the target hole, it means that the alignment of the target fastener and the target hole is completed, and the electronic device stops controlling the first robot to adjust the posture of the end effector.

[0129] In the embodiment of the present application, an image acquisition device may be configured on the first robot or in the assembly scene. The image acquisition device is installed above the target hole position and acquires a third image in a direction perpendicular to the target hole position. The electronic device recognizes the third image and determines whether the projection of the target fastener on the plane where the target hole position is located coincides with the target hole position.

[0130] For example, the robot controls the end effector to clamp the screw and insert the screw into the hole at an angle perpendicular to or nearly perpendicular to the plane where the hole is located. At this time, the projection of the target fastener on the plane where the target hole is located in the third image captured by the image acquisition device is as follows: Figure 6 As shown in (1). At this time, the screw is located within the area where the hole is located, butts against the edge of the hole or some areas inside the hole, and is not aligned. The robot adjusts the posture of the end effector. As the posture of the end effector is adjusted, the posture of the screw clamped by the end effector is adjusted accordingly. When the projection of the target fastener on the plane where the target hole is located in the third image captured by the image acquisition device is as shown in Figure 6 When (2) is shown in the figure, the projection of the target fastener on the plane where the target hole is located coincides with the target hole, indicating that the alignment between the target fastener and the target hole is completed, and the robot stops controlling and adjusting the posture of the end effector.

[0131] In the above step S202, the assembly state parameters of the fastener may include but are not limited to the contact position between the target fastener and the target hole, the distance the fastener moves toward the hole, the inclination angle of the fastener relative to the plane where the hole is located, etc. After the target fastener is aligned with the target hole, the electronic device starts to assemble the fastener. During the assembly process, the electronic device can continuously obtain the assembly state parameters of the target fastener at the current moment, that is, the current assembly state parameters.

[0132] In order to obtain the current assembly state parameters of the target fastener, multiple sensors can be installed in the robot or assembly scene, such as a first sensor for detecting the contact position, a second sensor for detecting the distance, a third sensor for detecting the angle, etc. The multiple sensors continuously detect the corresponding assembly state parameters and transmit the assembly state parameters to the electronic device.

[0133] In the above step S203, the prediction model is used to predict the assembly force parameters. The prediction model can be a convolutional neural network model, a deep neural network model or an artificial intelligence (AI) large model. The assembly force parameters can include but are not limited to the size of the assembly force, the law of change of the size of the assembly force, the direction of the assembly force, the law of change of the direction of the assembly force, etc.

[0134] After obtaining the current assembly state parameters of the target fastener each time, the electronic device can input the specification parameters and the current assembly state parameters of the target fastener as input data into the prediction model. The prediction model processes the input data and outputs the assembly force parameters at the current moment, namely, the current assembly force parameters.

[0135] To improve the prediction accuracy of the model, the electronic device can train the prediction model in advance.

[0136] In one example, an electronic device obtains a real data set, which includes specification parameters, assembly state parameters, and assembly force parameters of real fasteners; and uses the real data set to train a prediction model. Specifically, the electronic device uses the specification parameters and assembly state parameters of real fasteners as input data, outputs a prediction model, and obtains predicted assembly force parameters; determines the prediction loss based on the predicted assembly force parameters and the assembly force parameters of real fasteners; if the predicted loss is greater than or equal to the loss threshold, adjusts the network parameters of the prediction model, and re-executes the steps of using the specification parameters and assembly state parameters of real fasteners as input data, outputting the prediction model, and obtaining the predicted assembly force parameters; if the predicted loss is less than the loss threshold, ends the model training.

[0137] In order to ensure the prediction accuracy of the model, a large amount of data is needed to train the prediction model. However, if a large amount of real data is collected to train the prediction model, the labor cost is high and the data collection takes a long time. To overcome this problem, the present application embodiment provides a model training method, such as Figure 7 As shown, the following steps may be included:

[0138] Step S701, obtaining a real data set, where the real data set includes specification parameters, assembly state parameters, and assembly force parameters of real fasteners;

[0139] In the embodiment of the present application, for multiple sets of real fasteners and real hole positions, multiple postures and forces can be used to assemble the fasteners. The force is a vector, including magnitude and direction. For each set of real fasteners and real hole positions, multiple sensors collect assembly state parameters and assembly force parameters from the start of assembly to the completion of assembly to obtain a real data set.

[0140] In some embodiments, in order to save manpower, a robot can be used to obtain a real data set. For example, the electronic device can control the second robot to pick up a real fastener and align the real fastener with the real hole position, and the specification parameters of the real hole position match the specification parameters of the real fastener; in the process of controlling the second robot to assemble the real fastener into the real hole position, the assembly force parameters of the real fastener are continuously adjusted according to the activity freedom of the second robot, and the assembly state parameters of the real fastener are continuously obtained.

[0141] In the embodiment of the present application, the second robot can be any robot. The control of the second robot to pick up the real fastener and the method of aligning the real hole position can refer to the relevant description of the above step S201 part, which will not be repeated here. From the start of assembly to the completion of assembly, the electronic device can continuously adjust the posture of the end effector of the second robot according to the freedom of movement of the second robot, so as to achieve the purpose of continuously adjusting the assembly force parameters of the real fastener. A variety of sensors collect assembly state parameters from the start of assembly to the completion of assembly, so that the electronic device can continuously obtain the assembly state parameters of the real fastener. After the assembly is completed, the electronic device obtains a real data set.

[0142] The electronic device adjusts the posture of the end effector of the second robot according to the freedom of movement of the second robot, and can obtain assembly force parameters that meet the needs of subsequent application scenarios, avoiding the problem that the assembly force parameters output by the trained prediction model are difficult to be implemented by the robot due to the limitations of the rotation and holding force of the end effector. In addition, the electronic device continuously adjusts the assembly force parameters of the real fasteners and continuously obtains the assembly status parameters of the real fasteners, which can realize the comprehensive capture and collection of data and reduce the problem of insufficient training of the prediction model due to data omissions. In addition, the collection of data sets by robots reduces human participation and saves labor costs.

[0143] Step S702: inputting the real data set into the simulation model to obtain a simulation data set, wherein the simulation data set includes specification parameters, assembly state parameters, and assembly force parameters of the simulated fastener.

[0144] A simulation model is built in the electronic device, and the simulation model can build a simulation environment and obtain simulation data. After obtaining the real data set, the electronic device can input the real data such as specification parameters, assembly state parameters and assembly force parameters included in the real data set into the simulation model. In the simulation environment, the simulation model uses the real data set to simulate the assembly scene of the fastener and the hole position, generates simulation data of the simulated fastener, and obtains the simulation data set. The simulation data in the simulation data set is obtained using the real data set, which meets the data authenticity. Among them, the specification parameters of the simulated fastener can be the same as or different from the specification parameters of the real fastener.

[0145] Step S703: training a prediction model using a simulated data set and / or a real data set.

[0146] After obtaining the simulation data set, the electronic device can use any of the following data sets to train the prediction model.

[0147] Mode c1, using a simulation data set to train a prediction model. In this mode, the electronic device uses the simulation data set as an input data set, and inputs each data in the input data set into the prediction model to train the prediction model.

[0148] Mode c2, using a real data set to train a prediction model. In this mode, the electronic device uses the real data set as an input data set, and inputs each data in the input data set into the prediction model to train the prediction model.

[0149] Mode c3, using the simulation data set and the real data set to train the prediction model. In this mode, the electronic device uses the union of the simulation data set and the real data set as the input data set, and inputs each data in the input data set into the prediction model to train the prediction model.

[0150] Method c4, using part of the simulated data set and the real data set to train the prediction model. In this method, the electronic device extracts part of the data from the union of the simulated data set and the real data set, uses the extracted data as the input data set, and inputs each data in the input data set into the prediction model to train the prediction model.

[0151] In the above-mentioned methods c1 to c4, the process of the electronic device using various input data sets to train the prediction model can refer to the above-mentioned description of using the real data set to train the prediction model, which will not be repeated here.

[0152] In the embodiment of the present application, the electronic device for training the prediction model, the model for acquiring simulation data, and the electronic device for performing fastener assembly may be the same or different.

[0153] In the technical solution provided by the embodiment of the present application, the electronic device generates simulation data using the simulation model, and the simulation data is used to enrich the input data set of the training prediction model, thereby ensuring the prediction accuracy of the model. In addition, the simulation data is generated by the simulation model without manual participation, thus saving labor costs and reducing the time consumption of data collection.

[0154] In step S204, the current assembly force parameter can be understood as a force adjustment strategy. After obtaining the current assembly force parameter, the electronic device controls the end effector of the first robot to output force according to the current assembly force parameter to gradually assemble the target fastener into the target hole.

[0155] For example, the robot's end effector grips Figure 1a The pin 11 shown is aligned Figure 1aThe hole 12 shown. At time t1, the robot obtains the current assembly force parameters including: the force direction is perpendicular to the plane where the hole is located, the force direction is unchanged, the force magnitude is F1, and the force magnitude changes in accordance with δ / second (s). Then, at time t1, the robot controls the end effector to output a force perpendicular to the plane where the hole is located and magnitude F1, pushing the pin 11 into the hole 12, as shown in FIG. Figure 8 As time goes by, the robot increases the force output by the end effector according to the force change law of δ / second (s). For example, at time t2, the force becomes F2, and at time t2, the robot controls the end effector to output a force of F2 perpendicular to the plane where the hole is located, and continues to push the pin 11 into the hole 12. Figure 8 shown.

[0156] For example, the robot's end effector grips Figure 1c Screw 15 is aligned as shown Figure 1c The hole position 16 shown. At time t1, the robot obtains the current assembly force parameters including: the force direction is tangent to the circumscribed circle of the cross section of the screw 15 and parallel to the plane where the hole position is located, and the force direction changes in the clockwise direction by adjusting θ 。 / second (s), the force is F1, and the force changes according to δ / second (s). Then, at time t1, the robot controls the end effector to output a force of F1 that is tangent to the circumscribed circle of the cross section of the screw 15 and parallel to the plane where the hole is located, and rotates the screw 15 to tighten the screw 15 into the hole 16. Fig. 9 As time goes by, the robot increases the force output by the end effector according to the force change law of δ / s, and adjusts θ in the clockwise direction. 。 / second (s) force direction change law, adjust the force direction in the clockwise direction, such as at time t2 the force magnitude becomes F2, the force direction becomes Fig. 9 The direction shown, and at time t2 the robot controls the end effector to output as follows Fig. 9 The screw 15 is continuously rotated with a force of F2 in the direction shown to tighten the screw 15 into the hole 16 .

[0157] In the embodiment of the present application, the electronic device may use any of the following methods to perform the above step S204 to complete the fastener assembly.

[0158] In mode d1, the electronic device continuously detects the resistance from the target hole; if the resistance from the target hole is less than or equal to the resistance threshold, the first robot is controlled to push the target fastener toward the target hole according to the current assembly force parameter; if the resistance from the target hole is greater than the resistance threshold, the first robot is controlled to stop pushing the target fastener toward the target hole according to the current assembly force parameter.

[0159] In the embodiment of the present application, a sensor for detecting force (such as a fourth sensor) can be deployed in the robot or assembly scene. The fourth sensor continuously collects the resistance from the target hole; then the electronic device continuously obtains the resistance from the target hole collected by the fourth sensor, and compares the resistance from the target hole with the resistance threshold. If the resistance from the target hole is less than or equal to the resistance threshold, it means that the assembly of the target fastener is not completed, such as Figure 1c The screw 15 shown has not been tightened into the hole 16, and the electronic device controls the first robot to continue to push the target fastener toward the target hole. As the target fastener is pushed toward the target hole, the resistance from the target hole gradually increases, that is, the resistance from the target hole collected by the fourth sensor continuously obtained by the electronic device gradually increases. When the resistance from the target hole is greater than the resistance threshold, it means that the target fastener is assembled. Figure 1c The screw 15 shown has been tightened into the hole 16, and the electronic device controls the first robot to stop pushing the target fastener toward the target hole, thus ending the assembly work.

[0160] In mode d2, the electronic device continuously detects the resistance from the target hole; if the resistance is less than or equal to the resistance threshold, the first robot is controlled to push the target fastener toward the target hole according to the current assembly force parameter; if the resistance is greater than the resistance threshold, the first robot is controlled to push the target fastener toward the target hole by a third distance according to the current assembly force parameter.

[0161] In the embodiment of the present application, a fourth sensor for detecting force can be deployed in the robot or assembly scene. The fourth sensor continuously collects the resistance from the target hole; then the electronic device continuously obtains the resistance from the target hole collected by the fourth sensor, and compares the resistance from the target hole with the resistance threshold. If the resistance from the target hole is less than or equal to the resistance threshold, it means that the assembly of the target fastener is not completed, such as Figure 1c The screw 15 shown has not been tightened into the hole 16, and the electronic device controls the first robot to continue to push the target fastener in the direction of the target hole. As the target fastener is pushed in the direction of the target hole, the resistance from the target hole gradually increases, that is, the resistance from the target hole collected by the fourth sensor continuously acquired by the electronic device gradually increases. When the resistance from the target hole is greater than the resistance threshold, it means that the target fastener is about to be assembled, and the electronic device controls the first robot to continue to push the target fastener in the direction of the target hole. When it is detected that the target fastener is pushed forward by a small distance again (such as the third distance), it is considered that the target fastener is assembled. Figure 1c The screw 15 shown has been tightened into the hole 16, and the electronic device controls the first robot to stop pushing the target fastener toward the target hole, thus ending the assembly work.

[0162] For example, the third distance is d. The robot controls the end effector to output force F1 to rotate the screw 15 so as to screw the screw 15 into the hole 16. Fig.10 In the state shown in (1), the contact position between the screw 15 and the hole 16 is the contact position a. The robot controls the end effector to output force F2 to rotate the screw 15, so as to continue to screw the screw 15 into the hole 16. When the robot controls the end effector to screw the screw 15 into the hole 16 again by a distance d, as shown in FIG. Fig.10 In the state shown in (2), the contact position between the screw 15 and the hole 16 is the contact position b, and the distance between the contact position a and the contact position b is d. The robot is controlled to stop screwing the screw 15 into the hole 16, and the assembly work is completed.

[0163] The above force F1 and force F2 are both current assembly force parameters predicted by the robot using the prediction model and the current assembly state parameters continuously obtained. The force F2 can be less than the force F1, equal to the force F1, or greater than the force F1, and there is no limitation on this.

[0164] In mode d3, the electronic device continuously detects the fourth distance of the target fastener advancing toward the target hole; if the fourth distance is less than the second distance threshold, the first robot is controlled to advance the target fastener toward the target hole according to the current assembly force parameter; if the fourth distance is greater than or equal to the second distance threshold, the first robot is controlled to stop advancing the target fastener toward the target hole according to the current assembly force parameter. The second distance threshold can be set according to actual needs. For example, the second distance threshold can be the length of the fastener, such as Figure 1a The length L1 of the middle pin 11 in the installation direction, the fourth distance can also be the adaptation length of the fastener and the hole, such as Figure 1c The length L2 of the threaded end of the middle screw 15.

[0165] In an embodiment of the present application, a second sensor for detecting distance may be deployed in a robot or assembly scene. The second sensor may continuously collect the distance between the target fastener and the target hole position, and then the electronic device continuously obtains the distance collected by the second sensor, and determines the distance (such as the fourth distance) that the target fastener advances to the target hole position based on the distance collected by the second sensor and the distance between the target fastener and the target hole position when aligned with the target hole position, such as calculating the difference between the distance collected by the second sensor and the distance between the target fastener and the target hole position when aligned with the target hole position, and obtaining the fourth distance that the target fastener advances to the target hole position.

[0166] The electronic device compares the detected fourth distance of the target fastener advancing toward the target hole with the second distance threshold. If the fourth distance is less than the second distance threshold, it means that the assembly of the target fastener is not completed, and the electronic device controls the first robot to continue to advance the target fastener toward the target hole. As the target fastener advances toward the target hole, the distance collected by the second sensor continuously acquired by the electronic device gradually decreases, and the fourth distance of the target fastener advancing toward the target hole gradually increases. When the fourth distance is greater than or equal to the second distance threshold, it means that the assembly of the target fastener is completed, and the electronic device controls the first robot to stop advancing the target fastener toward the target hole, thereby ending this assembly work.

[0167] In the embodiment of the present application, the electronic device may also implement the above step S204 in other ways, which is not limited to this.

[0168] In some embodiments, Fig.11 As shown, a fastener assembly method is also provided, which may include the following steps:

[0169] Step S1101, controlling the first robot to pick up a target fastener and align the target fastener with a target hole position, wherein the specification parameters of the target hole position match the specification parameters of the target fastener; this is the same as the above step S201 and will not be described again here.

[0170] Step S1102, during the assembly process, continuously obtain the current assembly state parameters of the target fastener, the current assembly state parameters including the current contact position between the target fastener and the target hole; the same as the above step S202, no further description is given here.

[0171] Step S1103, continuously detecting a second distance between the current contact position and the last contact position; the last contact position is the contact position last input into the prediction model; if the second distance is greater than the first distance threshold, executing step S1104.

[0172] Step S1104, inputting the specification parameters of the target fastener and the current contact position into the prediction model to obtain the current assembly force parameters;

[0173] In the embodiment of the present application, the first distance threshold can be set according to actual needs.

[0174] The electronic device continuously obtains the contact position at the current moment, that is, the current contact position. After obtaining the current contact position, it detects the distance between the current contact position and the previous contact position, such as the second distance. The electronic device compares the second distance with the first distance threshold. When the second distance is greater than the first distance threshold, it means that the prediction condition is met. The electronic device inputs the specification parameters of the target fastener and the current contact position into the prediction model. The prediction model processes the input data and outputs new assembly force parameters as the current assembly force parameters. The assembly force parameters output at this time are assembly force parameters that are more in line with the current fastener assembly state, which facilitates the smooth assembly of the fastener into the hole.

[0175] In some embodiments, when the second distance is less than or equal to the first distance threshold, the electronic device may repeat step S1103 instead of step S1104, thereby saving computing resources of the device.

[0176] In some embodiments, a parameter adjustment strategy may be configured in the electronic device. When the second distance is less than or equal to the first distance threshold, the electronic device may also adjust the assembly force parameter output by the last prediction model according to the parameter adjustment strategy to obtain a new assembly force parameter as the current assembly force parameter, and then execute step S1105.

[0177] In the embodiment of the present application, when the second distance is less than or equal to the first distance threshold, the electronic device may no longer execute step S1104, thereby saving computing resources of the device.

[0178] Step S1105, controlling the first robot to assemble the target fastener into the target hole according to the current assembly force parameter, which is the same as the above step S204 and will not be described again.

[0179] Combine the following Fig.12 The signaling interaction diagram of the assembly system shown in the figure provides a detailed description of the fastener assembly method provided in the embodiment of the present application. Fig.12 In the assembly system, the image acquisition device, the end effector, various sensors and electronic devices can be integrated on the robot, such as Fig.12 As shown, it can also be deployed in an independent physical machine, which is not limited. The above-mentioned multiple sensors can be used to measure various assembly state parameters, such as the above-mentioned first sensor, second sensor, third sensor and fourth sensor. The specification parameters of the target fastener match the specification parameters of the target hole position. The process of the electronic device assembling the target fastener to the target hole position may include the following steps:

[0180] Step S1201: electronic device deployment prediction model. The prediction model can be trained using a simulation data set and a real data set. For details, see the above Figure 7 The relevant description of the part is not limited to this.

[0181] Step S1202: An image acquisition device acquires first images of a plurality of fasteners and a second image of a target hole position, and sends the first image and the second image to an electronic device.

[0182] Step S1203: the electronic device recognizes the first images of the plurality of fasteners and the second images of the target hole positions to obtain specification parameters of the plurality of fasteners and specification parameters of the target hole positions.

[0183] Step S1204: the electronic device determines, from the plurality of fasteners, a target fastener whose specification parameters match the specification parameters of the target hole position.

[0184] Step S1205 , the electronic device controls the end effector to pick up the target fastener and place the target fastener against the target hole.

[0185] Step S1206: the electronic device adjusts the posture of the end effector until the target fastener moves a first distance toward the target hole position, and then begins to perform assembly.

[0186] Step S1207: Multiple sensors continuously collect current assembly state parameters of the target fastener and send the current assembly state parameters to the electronic device. Then, the electronic device continuously receives the current assembly state parameters sent by multiple sensors.

[0187] In the embodiment of the present application, the execution order of step S1202 and step S1207 is not limited.

[0188] Step S1208, the electronic device detects whether the second distance between the current contact position and the previous contact position is greater than the first distance threshold; if so, execute step S1209; if not, repeat step S1208.

[0189] When the second distance between the previous contact position and the previous contact position is less than or equal to the first distance threshold, the electronic device continues to receive current assembly state parameters sent by multiple sensors, executes step S1208, skips step S1209, and executes step S1210.

[0190] Step S1209: the electronic device inputs the specification parameters of the target fastener and the current assembly state parameters into the prediction model to obtain the current assembly force parameters.

[0191] In step S1210, the electronic device continuously detects whether the resistance from the target hole position is less than or equal to the resistance threshold; if so, execute step S1211; if not, execute step S1212.

[0192] Step S1211: the electronic device controls the end effector to assemble the target fastener into the target hole according to the current assembly force parameter.

[0193] Step S1212, the electronic device controls the end effector to stop pushing the target fastener toward the target hole according to the current assembly force parameter. For example, the electronic device controls the end effector to release the target fastener.

[0194] In the technical solution provided by the embodiment of the present application, the model outputs the force adjustment strategy, that is, the current assembly force parameter, and the current assembly force parameter is updated differently to simulate human operation and assemble the fastener into the hole. In this solution, the electronic device can drive the robot to successfully complete the assembly of the fastener without infinite trial and error, thereby improving the assembly accuracy of the robot for the fastener.

[0195] Corresponding to the above-mentioned fastener assembly method, the embodiment of the present application also provides a fastener assembly device, such as Fig.13 As shown, the device comprises:

[0196] The first control module 1301 is used to control the first robot to pick up the target fastener and align the target fastener with a target hole position whose specification parameters match the specification parameters of the target fastener, and the specification parameters of the target hole position match the specification parameters of the target fastener;

[0197] An acquisition module 1302 is used to continuously acquire current assembly state parameters of the target fastener during the assembly process;

[0198] Prediction module 1303, used to input specification parameters of the target fastener and current assembly state parameters into the prediction model to obtain current assembly force parameters;

[0199] The second control module 1304 is used to control the first robot to assemble the target fastener into the target hole according to the current assembly force parameter.

[0200] In some embodiments, the fastener assembly device may further include a training module for training to obtain a prediction model, which may be specifically used for:

[0201] Acquire a real data set, where the real data set includes specification parameters, assembly state parameters, and assembly force parameters of real fasteners;

[0202] Inputting the real data set into the simulation model to obtain a simulation data set, wherein the simulation data set includes specification parameters, assembly state parameters, and assembly force parameters of the simulated fastener;

[0203] The prediction model is trained using a simulated data set, or the prediction model is trained using a simulated data set and a real data set.

[0204] In some embodiments, the training module may be specifically used to:

[0205] Controlling the second robot to pick up the real fastener and align the real fastener with the real hole position, wherein the specification parameters of the real hole position match the specification parameters of the real fastener;

[0206] In the process of controlling the second robot to assemble the real fastener into the real hole, the assembly force parameters of the real fastener are continuously adjusted according to the activity freedom of the second robot, and the assembly state parameters of the real fastener are continuously obtained.

[0207] In some embodiments, the first control module 1301 may be specifically used to:

[0208] Acquire images of target fasteners and target holes captured by an image acquisition device;

[0209] Recognize the image to obtain a first position of the target fastener and a second position of the target hole;

[0210] The first robot is controlled to pick up a target fastener at a first position, and align the target fastener with a target hole at a second position.

[0211] In some embodiments, the first control module 1301 may be specifically used to:

[0212] Acquire a first image of a target fastener and second images of a plurality of hole positions acquired by an image acquisition device;

[0213] Recognize the first image of the target fastener and the second image of the plurality of hole positions to obtain specification parameters of the target fastener and the specification parameters of the plurality of hole positions;

[0214] A target hole position having specification parameters matching the specification parameters of the target fastener is determined from the plurality of hole positions.

[0215] In some embodiments, the first control module 1301 may be specifically used to:

[0216] Acquire a first image of a plurality of fasteners and a second image of a target hole position acquired by an image acquisition device;

[0217] Recognize the first images of the plurality of fasteners and the second images of the target hole positions to obtain specification parameters of the plurality of fasteners and the specification parameters of the target hole positions;

[0218] A target fastener whose specification parameters match the specification parameters of the target hole position is determined from the plurality of fasteners.

[0219] In some embodiments, the first control module 1301 may be specifically used to:

[0220] Acquire a first image of a plurality of fasteners and a second image of a plurality of hole positions acquired by an image acquisition device;

[0221] Recognize the second images of the plurality of hole positions to obtain second positions of the plurality of hole positions;

[0222] A target hole position closest to the first robot is determined according to the second positions of the plurality of hole positions.

[0223] In some embodiments, the first control module 1301 may be specifically used to:

[0224] Acquire a first image of a plurality of fasteners and a second image of a plurality of hole positions acquired by an image acquisition device;

[0225] Recognize the second images of the plurality of hole positions to obtain second positions of the plurality of hole positions;

[0226] Connecting the plurality of hole positions according to the second positions of the plurality of hole positions to obtain an assembly path, wherein the starting point on the assembly path is the hole position closest to the first robot, and along the assembly path, the distance from the current hole position to the next hole position is less than the distance from the current hole position to other unreached hole positions;

[0227] Starting from the starting point on the assembly path, the current hole position on the assembly path is determined as the target hole position;

[0228] When the target hole position is assembled, the next hole position on the assembly path is updated as the current hole position, and the step of determining the current hole position on the assembly path as the target hole position is re-executed.

[0229] In some embodiments, the first control module 1301 may be specifically used to:

[0230] Controlling the first robot to pick up the target fastener and place the target fastener against the target hole;

[0231] The first robot is controlled to adjust the posture of the target fastener until it is detected that the target fastener moves a first distance toward the target hole position, or it is detected that the projection of the target fastener on the plane where the target hole position is located coincides with the target hole position.

[0232] In some embodiments, the current assembly state parameter includes a current contact position between the target fastener and the target hole position;

[0233] The prediction module 1303 may be specifically used for:

[0234] Continuously detecting a second distance between the current contact position and the last contact position; the last contact position is the last contact position input into the prediction model;

[0235] If the second distance is greater than the first distance threshold, the specification parameters of the target fastener and the current contact position are input into the prediction model to obtain the current assembly force parameters.

[0236] In some embodiments, the prediction module 1303 may also be used to:

[0237] If the second distance is less than or equal to the first distance threshold, the step of continuously detecting the second distance between the current contact position and the last contact position is repeated.

[0238] In some embodiments, the second control module 1304 may be specifically configured to:

[0239] Continuously detect the resistance from the target hole; if the resistance is less than or equal to the resistance threshold, control the first robot to push the target fastener toward the target hole according to the current assembly force parameter; if the resistance is greater than the resistance threshold, control the first robot to stop pushing the target fastener toward the target hole according to the current assembly force parameter; or

[0240] Continuously detecting the resistance from the target hole; if the resistance is less than or equal to the resistance threshold, controlling the first robot to push the target fastener toward the target hole according to the current assembly force parameter; if the resistance is greater than the resistance threshold, controlling the first robot to push the target fastener toward the target hole by a third distance according to the current assembly force parameter; or

[0241] Continuously detect a fourth distance of the target fastener advancing in the direction of the target hole; if the fourth distance is less than the second distance threshold, control the first robot to advance the target fastener in the direction of the target hole according to the current assembly force parameter; if the fourth distance is greater than or equal to the second distance threshold, control the first robot to stop advancing the target fastener in the direction of the target hole according to the current assembly force parameter.

[0242] In some embodiments, the assembly force parameters include the magnitude of the current assembly force, the law of change in the magnitude of the assembly force, the direction of the assembly force, and the law of change in the direction of the assembly force;

[0243] The assembly state parameters include the contact position between the fastener and the hole, the distance the fastener moves toward the hole, and the inclination angle of the fastener relative to the plane where the hole is located.

[0244] In the technical solution provided in the embodiment of the present application, the robot aligns the fasteners with matching specification parameters and the hole positions. Subsequently, the prediction model continuously uses the specification parameters and assembly state parameters of the fasteners to obtain the assembly force parameters of the fasteners, thereby enabling the robot to assemble the fasteners into the hole positions according to the assembly force parameters. In the embodiment of the present application, the prediction model assists the robot to continuously adjust the assembly force parameters. Each time the assembly force parameters are adjusted, the fit between the fasteners and the hole positions is improved. The assembly of the fasteners can be successfully completed without infinite trial and error, thereby improving the assembly accuracy of the robot for the fasteners.

[0245] Corresponding to the above-mentioned fastener assembly method, the embodiment of the present application also provides an electronic device, such as Fig.14 As shown, it includes a processor 1401 , a communication interface 1402 , a memory 1403 and a communication bus 1404 , wherein the processor 1401 , the communication interface 1402 , and the memory 1403 communicate with each other via the communication bus 1404 .

[0246] Memory 1403, used for storing computer programs;

[0247] The processor 1401 is used to implement the following steps when executing the program stored in the memory 1403:

[0248] Controlling the first robot to pick up a target fastener and align the target fastener with a target hole position, wherein specification parameters of the target hole position match specification parameters of the target fastener;

[0249] During the assembly process, the current assembly state parameters of the target fastener are continuously obtained;

[0250] Input specification parameters and current assembly state parameters of the target fastener into the prediction model to obtain current assembly force parameters;

[0251] The first robot is controlled to assemble the target fastener into the target hole according to the current assembly force parameter.

[0252] In the technical solution provided in the embodiment of the present application, the robot aligns the fasteners with matching specification parameters and the hole positions. Subsequently, the prediction model continuously uses the specification parameters and assembly state parameters of the fasteners to obtain the assembly force parameters of the fasteners, thereby enabling the robot to assemble the fasteners into the hole positions according to the assembly force parameters. In the embodiment of the present application, the prediction model assists the robot to continuously adjust the assembly force parameters. Each time the assembly force parameters are adjusted, the fit between the fasteners and the hole positions is improved. The assembly of the fasteners can be successfully completed without infinite trial and error, thereby improving the assembly accuracy of the robot for the fasteners.

[0253] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0254] The communication interface is used for communication between the above electronic device and other devices.

[0255] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0256] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0257] In another embodiment provided in the present application, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, any of the above-mentioned fastener assembly methods is implemented.

[0258] In another embodiment provided in the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any of the fastener assembly methods in the above embodiments.

[0259] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive Solid State Disk (SSD)), etc.

[0260] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0261] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, electronic device, storage medium, and program product embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0262] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. A fastener assembly method, characterized in that: The method comprises: Controlling the first robot to pick up a target fastener and align the target fastener with a target hole position, wherein specification parameters of the target hole position match specification parameters of the target fastener; During the assembly process, continuously obtaining current assembly state parameters of the target fastener; Inputting the specification parameters and current assembly state parameters of the target fastener into the prediction model to obtain current assembly force parameters; The first robot is controlled to assemble the target fastener into the target hole according to the current assembly force parameter.

2. The method according to claim 1, characterized in that The prediction model is trained using the following steps: Acquire a real data set, wherein the real data set includes specification parameters, assembly state parameters, and assembly force parameters of real fasteners; Inputting the real data set into a simulation model to obtain a simulation data set, wherein the simulation data set includes specification parameters, assembly state parameters, and assembly force parameters of the simulated fastener; Using the simulation data set to train the prediction model, or using the simulation data set and the real data set to train the prediction model; and / or, The step of obtaining a real data set includes: Controlling the second robot to pick up a real fastener and align the real fastener with a real hole position, wherein specification parameters of the real hole position match specification parameters of the real fastener; During the process of controlling the second robot to assemble the real fastener into the real hole, the assembly force parameters of the real fastener are continuously adjusted according to the second robot's degree of freedom, and the assembly state parameters of the real fastener are continuously obtained.

3. The method according to claim 1, characterized in that The step of controlling the first robot to pick up the target fastener and aligning the target fastener with the target hole position includes: Acquire an image of a target fastener and a target hole position captured by an image acquisition device; recognize the image to obtain a first position of the target fastener and a second position of the target hole position; control a first robot to pick up the target fastener at the first position and align the target fastener with the target hole position at the second position; and / or, The first robot is controlled to pick up a target fastener and place the target fastener against a target hole position; the first robot is controlled to adjust the posture of the target fastener until it is detected that the target fastener moves a first distance toward the target hole position, or it is detected that a projection of the target fastener on the plane where the target hole position is located coincides with the target hole position.

4. The method according to claim 3, characterized in that The step of acquiring images of the target fastener and the target hole position acquired by the image acquisition device includes: Acquire a first image of a target fastener and second images of a plurality of hole positions captured by an image acquisition device; identify the first image of the target fastener and the second images of the plurality of hole positions to obtain specification parameters of the target fastener and specification parameters of the plurality of hole positions; determine a target hole position from the plurality of hole positions whose specification parameters match the specification parameters of the target fastener; or, Acquire first images of a plurality of fasteners and a second image of a target hole position captured by an image capture device; identify the first images of the plurality of fasteners and the second image of the target hole position to obtain specification parameters of the plurality of fasteners and specification parameters of the target hole position; and determine, from the plurality of fasteners, a target fastener whose specification parameters match the specification parameters of the target hole position.

5. The method according to claim 4, characterized in that The step of acquiring first images of a plurality of fasteners and second images of target hole positions captured by an image capture device comprises: Acquire a plurality of first images of fasteners and a plurality of second images of hole positions captured by an image acquisition device; identify the second images of the plurality of hole positions to obtain second positions of the plurality of hole positions; determine a target hole position closest to the first robot based on the second positions of the plurality of hole positions; or, Acquire first images of multiple fasteners and second images of multiple hole positions captured by an image acquisition device; identify the second images of the multiple hole positions to obtain second positions of the multiple hole positions; connect the multiple hole positions according to the second positions of the multiple hole positions to obtain an assembly path, wherein the starting point on the assembly path is the hole position closest to the first robot, and along the assembly path, the distance from the current hole position to the next hole position is less than the distance from the current hole position to other unreached hole positions; starting from the starting point on the assembly path, determine the current hole position on the assembly path as the target hole position; when the assembly of the target hole position is completed, update the next hole position on the assembly path to the current hole position, and re-execute the step of determining the current hole position on the assembly path as the target hole position.

6. The method according to claim 1, wherein The current assembly state parameter includes a current contact position between the target fastener and the target hole position; The step of inputting the specification parameters and current assembly state parameters of the target fastener into the prediction model to obtain the current assembly force parameters includes: Continuously detecting a second distance between the current contact position and a previous contact position; the previous contact position being the contact position last input into the prediction model; If the second distance is greater than the first distance threshold, inputting the specification parameters of the target fastener and the current contact position into the prediction model to obtain the current assembly force parameter; And / or, the method further comprises: If the second distance is less than or equal to the first distance threshold, the step of continuously detecting the second distance between the current contact position and the previous contact position is repeated.

7. The method according to any one of claims 1 to 6, characterized in that The step of controlling the first robot to assemble the target fastener into the target hole according to the current assembly force parameter includes: Continuously detecting the resistance from the target hole; if the resistance is less than or equal to a resistance threshold, controlling the first robot to advance the target fastener toward the target hole according to the current assembly force parameter; if the resistance is greater than the resistance threshold, controlling the first robot to stop advancing the target fastener toward the target hole according to the current assembly force parameter; or Continuously detecting resistance from the target hole; if the resistance is less than or equal to a resistance threshold, controlling the first robot to advance the target fastener toward the target hole according to the current assembly force parameter; if the resistance is greater than the resistance threshold, controlling the first robot to advance the target fastener toward the target hole by a third distance according to the current assembly force parameter; or and continuously detecting a fourth distance that the target fastener is advanced toward the target hole position; if the fourth distance is less than a second distance threshold, controlling the first robot to advance the target fastener toward the target hole position according to a current assembly force parameter; and if the fourth distance is greater than or equal to the second distance threshold, controlling the first robot to stop advancing the target fastener toward the target hole position according to the current assembly force parameter.

8. The method according to any one of claims 1 to 6, characterized in that The assembly force parameters include the size of the assembly force, the law of change of the size of the assembly force, the direction of the assembly force, and the law of change of the direction of the assembly force; The assembly state parameters include the contact position between the fastener and the hole, the distance the fastener moves toward the hole, and the inclination angle of the fastener relative to the plane where the hole is located.

9. A fastener assembly device, characterized in that: The device comprises: a first control module, configured to control the first robot to pick up a target fastener and align the target fastener with a target hole position, wherein specification parameters of the target hole position match specification parameters of the target fastener; An acquisition module, configured to continuously acquire current assembly state parameters of the target fastener during the assembly process; A prediction module, configured to input specification parameters of the target fastener and current assembly state parameters into a prediction model to obtain current assembly force parameters; The second control module is used to control the first robot to assemble the target fastener into the target hole according to the current assembly force parameter.

10. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; Memory for storing computer programs; A processor, configured to implement the method steps described in any one of claims 1 to 8 when executing a program stored in a memory.