FPC automatic assembly control method, device and system, and electronic equipment

Abstract

The embodiments of the present application disclose a method, device and system for controlling the automated assembly of FPC, and an electronic device. The control method comprises: establishing communication connections with a lower computer and a 3D camera respectively; in response to an assembly request of a certain FPC from the lower computer, controlling the 3D camera to collect a point cloud image of the FPC before assembly, and displaying the point cloud image of the FPC before assembly on a display screen; calculating the grasping position of the FPC according to the point cloud image before assembly of the FPC, and returning it to the lower computer together with the assembly point position information of the FPC, so as to control the lower computer to guide the manipulator to complete the grasping and assembly of the FPC according to the grasping position of the FPC and the assembly point position information of the FPC. The scheme of the present application realizes 3D visualization in the assembly process, can eliminate the recognition deviation of the position of the FPC flexible cable on the motherboard, improve the position accuracy of the manipulator grasping the FPC, and reduce the number of grasping times.

Description

Technical Field

[0001] The present application belongs to the technical field of FPC automated assembly, and in particular, relates to an FPC automated assembly control method, device and system, and electronic equipment. Background Art

[0002] With the rise of the concept of the metaverse, the market demand for VR devices has also risen. In the manufacturing process of VR devices, the assembly of flexible printed circuits (FPCs) is crucial. As consumer electronics become increasingly lightweight and the functions of VR devices become increasingly rich, the number of FPCs on VR device motherboards is increasing, and they are also getting smaller and smaller. The difficulty and requirements of FPC assembly are also rising. Traditional manual assembly is not only difficult to quantify, but also difficult to ensure quality. Using robots to replace manual labor and realize FPC automated assembly is an excellent solution.

[0003] However, due to the large number of FPCs on the VR device motherboard and their small size, there is currently a deviation in the FPC soft cable position recognition during the FPC automated assembly process, and the robot is not accurate enough in grasping the FPC position, and sometimes multiple grasps are required. Summary of the invention

[0004] The present application provides an FPC automated assembly control method, device and system, and electronic equipment, which can overcome the problems of FPC soft cable position recognition deviation and inaccurate FPC position grasping.

[0005] According to a first aspect of the present application, there is provided a FPC automated assembly control method, comprising:

[0006] Establish communication connections with the lower computer and 3D camera respectively;

[0007] In response to an assembly request of a certain FPC by the lower computer, controlling the 3D camera to collect a point cloud image of the FPC before assembly, and displaying the point cloud image of the FPC before assembly on a display screen;

[0008] The grasping position of the FPC is calculated based on the point cloud image before the FPC is assembled, and returned to the lower computer together with the assembly point position information of the FPC, so as to control the lower computer to guide the robot arm to complete the grasping and assembly of the FPC according to the grasping position of the FPC and the assembly point position information of the FPC.

[0009] According to a second aspect of the present application, there is provided an FPC automated assembly control device, comprising:

[0010] A communication connection unit, used to establish communication connections with the lower computer and the 3D camera respectively;

[0011] The pre-assembly point cloud image acquisition and display unit is used to control the 3D camera to acquire the point cloud image of the FPC before assembly in response to the assembly request of the lower computer for a certain FPC, and display the point cloud image of the FPC before assembly on the display screen;

[0012] A grasping position calculation unit, used to calculate the grasping position of the FPC according to the point cloud image before the FPC is assembled;

[0013] The position data return unit is used to return the grasping position of the FPC together with the assembly point position information of the FPC to the lower computer, so as to control the lower computer to guide the robot to complete the grasping and assembly of the FPC according to the grasping position of the FPC and the assembly point position information of the FPC.

[0014] According to a third aspect of the present application, there is provided an FPC automated assembly control system, comprising an upper computer, a lower computer, a 3D camera, a 2D camera and a manipulator, wherein the upper computer establishes communication connections with the lower computer, the 3D camera and the 2D camera respectively, and the lower computer is communicatively connected to the manipulator;

[0015] The host computer is used to run the aforementioned FPC automated assembly control method;

[0016] The lower computer is used to send an assembly request for a certain FPC to the upper computer, and guide the robot to complete the grasping and assembly of the FPC according to the grasping position of the FPC and the assembly point position information of the FPC returned by the upper computer; and send a detection request for a certain FPC to the upper computer, and receive the assembly correct or incorrect result data returned by the upper computer.

[0017] According to a fourth aspect of the present application, there is provided an electronic device, comprising: a processor and a memory storing computer instructions;

[0018] The processor executes the aforementioned FPC automated assembly control method when running the computer instructions.

[0019] According to a fifth aspect of the present application, a computer-readable storage medium is provided, which stores one or more programs. When the one or more programs are executed by an electronic device, the electronic device executes the aforementioned FPC automated assembly control method.

[0020] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects:

[0021] During FPC assembly, the embodiment of the present application controls a 3D camera to collect a point cloud image before the FPC is assembled, displays the point cloud image before the FPC is assembled on a display screen, and calculates the grasping position of the FPC based on the point cloud image before the FPC is assembled, and then returns the grasping position of the FPC together with the assembly point position information of the FPC to the lower computer, so as to control the lower computer to guide the manipulator to complete the grasping and assembly of the FPC, thereby realizing 3D visualization in the assembly process, and being able to eliminate the recognition deviation of the position of the FPC flexible cable on the motherboard, and the grasping position of the FPC is calculated based on the 3D point cloud image, so the calculated grasping position of the FPC has 3D spatial information, which is convenient for controlling the lower computer to adjust the spatial grasping posture of the manipulator according to the FPC grasping position, thereby improving the position accuracy of the manipulator grasping the FPC and reducing the number of grasping times. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in this application, the following briefly introduces the drawings required for the description of the embodiments. Obviously, the drawings described below are only some embodiments recorded in this application. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings. In the entire drawings, the same reference symbols represent the same components.

[0023] Figure 1 A schematic diagram of a flow chart of an FPC automated assembly control method provided in an embodiment of the present application;

[0024] Figure 2 is a schematic diagram of a point cloud image before FPC assembly given in an embodiment of the present application;

[0025] Figure 3 It is a schematic diagram of a time axis curve of force control data given in the first preferred embodiment of the present application;

[0026] Figure 4 It is a schematic diagram of a point cloud image after FPC assembly given in the second preferred embodiment of the present application;

[0027] Figure 5 It is a flowchart of algorithm processing of a point cloud image input by a 3D camera according to the second preferred embodiment of the present application;

[0028] Figure 6 is a schematic diagram of a parameter configuration interface provided in the fourth preferred embodiment of the present application;

[0029] Figure 7 is a schematic diagram of directly performing parameter configuration according to the fourth preferred embodiment of the present application;

[0030] Figure 8 is a schematic diagram of visualization parameter configuration provided in the fourth preferred embodiment of the present application;

[0031] Fig. 9 is a schematic diagram of a teaching scenario provided in an embodiment of the present application;

[0032] Fig.10 Schematic diagram of the front and back of the FPC provided in the embodiment of the present application;

[0033] Fig.11 This is a schematic diagram of the respective permissions of the administrator and operator given by this application software;

[0034] Fig.12 It is a schematic diagram of the manual test interface provided by the application software;

[0035] Fig.13 It is a schematic diagram of the automatic test interface provided by the application software;

[0036] Fig.14 It is the operation flow chart of this application software;

[0037] Fig.15 It is a structural schematic diagram of an FPC automatic assembly control device provided in an embodiment of the present application;

[0038] Fig.16 It is a structural schematic diagram of an FPC automated assembly control system provided in an embodiment of the present application;

[0039] Fig.17 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0041] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0042] The FPC automated assembly control method of the embodiment of the present application is executed by a host computer. The host computer refers to a computer that can directly issue operation control commands, and the slave computer is a computer that directly controls the manipulator or terminal actuator and obtains its status. Usually, an industrial computer, a workstation, or an operating platform is used as a host computer; a programmable logic controller (PLC), a single-chip microcomputer, etc. is used as a slave computer.

[0043] Figure 1 A schematic diagram of a flow chart of an FPC automated assembly control method provided in an embodiment of the present application. Figure 1 As shown, the FPC automated assembly control method provided in the embodiment of the present application at least includes the following steps S110 to S130:

[0044] Step S110, establishing communication connections with the lower computer and the 3D camera respectively.

[0045] Before FPC automated assembly, the upper computer needs to send communication connection requests to the lower computer and 3D camera respectively. After receiving their connection responses, it needs to establish communication connections with the lower computer and 3D camera respectively, so as to send control instructions to the lower computer and 3D camera respectively during the assembly process, and receive the results of the lower computer and 3D camera executing the control instructions.

[0046] The communication connection established may be a wired connection or a wireless connection, or a wired or wireless connection may be established separately according to actual conditions. This application does not limit the communication connection method.

[0047] Step S120, in response to the lower computer's request for assembling a certain FPC, controlling the 3D camera to collect the point cloud image of the FPC before assembly, and displaying the point cloud image of the FPC before assembly on a display screen.

[0048] There are usually multiple FPCs to be assembled on the motherboard. When the lower computer sends a PLC assembly request to the upper computer, it needs to carry the information of a specific FPC. In order to distinguish the multiple FPCs on the motherboard, these multiple FPCs can be numbered or named, or the assembly order of these multiple FPCs can be determined in advance, and the specific FPC to be assembled each time can be determined according to the assembly order.

[0049] In response to the assembly request of a certain FPC from the lower computer, the upper computer controls the 3D camera to collect the point cloud image of the FPC before assembly. Specifically, the upper computer controls the 3D camera to move to the top of a specific FPC, determines the shooting posture of the 3D camera according to the pre-set parameters, and sends a shooting request of the assembly position to the 3D camera, and the shooting request carries the information of a specific FPC.

[0050] The 3D camera returns the point cloud data obtained by taking pictures to the host computer. These point cloud data represent a set of vectors for each sampling point on the surface of the object in a three-dimensional coordinate system. These vectors are usually expressed in the form of X, Y, and Z three-dimensional coordinates, and some may also contain color information (RGB) or reflection intensity information (Intensity).

[0051] At this time, the 3D camera inputs the point cloud image before FPC assembly to the host computer. Figure 2This is a schematic diagram of a point cloud image of an FPC before assembly given in an embodiment of the present application. The host computer displays the point cloud image of the FPC before assembly on the display screen of the host computer, realizing 3D visualization of the assembly process, facilitating users to observe the position and posture of the FPC before assembly in multiple dimensions, and eliminating the recognition deviation of the position of the FPC flexible cable on the motherboard.

[0052] Step S130, calculate the grasping position of the FPC based on the point cloud image before assembly of the FPC, and return it together with the assembly point position information of the FPC to the lower computer, so as to control the lower computer to guide the robot arm to complete the grasping and assembly of the FPC according to the grasping position of the FPC and the assembly point position information of the FPC.

[0053] When the host computer calculates the grabbing position of the FPC based on the point cloud image before FPC assembly, it needs to use some pre-configured parameters. For example, in order to determine the grabbing position of the FPC in the point cloud image, it is necessary to configure the FPC grabbing range box parameters in advance. This parameter is used to make a rough judgment and cut on the point cloud image to facilitate the calculation of subsequent algorithms.

[0054] When calculating the grasping position of the FPC based on the point cloud image before the FPC is assembled, the host computer uses a grasping position detection algorithm, which involves three levels of three-dimensional image processing: the low level includes basic operations such as image enhancement, filtering, key point / edge detection, etc.; the middle level includes operations such as connected domain labeling (label), image segmentation, etc.; the high level includes operations such as object recognition and scene analysis. The grasping position detection algorithm of this application requires the use of multiple levels of image processing methods, and its specific implementation is not the focus of this application. It can be implemented using a variety of existing mature algorithms.

[0055] In order to realize the assembly of FPC by the manipulator, the position information of the assembly point of FPC is also required. The upper computer returns the position information of the assembly point of FPC and the calculated grasping position of FPC to the lower computer, so as to control the lower computer to guide the manipulator to complete the grasping and assembly of FPC according to these two position information.

[0056] In summary, the FPC automated assembly control method of the embodiment of the present application displays the point cloud image before the FPC assembly on the display screen, realizes 3D visualization during the assembly process, can eliminate the recognition deviation of the position of the FPC soft cable on the motherboard, and the grasping position of the FPC is calculated based on the 3D point cloud image, so the calculated FPC grasping position has 3D spatial information, which is convenient for the lower computer to adjust the spatial grasping posture of the manipulator according to the FPC grasping position, thereby improving the position accuracy of the manipulator grasping the FPC and reducing the number of grasping times.

[0057] In a first preferred embodiment, the FPC automated assembly control method of the present application further includes:

[0058] In the process of controlling the lower computer to guide the manipulator to complete the grasping and assembly of the FPC, the force control data of the pressure sensor arranged on the manipulator is obtained, and the time axis curve of the force control data is synchronously displayed on the display screen.

[0059] Figure 3 This is a schematic diagram of a time axis curve of force control data given in the first preferred embodiment of the present application. By synchronously displaying the time axis curve of the force control data on the display screen, a 2D visualization of the force control data of the manipulator grabbing the FPC during FPC assembly is achieved, which is helpful for users to analyze the specific behavior of the manipulator.

[0060] In a second preferred embodiment, the FPC automated assembly control method of the present application further includes:

[0061] In response to a detection request of a certain FPC from a lower computer, a 3D camera is controlled to collect a point cloud image of the FPC after assembly, and the point cloud image of the FPC after assembly is displayed on a display screen;

[0062] Calculating the current posture of the FPC according to the point cloud image after the FPC is assembled, and comparing the current posture of the FPC with a pre-configured normal posture range of the FPC to determine whether the current posture of the FPC is normal;

[0063] When it is determined that the current posture of the FPC is normal, further combining the force control data of the manipulator in the process of grasping and assembling the FPC to determine whether the FPC is assembled in place;

[0064] When there is a sudden change in force in the force control data, it is determined that the FPC is assembled in place, and correct assembly result data is returned to the subordinate computer, otherwise, incorrect assembly result data is returned to the subordinate computer.

[0065] Figure 4 This is a schematic diagram of a point cloud image after FPC assembly given in the second preferred embodiment of the present application. When performing FPC assembly result detection, the host computer displays the point cloud image of the FPC after assembly collected by the 3D camera on the display screen, realizing 3D visualization of the assembly result, and at the same time, it can also eliminate the recognition deviation of the FPC flexible cable position on the motherboard.

[0066] When calculating the current posture of the FPC based on the point cloud image after FPC assembly, some pre-configured parameters are also needed. For example, in order to determine the assembly position of the FPC in the point cloud image, it is necessary to configure the FPC end detection range parameter in advance. This parameter is used to make a rough judgment and cut on the point cloud image to facilitate the calculation of subsequent algorithms.

[0067] Figure 5 FIG. 1 is a flow chart of the algorithm processing of the point cloud image input by the 3D camera according to the second preferred embodiment of the present application. The entire algorithm processing is executed by the host computer, such as Figure 5 As shown, the following steps S510 to S560 are included:

[0068] Step S510: receiving a point cloud image input by a 3D camera.

[0069] Step S520, determine whether the current state is capture or detection. If it is capture, proceed to step S530, if it is detection, proceed to step S540.

[0070] Step S530, using a grasping position detection algorithm to calculate the grasping position of the FPC, and outputting the calculation result of the FPC grasping position to the lower computer.

[0071] Step S540, using the posture analysis algorithm in the assembly result detection algorithm to first calculate the current posture of the FPC, and determine whether the calculated current posture of the FPC is normal;

[0072] The judgment method can be: compare the calculated current posture of the FPC with the pre-set normal posture range of the FPC. If the error is within the result judgment threshold, the current posture of the FPC is judged to be normal, and enter step S550. If the error is outside the result judgment threshold, the current posture of the FPC is judged to be abnormal, and the detection result of the abnormal current posture of the FPC is output to the lower computer.

[0073] Step S550, acquiring force control data of the pressure sensor on the manipulator, and proceeding to step S560.

[0074] Step S560, using the pressure data analysis algorithm in the assembly result detection algorithm, analyze whether there is a sudden change in force in the force control data. If there is a sudden change in force in the force control data, it is determined that the FPC is assembled in place, and the detection result of the correct FPC assembly is output to the subordinate computer; if there is no sudden change in force in the force control data, it is determined that the FPC is not assembled in place, and the detection result of the FPC not being assembled in place is output to the subordinate computer.

[0075] The assembly result detection algorithm used in the second preferred embodiment of the present application, in addition to involving the processing of 3D point cloud data, also needs to be further analyzed in conjunction with the force control data of the pressure sensor on the manipulator. Only when it is determined that the current posture of the FPC is normal and the FPC is in place, will the detection result of the correct FPC assembly be returned to the subordinate computer; when it is determined that the current posture of the FPC is abnormal or the FPC is not assembled in place, the upper computer will return the detection result of the FPC assembly error to the subordinate computer. Of course, the specific cause of the error can also be given in the returned detection result of the FPC assembly error; thereby ensuring the accuracy of the upper computer's detection of the FPC assembly result from the two aspects of normal posture and in-place assembly.

[0076] It should be noted that in the FPC automated assembly process,

[0077] In a third preferred embodiment, the FPC automated assembly control method of the present application further includes:

[0078] Establish a communication connection with the 2D camera;

[0079] After the mainboard is placed on the storage table, the 2D camera is controlled to collect a 2D image of the mainboard;

[0080] Calculating the current position of a pre-calibrated marking point on the mainboard according to the 2D image of the mainboard;

[0081] Determine the deviation of the placement position of the mainboard according to the pre-calibrated mark point position on the mainboard and the calculated current position; and

[0082] According to the deviation of the placement position of the mainboard, the pre-designed assembly point position of each FPC on the mainboard is corrected, and the position information of the assembly point of each FPC after the correction is determined.

[0083] The “returning to the lower computer together with the assembly point position information of the FPC” in the above step S130 specifically means: returning to the lower computer together with the assembly point position information of the FPC after correction.

[0084] The third preferred embodiment of the present application involves a 2D vision algorithm. The 2D vision algorithm plays a very important role in correcting the position during the teaching process and the actual use process. The main logic of the 2D vision algorithm is to take a picture of the mainboard with a 2D camera after the mainboard is fixed on the storage table, and then identify the pre-calibrated marking point (Mark point) through the 2D vision algorithm. Although the position of the Mark point on the mainboard is fixed, there will still be some errors when the mainboard is fixed on the storage table, so the calculated Mark point position is corrected according to the pre-calibrated Mark point position to eliminate the above-mentioned error of the Mark point.

[0085] The third preferred embodiment of the present application utilizes the correction logic of the 2D vision algorithm. After the mainboard is placed on the storage table, the upper computer corrects the assembly point position of each FPC on the mainboard. The correction calculation process is: first determine the deviation of the mainboard placement position based on the pre-calibrated Mark point position and the calculated Mark point position on the mainboard; then, based on the deviation of the mainboard placement position, correct the pre-designed assembly point position of each FPC on the mainboard to determine the position information of the assembly point after correction of each FPC. Afterwards, upon receiving the assembly request of a certain FPC from the lower computer, the upper computer returns the assembly point position information of the corresponding FPC after correction to the lower computer, so as to control the lower computer to guide the robot to complete the assembly of the FPC according to the assembly point position information of the FPC after correction.

[0086] Obviously, the third preferred embodiment of the present application eliminates the deviation of the mainboard placement position by returning the assembly point position information of the FPC after correction to the lower computer, which can further improve the position accuracy of the robot grasping the FPC and reduce the number of grasping times.

[0087] In a fourth preferred embodiment, the FPC automated assembly control method of the present application further includes:

[0088] Determine all hyperparameters to complete the automated assembly of each FPC;

[0089] All the hyperparameter items are designed in the same parameter table, and the parameter table is displayed in a parameter configuration interface, wherein each hyperparameter in the parameter table can be directly configured in the parameter configuration interface, and when some hyperparameter items with spatial significance are configured, point cloud images are used in the parameter configuration interface to intuitively display the parameter configuration results.

[0090] In order to complete the FPC flexible cable automated assembly task, in addition to some key parameters that need to be calculated through the algorithm, there are also some hyperparameters that need to be configured in advance. The configuration of these hyperparameters directly affects the accuracy and ease of use of the automated assembly process.

[0091] The fourth preferred embodiment of the present application designs a parameter table, which includes all the hyperparameters that need to be configured in advance to complete each FPC automated assembly. Table 1 is a parameter table designed in the fourth preferred embodiment of the present application:

[0092] Table 1 Parameters

[0093]

[0094]

[0095] As can be seen from Table 1 above, even the hyperparameters are very complex. In order to facilitate debugging and make the software have good portability and ease of use, the fourth preferred embodiment of the present application designs a set of parameter setting functions, which can display the above parameter table in the parameter configuration interface. Figure 6 It is a schematic diagram of the parameter configuration interface given in the fourth preferred embodiment of the present application. Multiple FPCs share the same parameter configuration interface, and each FPC corresponds to a column in the parameter configuration interface.

[0096] Each hyperparameter in the parameter table can be directly configured in the parameter configuration interface. Figure 7 This is a schematic diagram of directly configuring parameters according to the fourth preferred embodiment of the present application. In the parameter configuration interface, double-click the corresponding parameter item in the parameter table to directly configure or fine-tune.

[0097] Some parameter values ​​in the parameter table are difficult to understand intuitively, such as the sixth item in Table 1: Capture - Range box in image, which indicates the specific range of FPC in the point cloud. The fourth preferred embodiment of the present application designs a visual setting method. When configuring some hyperparameter items with spatial significance, point cloud images are also used in the parameter configuration interface to intuitively display the parameter configuration results. Figure 8 Schematic diagram of visualization parameter configuration given in the fourth preferred embodiment of the present application. Figure 8 As shown in the figure, the actual meaning of the parameters is directly represented in the point cloud image, so that the parameters with spatial significance can be displayed intuitively, thereby reducing the difficulty of using the software and improving the user experience by visualizing the parameter configuration process.

[0098] It should be noted that the configuration parameters of some hyperparameter items in the parameter table of Table 1 above can be determined or adjusted through the teaching process of the teach pendant. Fig. 9 The teaching process is to manually control the manipulator through the teaching pendant, move it to a suitable position, and record the manipulator position information and FPC-related spatial information.

[0099] The specific teaching process is as follows:

[0100] 901 Use the teaching pendant to move the 2D camera to the top of the FPC;

[0101] 902 controls the 2D camera to shoot the FPC slot position, and records the current position as the shooting position, and determines the "5 Grasping - Camera Shooting Posture" parameter in Table 1 according to the shooting position;

[0102] 903, the front side of the FPC cable at the current position is facing upward, and the 2D camera is controlled to take a photo to obtain a point cloud image; Fig.10 Schematic diagram of the front and back of the FPC provided in the embodiment of the present application;

[0103] 904 Through the parameter configuration function provided in the fourth preferred embodiment of the present application, the FPC end position is selected in the point cloud, and the "6 Capture--Range Box in Image" parameter in Table 1 is determined according to the current position of the box, and the specific information of the FPC end reinforcement plate is recorded as the "2 End Reinforcement Plate Size" parameter in Table 1;

[0104] 905 turns the FPC over, with the reverse side facing upward, controls the 2D camera to take a photo, and calculates the offset from the center position of the connector to the end of the FPC as the parameter of "7 Grasping--Nose Center Offset Distance" in Table 1;

[0105] 906 manually fastens the FPC to the mainboard, moves the 2D camera to the top of the FPC through the teaching pendant, and takes a photo to obtain a point cloud image;

[0106] 907 Through the parameter configuration function provided in the fourth preferred embodiment of the present application, the FPC end position is selected in the point cloud, and the "10 Detection--End Range in Image" parameter in Table 1 is determined according to the current position of the box, and the "9 Detection--Camera Photo Posture" parameter in Table 1 is determined according to the current position of the 2D camera;

[0107] 908 Move the 2D camera to the next FPC position through the teaching pendant and repeat the above steps 901 to 907.

[0108] It should also be noted that the software of this application is designed based on the FPC automated assembly control method provided in the above-mentioned embodiments of this application, and the software of this application runs in a host computer.

[0109] When the software of this application controls the FPC automated assembly process, there may be situations where the spatial point cloud data cannot meet the conditions during grabbing or detection. One situation may be caused by unreasonable configuration of super three. For example, if the parameter configuration of "6 grabbing--range box in the image" in Table 1 is unreasonable, it will cause part of the point cloud image displayed on the display screen to be missing; for example, if the parameter configuration of "10 detection--end range in the image" in Table 1 is unreasonable, the end of the FPC will not be within the display range box; another situation may be that the point cloud image collected by the 3D camera does not meet the minimum requirements of the algorithm. The software of this application takes this issue into consideration and will provide eye-catching prompts for various situations during the software operation process.

[0110] It should also be noted that the software in this application provides two different permissions for administrators and operators. The administrator's permissions include: parameter configuration, manual testing and automatic testing, and the operator's permissions are limited to automatic testing. Fig.11 This is a schematic diagram of the respective permissions for administrators and operators provided by this application software.

[0111] Corresponding to manual testing and automatic testing, this application software has designed two different UI interfaces for front-end display, and also controls which working mode to use in the back-end logic. The manual testing mode requires the administrator to actively send commands to run the work process, while the automatic testing mode is fully controlled by the work thread.

[0112] Fig.12 This is a schematic diagram of the manual test interface provided by this application software. Fig.12 , the following are displayed in the manual test interface: point cloud images before or after assembly, the position and number of the FPC on the mainboard, parameter configuration results of FPCs with different numbers, etc. Fig.13 This is a schematic diagram of the automatic test interface provided by the application software. Fig.13 , the automatic test interface displays: point cloud images before or after assembly, time axis curves of force control data, and real-time output of automatic test results, etc.

[0113] Fig.14 This is the operation flow chart of this application software. Fig.14 , the upper computer will automatically connect to the lower computer (taking PLC as an example) and the 3D camera when it starts, and will give a corresponding prompt when the connection fails. In administrator mode, after successfully connecting the PLC and the 3D camera, you will enter the administrator's main interface to configure the parameters, and you can adjust the parameters in conjunction with the teaching process. Usually, when the upper computer runs this application software for the first time or the software has been partially changed, you need to use the administrator account to enter the main interface to set the parameters and perform manual testing to ensure the reliability of the parameters. In operator mode, after successfully connecting the PLC and the camera, the operator places the main board on the storage table, and the application software will run automatically and give a real-time status diagram of the assembly process.

[0114] The present invention also provides an FPC automatic assembly control device, which is the same inventive concept as the above-mentioned method embodiment. Fig.15 : is a structural schematic diagram of an FPC automatic assembly control device provided in an embodiment of the present application, such as Fig.15 As shown, the device of the embodiment of the present application includes:

[0115] The communication connection unit 151 is used to establish communication connections with the lower computer and the 3D camera respectively;

[0116] The pre-assembly point cloud image acquisition and display unit 152 is used to control the 3D camera to acquire the point cloud image of the FPC before assembly in response to the lower computer's assembly request for a certain FPC, and display the point cloud image of the FPC before assembly on the display screen;

[0117] A grasping position calculation unit 153, used to calculate the grasping position of the FPC according to the point cloud image before the FPC is assembled;

[0118] The position data return unit 154 is used to return the grasping position of the FPC together with the assembly point position information of the FPC to the lower computer, so as to control the lower computer to guide the robot arm to complete the grasping and assembly of the FPC according to the grasping position of the FPC and the assembly point position information of the FPC.

[0119] In a preferred embodiment, the device of the embodiment of the present application further includes:

[0120] The force control data acquisition and display unit is used to obtain the force control data of the pressure sensor arranged on the manipulator while controlling the lower computer to guide the manipulator to complete the grasping and assembly of the FPC, and synchronously display the time axis curve of the force control data on the display screen.

[0121] In a preferred embodiment, the device of the embodiment of the present application further includes:

[0122] The assembled point cloud image acquisition and display unit is used to control the 3D camera to acquire the point cloud image of the FPC after assembly in response to the detection request of the lower computer for a certain FPC, and display the point cloud image of the FPC after assembly on the display screen;

[0123] An assembly posture judgment unit, used to calculate the current posture of the FPC according to the point cloud image after the FPC is assembled, and compare the current posture of the FPC with a pre-configured normal posture range of the FPC to determine whether the current posture of the FPC is normal;

[0124] An assembly in place judgment unit, used to judge whether the FPC is assembled in place by further combining the force control data of the manipulator in the process of grasping and assembling the FPC when judging that the current posture of the FPC is normal;

[0125] The assembly result returning unit is used to determine that the FPC is assembled in place when there is a sudden change of force in the force control data, and return the result data of correct assembly to the subordinate computer, otherwise return the result data of wrong assembly to the subordinate computer.

[0126] In a preferred embodiment, the device of the embodiment of the present application further includes:

[0127] The assembly point position correction unit is used to control the 2D camera to collect the 2D image of the mainboard after the mainboard is placed on the storage table; calculate the current position of the pre-calibrated mark point on the mainboard according to the 2D image of the mainboard; determine the deviation of the mainboard placement position according to the pre-calibrated mark point position on the mainboard and the calculated current position; and correct the pre-designed assembly point position of each FPC on the mainboard according to the deviation of the mainboard placement position, and determine the position information of the assembly point of each FPC after correction;

[0128] The position data returning unit 154 returns the assembly point position information of the FPC to the lower computer together, specifically, returns the assembly point position information of the FPC after correction to the lower computer together.

[0129] In a preferred embodiment, the device of the embodiment of the present application further includes:

[0130] The hyperparameter configuration unit is used to determine all the hyperparameter items required to complete the automated assembly of each FPC; all the hyperparameter items are designed in the same parameter table, and the parameter table is displayed in a parameter configuration interface, wherein each hyperparameter in the parameter table can be directly configured in the parameter configuration interface, and when configuring some hyperparameter items with spatial significance, point cloud images are used in the parameter configuration interface to intuitively display the parameter configuration results.

[0131] It can be understood that the above-mentioned FPC automated assembly control device can implement each step of the FPC automated assembly control method provided in the aforementioned embodiment, and the relevant explanations about the FPC automated assembly control method are applicable to the FPC automated assembly control device and will not be repeated here.

[0132] The present invention also provides an FPC automatic assembly control system, which is the same invention concept as the above method embodiment. Fig.16 : is a structural diagram of an FPC automated assembly control system provided in an embodiment of the present application, such as Fig.16 As shown, the system of the embodiment of the present application includes: a host computer 160, a slave computer 161, a 3D camera 162, a 2D camera 163 and a manipulator 169, wherein the host computer 160 establishes communication connections with the slave computer 161, the 3D camera 162 and the 2D camera 163 respectively, and the slave computer 161 is communicatively connected to the manipulator 169;

[0133] The host computer 160 is used to run the aforementioned FPC automated assembly control method;

[0134] The lower computer 161 is used to send an assembly request for a certain FPC to the upper computer 160, and guide the robot 169 to complete the grasping and assembly of the FPC according to the grasping position of the FPC and the assembly point position information of the FPC returned by the upper computer 160; and send a detection request for a certain FPC to the upper computer 160, and receive the assembly correct or incorrect result data returned by the upper computer 160.

[0135] It can be understood that the above-mentioned FPC automated assembly control system can implement the various steps of the FPC automated assembly control method provided in the aforementioned embodiments, and the relevant explanations on the FPC automated assembly control method are applicable to the FPC automated assembly control system and will not be repeated here.

[0136] The embodiment of the present application also provides an electronic device, which is based on the same technical concept as the aforementioned FPC automated assembly control method. Fig.17 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Fig.17 At the hardware level, the electronic device includes a memory and a processor, and optionally also includes a display panel, an interface module, a communication module, etc. Of course, the electronic device may also include hardware required for other services.

[0137] The processor, display panel, interface module, communication module and memory can be interconnected through an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.17 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0138] The memory is used to store computer executable instructions. The memory may include internal memory, such as high-speed random access memory (RAM), and may also include non-volatile memory (non-volatile memory), such as at least one disk storage. The memory provides computer executable instructions to the processor through an internal bus.

[0139] The processor executes the computer executable instructions stored in the memory and is specifically used to implement the aforementioned FPC automated assembly control method. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor or the instructions in the form of software. The above processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc.

[0140] An embodiment of the present application also proposes a computer-readable storage medium, which stores one or more computer programs. When the one or more computer programs are executed by an electronic device including multiple application programs, the aforementioned FPC automated assembly control method is implemented.

[0141] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program codes.

[0142] Computer-readable storage media include permanent and non-permanent, removable and non-removable media, and can be implemented by any method or technology to store information. Information can be computer-readable instructions, data structures, modules of programs or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable storage media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0143] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0144] The above is only a specific implementation of the present application. Under the above teachings of the present application, those skilled in the art can make other improvements or modifications based on the above embodiments. Those skilled in the art should understand that the above specific description is only to better explain the purpose of the present application, and the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A FPC automated assembly control method, characterized in that: include: Establish communication connections with the lower computer and 3D camera respectively; In response to an assembly request of a certain FPC by the lower computer, controlling the 3D camera to collect a point cloud image of the FPC before assembly, and displaying the point cloud image of the FPC before assembly on a display screen; The grabbing position of the FPC is calculated according to the point cloud image before the FPC is assembled, and is returned to the lower computer together with the assembly point position information of the FPC, so as to control the lower computer to guide the manipulator to complete the grabbing and assembly of the FPC according to the grabbing position of the FPC and the assembly point position information of the FPC; The grabbing position of the FPC is calculated according to the point cloud image before the FPC is assembled, specifically comprising: configuring the FPC grabbing range frame parameters in advance as needed to cut the point cloud image before the FPC is assembled, and calculating the cut point cloud image according to the grabbing position detection algorithm to obtain the grabbing position of the FPC, wherein the grabbing range frame parameters represent the range of the FPC in the point cloud image, and are determined by selecting a box in the point cloud image to define the end position of the FPC; The method further comprises: Establishing a communication connection with a 2D camera; after the mainboard is placed on the storage table, controlling the 2D camera to collect a 2D image of the mainboard; calculating the current position of a pre-calibrated mark point on the mainboard according to the 2D image of the mainboard; determining the deviation of the mainboard placement position according to the pre-calibrated mark point position on the mainboard and the calculated current position; and, according to the deviation of the mainboard placement position, correcting the pre-designed assembly point position of each FPC on the mainboard, and determining the position information of the assembly point of each FPC after correction; The returning to the lower computer together with the assembly point position information of the FPC specifically refers to: returning to the lower computer together with the assembly point position information of the FPC after correction.

2. The method according to claim 1, characterized in that The method further comprises: In the process of controlling the lower computer to guide the manipulator to complete the grasping and assembly of the FPC, the force control data of the pressure sensor arranged on the manipulator is obtained; The time axis curve of the force control data is synchronously displayed on the display screen.

3. The method according to claim 2, characterized in that The method further comprises: In response to a detection request of a certain FPC from a lower computer, a 3D camera is controlled to collect a point cloud image of the FPC after assembly, and the point cloud image of the FPC after assembly is displayed on a display screen; Calculating the current posture of the FPC according to the point cloud image after the FPC is assembled, and comparing the current posture of the FPC with a pre-configured normal posture range of the FPC to determine whether the current posture of the FPC is normal; When it is determined that the current posture of the FPC is normal, further combining the force control data of the manipulator in the process of grasping and assembling the FPC to determine whether the FPC is assembled in place; When there is a sudden change in force in the force control data, it is determined that the FPC is assembled in place, and correct assembly result data is returned to the subordinate computer, otherwise, incorrect assembly result data is returned to the subordinate computer.

4. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: Determine all hyperparameters to complete the automated assembly of each FPC; All the hyperparameter items are designed in the same parameter table, and the parameter table is displayed in a parameter configuration interface, wherein each hyperparameter in the parameter table can be directly configured in the parameter configuration interface, and when some hyperparameter items with spatial significance are configured, point cloud images are used in the parameter configuration interface to intuitively display the parameter configuration results.

5. An FPC automated assembly control device, characterized in that: include: A communication connection unit, used to establish communication connections with the lower computer and the 3D camera respectively; The pre-assembly point cloud image acquisition and display unit is used to control the 3D camera to acquire the point cloud image of the FPC before assembly in response to the assembly request of the lower computer for a certain FPC, and display the point cloud image of the FPC before assembly on the display screen; a grab position calculation unit, used to calculate the grab position of the FPC according to the point cloud image before the FPC is assembled, specifically used to configure the FPC grab range frame parameters in advance as needed to cut the point cloud image before the FPC is assembled, and calculate the cut point cloud image according to the grab position detection algorithm to obtain the grab position of the FPC, the grab range frame parameters represent the range of the FPC in the point cloud image, and are determined by a box that selects the end position of the FPC in the point cloud image; A position data returning unit, used to return the grasping position of the FPC together with the assembly point position information of the FPC to the lower computer, so as to control the lower computer to guide the manipulator to complete the grasping and assembly of the FPC according to the grasping position of the FPC and the assembly point position information of the FPC; The assembly point position correction unit is used to control the 2D camera to collect the 2D image of the mainboard after the mainboard is placed on the storage table; calculate the current position of the pre-calibrated marking point on the mainboard according to the 2D image of the mainboard; determine the deviation of the mainboard placement position according to the pre-calibrated marking point position on the mainboard and the calculated current position; And, according to the deviation of the placement position of the mainboard, the pre-designed assembly point position of each FPC on the mainboard is corrected, and the position information of the assembly point of each FPC after the correction is determined; The position data returning unit returns the assembly point position information of the FPC to the lower computer together, specifically, returns the assembly point position information of the FPC after correction to the lower computer together.

6. The device according to claim 5, characterized in that The device also includes: The force control data acquisition and display unit is used to obtain the force control data of the pressure sensor arranged on the manipulator while controlling the lower computer to guide the manipulator to complete the grasping and assembly of the FPC, and synchronously display the time axis curve of the force control data on the display screen.

7. The device according to claim 6, characterized in that The device also includes: The assembled point cloud image acquisition and display unit is used to control the 3D camera to acquire the point cloud image of the FPC after assembly in response to the detection request of the lower computer for a certain FPC, and display the point cloud image of the FPC after assembly on the display screen; An assembly posture judgment unit, used to calculate the current posture of the FPC according to the point cloud image after the FPC is assembled, and compare the current posture of the FPC with a pre-configured normal posture range of the FPC to determine whether the current posture of the FPC is normal; An assembly in place judgment unit, used to judge whether the FPC is assembled in place by further combining the force control data of the manipulator in the process of grasping and assembling the FPC when judging that the current posture of the FPC is normal; The assembly result returning unit is used to determine that the FPC is assembled in place when there is a sudden change of force in the force control data, and return the result data of correct assembly to the subordinate computer, otherwise return the result data of wrong assembly to the subordinate computer.

8. The device according to any one of claims 5 to 7, characterized in that: The device also includes: The hyperparameter configuration unit is used to determine all the hyperparameter items required to complete the automated assembly of each FPC; all the hyperparameter items are designed in the same parameter table, and the parameter table is displayed in a parameter configuration interface, wherein each hyperparameter in the parameter table can be directly configured in the parameter configuration interface, and when configuring some hyperparameter items with spatial significance, point cloud images are used in the parameter configuration interface to intuitively display the parameter configuration results.

9. An FPC automated assembly control system, characterized in that: It includes an upper computer, a lower computer, a 3D camera, a 2D camera and a manipulator, wherein the upper computer establishes communication connections with the lower computer, the 3D camera and the 2D camera respectively, and the lower computer is communicatively connected to the manipulator; The host computer is used to run the FPC automated assembly control method according to any one of claims 1 to 4; The lower computer is used to send an assembly request for a certain FPC to the upper computer, and guide the robot to complete the grasping and assembly of the FPC according to the grasping position of the FPC and the assembly point position information of the FPC returned by the upper computer; and send a detection request for a certain FPC to the upper computer, and receive the assembly correct or incorrect result data returned by the upper computer.

10. An electronic device, comprising: a processor and memory for storing computer instructions; The processor executes the FPC automated assembly control method described in any one of claims 1-4 when running the computer instructions.

Images