Component position coordinate acquisition method and device, and equipment assembly system

Abstract

The invention provides a position coordinate acquisition method and device of a component and an equipment assembly system. The method comprises the steps that a panoramic image corresponding to a component tray is acquired through first camera equipment at a preset position; inputting the panoramic image into a component identification model to obtain types and initial position coordinates of components in a component tray; after a mechanical arm is controlled to move to a target position according to the type of the component and the initial position coordinate, an overall image of the component is obtained through second camera equipment arranged on the mechanical arm; and inputting the overall image of the component into the component positioning model to obtain a final position coordinate of the component in the component tray. According to the invention, the position coordinates of the component are obtained through the camera device, the component identification model and the component positioning model, so that the position identification accuracy of the component is improved.

Description

Technical Field

[0001] The present invention relates to the field of automated equipment assembly, and in particular to a method and device for acquiring position coordinates of components, and an equipment assembly system. Background Art

[0002] In the automated assembly process of the equipment, the position information of the components to be assembled is first identified, and then the robotic arm is controlled to grab the components based on the identified position information and assemble the components onto the equipment.

[0003] In the prior art, the location information of components is mostly identified based on devices such as laser radar and ultrasonic sensors. However, both laser radar and ultrasonic sensors are easily affected by environmental reflection interference, which reduces the accuracy of location information identification. Summary of the invention

[0004] In view of this, the purpose of the present application is to provide a method, device, and equipment assembly system for obtaining the position coordinates of components, which can obtain the position coordinates of components through a camera device, a component recognition model, and a component positioning model, thereby improving the accuracy of component position recognition.

[0005] In a first aspect, an embodiment of the present application provides a method for obtaining position coordinates of a component, the method comprising: Acquire a panoramic image corresponding to a component tray by a first camera device at a predetermined position; components to be assembled are placed in the component tray; Inputting the panoramic image into a component recognition model to obtain the type and initial position coordinates of the components in the component tray; After controlling the mechanical arm to move to the target position according to the type of the component and the initial position coordinates, a second camera device provided on the mechanical arm is used to obtain an overall image of the component, wherein the overall image of the component includes an overall image of the component to be assembled; The overall component image is input into a component positioning model to obtain the final position coordinates of the component in the component tray.

[0006] In a possible implementation, after controlling the mechanical arm to move to the target position according to the type and initial position coordinates of the component, acquiring the overall image of the component by a second camera device provided on the mechanical arm includes: The shooting height of the second camera device is determined by the type of the component; the second camera device is installed at the upper end of the end effector of the mechanical arm; Determining a moving path of the robotic arm according to the shooting height, the initial position coordinates and the current position coordinates of the robotic arm; After the mechanical arm is controlled to move to the target position according to the moving path, the overall image of the component is acquired through the second camera device arranged on the mechanical arm.

[0007] In a possible implementation, determining the moving path of the robotic arm according to the shooting height, the initial position coordinates, and the current position coordinates of the robotic arm includes: The sum of the height coordinate value in the initial position coordinate and the shooting height is determined as the target height; Modify the height coordinate value in the current position coordinate of the robot arm to the target height to obtain the first moving position coordinate; Modify the height coordinate value in the initial position coordinate to the target height to obtain a second moving position coordinate; Input the current position coordinates of the robot arm as a starting point and the first moving position coordinates as an end point into a first path planning model to obtain a first moving path; and inputting the first moving position coordinates of the robot arm as a starting point and the second moving position coordinates as an end point into the first path planning model to obtain a second moving path; The first moving path and the second moving path are connected to obtain a moving path of the robot arm.

[0008] In a possible implementation manner, determining the moving path of the robotic arm according to the shooting height, the initial position coordinates, and the current position coordinates of the robotic arm further includes: The shooting height, the initial position coordinates and the current position coordinates of the robotic arm are input into a second path planning model to obtain the moving path of the robotic arm; the starting point in the moving path is the current position coordinates of the robotic arm, the sum of the height coordinate value in the initial position coordinates and the shooting height is the height coordinate value of the end point in the moving path, and the distance between the end point in the moving path and the corresponding position of the initial position coordinates is less than a preset distance.

[0009] In a possible implementation, the method further includes: Controlling the robot arm to move to the final position coordinates to grab the components; Controlling the movement of the robotic arm according to the preset assembly position coordinates corresponding to the component until the current position coordinates of the robotic arm and the preset assembly position coordinates meet a preset numerical relationship condition, and rotating the second camera device according to a preset assembly shooting angle corresponding to the type of the component; Acquire an assembly scene image by means of the rotated second camera device; the assembly scene image includes a position scene corresponding to the preset assembly position coordinates; The robot arm is controlled to assemble the components according to the assembly scene image and the preset assembly position coordinates.

[0010] In a possible implementation manner, the component recognition model is trained through the following steps: Acquire a panoramic sample image corresponding to the component tray, and a position coordinate and type corresponding to a first component sample in the component tray; The panoramic sample image is used as a sample, and the position coordinates and type corresponding to the first component sample are used as labels to perform model training on the component recognition model.

[0011] In a possible implementation, the method further includes: Calculating the difference between the final position coordinates of the component and the initial position coordinates; If the difference satisfies a preset difference condition, the panoramic image is used as sample data, and the final position coordinates and actual type of the component are used as labels corresponding to the panoramic image, and are added to the sample data set accordingly; When the number of sample data in the sample data set is greater than a preset number, or the non-updated time length corresponding to the component identification model is greater than a preset update time length, the component identification model is updated according to the sample data in the sample data set and the corresponding labels.

[0012] In a possible implementation manner, the component positioning model is trained through the following steps: Acquire the component overall sample image and the position coordinates corresponding to the second component sample in the component overall sample image; The component positioning model is trained by taking the overall sample image of the component as a sample and the position coordinates corresponding to the second component sample as a label.

[0013] In a second aspect, an embodiment of the present application further provides a device for obtaining position coordinates of a component, the device comprising: An acquisition module, used for acquiring a panoramic image corresponding to a component tray through a first camera device at a predetermined position; the component tray contains components to be assembled; An input module, used to input the panoramic image into a component recognition model to obtain the type and initial position coordinates of the components in the component tray; The acquisition module is further used to control the mechanical arm to move to the target position according to the type and initial position coordinates of the component, and then acquire the overall image of the component through the second camera device provided on the mechanical arm, wherein the overall image of the component includes the overall image of the component to be assembled; The input module is also used to input the overall image of the component into the component positioning model to obtain the final position coordinates of the component in the component tray.

[0014] In the third aspect, an embodiment of the present application also provides an electronic device, comprising: a processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium through the bus, and the processor executes the machine-readable instructions to perform the steps of the method for obtaining position coordinates of components as described in any one of the first aspects.

[0015] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method for obtaining the position coordinates of components as described in any one of the first aspects are executed.

[0016] In a fifth aspect, an embodiment of the present application further provides a device assembly system, the system comprising: a controller, a robotic arm, a first camera device, a second camera device, and an assembly operation table; Wherein, the second camera device is installed at the upper end of the end effector of the robotic arm; the assembly operation table is used to place a component tray, and the component tray contains components to be assembled; The controller is used for the method for obtaining position coordinates of components according to any one of claims 1 to 8 to obtain the final position coordinates of the components in the component tray; The controller is also used to control the robot arm to assemble the components according to the final position coordinates.

[0017] The embodiment of the present application provides a method, device, and equipment assembly system for obtaining the position coordinates of components, the method comprising: obtaining a panoramic image corresponding to a component tray through a first camera device at a predetermined position; inputting the panoramic image into a component recognition model to obtain the type and initial position coordinates of the components in the component tray; after controlling the mechanical arm to move to the target position according to the type and initial position coordinates of the components, obtaining the overall image of the components through a second camera device set on the mechanical arm; inputting the overall image of the components into a component positioning model to obtain the final position coordinates of the components in the component tray. The present application obtains the position coordinates of components through a camera device, a component recognition model, and a component positioning model, thereby improving the accuracy of component position recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 A flow chart showing a method for obtaining position coordinates of a component provided in an embodiment of the present application is shown; Figure 2 A schematic diagram of a process for controlling the movement of a mechanical arm provided in an embodiment of the present application is shown; Figure 3 A schematic diagram of a process of updating a model provided in an embodiment of the present application is shown; Figure 4 A structural diagram of a device for obtaining position coordinates of a component provided in an embodiment of the present application is shown; Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown; Figure 6 A schematic structural diagram of a device assembly system provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0020] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of explanation and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in this application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can be implemented out of sequence, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart under the guidance of the content of the present application, or remove one or more operations from the flowchart.

[0021] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0022] In order to enable those skilled in the art to use the content of this application, the following implementation is provided in conjunction with the specific application scenario "automation equipment assembly field". For those skilled in the art, the general principles defined herein can be applied to other embodiments and application scenarios without departing from the spirit and scope of this application. Although this application is mainly described around the "automation equipment assembly field", it should be understood that this is only an exemplary embodiment.

[0023] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.

[0024] The following is a detailed description of a method for obtaining the position coordinates of a component provided in an embodiment of the present application.

[0025] Reference Figure 1 As shown, it is a flow chart of a method for obtaining the position coordinates of a component provided in an embodiment of the present application. The exemplary steps of the embodiment of the present application are described below: S101, acquiring a panoramic image corresponding to a component tray through a first camera device at a predetermined position; components to be assembled are placed in the component tray.

[0026] In an embodiment of the present application, the panoramic image includes the entire area of ​​the component tray, which is used to identify the type and initial position coordinates of the components in the component tray. The predetermined position refers to a predetermined fixed position of the first camera device, which can capture the entire area of ​​the component tray. The components to be assembled refer to the parts or components of the current assembly device (which can be a server). When there are components to be assembled in the component tray, a panoramic image corresponding to the component tray captured by the first camera device is obtained through a data transmission device such as USB; the components to be assembled are placed in the component tray.

[0027] S102: Input the panoramic image into a component recognition model to obtain the types and initial position coordinates of the components in the component tray.

[0028] In the implementation mode of the present application, the component recognition model is used to obtain the type and initial position coordinates of the components in the component tray, which can be obtained by pre-training through the panoramic sample image corresponding to the component tray, the position coordinates and type corresponding to the first component sample in the component tray, or by pre-training through the panoramic sample image corresponding to the component tray, the position coordinates, type and angle corresponding to the first component sample in the component tray; the angle refers to the placement angle of the first component sample in the component tray. In other words, the input of the component recognition model is the panoramic image corresponding to the component tray, the output of the component recognition model can be the type and initial position coordinates of the components in the component tray, and the output of the component recognition model can also be the initial position coordinates, type and angle of the components in the component tray.

[0029] The type of the component may be the name and number of the component, etc. The component recognition model may be a deep learning model such as YOLOv8 and CNN.

[0030] In addition, when the output of the component recognition model is the type and initial position coordinates of the component in the component tray, the component recognition model is trained by the following steps: Step 1: Obtain a panoramic sample image corresponding to the component tray, and the position coordinates and type corresponding to the first component sample in the component tray.

[0031] In an embodiment of the present application, the panoramic sample image includes the entire area of ​​the component tray, which is used to train the component recognition model. The first component sample in the component tray refers to the component in the component tray in the panoramic sample image, which is used to train the component recognition model. The position coordinates corresponding to the first component sample can be manually marked. The type corresponding to the first component sample is the same as the type of the aforementioned component.

[0032] Step 2: Use the panoramic sample image as a sample, and the position coordinates and type corresponding to the first component sample as a label to train the component recognition model.

[0033] In an implementation manner of the present application, a panoramic sample image is input into a component recognition model to obtain predicted position coordinates and predicted type corresponding to the first component sample; the component recognition model is trained according to the position coordinates and type corresponding to the first component sample, and the predicted position coordinates and predicted type corresponding to the first component sample.

[0034] In addition, when the output of the component recognition model is the initial position coordinates, type and angle of the component in the component tray, the component recognition model is trained by the following steps: Step 1: Obtain a panoramic sample image corresponding to the component tray, and the position coordinates, type, and angle corresponding to the first component sample in the component tray.

[0035] In an embodiment of the present application, the panoramic sample image includes the entire area of ​​the component tray, which is used to train the component recognition model. The first component sample in the component tray refers to the component in the component tray in the panoramic sample image, which is used to train the component recognition model. The position coordinates and angles corresponding to the first component sample can be manually marked. The type corresponding to the first component sample is the same as the type of the aforementioned component.

[0036] Step 2: Use the panoramic sample image as a sample, and the position coordinates, type and angle corresponding to the first component sample as labels to train the component recognition model.

[0037] In an implementation manner of the present application, a panoramic sample image is input into a component recognition model to obtain the predicted position coordinates, predicted type and predicted angle corresponding to the first component sample; the component recognition model is trained according to the position coordinates, type and angle corresponding to the first component sample, and the predicted position coordinates, predicted type and predicted angle corresponding to the first component sample.

[0038] Furthermore, the method also includes: adjusting the posture of the robot arm according to the angle of the component, so that the robot arm can smoothly grasp the component.

[0039] Here, the application uses the initial position coordinates of the components as the center point, and uses the "center point + angle" method to ensure that the robot arm can smoothly grasp the components and assemble them in the future.

[0040] S103 , after controlling the robot arm to move to the target position according to the type of the component and the initial position coordinates, a second camera device arranged on the robot arm acquires an overall image of the component, wherein the overall image of the component includes an overall image of the component to be assembled.

[0041] In the implementation of the present application, the target position refers to the lowest position at which the entire image of the component can be captured, and the absolute value of the difference between the height coordinate value of the target position and the height coordinate value of the component is the corresponding value of the shooting height. The shooting height is pre-set according to the size of the component. Different types of components correspond to different target positions.

[0042] Specifically, after controlling the mechanical arm to move to the target position according to the type of the component and the initial position coordinates, the overall image of the component is obtained by a second camera device arranged on the mechanical arm, including: Step 1: Determine the shooting height of the second camera device according to the type of components; the second camera device is installed at the upper end of the end effector (such as a gripper) of the robot arm.

[0043] In an implementation manner of the present application, a shooting height corresponding to the type of component is queried in a preset shooting height table, and the queried shooting height is determined as the shooting height of the second camera device; wherein the shooting height table is preset with corresponding storage of the type of component and the shooting height of the component.

[0044] Here, the shooting height table may also store the size of components. The shooting height of a new component may be added to the shooting height table through the following steps, specifically: obtaining the type and size of the component to be added; and determining the shooting height of the component to be added according to the shooting heights of all components in the shooting height table that have the same size as the component to be added.

[0045] Furthermore, the shooting height of the component to be added is determined according to the shooting heights of all components with the same size as the component to be added in the shooting height table, including: determining the shooting height of the component to be added as the average value of the shooting heights of all components with the same size as the component to be added in the shooting height table; or, determining the shooting height of the component to be added as the mode of the shooting heights of all components with the same size as the component to be added in the shooting height table.

[0046] For example, the components in the shooting height table that are the same size as the components to be added include components A, components B, components C, and components D. The shooting height of component A is 4 cm, the shooting height of component B is 4 cm, the shooting height of component C is 3 cm, and the shooting height of component C is 2 cm. The shooting height of the components to be added can be (4+4+3+2) / 4=3.25 cm (i.e., the average of the shooting heights of all components in the shooting height table that are the same size as the components to be added is determined as the shooting height of the components to be added), and can also be 4 cm (i.e., the mode of the shooting heights of all components in the shooting height table that are the same size as the components to be added is determined as the shooting height of the components to be added).

[0047] Step 2: Determine the moving path of the robotic arm based on the shooting height, initial position coordinates and the current position coordinates of the robotic arm.

[0048] In the implementation manner of the present application, the end point in the moving path is the target position. The embodiment of the present application provides two methods for determining the moving path of the robot arm.

[0049] The specific steps of the first method are as follows: i. The sum of the height coordinate value in the initial position coordinate and the shooting height is determined as the target height.

[0050] For example, the initial position coordinates are (2, 3, 6), and the shooting height is 2, then the target height is 6+2=8.

[0051] ii. Modify the height coordinate value in the current position coordinate of the robot arm to the target height to obtain the first moving position coordinate.

[0052] In the implementation manner of the present application, the current position coordinates of the end effector of the robot arm are used as the current position coordinates of the robot arm.

[0053] For example, the target height is 8, and the current position coordinates of the robot are (3, 4, 2). Then the height coordinate value in the current position coordinates of the robot is modified to the target height, and the first moving position coordinates are obtained (3, 4, 8).

[0054] iii. Modify the height coordinate value in the initial position coordinate to the shooting height to obtain the second moving position coordinate.

[0055] For example, the target height is 8, and the initial position coordinates are (2, 3, 6). Then the height coordinate value in the initial position coordinates is modified to the shooting height, and the second moving position coordinates are obtained (2, 3, 8).

[0056] iv. Input the current position coordinates of the robot arm as the starting point and the first moving position coordinates as the end point into the first path planning model to obtain the first moving path.

[0057] In an embodiment of the present application, the current position coordinates of the robot arm are input into the first path planning model as the starting point and the first moving position coordinates are input as the end point to predict possible situations to generate an optimal first moving path.

[0058] v. Input the first moving position coordinates of the robot arm as the starting point and the second moving position coordinates as the end point into the first path planning model to obtain the second moving path.

[0059] In an embodiment of the present application, the first moving position coordinates of the robot arm are input into the first path planning model as the starting point and the second moving position coordinates are input into the first path planning model as the end point to predict possible situations and generate an optimal second moving path.

[0060] vi. Connect the first moving path and the second moving path to obtain the moving path of the robot arm.

[0061] In the embodiment of the present application, the end point of the first moving path is the tangent point of the second moving path. Therefore, the moving path of the robot arm is obtained by connecting the first moving path and the second moving path through the first moving position.

[0062] The specific steps of the second method include: inputting the shooting height, initial position coordinates and current position coordinates of the robot into the second path planning model to obtain the moving path of the robot; the starting point in the moving path is the current position coordinates of the robot, the sum of the height coordinate value in the initial position coordinates and the shooting height is the height coordinate value of the end point in the moving path, and the distance between the end point in the moving path and the corresponding position of the initial position coordinates is less than the preset distance.

[0063] Here, both the first path planning model and the second path planning model are trained by taking whether the moving path has a collision and the moving time as rewards.

[0064] Step 3: After controlling the robot arm to move to the target position according to the moving path, the overall image of the component is obtained through the second camera device arranged on the robot arm.

[0065] In the implementation mode of the present application, the monitoring data of the robot arm during movement is read in real time through sensors (such as cameras, laser radars, etc.), the state and environmental information of the robot arm are monitored, and the movement is ensured to proceed smoothly. If an abnormality occurs during the movement, the abnormality is eliminated and then updated to the first path planning model.

[0066] In addition, the present application also provides another method for controlling the robot arm to move to the target position according to the type of component and the initial position coordinates, referring to Figure 2 As shown, it is a schematic diagram of the process of controlling the movement of the robotic arm provided in an embodiment of the present application, wherein the initial position coordinates of the component are obtained based on the component recognition model; the PID (Proportional-Integral-Derivative) controller calculates the offset of the component relative to the robotic arm according to the initial position coordinates and the current position coordinates of the robotic arm; the Coordinate Transformation coordinate transformation module determines the coordinates of the component in the robotic arm coordinate system according to the offset and the current position coordinates of the robotic arm; the movement direction of the robotic arm is controlled based on the coordinates obtained after the conversion, and at the same time, during the movement, it is determined whether the target position has been reached through the image of the front of the robotic arm taken by the second camera device; if the target position has not been reached, the initial position coordinates of the component continue to be obtained based on the component recognition model to continue to control the robotic arm.

[0067] It should be noted that the following algorithms are used in the process of controlling the robotic arm: (1) tracking control, closed-loop control, and feedback control; (2) visual servoing-closed-loop control; (3) AI target detection with angle detection; (4) target detection and angle detection, etc.

[0068] S104 , inputting the entire component image into the component positioning model to obtain the final position coordinates of the component in the component tray.

[0069] In the implementation manner of the present application, the component positioning model is used to determine the final position coordinates of the components in the component tray, and is obtained by pre-training with the position coordinates corresponding to the overall component sample image and the second component sample in the overall component sample image.

[0070] Among them, the component positioning model can be a deep learning model such as YOLOv8 and CNN.

[0071] Here, the components in the component tray are positioned by using the first camera device, the second camera device, the component recognition model and the component positioning model, which can be locked by looking, dynamically corrected and tracked. Compared with the position of fixed components in traditional technology, it has strong adaptability, more flexible application, greater elasticity, and removes the traditional concept of blind insertion of positioning, which greatly improves the efficiency of line change. Using basic movement with optical feedback, learning to obtain spatial concepts, achieving intelligent initialization, and saving a lot of deployment time. In the field of precision operation of industrial automation, we often involve the precise grasping and positioning of target objects. At this time, it is particularly important to determine a suitable reference point. Compared with the traditional positioning method that relies on absolute zero point, this application uses a center point with unique advantages. This center point is not the zero point in the traditional sense, but a grasping zero point determined based on optical technology. Compared with traditional zero-point positioning, the optical grasping zero point does not require cumbersome mechanical calibration to determine the position of the absolute zero point, which greatly saves the time cost of equipment debugging and maintenance.

[0072] In addition, the component location model is trained by the following steps: Step 1: Obtain the component overall sample image and the position coordinates corresponding to the second component sample in the component overall sample image.

[0073] In the embodiment of the present application, the whole sample image of the components includes the whole area of ​​the two component samples, which is used to train the component positioning model. The second component sample is used to train the component positioning model. The position coordinates corresponding to the second component sample can be manually marked.

[0074] Step 2: Use the overall sample image of the component as a sample and the position coordinates corresponding to the second component sample as a label to train the component positioning model.

[0075] In an implementation manner of the present application, the overall sample image of the component is input into the component positioning model to obtain the predicted position coordinates corresponding to the second component sample; the component positioning model is trained based on the position coordinates corresponding to the second component sample and the predicted position coordinates corresponding to the second component sample.

[0076] In addition, refer to Figure 3As shown, it is a flow chart of the model update provided in the embodiment of the present application, in which a plurality of model optimization conditions are set in advance; during the application of the component identification model, when the preset model optimization conditions are triggered, the reward mechanism feeds back a penalty signal to the component identification model; after receiving the penalty signal, the component identification model updates the component identification model according to the sample data in the sample data set and the corresponding labels.

[0077] Among them, the model optimization conditions may include: (1) customer feedback that component assembly failed; (2) the time required to determine the initial position coordinates of the component exceeds a preset threshold; (3) the component cannot be identified; (4) the number of sample data in the sample data set is greater than a preset number; (5) the non-update time corresponding to the component identification model is greater than the preset update time.

[0078] The embodiment of the present application uses the prediction results of the component positioning model to update the component identification model, specifically: Step 1: Calculate the difference between the final position coordinates and the initial position coordinates of the component.

[0079] In an embodiment of the present application, for each coordinate axis, the absolute value of the difference between the coordinate value of the coordinate axis in the final position coordinates of the component and the coordinate value of the coordinate axis in the initial position coordinates is calculated to obtain the difference corresponding to the coordinate axis; the sum of the differences corresponding to all coordinate axes is taken as the difference between the final position coordinates and the initial position coordinates of the component; or, the weighted sum of the differences corresponding to all coordinate axes is taken as the difference between the final position coordinates and the initial position coordinates of the component.

[0080] Step 2: If the difference meets the preset difference condition, the panoramic image is used as sample data, and the final position coordinates and actual types of the components are used as labels corresponding to the panoramic image, and are added to the sample data set accordingly.

[0081] In the implementation manner of the present application, the larger the difference, the less accurate the prediction result of the component positioning model. The preset difference condition can be that the difference is greater than or equal to the preset difference. If the difference is greater than or equal to the preset difference, the panoramic image is used as sample data, and the final position coordinates and actual type of the component are used as labels corresponding to the panoramic image, and are added to the sample data set accordingly.

[0082] Here, through the method of the present application, image resources in actual applications can be added to the sample data set to update the component recognition model.

[0083] Step 3: When the number of sample data in the sample data set is greater than the preset number, or the non-updated time length corresponding to the component recognition model is greater than the preset update time length, the component recognition model is updated according to the sample data in the sample data set and the corresponding labels.

[0084] In the implementation manner of the present application, the non-updated time period refers to the time period between the current time and the last update time of the component identification model.

[0085] Here, when the model is updated, the offset designed by the component recognition model can be adjusted to improve the accuracy of the model.

[0086] Furthermore, the embodiment of the present application also controls the robot arm to assemble components according to the final position coordinates, specifically including: Step 1: Control the robot arm to move to the final position coordinates and grab the components; Step 2: Control the movement of the robotic arm according to the preset assembly position coordinates corresponding to the components until the current position coordinates of the robotic arm and the preset assembly position coordinates meet the preset numerical relationship conditions, and rotate the second camera device according to the preset assembly shooting angle corresponding to the type of component.

[0087] In the implementation manner of the present application, the preset assembly position refers to the position of the preset assembly components, and the components correspond one to one with the preset assembly position coordinates. The preset numerical relationship condition may be that the distance difference between the current position coordinates of the robot and the preset assembly position coordinates is less than the preset distance. When the distance difference between the current position coordinates of the robot and the preset assembly position coordinates is less than the preset distance, the second camera device is rotated according to the preset assembly shooting angle corresponding to the type of component, so that the second camera device can obtain the assembly scene image.

[0088] Here, in contrast to the one-to-one correspondence between components and preset assembly position coordinates, the component tray used to hold these components is in a state where they can be placed arbitrarily. This is because the flexibility of placing the component tray is taken into account in the design of the entire production process. Through this method, even if the placement position of the component tray is not fixed, the components in it can be quickly and accurately grabbed.

[0089] Among them, the preset assembly shooting angle corresponding to the type of component is determined according to the assembly direction of the component (from top to bottom, from left to right, etc.).

[0090] Step three: acquiring an assembly scene image through the rotated second camera device; the assembly scene image includes a position scene corresponding to a preset assembly position coordinate.

[0091] Step 4: Control the robotic arm to assemble components according to the assembly scene image and preset assembly position coordinates.

[0092] In addition, the deep learning model involved in this application can be transmitted back to the data center and shared with other robotic arms to share resources. Through this multi-brain and multi-learning parallel replication, iteration can be facilitated and the deployment scope can be expanded.

[0093] Here, the present application only needs to define the preset assembly position of the components to guide the action path of the robot arm after grasping the components. Through thousands of reinforcement learning in the virtual environment, various conditions before application are simulated. Engineers do not need to calibrate online, which saves the manpower configuration of engineers. In the process of motion control of the robot arm, it is crucial to ensure that its posture does not hit the casing (collision learning path). A variety of different posture combinations can be considered to select a relatively safe solution. To this end, the present application uses a virtual environment for learning and simulation, uses ROS (Robot Operating System) simulation as an interface, and uses the reward mechanism of reinforcement learning to accurately determine whether the robot arm will collide and interfere during movement. In this way, it is possible to continuously trial and error and optimize in the virtual environment, so that the robot arm can perform tasks more stably and safely during actual operation, avoiding collision problems caused by improper posture, thereby improving the reliability and safety of the present application and providing a strong guarantee for the efficient operation of the robot arm.

[0094] In addition, this method requires assembly simulation testing in a virtual environment before application. If it is a mechanism test and process test of the entire line, the digital twin system will be tested first, and when controlling the robot arm, ROS (Robot Operating System) will be used for simulation.

[0095] The embodiment of the present application provides a method for obtaining the position coordinates of components, the method comprising: obtaining a panoramic image corresponding to a component tray through a first camera device at a predetermined position; inputting the panoramic image into a component recognition model to obtain the type and initial position coordinates of the components in the component tray; after controlling the mechanical arm to move to the target position according to the type and initial position coordinates of the components, obtaining the overall image of the components through a second camera device set on the mechanical arm; inputting the overall image of the components into a component positioning model to obtain the final position coordinates of the components in the component tray. The present application obtains the position coordinates of components through a camera device, a component recognition model, and a component positioning model, thereby improving the accuracy of component position recognition.

[0096] Based on the same inventive concept, an embodiment of the present application also provides a component position coordinate acquisition device corresponding to the component position coordinate acquisition method. Since the principle of solving the problem by the device in the embodiment of the present application is similar to the component position coordinate acquisition method mentioned above in the embodiment of the present application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0097] Reference Figure 4FIG. 1 is a schematic diagram of a device for obtaining position coordinates of a component provided in an embodiment of the present application, the device comprising: An acquisition module 401 is used to acquire a panoramic image corresponding to a component tray through a first camera device at a predetermined position; components to be assembled are placed in the component tray; An input module 402 is used to input the panoramic image into a component recognition model to obtain the type and initial position coordinates of the components in the component tray; The acquisition module 403 is further used to control the robot arm to move to the target position according to the type and initial position coordinates of the component, and then acquire the overall image of the component through the second camera device provided on the robot arm, wherein the overall image of the component includes the overall image of the component to be assembled; The input module 404 is further used to input the overall component image into the component positioning model to obtain the final position coordinates of the component in the component tray.

[0098] An embodiment of the present application provides a device for acquiring the position coordinates of a component, which acquires the position coordinates of a component through a camera device, a component recognition model, and a component positioning model, thereby improving the accuracy of component position recognition.

[0099] like Figure 5 As shown, an electronic device 500 provided in an embodiment of the present application includes: a processor 501, a memory 502 and a bus, wherein the memory 502 stores machine-readable instructions executable by the processor 501. When the electronic device is running, the processor 501 communicates with the memory 502 through the bus, and the processor 501 executes the machine-readable instructions to perform the steps of the method for obtaining the position coordinates of components as described above.

[0100] Specifically, the memory 502 and the processor 501 can be general-purpose memories and processors, which are not specifically limited here. When the processor 501 runs the computer program stored in the memory 502, the method for obtaining the position coordinates of the components can be executed.

[0101] Corresponding to the above-mentioned method for obtaining the position coordinates of components, an embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method for obtaining the position coordinates of components are executed.

[0102] like Figure 6 As shown, an embodiment of the present application provides an equipment assembly system, the system comprising: a controller, a robotic arm, a first camera device, a second camera device, and an assembly operation table; The second camera device is installed at the upper end of the end effector of the robot arm; the assembly operation table is used to place the component tray, and the component tray contains the components to be assembled; In the implementation mode of the present application, the end effector must have high repeatability and must be stable, and must not damage components and assembly equipment.

[0103] A controller, used in a component position coordinate acquisition method, to acquire the final position coordinates of the components in the component tray; The controller is also used to control the robot arm to assemble components according to the final position coordinates.

[0104] An embodiment of the present application provides a device assembly system, which obtains the position coordinates of components through a camera device, a component recognition model and a component positioning model, thereby improving the accuracy of component position recognition, and controls a robotic arm to assemble components according to the final position coordinates, thereby improving assembly efficiency.

[0105] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the method embodiment, and will not be repeated in this application. In the several embodiments provided in this application, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0106] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0107] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0108] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the information processing method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks, or optical disks.

[0109] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for obtaining the position coordinates of a component, characterized in that: The method comprises: Acquire a panoramic image corresponding to a component tray by a first camera device at a predetermined position; components to be assembled are placed in the component tray; Inputting the panoramic image into a component recognition model to obtain the type and initial position coordinates of the components in the component tray; After controlling the mechanical arm to move to the target position according to the type of the component and the initial position coordinates, a second camera device provided on the mechanical arm is used to obtain an overall image of the component, wherein the overall image of the component includes an overall image of the component to be assembled; The overall component image is input into a component positioning model to obtain the final position coordinates of the component in the component tray.

2. The method for obtaining the position coordinates of a component according to claim 1, characterized in that: After controlling the mechanical arm to move to the target position according to the type and initial position coordinates of the component, obtaining the overall image of the component through a second camera device provided on the mechanical arm, comprises: The shooting height of the second camera device is determined by the type of the component; the second camera device is installed at the upper end of the end effector of the mechanical arm; Determining a moving path of the robotic arm according to the shooting height, the initial position coordinates and the current position coordinates of the robotic arm; After the mechanical arm is controlled to move to the target position according to the moving path, the overall image of the component is acquired through the second camera device arranged on the mechanical arm.

3. The method for obtaining the position coordinates of a component according to claim 2, characterized in that: The step of determining the moving path of the robotic arm according to the shooting height, the initial position coordinates and the current position coordinates of the robotic arm comprises: The sum of the height coordinate value in the initial position coordinate and the shooting height is determined as the target height; Modify the height coordinate value in the current position coordinate of the robot arm to the target height to obtain the first moving position coordinate; Modify the height coordinate value in the initial position coordinate to the target height to obtain a second moving position coordinate; Input the current position coordinates of the robot arm as a starting point and the first moving position coordinates as an end point into a first path planning model to obtain a first moving path; and inputting the first moving position coordinates of the robot arm as a starting point and the second moving position coordinates as an end point into the first path planning model to obtain a second moving path; The first moving path and the second moving path are connected to obtain a moving path of the robot arm.

4. The method for obtaining the position coordinates of a component according to claim 2, characterized in that: The step of determining the moving path of the robotic arm according to the shooting height, the initial position coordinates and the current position coordinates of the robotic arm further includes: The shooting height, the initial position coordinates and the current position coordinates of the robotic arm are input into a second path planning model to obtain the moving path of the robotic arm; the starting point in the moving path is the current position coordinates of the robotic arm, the sum of the height coordinate value in the initial position coordinates and the shooting height is the height coordinate value of the end point in the moving path, and the distance between the end point in the moving path and the corresponding position of the initial position coordinates is less than a preset distance.

5. The method for obtaining the position coordinates of a component according to claim 1, characterized in that: The method further comprises: Controlling the robot arm to move to the final position coordinates to grab the components; Controlling the movement of the robotic arm according to the preset assembly position coordinates corresponding to the component until the current position coordinates of the robotic arm and the preset assembly position coordinates meet a preset numerical relationship condition, and rotating the second camera device according to a preset assembly shooting angle corresponding to the type of the component; Acquire an assembly scene image by means of the rotated second camera device; the assembly scene image includes a position scene corresponding to the preset assembly position coordinates; The robot arm is controlled to assemble the components according to the assembly scene image and the preset assembly position coordinates.

6. The method for obtaining the position coordinates of a component according to claim 1, characterized in that: The component recognition model is trained by the following steps: Acquire a panoramic sample image corresponding to the component tray, and a position coordinate and type corresponding to a first component sample in the component tray; The panoramic sample image is used as a sample, and the position coordinates and type corresponding to the first component sample are used as labels to perform model training on the component recognition model.

7. The method for obtaining the position coordinates of a component according to claim 1, characterized in that: The method further comprises: Calculating the difference between the final position coordinates of the component and the initial position coordinates; If the difference satisfies a preset difference condition, the panoramic image is used as sample data, and the final position coordinates and actual type of the component are used as labels corresponding to the panoramic image, and are added to the sample data set accordingly; When the number of sample data in the sample data set is greater than a preset number, or the non-updated time length corresponding to the component identification model is greater than a preset update time length, the component identification model is updated according to the sample data in the sample data set and the corresponding labels.

8. The method for obtaining the position coordinates of a component according to claim 1, characterized in that: The component positioning model is trained by the following steps; Acquire the component overall sample image and the position coordinates corresponding to the second component sample in the component overall sample image; The component positioning model is trained by taking the overall sample image of the component as a sample and the position coordinates corresponding to the second component sample as a label.

9. A device for obtaining the position coordinates of a component, characterized in that: The device comprises: An acquisition module, used for acquiring a panoramic image corresponding to a component tray through a first camera device at a predetermined position; the component tray contains components to be assembled; An input module, used to input the panoramic image into a component recognition model to obtain the type and initial position coordinates of the components in the component tray; The acquisition module is further used to control the mechanical arm to move to the target position according to the type and initial position coordinates of the component, and then acquire the overall image of the component through the second camera device provided on the mechanical arm, wherein the overall image of the component includes the overall image of the component to be assembled; The input module is also used to input the overall image of the component into the component positioning model to obtain the final position coordinates of the component in the component tray.

10. An electronic device, characterized in that: include: A processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of the method for obtaining the position coordinates of components as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method for obtaining the position coordinates of a component as claimed in any one of claims 1 to 8 are executed.

12. A device assembly system, the system comprising: Controller, robotic arm, first camera device, second camera device, assembly operation table; Wherein, the second camera device is installed at the upper end of the end effector of the robotic arm; the assembly operation table is used to place a component tray, and the component tray contains components to be assembled; The controller is used for the method for obtaining position coordinates of components according to any one of claims 1 to 8 to obtain the final position coordinates of the components in the component tray; The controller is also used to control the robot arm to assemble the components according to the final position coordinates.

Images