A plate correction method and system based on robot vision

Abstract

The present application discloses a panel correction method and system based on robot vision, including: a visual correction system takes a picture of a target panel to obtain a label picture, and the robot system sends the panel theoretical information to the visual correction system; the visual correction system calculates the target barcode coordinate information according to the target feature information of the barcode on the label picture; the actual coordinates of the label in the robot coordinate system are calculated according to the conversion relationship matrix, the reference point coordinates of the calibration picture in the robot coordinate system and the target barcode coordinate information; the positioning offset is calculated according to the reference point coordinates of the calibration picture in the robot coordinate system, the actual coordinates of the label in the robot coordinate system, the panel theoretical coordinates and the label theoretical coordinates; the target panel center coordinates are calculated according to the positioning offset, and the target panel center coordinates are sent to the robot system so that the robot system controls the robot to adsorb the target panel, which can improve the accuracy and production efficiency of automated operations and can be widely used in the field of automated control technology.

Description

Technical Field

[0001] The present application relates to the field of automation control technology, and in particular to a panel correction method and system based on robot vision. Background Art

[0002] At present, in the production process of smart home, the milling processing center has become one of the most popular processing equipment in the home furnishing manufacturing industry in recent years because of its high optimization utilization rate of plate materials and its ability to simultaneously realize cutting and milling. However, when the plate materials are pushed to the unloading area after processing, the uneven force on some plates may cause the rotation angle and unpredictable deviation of the unloading position due to the squeezing of the processing gap of the plate parts.

[0003] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the invention

[0004] The embodiments of the present application aim to solve at least one of the technical problems in the related art to a certain extent. To this end, the main purpose of the embodiments of the present application is to propose a panel correction method and system based on robot vision, which can improve the accuracy and production efficiency of automated operations.

[0005] To achieve the above-mentioned purpose, one aspect of an embodiment of the present application provides a plate correction method based on robot vision, the method comprising:

[0006] The robot system controls the robot to send a shooting instruction to the visual correction system, so that the visual correction system takes a picture of the label of the target panel to obtain a target label image, and sends the panel theoretical information of the target panel to the visual correction system through the robot system; the panel theoretical information includes the panel theoretical coordinates and the label theoretical coordinates;

[0007] Calculating target barcode coordinate information of the target barcode according to target feature information of the target barcode on the target label image by the visual correction system;

[0008] The visual correction system calculates the actual coordinates of the label in the robot coordinate system according to the target conversion relationship matrix, the reference point coordinates of the target calibration image in the robot coordinate system, and the target barcode coordinate information;

[0009] The positioning offset of the robot is calculated by the visual correction system according to the reference point coordinates of the target calibration image in the robot coordinate system, the actual coordinates of the label in the robot coordinate system, the theoretical coordinates of the panel and the theoretical coordinates of the label;

[0010] The target panel center coordinates of the target panel are calculated by the visual correction system according to the positioning offset, and the target panel center coordinates are sent to the robot system so that the robot system controls the robot to adsorb the target panel at the target panel center coordinates.

[0011] In some embodiments, before the robot system controls the robot to send a shooting instruction to the visual correction system so that the visual correction system takes a picture of the label of the target panel to obtain a target label image, and before the robot system sends the panel theoretical information of the target panel to the visual correction system, the method further includes:

[0012] The panel theoretical information of the target panel is acquired through the software control system, and the panel theoretical information is sent to the robot system.

[0013] In some embodiments, the robot system controls the robot to send a shooting instruction to the visual correction system, so that the visual correction system takes a picture of the label of the target panel to obtain a target label picture, and sends the panel theoretical information of the target panel to the visual correction system through the robot system, including:

[0014] The robot system controls the robot to move to the label theoretical coordinates, and controls the robot to issue a shooting instruction to the visual correction system, so that the visual correction system takes a picture of the label of the target panel at the label theoretical coordinates to obtain the target label image, and sends the panel theoretical information of the target panel to the visual correction system through the robot system.

[0015] In some embodiments, the panel theoretical information further includes original barcode information, and the target barcode coordinate information includes the target barcode center point coordinates and the target barcode angle; the target barcode coordinate information of the target barcode is calculated by the visual correction system according to the target feature information of the target barcode on the target label image, including:

[0016] The target barcode on the target label image is identified by the visual correction system to obtain target barcode information;

[0017] Calculate the target barcode center point coordinates of the target barcode according to the target barcode information by the visual correction system;

[0018] Matching the target barcode information with the original barcode information through the visual correction system to obtain a target matching result;

[0019] Acquiring target edge information of the target barcode according to the target matching result through the visual correction system;

[0020] The visual correction system calculates a target barcode angle of the target barcode according to the target edge information.

[0021] In some embodiments, the method further comprises:

[0022] Matching the target barcode information with the original barcode information through the visual correction system to obtain an abnormal matching result;

[0023] The abnormal matching result is sent to the robot system through the visual correction system.

[0024] In some embodiments, the positioning offset includes a translation offset and a rotation offset, and the positioning offset of the robot is calculated by the visual correction system according to the reference point coordinates of the target calibration image in the robot coordinate system, the actual coordinates of the label in the robot coordinate system, the theoretical coordinates of the panel, and the theoretical coordinates of the label, including:

[0025] Calculating the translation offset of the robot according to the reference point coordinates of the target calibration image in the robot coordinate system and the actual coordinates of the label in the robot coordinate system by the visual correction system;

[0026] The visual correction system calculates the rotation offset of the robot according to the theoretical coordinates of the plate and the theoretical coordinates of the label.

[0027] In some embodiments, the target barcode angle in the target barcode coordinate information is the rotation angle of the robot adsorbing the target panel; the target panel center coordinates of the target panel are calculated by the visual correction system according to the positioning offset, and the target panel center coordinates are sent to the robot system so that the robot system controls the robot to adsorb the target panel at the target panel center coordinates, including:

[0028] Calculating the target panel center coordinates of the target panel according to the positioning offset by the visual correction system, and sending the target panel center coordinates to the robot system;

[0029] Control the robot by the robot system to move the robot suction cup to the target panel center coordinate, and control the robot by the robot system to rotate the robot suction cup to the target barcode angle;

[0030] The robot system controls the robot to use the robot suction cup to absorb the target panel at the center coordinate of the target panel.

[0031] To achieve the above-mentioned purpose, another aspect of the embodiment of the present application provides a panel correction system based on robot vision, the system comprising:

[0032] The image acquisition module is used to control the robot through the robot system to send a shooting instruction to the visual correction system, so that the visual correction system takes a picture of the label of the target panel to obtain the target label picture, and sends the panel theoretical information of the target panel to the visual correction system through the robot system; the panel theoretical information includes the panel theoretical coordinates and the label theoretical coordinates;

[0033] A barcode coordinate information acquisition module, used to calculate the target barcode coordinate information of the target barcode according to the target feature information of the target barcode on the target label image through the visual correction system;

[0034] A label coordinate calculation module, used to calculate the actual coordinates of the label in the robot coordinate system according to the target conversion relationship matrix, the reference point coordinates of the target calibration image in the robot coordinate system and the target barcode coordinate information through the visual correction system;

[0035] A positioning offset calculation module, used to calculate the positioning offset of the robot according to the reference point coordinates of the target calibration image in the robot coordinate system, the actual coordinates of the label in the robot coordinate system, the theoretical coordinates of the panel and the theoretical coordinates of the label through the visual correction system;

[0036] The target panel center calculation module is used to calculate the target panel center coordinates of the target panel according to the positioning offset through the visual correction system, and send the target panel center coordinates to the robot system so that the robot system controls the robot to adsorb the target panel at the target panel center coordinates.

[0037] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above-mentioned method when executing the computer program.

[0038] To achieve the above objective, another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described above is implemented.

[0039] The embodiments of the present application include at least the following beneficial effects: The present application provides a panel correction method and system based on robot vision, which controls the robot through the robot system to send a shooting instruction to the visual correction system, so that the visual correction system takes a picture of the label of the target panel to obtain the target label image, and sends the panel theoretical information of the target panel to the visual correction system through the robot system; the panel theoretical information includes the panel theoretical coordinates and the label theoretical coordinates; the visual correction system calculates the target barcode coordinate information of the target barcode according to the target feature information of the target barcode on the target label image; the visual correction system calculates the actual coordinates of the label in the robot coordinate system according to the target conversion relationship matrix, the reference point coordinates of the target calibration image in the robot coordinate system, and the target barcode coordinate information; the visual correction system calculates the positioning offset of the robot according to the reference point coordinates of the target calibration image in the robot coordinate system, the actual coordinates of the label in the robot coordinate system, the panel theoretical coordinates, and the label theoretical coordinates; the visual correction system calculates the target panel center coordinates of the target panel according to the positioning offset, and sends the target panel center coordinates to the robot system, so that the robot system controls the robot to adsorb the target panel at the target panel center coordinates. The embodiment of the present application calculates the positioning offset of the robot based on the visual correction system to provide data support for subsequent correction; the target panel center coordinates of the target panel are calculated by the visual correction system according to the positioning offset, and the target panel center coordinates are sent to the robot system, that is, the robot can be guided to perform actions through the corrected coordinate information, so that the robot can accurately find and adsorb the center position of the target panel and then perform unloading operations, so as to improve the accuracy and production efficiency of automated operations, while reducing the labor intensity of employees. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a flowchart of the steps of a plate correction method based on robot vision provided in an embodiment of the present application;

[0041] Figure 2 It is a barcode schematic diagram provided in an embodiment of the present application;

[0042] Figure 3 It is a flowchart of a visual calibration method provided in an embodiment of the present application;

[0043] Figure 4 is a flowchart of the steps of a visual calibration method provided in an embodiment of the present application;

[0044] Figure 5 It is a flowchart of a panel correction method based on robot vision provided in an embodiment of the present application;

[0045] Figure 6It is a schematic flow chart of a panel deviation correction method based on robot vision provided in an embodiment of the present application;

[0046] Figure 7 It is a structural schematic diagram of a panel correction system based on robot vision provided in an embodiment of the present application;

[0047] Figure 8 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of systems and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0049] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0050] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0051] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0052] Before describing the embodiments of the present application in detail, some nouns and terms involved in the embodiments of the present application are first described. The nouns and terms involved in the embodiments of the present application are subject to the following explanations.

[0053] At present, in the production process of smart home, the milling processing center has become one of the most popular processing equipment in the home furnishing manufacturing industry in recent years because of its high optimization utilization rate of plate materials and its ability to simultaneously realize cutting and milling. However, when the plate materials are pushed to the unloading area after processing, the uneven force on some plates may cause the rotation angle and unpredictable deviation of the unloading position due to the squeezing of the processing gap of the plate parts.

[0054] In view of this, a panel correction method and system based on robot vision are provided in an embodiment of the present application. The scheme controls the robot through the robot system to send a shooting instruction to the visual correction system, so that the visual correction system takes a picture of the label of the target panel to obtain the target label image, and sends the panel theoretical information of the target panel to the visual correction system through the robot system; the panel theoretical information includes the panel theoretical coordinates and the label theoretical coordinates; the visual correction system calculates the target barcode coordinate information of the target barcode according to the target feature information of the target barcode on the target label image; the visual correction system calculates the actual coordinates of the label in the robot coordinate system according to the target conversion relationship matrix, the reference point coordinates of the target calibration image in the robot coordinate system and the target barcode coordinate information; the visual correction system calculates the positioning offset of the robot according to the reference point coordinates of the target calibration image in the robot coordinate system, the actual coordinates of the label in the robot coordinate system, the panel theoretical coordinates and the label theoretical coordinates; the visual correction system calculates the target panel center coordinates of the target panel according to the positioning offset, and sends the target panel center coordinates to the robot system, so that the robot system controls the robot to adsorb the target panel at the target panel center coordinates. The embodiment of the present application calculates the positioning offset of the robot based on the visual correction system to provide data support for subsequent correction; the target panel center coordinates of the target panel are calculated by the visual correction system according to the positioning offset, and the target panel center coordinates are sent to the robot system, that is, the robot can be guided to perform actions through the corrected coordinate information, so that the robot can accurately find and adsorb the center position of the target panel and then perform unloading operations, so as to improve the accuracy and production efficiency of automated operations, while reducing the labor intensity of employees.

[0055] The panel correction method based on robot vision provided in the embodiment of the present application relates to the field of automation control technology. The panel correction method based on robot vision provided in the embodiment of the present application can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or it can be configured as a server cluster or a distributed system composed of multiple physical servers, and can also be configured to provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network) and big data and artificial intelligence platforms and other basic cloud computing services. The cloud server, the server can also be a node server in the blockchain network; the software can be an application that implements a panel correction method based on robot vision, etc., but is not limited to the above forms.

[0056] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs (personal computers), minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0057] See also Figure 1 , Figure 1 is an optional step flow chart of a panel deviation correction method based on robot vision provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S105.

[0058] Step S101, controlling the robot through the robot system to send a shooting instruction to the visual correction system, so that the visual correction system takes a picture of the label of the target panel to obtain a target label image, and sends the panel theoretical information of the target panel to the visual correction system through the robot system; the panel theoretical information includes the panel theoretical coordinates and the label theoretical coordinates;

[0059] For the robot system, it is a system used to control the robot to perform relevant actions. The robot system is mainly composed of a PLC (Programmable Logic Controller). In the robot control system, the PLC is usually used to process data from sensors and control the movement and operation of the robot based on these data. For example, the PLC can receive signals from position sensors, and then control the robot's motor based on these signals to move the robot to a specified position.

[0060] For the target panel, it is the panel to be unloaded and palletized. Specifically, in the process of automated production and palletizing of customized home furnishings, loading refers to loading raw materials or semi-finished products to the starting position of the production line to prepare for processing; processing refers to performing the required processing operations on raw materials or semi-finished products, such as cutting, grinding, and forming; pushing refers to pushing the finished product or the raw materials required for the next round of processing from the processing area to the position of the next processing step after the processing is completed; unloading refers to removing the processed finished product from the production equipment and placing it in a designated location, such as the palletizing area, for storage or further logistics operations; palletizing refers to stacking or placing the finished products in an orderly manner in a designated area, usually for the convenience of storage, transportation or further processing. It should be noted that the above example is only a simple example of the automated production and palletizing of customized home furnishings. In actual applications, there may be more complex processing flows, which will not be described in detail in the embodiments of this application.

[0061] In some embodiments, before step S101, the method may further include: acquiring theoretical panel information of the target panel through a software control system, and sending the theoretical panel information to the robot system.

[0062] Among them, the software control system can also be called the master control system or upstream control system, which is mainly used to provide a data source for the robot system, and subsequently used for the correction calculation of the visual correction system. The data source is mainly the theoretical data value of the target panel, that is, the theoretical information of the panel.

[0063] The theoretical information of the panel in the software control system (the data transmitted from the PLC to the visual correction system) may include but is not limited to the following examples:

[0064] Trigger: Trigger logic algorithm, with rising edge signal as trigger. The visual correction system will keep reading the signal value of PLC. When the Trigger value of PLC changes from 0 to 1 (i.e. rising edge signal. 0 means the robot is in standby state, 1 means the robot has reached the visual photography position), the image acquisition step is triggered.

[0065] Barcode: Photographed barcode, data provided by the upstream control system. In the specific implementation, the camera field of view may capture multiple DataMatrix (two-dimensional barcode) information. Through the photographed barcode transmitted by the PLC, the camera finds the barcode data consistent with the photographed barcode provided by the upstream control system within the field of view for visual correction calculation. If there is no matching barcode, an abnormal state is returned.

[0066] CaptureX: The X coordinate of the robot when taking a photo, that is, the theoretical value of the tag's X coordinate (label theoretical coordinate), data provided by the upstream control system.

[0067] CaptureY: The Y coordinate of the robot when taking a photo, that is, the theoretical value of the Y coordinate of the tag (label theoretical coordinate), data provided by the upstream control system.

[0068] CaptureR: The R angle of the robot's posture when taking pictures. During the production process, due to the limitation of the labeling mechanism, the label can only be pasted parallel to the long side (0°) or short side (90°) of the plate. In order to reduce the calculation error, the coordinates at two different angles are established during calibration, that is, the calibration relationship between the robot and the vision. In the actual production process, the upstream control system provides the labeling angle to control the robot's shooting posture and retrieves the calibration results of the corresponding angle.

[0069] PanelCenterX: The theoretical value of the X coordinate of the panel center (theoretical panel coordinate), data provided by the upstream control system.

[0070] PanelCenterY: The theoretical value of the Y coordinate of the panel center (theoretical panel coordinate), data provided by the upstream control system.

[0071] In some embodiments, step S101 may include: controlling the robot to move to the theoretical coordinates of the label through the robot system, and controlling the robot to issue a shooting instruction to the visual correction system, so that the visual correction system takes a picture of the label of the target panel on the theoretical coordinates of the label to obtain the target label image, and sending the theoretical panel information of the target panel to the visual correction system through the robot system.

[0072] Among them, the label on the board is affixed to the surface of the board, and can be automatically labeled using equipment. The upstream system (software control system) generally sets parameters to control the labeling position, such as setting the label in the upper left / upper right / lower left / lower right corner of the board or in the center (centering means overlapping with the center point of the board).

[0073] With the continuous development of technology and the promotion of its application, machine vision brings more intelligent and automated solutions to home furnishing manufacturing, thereby improving production efficiency, quality and customer satisfaction. In recent years, the use of vision-guided robots has been a popular direction for image recognition, and it is also an effective way to transform traditional production workshop operations into intelligent and automated ones. The positioning and alignment of parts is a scenario for the application of vision-guided robots in the panel home furnishing manufacturing industry. However, there are currently 2D (Two Dimensions) or 3D (Three Dimensions) vision-guided systems for different application scenarios on the market. Compared with 3D vision-guided systems, 2D vision is more economical and has a faster processing speed.

[0074] In the embodiment of the present application, a 2D array camera is used, the label size is set to 100*40mm, a 200mm material pushing error is reserved, and the camera field of view only needs to be 300*250mm. It should be noted that in the specific implementation, the setting of each size range or camera field of view can be adjusted according to the actual situation, and the embodiment of the present application does not limit this.

[0075] In the specific implementation, a 2D black and white area array camera is used to collect label images. During the calibration process, the calibration plate is kept in place, the robot is moved manually, 12-point translation and rotation calibration is performed, and the corresponding image is collected. During the correction process, the robot receives the theoretical x and y coordinates (label theoretical coordinates) transmitted by the upstream control system through the PLC and moves the camera position to the corresponding position for image acquisition. It can be understood that in the embodiment of the present application, there are two picture collections. The first picture collection is performed before the equipment is put into production in order to determine the relationship between the image coordinates and the robot coordinates, that is, to determine the relationship matrix between the robot's coordinate system and the image coordinate system of the visual system. The second picture collection is performed in the actual production process, and each plate needs to be executed. The position correction is performed by calculating the relationship matrix between the robot's coordinates and the target label image's coordinates and related parameters.

[0076] Among them, the target label image is only an image of the label part on the target board. Since the board size is not fixed, the label size is fixed, and the camera coverage is limited, the embodiment of the present application mainly performs subsequent correction calculations on the acquired label image.

[0077] Among them, the visual correction system can be used to correct the theoretical value of the theoretical adsorption point position of the panel to the actual value, and the robot is responsible for transporting the panel from the adsorption point to the stacking point.

[0078] In the embodiment of the present application, the robot is controlled by the robot system to move to the label theoretical coordinates, and the robot is controlled to send a shooting command to the visual correction system, so that the visual correction system takes a picture of the label of the target panel on the label theoretical coordinates, obtains the target label image, and sends the panel theoretical information of the target panel to the visual correction system through the robot system. By capturing the image of the target label with a high-definition camera, the recognition accuracy of subsequent processing can be improved; obtaining actual image information and theoretical data, providing basic data for subsequent visual processing, can help the visual correction system correct errors in actual operations.

[0079] Step S102, calculating the target barcode coordinate information of the target barcode according to the target feature information of the target barcode on the target label image by the visual correction system;

[0080] In some embodiments, step S102 may include: identifying the target barcode on the target label image through a visual correction system to obtain target barcode information; calculating the target barcode center point coordinates of the target barcode based on the target barcode information through the visual correction system; matching the target barcode information with the original barcode information through the visual correction system to obtain a target matching result; obtaining the target edge information of the target barcode based on the target matching result through the visual correction system; calculating the target barcode angle of the target barcode based on the target edge information through the visual correction system.

[0081] In the embodiment of the present application, the target barcode is a DataMatrix barcode. The target feature information can be understood as the characteristics of the DataMatrix barcode on the panel label. The outer frame of the DataMatrix barcode has an L-shaped solid line frame. Figure 2 , Figure 2 It is a barcode schematic diagram provided in an embodiment of the present application.

[0082] The theoretical information of the panel also includes the original barcode information, that is, the photographed barcode data provided by the above-mentioned software control system, which is used for matching with the content of the two-dimensional code string obtained by the subsequent recognition of the DataMatrix barcode.

[0083] The target barcode coordinate information includes the target barcode center point coordinates and the target barcode angle. The target barcode center point coordinates are the center position of the DataMatrix barcode on the panel label. The target barcode angle refers to the tilt angle of the DataMatrix barcode in the DataMatrix barcode image, that is, the rotation angle of the DataMatrix barcode in the image. In the DataMatrix barcode, it is the tilt angle of the two edge straight lines of the L-shaped feature corner point.

[0084] The target barcode information may be the two-dimensional code string content of the DataMatrix barcode, and may include the corner point coordinates of the L-shaped characteristic edge of the DataMatrix barcode, and the width and height information of the rectangle formed by the corner point and the diagonally opposite vertices of the DataMatrix barcode.

[0085] For example, the process of recognizing a DataMatrix barcode is as follows: with the help of the libdmtx tool library, the DataMatrix barcode is recognized and the recognition result is returned. The recognition result is the content of the QR code string, such as Figure 2 As shown, the recognition result may include the corner point coordinates P1(x dm ,y dm ), and the width Wd of the rectangle formed by the corner point and the diagonally opposite vertex of the DataMatrix barcode m and height ha m Information, according to the corner point coordinates P1(x dm ,y dm ), the width W of the rectangle formed by the corner point and the diagonally opposite vertex of the DataMatrix barcode dm and height h dm Information, we can calculate the coordinates of the diagonal vertex of the DataMatrix as P3(x dm +W dm ,y dm +h dm ), the center point coordinates of the DataMatrix barcode are If there are multiple DataMatrix two-dimensional codes, this tool library function can return the information corresponding to multiple barcodes.

[0086] Among them, libdmtx is an open source library for decoding DMTX (DataMatrix) code, a two-dimensional barcode standard for data storage. For the definition and technical principles of libdmtx, please refer to the relevant technical content, and the embodiments of this application will not be repeated here.

[0087] For the target matching result, it is defined here as the result of a successful match, that is, the DataMatrix barcode is recognized with the help of the libdmtx tool library, and the recognition result of the QR code string content can be returned. According to the Barcode (photographed barcode) data transmitted to the visual correction system by the PLC (robot system), if a DataMatrix barcode with the same QR code string content is matched, it is marked as the target matching result.

[0088] In some embodiments, step S102 may further include: matching the target barcode information with the original barcode information through a visual correction system to obtain an abnormal matching result; and sending the abnormal matching result to the robot system through the visual correction system.

[0089] Among them, the abnormal matching result is the result of the matching failure, that is, the DataMatrix barcode is recognized with the help of the l.bdmtx tool library, and the recognition result of the QR code string content can be returned. According to the Barcode (photographed barcode) data transmitted to the visual correction system by the PLc (robot system), if the DataMatrix barcode that is consistent with the QR code string content cannot be matched, it is marked as an abnormal matching result. Finally, the abnormal matching result is sent to the robot system through the visual correction system.

[0090] The target edge information may be the x and y coordinate values ​​of two end points of the L-shaped edge line of the DataMatrix barcode.

[0091] In a specific implementation, the process of calculating the target barcode coordinate information of the target barcode according to the target feature information of the target barcode on the target label image through the visual correction system can be:

[0092] First, with the help of the libdmtx tool library, the DataMatrix barcode is recognized and the recognition result of the QR code string content is returned. The QR code string content can include the corner point coordinates P1 (x dm ,y am ), and the width W of the rectangle formed by the corner point and the diagonally opposite vertex of the DataMatrix barcode dm and height h dm Based on the obtained corner point coordinates and the width and height information of the rectangle, the coordinates of the diagonal vertex of the DataMatrix barcode can be calculated as P3(x dm +W dm ,y dm +h dm ), the center point coordinates of the DataMatrix barcode are If there are multiple DataMatrix barcodes, this tool library function can return the information corresponding to multiple barcodes.

[0093] Secondly, in the step of QR code recognition, multiple DataMatrix barcode information may be recognized in the target label image. According to the original photographed barcode data transmitted to the visual correction system by the robot system (PLC), the DataMatrix barcode consistent with the QR code string content is matched. If a consistent result cannot be matched, it is fed back as an abnormal state (State output parameter) to the robot system (PLC); if a consistent result exists, the corner point coordinates P1 and the rectangle width W of the DataMatrix barcode feature edge obtained above are used. dm and height h dm , use a rectangular box to mark the matching DataMatrix barcode on the target label image.

[0094] Finally, with the help of the Hough line detection operator, two straight lines passing through the L-shaped feature corner point P1 of the DataMatrix barcode are found from the center point of the DataMatrix barcode to the surrounding areas, that is, the L-shaped edge straight lines of the DataMatrix barcode. The Hough operator returns the x and y coordinate values ​​of the two endpoints of each straight line. For example, assume that the coordinates of the two endpoints returned by line 1 are P 11 (x 11 ,y 11 ), P 12 (x 12 ,y 12 ), Line 2 returns the coordinates of the two endpoints as P 21 (x 21 ,y 21 ), P 22 (x 22 ,y 22 ), the inclination angles of the two straight lines are calculated by the following formulas:

[0095]

[0096]

[0097] Wherein, θ1 is the inclination angle of line 1, and θ2 is the inclination angle of line 2.

[0098] The positional relationship between the two straight lines (line 1 and line 2) in the L-shaped DataMatrix barcode can be solved according to the vector cross multiplication right-hand rule. The two straight line vectors passing through the L-shaped feature corner point P1 are:

[0099]

[0100]

[0101] The positional relationship between the two straight lines (straight line 1 and straight line 2) in the L-shaped DataMatrix barcode is calculated using the vector cross multiplication right-hand rule as follows:

[0102]

[0103] when When the straight line P 21 P 22 (Line 2) on Line P 11 P 12 (Line 1) to the right, then P 21 P 22 is the lower end straight line of the L-shaped DataMatrix barcode, and θ2 is taken as the DataMatrix barcode angle; when When the straight line P 21 P 22 On the straight line P 11 P 12 On the left, P 21 P 22 is the left end straight line of the L-shaped DataMatrix barcode, and θ1 is taken as the DataMatrix barcode angle; when When , the two straight lines are parallel, this situation does not exist.

[0104] Among them, the Hough line detection operator is an algorithm used in image processing to detect straight line structures in an image. Its basic idea is to convert each straight line in the image into a point in the parameter space. The vector cross multiplication (also called vector product, outer product or cross product) right-hand rule is a rule used to determine the result vector of the cross multiplication of two three-dimensional vectors. It describes how to determine the direction of the generated result vector by the right-hand rule of two vectors. For the specific implementation principles of the Hough line detection operator and the vector cross multiplication right-hand rule, reference can be made to the contents of the relevant technical solutions, and the embodiments of this application will not be repeated here.

[0105] In an embodiment of the present application, the target barcode coordinate information of the target barcode is calculated based on the target feature information of the target barcode on the target label image through the visual correction system; the target barcode coordinate information is accurately obtained through feature recognition, providing an important basis for subsequent coordinate conversion and positioning.

[0106] Step S103, calculating the actual coordinates of the label in the robot coordinate system through the visual correction system according to the target conversion relationship matrix, the reference point coordinates of the target calibration image in the robot coordinate system, and the target barcode coordinate information;

[0107] Among them, the reference point coordinates of the target transformation relationship matrix and the target calibration image in the robot coordinate system are obtained based on the target calibration method before the equipment is put into production. The target calibration method (i.e., the visual calibration method) is a method that uses a 12-point translation and rotation (9-point translation + 3-point rotation) on the plane to calibrate the robot hand and eye.

[0108] Among them, from the content of step S101, it can be seen that there are two image collections. The first image collection is performed before the equipment is put into production, in order to determine the relationship between the image coordinates and the robot coordinates, that is, to determine the relationship matrix between the robot's coordinate system and the visual system image coordinate system. The second image collection is performed during the actual production process, and each panel needs to be executed. The position correction is performed by calculating the relationship matrix between the robot's coordinates and the coordinates of the target label image and related parameters.

[0109] Among them, the target conversion relationship matrix reflects the conversion relationship between the robot coordinate system and the visual system image coordinate system. The target calibration image is the image obtained during visual calibration. Specifically, the process of obtaining the target calibration image based on the visual calibration method is to use a fixed label (not the label generated during actual production) to perform hand-eye calibration using multiple points, thereby forming multiple images. The multiple images formed are the target calibration images.

[0110] Specifically, the transformation relationship matrix is ​​used to represent the transformation relationship between the image coordinate system and the robot coordinate system, and the reference point coordinates of the target calibration image in the robot coordinate system are taken as a reference to match the robot coordinates at a specific point on the image. For example, assuming that the robot coordinates move n millimeters along the X / Y direction, the same object point moves n pixels along the X / Y axis on the image, and the transformation relationship matrix records the transformation relationship between the image coordinate system and the robot coordinate system. According to the reference point relationship between the image and the robot and the transformation relationship matrix, the image pixel points can be converted to the robot coordinate points.

[0111] It is understandable that before the equipment is put into production, the visual calibration method can be used to obtain the conversion relationship matrix and the reference point coordinates of the image in the robot coordinate system. The conversion relationship matrix records that the n pixels of the image translated in the X-axis or Y-axis direction correspond to the robot translated in the X-axis or Y-axis direction by n millimeters, while the reference point records a certain coordinate point on the image (x i ,y i ) corresponds to the coordinate point (x r ,y r). When the equipment is put into production, assuming that the pixel point of the label image is (x1, y1), the movement of the pixel point relative to the image reference point in the X-axis and Y-axis directions can be obtained, and then the movement of the robot world coordinate system in the X-axis and Y-axis directions can be obtained according to the transformation relationship matrix. Finally, the movement of the robot world coordinate system in the X-axis and Y-axis directions obtained by the transformation relationship matrix is ​​superimposed on the robot reference point coordinates (x r ,y r ), then the actual coordinate point of the robot corresponding to the label image point (x r1 ,y r1 ).

[0112] In the specific implementation, the process of obtaining the target transformation relationship matrix is ​​as follows: the visual calibration uses a 12-point translation and rotation (9-point translation + 3-point rotation) to calibrate the robot hand and eye. The camera is fixed to the robot arm and moves with the movement of the robot arm. Figure 3 , Figure 3 The process of visual and robot calibration can be as follows: first, the rotation angle R remains unchanged, the robot is translated 9 times along the X-axis or Y-axis direction, and the 5th point is used as the reference, the X-coordinate value and Y-coordinate value of the robot arm are kept unchanged, the rotation angle R is added or subtracted by a certain angle to obtain the 10th / 12th point after rotation, and the X-coordinate information x of the robot at the 12 points is recorded respectively. rl , x r2 , ..., x r12 , Y coordinate information y r1 ,y r2 , ..., y r12 and the rotation angle r r1 , r r2 , ..., r r12 , and the X coordinate x of the center point of the above DataMatrix barcode on the image c1 , x c2 , ..., x c12 , Y coordinate y c1 ,y c2 , ..., y c12 and the rotation angle r cl , r c2 , ..., r c12, where 1 to 9 are translation coordinate point information, and 10 to 12 are rotation coordinate point information. Then, with the help of OpenCV's calibrateHandEye() function, as well as the robot coordinate point information and image coordinate point information obtained by visual calibration, the function solves the conversion relationship matrix between the robot coordinates and the image coordinates, that is, the calibration scheme data matrix. For example, assuming that the robot coordinate matrix is ​​A and the image coordinate matrix is ​​B, the relationship matrix X between AX=XB can be solved.

[0113] In the embodiment of the present application, the reference points of the robot and the image are first bound, and the X coordinate, Y coordinate and rotation angle R of the robot are manually adjusted so that the center point of the DataMatrix barcode overlaps with the center point of the image, and the angle is close to being parallel or perpendicular to the bottom of the image. Then the reference point of the image is approximately Where w represents the image width, h represents the image height, and the robot coordinates P at this time are recorded ref (x rr ,y rr ) is used as the reference point coordinates of the target calibration image in the robot coordinate system.

[0114] It should be noted that visual calibration is to determine the relationship between image coordinates and robot coordinates, that is, to determine the relationship matrix between the robot's coordinate system and the visual system's image coordinate system. The calibration process is to manually move the robot to obtain 12-point coordinates, and obtain the relationship matrix X of AX=XB. During the correction process, that is, the actual production operation, it is fully automatic and does not require human intervention. The visual correction system can convert the robot's real coordinates based on the X relationship matrix without repeating the calibration action. In addition, based on visual calibration, the coordinates of the reference points of the image in the robot coordinate system can also be obtained.

[0115] Among them, OpenCV (Open Source Computer Vision Library) is an open source computer vision and machine learning software library that provides a series of functions and tools for image processing, computer vision and machine learning. calibrateHandEye() is a function in OpenCV (Open Source Computer Vision Library) that is used for hand-eye calibration between the robot and the camera, allowing the estimation of the relative transformation between the robot and the camera. For the implementation principle of OpenCV and its functions, please refer to the content of the relevant technical solution, and the embodiments of this application will not be repeated here.

[0116] See also Figure 4 , Figure 4 is a flowchart of a visual calibration method provided in an embodiment of the present application. Figure 4As shown, visual calibration can be used to determine the conversion relationship between the image coordinate system and the robot world coordinate system. Specifically: first, obtain the target label image corresponding to the label on the target panel, use the libdmtx tool library to identify the DataMatrix barcode on the target label image, and return the recognition result of the two-dimensional code string content, then match it with the original barcode information based on the recognition result, capture the edge information of the DataMatrix barcode based on the successful matching result and the recognition result, and determine the barcode angle of the DataMatrix barcode based on the edge information of the DataMatrix barcode. Finally, based on the recognition result of the two-dimensional code string content obtained above, the edge information of the DataMatrix barcode and the barcode angle of the DataMatrix barcode, perform vision and robot calibration, that is, based on the target transformation relationship matrix and the reference point coordinates of the target calibration image in the robot coordinate system, the actual coordinates of the label in the robot coordinate system can be converted.

[0117] Specifically, when the equipment is put into production, assuming that the target barcode coordinate information of the target barcode on the target label image is obtained, the target barcode coordinate information can be input into the target transformation relationship matrix, and the target transformation relationship matrix outputs the movement of the target label image in the X-axis or Y-axis direction of the robot coordinate system. Finally, the movement of the target label image converted by the target transformation relationship matrix in the X-axis or Y-axis direction of the robot coordinate system is superimposed with the reference point coordinates of the target calibration image obtained before the equipment is put into production in the robot coordinate system, and the actual coordinates of the label in the robot coordinate system can be obtained.

[0118] In the specific implementation, the visual correction system obtains the reference point coordinates of the target calibration image in the robot coordinate system based on the target calibration method. After obtaining the relationship between the image coordinate system and the robot world coordinate system, this relationship can be called in the actual production process for calculation to achieve position correction. At the same time, obtaining the reference point coordinates of the target calibration image in the robot coordinate system is conducive to correcting the system deviation caused by the robot or other factors, and providing a data basis for subsequent actual coordinate calculation and offset calculation.

[0119] In the specific implementation, after collecting the real image based on robot vision, it is assumed that the center point coordinates of the matching DataMatrix barcode are P c (x c ,y c ), angle r c , through AX = XB, we can solve the actual coordinates P of the label in the robot coordinate system r (x r ,y r ). Since the long side of the panel label is parallel to the long side of the panel, the angle of the DataMatrix barcode is rc is the angle of the robot's board suction point, that is, the rotation angle of the suction cup when the robot finally adsorbs the board.

[0120] In an embodiment of the present application, a visual correction system calculates the actual coordinates of the label in the robot coordinate system based on the target transformation relationship matrix, the reference point coordinates of the target calibration image in the robot coordinate system, and the target barcode coordinate information to achieve mapping from image space to robot space. Accurate actual coordinates of the label in the robot coordinate system are conducive to optimizing the robot's motion path and improving the efficiency of automated operations.

[0121] Step S104, calculating the positioning offset of the robot by the visual correction system according to the reference point coordinates of the target calibration image in the robot coordinate system, the actual coordinates of the label in the robot coordinate system, the theoretical coordinates of the panel and the theoretical coordinates of the label;

[0122] Since the rotation center of the robot suction cup is not coaxial with the camera center, when the angle of the panel changes, if the robot rotates a certain angle, the camera center or the suction cup center cannot be aligned with the center of the panel, resulting in positioning failure. Therefore, in the embodiment of the present application, the robot positioning offset is split into translation offset and rotation offset for ease of calculation and analysis.

[0123] In some embodiments, step S104 may include: calculating the translation offset of the robot according to the reference point coordinates of the target calibration image in the robot coordinate system and the actual coordinates of the label in the robot coordinate system through the visual correction system; calculating the rotation offset of the robot according to the theoretical coordinates of the panel and the theoretical coordinates of the label through the visual correction system.

[0124] In the specific implementation, the translation offset (x offset ,y offset ) is the actual coordinate P of the tag in the robot coordinate system r (x r ,y r ) and the reference point coordinates P of the target calibration image in the robot coordinate system ref (x rr ,y rr ) is calculated, and the calculation formula is as follows:

[0125] x offset =x r -x rr

[0126] y offset =y r -y rr

[0127] When calculating the rotation offset, the rotation angle offset also needs to be added for calculation. The calculation formula for the rotation angle offset is: offset =r c -r capture , where r capture The theoretical value of the labeling angle provided to the upstream control system is the R angle of the robot when taking the photo.

[0128] Rotation offset (x rotate ,y rotate ) is obtained by the camera robot coordinates (label theoretical coordinates) P capture (x capture ,y capture ) and the coordinates of the center point of the plate (theoretical coordinates of the plate) P panel (x panel ,y panel ) and the rotation angle offset r offset The calculation formula is as follows:

[0129] x rotate =(x panel -x capture )*COs offset +(y panel -y captute )*sinr offset

[0130] y rotate =(x panel -x capture )*Sin r offset +(y panel -y capture )*cos r offset

[0131] In an embodiment of the present application, the visual correction system calculates the positioning offset of the robot based on the reference point coordinates of the target calibration image in the robot coordinate system, the actual coordinates of the label in the robot coordinate system, the theoretical coordinates of the panel, and the theoretical coordinates of the label, and provides data support for subsequent correction based on the deviation between the actual coordinates and the theoretical coordinates.

[0132] Step S105, calculating the target panel center coordinates of the target panel according to the positioning offset through the visual correction system, and sending the target panel center coordinates to the robot system, so that the robot system controls the robot to adsorb the target panel at the target panel center coordinates.

[0133] The target panel center coordinates are the center position of the panel that the robot finally adsorbs. The target barcode angle in the target barcode coordinate information is the rotation angle of the robot when adsorbing the target panel.

[0134] In some embodiments, step S105 may include: calculating the target panel center coordinates of the target panel according to the positioning offset through the visual correction system, and sending the target panel center coordinates to the robot system; controlling the robot through the robot system to move the robot suction cup to the target panel center coordinates, and controlling the robot through the robot system to rotate the robot suction cup to the target barcode angle; controlling the robot through the robot system to use the robot suction cup to absorb the target panel at the target panel center coordinates.

[0135] Among them, the robot suction cup is a component of the robot used to adsorb panels.

[0136] In a specific implementation, the visual correction system needs to write data to the robot system (PLC). The data written by the visual correction system to the robot system (PLC) may include but is not limited to:

[0137] ActualX: data after visual correction, that is, the actual value of the X coordinate of the panel.

[0138] ActualY: data after visual correction, that is, the actual value of the Y coordinate of the panel.

[0139] ActualR: data after visual correction, that is, the actual value of the plate rotation angle.

[0140] State: The visual correction calculation state. If the barcode data can be matched and the result can be calculated, the feedback is normal. Otherwise, the feedback is abnormal.

[0141] After the visual correction returns the result to the robot system, the robot receives the corresponding data and moves the suction cup mechanism, moves the center point of the suction cup mechanism to the coordinates ActualX and ActualY, and rotates to the angle ActualR to implement the action of adsorbing the panel.

[0142] Specifically, the calculation formula of the robot's final plate suction point coordinates (x, y) is as follows:

[0143] x=x capture +x offset +x rotate

[0144] y=y capture +y offset +y rotate

[0145] After calculating the final coordinates of the robot's suction point, the visual correction system uses the calculated x as the PLC input parameter ActualX, y as ActualY, and r cAs ActualR writes data to the robot system, the robot receives the data through the robot system, controls the robot suction cup to move the corresponding coordinates (x, y), and rotates the suction cup to the corresponding angle r c , and finally the plate suction action is performed to complete the overall process.

[0146] In the embodiment of the present application, the target panel center coordinates of the target panel are calculated by the visual correction system according to the positioning offset, and the target panel center coordinates are sent to the robot system, so that the robot system controls the robot to absorb the target panel at the target panel center coordinates. The robot action is guided by the coordinate information after correction, so that the robot can accurately find and absorb the center position of the target panel, thereby quickly and accurately absorbing the panel, and improving the accuracy and efficiency of the automated operation.

[0147] In order to explain the principle of the technical solution of the present invention in detail, the overall process of the present invention is described below in combination with some specific embodiments. It is easy to understand that the following is an explanation of the technical principle of the present invention and cannot be regarded as a limitation of the present invention.

[0148] See also Figure 5 , Figure 5 is a flowchart of a panel correction method based on robot vision provided in an embodiment of the present application; Figure 5 As shown, the workflow of the vision-guided unloading system of the nesting milling machining center can be mainly divided into four modules: nesting milling machining center cutting, camera image acquisition, visual correction and robot unloading and stacking. Specifically, the panel is first placed on the nesting milling machining center for cutting. After the panel is processed in the nesting milling machining center, the equipment pushes and unloads the panel. During the process of pushing and unloading, the position of the panel may be displaced. Then, a 2D area array camera is used to collect the image data of the panel. Through image recognition technology, the actual displacement size and rotation angle of the panel are calculated to realize visual position correction. Finally, according to the corrected data, the robot is guided to move the suction cup to the center point of the panel for adsorption and unloading, thereby improving the stability of unloading and the accuracy of the stacking position of the panel.

[0149] Specifically, see Figure 6 , Figure 6 is a schematic diagram of a process flow of a panel correction method based on robot vision provided in an embodiment of the present application; Figure 6As shown in the figure, the method flow of panel correction based on robot vision can be as follows: first, the visual correction system will keep reading the trigger signal value of the PLC. When the visual correction system monitors that the Trigger value of the PLC changes from 0 to 1 (i.e., a rising edge signal, 0 indicates that the robot is in standby mode, and 1 indicates that the robot has reached the visual photography position), the image acquisition step is triggered; otherwise, the trigger signal value of the PLC continues to be read until the Trigger value of the PLC changes from 0 to 1. Then, the panel image data is collected using a 2D array camera, and the DataMatrix barcode in the panel image data is recognized with the help of the libdmtx tool library, and the recognition result of the QR code string content is returned. Next, in the step of QR code recognition, multiple DataMatrix barcode information may be recognized in the target label image. According to the original photographed barcode data transmitted by the robot system to the visual correction system, the DataMatrix barcode consistent with the QR code string content is matched. If a consistent result cannot be matched, it is fed back as an abnormal state (State output parameter) to the robot system. If there is a consistent match result, according to the QR code string content obtained above, a rectangular box is used to mark the matching DataMatrix QR code on the target label image. Furthermore, with the help of the Hough line detection operator, the center point of the DataMatrix barcode is diverged to the surroundings to find two straight lines passing through the L-shaped feature corner points of the DataMatrix barcode, that is, the L-shaped edge straight line of the DataMatrix barcode. The Hough operator returns the coordinate values ​​of the two endpoints of each straight line. The DataMatrix barcode angle is calculated based on the coordinate values ​​of the two endpoints of the two straight lines and the QR code string content. Then, based on the recognition results of the QR code string content obtained above, the edge information of the DataMatrix barcode and the barcode angle of the DataMatrix barcode, visual and robot calibration is performed, that is, the actual coordinates of the label in the robot coordinate system can be calculated based on the reference point coordinates of the target calibration image in the robot coordinate system obtained in advance by the calibration scheme, the pre-acquired conversion relationship matrix and the barcode coordinate information obtained above. Then, the positioning offset of the robot is calculated by the visual correction system according to the reference point coordinates of the target calibration image in the robot coordinate system, the actual coordinates of the label in the robot coordinate system, the theoretical coordinates of the plate and the theoretical coordinates of the label. Finally, the center coordinates of the plate are calculated by the visual correction system according to the positioning offset, and the center coordinates of the plate are sent to the robot system so that the robot system controls the robot to absorb the plate at the center coordinates of the plate, and uses the DataMatrix barcode angle calculated above as the rotation angle when the robot absorbs the plate.After calculating the robot's final plate suction point coordinates and rotation angle, the visual correction system writes the calculated plate center coordinates and rotation angles as input data into the robot system (PLC). After the robot receives the data through the robot system, it controls the robot suction cup to move the corresponding plate center coordinates and rotate the suction cup to the corresponding rotation angle, and finally performs the plate suction action to unload the material to complete the overall process. After executing the entire process, it is necessary to count the execution time of the entire process and store the log.

[0150] In the specific implementation, the plate is first placed on the nesting milling machining center for cutting. After the plate is processed in the nesting milling machining center, the equipment pushes and unloads the plate. During the process of pushing and unloading, the position of the plate may be displaced. Then, a 2D area array camera is used to collect the image data of the plate. Through image recognition technology, the actual displacement size and rotation angle of the plate are calculated to achieve visual position correction. Finally, according to the corrected data, the robot is guided to move the suction cup to the center point of the plate for adsorption and unloading, so as to realize the automated and intelligent plate unloading of the nesting milling machining center, while reducing the labor intensity of employees; improving the stability of unloading and the accuracy of the stacking position of the plates.

[0151] Specifically, the upstream control system provides the theoretical coordinates of the panel label to the PLC. After the robot receives the theoretical values ​​of the panel label from the PLC, it moves the visual camera to the theoretical coordinate position of the panel label and triggers the visual photography. Through the DataMatrix barcode characteristics on the panel label of the visual correction system, combined with the theoretical values ​​of the panel coordinates and the theoretical values ​​of the label coordinates provided by the upstream control system, the label offset position and the panel suction coordinates after correction, as well as the panel rotation angle, are solved, and these data are transmitted to the PLC. After receiving the results, the PLC adjusts the robot suction cup posture, aligns the center point of the suction cup with the center point of the panel, and the long side of the suction cup is parallel to the long side of the panel, guiding the robot to move to the real position for unloading. In order to achieve accurate adsorption of the panel and keep the stacking shape stable during unloading and stacking.

[0152] It should be pointed out that this embodiment only provides a brief schematic description of the general process of the panel correction method based on robot vision. The detailed description of each step can refer to the relevant content in the aforementioned embodiment and will not be repeated here. It can be understood that the present invention is not limited to this.

[0153] In the embodiment of the present application, the robot system controls the robot to send a shooting instruction to the visual correction system, so that the visual correction system takes a picture of the label of the target panel to obtain the target label image, and sends the panel theoretical information of the target panel to the visual correction system through the robot system; the panel theoretical information includes the panel theoretical coordinates and the label theoretical coordinates; the visual correction system calculates the target barcode coordinate information of the target barcode according to the target feature information of the target barcode on the target label image; the visual correction system calculates the actual coordinates of the label in the robot coordinate system according to the target transformation relationship matrix, the reference point coordinates of the target calibration image in the robot coordinate system and the target barcode coordinate information; the visual correction system calculates the positioning offset of the robot according to the reference point coordinates of the target calibration image in the robot coordinate system, the actual coordinates of the label in the robot coordinate system, the panel theoretical coordinates and the label theoretical coordinates; the visual correction system calculates the target panel center coordinates of the target panel according to the positioning offset, and sends the target panel center coordinates to the robot system, so that the robot system controls the robot to adsorb the target panel at the target panel center coordinates. The embodiment of the present application calculates the positioning offset of the robot based on the visual correction system to provide data support for subsequent correction; the target panel center coordinates of the target panel are calculated by the visual correction system according to the positioning offset, and the target panel center coordinates are sent to the robot system, that is, the robot can be guided to perform actions through the corrected coordinate information, so that the robot can accurately find and adsorb the center position of the target panel and then perform unloading operations, so as to improve the accuracy and production efficiency of automated operations, while reducing the labor intensity of employees.

[0154] See also Figure 7 The embodiment of the present application further provides a panel correction system 700 based on robot vision, which can implement the above-mentioned panel correction method based on robot vision. The system 700 includes:

[0155] The image acquisition module 701 is used to control the robot through the robot system to send a shooting instruction to the visual correction system, so that the visual correction system takes a picture of the label of the target panel to obtain a target label picture, and sends the panel theoretical information of the target panel to the visual correction system through the robot system; the panel theoretical information includes the panel theoretical coordinates and the label theoretical coordinates;

[0156] The barcode coordinate information acquisition module 702 is used to calculate the target barcode coordinate information of the target barcode according to the target feature information of the target barcode on the target label image through the visual correction system;

[0157] The label coordinate calculation module 703 is used to calculate the actual coordinates of the label in the robot coordinate system according to the target conversion relationship matrix, the reference point coordinates of the target calibration image in the robot coordinate system and the target barcode coordinate information through the visual correction system;

[0158] A positioning offset calculation module 704 is used to calculate the positioning offset of the robot according to the reference point coordinates of the target calibration image in the robot coordinate system, the actual coordinates of the label in the robot coordinate system, the theoretical coordinates of the panel and the theoretical coordinates of the label through the visual correction system;

[0159] The target panel center calculation module 705 is used to calculate the target panel center coordinates of the target panel according to the positioning offset through the visual correction system, and send the target panel center coordinates to the robot system so that the robot system controls the robot to adsorb the target panel at the target panel center coordinates.

[0160] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0161] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned board correction method based on robot vision when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0162] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0163] See also Figure 8 , Figure 8 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0164] The processor 801 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0165] The memory 802 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 802 can store an operating system and other applications. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 802, and the processor 801 calls and executes the board correction method based on robot vision in the embodiment of this application;

[0166] Input / output interface 803, used to implement information input and output;

[0167] The communication interface 804 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0168] A bus 805 that transmits information between the various components of the device (e.g., the processor 801, the memory 802, the input / output interface 803, and the communication interface 804);

[0169] The processor 801 , the memory 802 , the input / output interface 803 and the communication interface 804 are connected to each other in communication within the device via a bus 805 .

[0170] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned panel correction method based on robot vision is implemented.

[0171] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0172] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0173] The embodiments of the present application provide a panel correction method based on robot vision, a panel correction system based on robot vision, an electronic device and a storage medium, which control a robot through a robot system to send a shooting instruction to a visual correction system, so that the visual correction system takes a picture of the label of a target panel to obtain a target label image, and sends the panel theoretical information of the target panel to the visual correction system through the robot system; the panel theoretical information includes the panel theoretical coordinates and the label theoretical coordinates; the visual correction system calculates the target barcode coordinate information of the target barcode according to the target feature information of the target barcode on the target label image; and the visual correction system The actual coordinates of the label in the robot coordinate system are calculated according to the target conversion relationship matrix, the reference point coordinates of the target calibration image in the robot coordinate system, and the target barcode coordinate information; the positioning offset of the robot is calculated by the visual correction system according to the reference point coordinates of the target calibration image in the robot coordinate system, the actual coordinates of the label in the robot coordinate system, the theoretical coordinates of the panel, and the theoretical coordinates of the label; the target panel center coordinates of the target panel are calculated by the visual correction system according to the positioning offset, and the target panel center coordinates are sent to the robot system, so that the robot system controls the robot to absorb the target panel at the target panel center coordinates. The embodiment of the present application calculates the positioning offset of the robot based on the visual correction system to provide data support for subsequent correction; the target panel center coordinates of the target panel are calculated by the visual correction system according to the positioning offset, and the target panel center coordinates are sent to the robot system, that is, the robot can be guided to perform actions through the coordinate information after correction, so that the robot can accurately find and absorb the center position of the target panel and then perform material unloading operations, so as to improve the accuracy and production efficiency of automated operations, while reducing the labor intensity of employees.

[0174] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0175] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0176] The system embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0177] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0178] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0179] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0180] In the several embodiments provided in the present application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of systems or units, which can be electrical, mechanical or other forms.

[0181] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0182] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0183] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0184] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A plate correction method based on robot vision, characterized in that: The method comprises: The robot system controls the robot to send a shooting instruction to the visual correction system, so that the visual correction system takes a picture of the label of the target panel to obtain a target label image, and sends the panel theoretical information of the target panel to the visual correction system through the robot system; the panel theoretical information includes the panel theoretical coordinates and the label theoretical coordinates; Calculating target barcode coordinate information of the target barcode according to target feature information of the target barcode on the target label image by the visual correction system; The visual correction system calculates the actual coordinates of the label in the robot coordinate system according to the target conversion relationship matrix, the reference point coordinates of the target calibration image in the robot coordinate system, and the target barcode coordinate information; The positioning offset of the robot is calculated by the visual correction system according to the reference point coordinates of the target calibration image in the robot coordinate system, the actual coordinates of the label in the robot coordinate system, the theoretical coordinates of the panel and the theoretical coordinates of the label; The target panel center coordinates of the target panel are calculated by the visual correction system according to the positioning offset, and the target panel center coordinates are sent to the robot system, so that the robot system controls the robot to adsorb the target panel at the target panel center coordinates; the target barcode angle in the target barcode coordinate information is the rotation angle of the robot adsorbing the target panel; The positioning offset includes a translation offset and a rotation offset. The positioning offset of the robot is calculated by the visual correction system according to the reference point coordinates of the target calibration image in the robot coordinate system, the actual coordinates of the label in the robot coordinate system, the theoretical coordinates of the panel and the theoretical coordinates of the label, including: Calculating the translation offset of the robot according to the reference point coordinates of the target calibration image in the robot coordinate system and the actual coordinates of the label in the robot coordinate system by the visual correction system; Calculating the rotation offset of the robot according to the theoretical coordinates of the plate and the theoretical coordinates of the label by the visual correction system; The calculation formula of the rotation offset is: x rotate (x) panel -x capture )*cosr offset +(and panel -and capture )*without offset and rotate (x) panel -x capture )*without offset +(and panel -and capture )*cosr offset Among them, x rotate The horizontal coordinate of the rotation offset, y rotate The ordinate of the rotation offset, x panel It is expressed as the abscissa of the theoretical coordinate of the plate, y panel It is expressed as the ordinate of the theoretical coordinate of the plate, x capture Expressed as the abscissa of the label's theoretical coordinates, y capture Expressed as the ordinate of the label's theoretical coordinates, r offset Expressed as rotation angle offset; The calculation formula of the target plate center coordinates is: x=x capture +x offset +x rotate y=y capture +y offset +y rotate Wherein, x represents the horizontal coordinate of the center coordinate of the target plate, y represents the vertical coordinate of the center coordinate of the target plate, and x offset Represents the horizontal coordinate of the translation offset, y offset Expressed as the ordinate of the translation offset.

2. The method according to claim 1, characterized in that Before the robot system is used to control the robot to send a shooting instruction to the visual correction system so that the visual correction system takes a picture of the label of the target panel to obtain a target label image, and before the robot system sends the panel theoretical information of the target panel to the visual correction system, the method further includes: The panel theoretical information of the target panel is acquired through the software control system, and the panel theoretical information is sent to the robot system.

3. The method according to claim 1, characterized in that The method of controlling the robot through the robot system to send a shooting instruction to the visual correction system so that the visual correction system takes a picture of the label of the target panel to obtain a target label picture, and sending the panel theoretical information of the target panel to the visual correction system through the robot system includes: The robot system controls the robot to move to the label theoretical coordinates, and controls the robot to issue a shooting instruction to the visual correction system, so that the visual correction system takes a picture of the label of the target panel at the label theoretical coordinates to obtain the target label image, and sends the panel theoretical information of the target panel to the visual correction system through the robot system.

4. The method according to claim 1, characterized in that The panel theoretical information also includes original barcode information, and the target barcode coordinate information includes the target barcode center point coordinates and the target barcode angle; the target barcode coordinate information of the target barcode is calculated by the visual correction system according to the target feature information of the target barcode on the target label image, including: The target barcode on the target label image is identified by the visual correction system to obtain target barcode information; Calculate the target barcode center point coordinates of the target barcode according to the target barcode information by the visual correction system; Matching the target barcode information with the original barcode information through the visual correction system to obtain a target matching result; Acquiring target edge information of the target barcode according to the target matching result by the visual correction system; The visual correction system calculates a target barcode angle of the target barcode according to the target edge information.

5. The method according to claim 4, characterized in that The method further comprises: Matching the target barcode information with the original barcode information through the visual correction system to obtain an abnormal matching result; The abnormal matching result is sent to the robot system through the visual correction system.

6. The method according to claim 1, characterized in that The method of calculating the target panel center coordinates of the target panel according to the positioning offset by the visual correction system, and sending the target panel center coordinates to the robot system so that the robot system controls the robot to absorb the target panel at the target panel center coordinates includes: Calculating the target panel center coordinates of the target panel according to the positioning offset by the visual correction system, and sending the target panel center coordinates to the robot system; Control the robot by the robot system to move the robot suction cup to the target panel center coordinate, and control the robot by the robot system to rotate the robot suction cup to the target barcode angle; The robot system controls the robot to use the robot suction cup to absorb the target panel at the center coordinate of the target panel.

7. A panel correction system based on robot vision, characterized in that: The system comprises: The image acquisition module is used to control the robot through the robot system to send a shooting instruction to the visual correction system, so that the visual correction system takes a picture of the label of the target panel to obtain the target label picture, and sends the panel theoretical information of the target panel to the visual correction system through the robot system; the panel theoretical information includes the panel theoretical coordinates and the label theoretical coordinates; A barcode coordinate information acquisition module, used to calculate the target barcode coordinate information of the target barcode according to the target feature information of the target barcode on the target label image through the visual correction system; A label coordinate calculation module, used to calculate the actual coordinates of the label in the robot coordinate system according to the target conversion relationship matrix, the reference point coordinates of the target calibration image in the robot coordinate system and the target barcode coordinate information through the visual correction system; A positioning offset calculation module, used to calculate the positioning offset of the robot according to the reference point coordinates of the target calibration image in the robot coordinate system, the actual coordinates of the label in the robot coordinate system, the theoretical coordinates of the panel and the theoretical coordinates of the label through the visual correction system; A target panel center calculation module is used to calculate the target panel center coordinates of the target panel according to the positioning offset through the visual correction system, and send the target panel center coordinates to the robot system, so that the robot system controls the robot to adsorb the target panel at the target panel center coordinates; the target barcode angle in the target barcode coordinate information is the rotation angle of the robot adsorbing the target panel; The positioning offset includes a translation offset and a rotation offset, and the positioning offset calculation module is specifically used for: Calculating the translation offset of the robot according to the reference point coordinates of the target calibration image in the robot coordinate system and the actual coordinates of the label in the robot coordinate system by the visual correction system; Calculating the rotation offset of the robot according to the theoretical coordinates of the plate and the theoretical coordinates of the label by the visual correction system; The calculation formula of the rotation offset is: x rotate (x) panel -x capture )*cosr offset +(and panel -and capture )*without offset and rotate (x) panel -x capture )*without offset +(and panel -and capture )*cosr offset Among them, x rotate The horizontal coordinate of the rotation offset, y rotate The ordinate of the rotation offset, x panel It is expressed as the abscissa of the theoretical coordinate of the plate, y panel It is expressed as the ordinate of the theoretical coordinate of the plate, x capture Expressed as the abscissa of the label's theoretical coordinates, y capture Expressed as the ordinate of the label's theoretical coordinates, r offset Expressed as rotation angle offset; The calculation formula of the target plate center coordinates is: x=x capture +x offset +x rotate y=y capture +y offset +y rotate Wherein, x represents the horizontal coordinate of the center coordinate of the target plate, y represents the vertical coordinate of the center coordinate of the target plate, and x offset Represents the horizontal coordinate of the translation offset, y offset Expressed as the ordinate of the translation offset.

8. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images