Fitting alignment guiding method and device, equipment, storage medium and program product

Abstract

The invention discloses a fitting alignment guiding method and device, equipment, a storage medium and a program product. The method comprises the following steps: acquiring first image data of a target product and second image data of a to-be-fitted product according to a calibration camera; acquiring a position offset error of the calibration camera moving from the acquisition position of the first image data to the acquisition position of the second image data, and adjusting the second image data according to the position offset error; obtaining first coordinate information and second coordinate information of product marks in the first image data and the adjusted second image data, and determining an adjustment offset angle and an adjustment offset according to the first coordinate information and the second coordinate information; and controlling the to-be-laminated product to be laminated with the target product according to the adjustment deviation angle and the adjustment deviation amount. The position deviation caused by camera movement calibration can be effectively eliminated through the position deviation error, the product lamination precision is improved, product lamination can be automatically controlled according to the deviation amount, manual intervention is reduced, and the lamination stability is ensured.

Description

Technical Field

[0001] The present invention relates to the field of camera-guided positioning, and particularly to a method, device, equipment, storage medium and program product for fitting and alignment guidance. Background Art

[0002] In modern precision manufacturing, the alignment and fitting process of double-layer or even multi-layer products requires extremely high alignment accuracy to ensure excellent performance and high reliability of the final product. As a key means to achieve this high-precision goal, the camera alignment guidance technology can capture the image information of the product, thereby guiding the fitting equipment to complete high-precision alignment operations. Utilizing the camera alignment technology can significantly improve the production efficiency of double-layer and multi-layer transparent products and optimize product quality. However, when dealing with the alignment of double-layer or multi-layer products with a large thickness, a common practice in traditional camera alignment technology is to customize a large-depth-of-field lens to meet the product thickness requirements. However, customizing the lens is costly, and large-depth-of-field lenses often suffer from the problem of blurred imaging edges, which affects the positioning accuracy. Another common practice is to use multiple cameras with a small depth of field for collaborative alignment guidance. However, for products beyond the depth-of-field range, operators still need to manually debug based on experience, increasing the debugging difficulty and thus resulting in unstable alignment accuracy. Therefore, providing a camera fitting and alignment guidance method with low debugging difficulty, high and stable positioning accuracy for products with a large thickness has become an urgent problem to be solved in the field of camera-guided positioning. Summary of the Invention

[0003] The present invention provides a method, device, equipment, storage medium and program product for fitting and alignment guidance, which improves the product fitting accuracy, reduces manual intervention, and ensures fitting stability.

[0004] On the one hand, an embodiment of the present invention provides a method for fitting and alignment guidance, including:

[0005] Obtaining first image data of a target product and second image data of a product to be fitted by a calibrated camera;

[0006] Obtaining the position offset error of the calibrated camera moving from the acquisition position of the first image data to the acquisition position of the second image data, and adjusting the second image data according to the position offset error;

[0007] Obtaining first coordinate information and second coordinate information of product markers in the first image data and the adjusted second image data, and determining an adjustment offset angle and an adjustment offset amount according to the first coordinate information and the second coordinate information;

[0008] Controlling the product to be fitted to fit the target product according to the adjustment offset angle and the adjustment offset amount.

[0009] On the other hand, an embodiment of the present invention provides a fitting alignment guiding device, including:

[0010] A data acquisition module, configured to acquire first image data of a target product and second image data of a product to be fitted according to a calibration camera;

[0011] An error acquisition module, configured to acquire a position offset error of the calibration camera moving from the acquisition position of the first image data to the acquisition position of the second image data, and adjust the second image data according to the position offset error;

[0012] An offset amount determination module, configured to acquire first coordinate information and second coordinate information of product markings in the first image data and the adjusted second image data, and determine an adjustment offset angle and an adjustment offset amount according to the first coordinate information and the second coordinate information;

[0013] A fitting control module, configured to control the product to be fitted to fit the target product according to the adjustment offset angle and the adjustment offset amount.

[0014] On the other hand, an embodiment of the present invention provides a device, including:

[0015] At least one processor;

[0016] And a memory communicatively connected to the at least one processor;

[0017] Wherein, the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the fitting alignment guiding method of any embodiment of the embodiments of the present invention.

[0018] On the other hand, an embodiment of the present invention provides a computer-readable storage medium, including: computer instructions, and the computer instructions are used to enable a processor to execute the fitting alignment guiding method of any embodiment of the embodiments of the present invention.

[0019] On the other hand, an embodiment of the present invention provides a computer program product, including:

[0020] A computer program, and the computer program executes the fitting alignment guiding method of any embodiment of the embodiments of the present invention when being executed by a processor.

[0021] In the embodiments of the present invention, a calibration camera can be used to photograph a target product and a product to be attached. The target product image and the product-to-be-attached image captured by the calibration camera can be obtained from the calibration camera. Image processing can be performed on the obtained target product image and the product-to-be-attached image to obtain the first image data of the target product and the second image data of the product to be attached. The position offset error when the calibration camera moves to the acquisition position of the second image data can be obtained. The second image data collected by the calibration camera can be adjusted according to the obtained position offset error. The first coordinate information for determining the position of the product mark in the target product can be extracted from the first image data, and the second coordinate information for determining the position of the product mark in the product to be attached can be extracted from the adjusted second image data. The angle difference in the rotation direction and the distance difference in the translation direction of the product to be attached relative to the target product during the attachment process, that is, the adjustment offset angle and the adjustment offset amount of the product to be attached relative to the target product, can be determined according to the extracted first coordinate information and second coordinate information. The product to be attached can be adjusted based on the obtained adjustment offset angle and adjustment offset amount so that the product to be attached is attached to the target product. In the embodiments of the present invention, a calibration camera with a small depth of field characteristic can be used to photograph the target product and the product to be attached, and a product image with clear edges can be obtained, ensuring that the position of the product mark can be determined in the product image data and improving the accuracy of product attachment. By obtaining the position offset error of the calibration camera when moving from the acquisition position of the first image data to the acquisition position of the second image data, the deviation of the product mark position information in the second image data caused by factors such as the movement of the calibration camera can be effectively eliminated, further improving the product attachment accuracy. During the subsequent process of attaching multiple products to be attached to the target product, the shooting position of the calibration camera can be automatically adjusted according to the determined position offset error, simplifying the debugging process of the calibration camera and reducing the difficulty of guiding product attachment based on the calibration camera. Through the calculated adjustment offset angle and adjustment offset amount, automatic attachment of the product to be attached to the target product can be achieved, reducing manual intervention and ensuring the stability of the attachment.

[0022] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present invention, nor is it used to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0024] Figure 1 It is a flowchart of a fitting alignment guiding method provided according to Embodiment 1 of the present invention;

[0025] Figure 2 It is another flowchart of a fitting alignment guiding method provided according to Embodiment 2 of the present invention;

[0026] Figure 3 It is a schematic diagram showing the relationship of the movement error of a calibration camera provided according to Embodiment 3 of the present invention;

[0027] Figure 4 It is a schematic diagram of the relative position between a calibration camera and a target product provided according to Embodiment 3 of the present invention;

[0028] Figure 5 It is a schematic diagram of the structure of a fitting alignment guiding device provided according to Embodiment 4 of the present invention;

[0029] Figure 6 It is a block diagram of a device for performing the fitting alignment guiding method provided according to Embodiment 5 of the present invention. Detailed implementation manners

[0030] In order to enable those skilled in the art to better understand the solution of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0031] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily need to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or units does not necessarily need to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0032] Embodiment 1

[0033] Figure 1FIG. 0 is a flowchart of a method for guiding alignment of bonding provided by Embodiment 1 of the present invention. The embodiments of the present invention are applicable to the case where the product has a relatively large thickness, and the cost of bonding guidance is limited but the bonding accuracy requirement is relatively high. This method can be executed by a bonding alignment guiding device, which can be implemented in the form of hardware and / or software, and the bonding alignment guiding device can be configured in a device. As Figure 1 shown, the method includes:

[0034] S101. Obtain first image data of a target product and second image data of a product to be bonded according to a calibrated camera.

[0035] Among them, the calibrated camera can be understood as a device for capturing product image data. For example, the calibrated camera can be associated with a motion control unit, and the motion control unit can be responsible for controlling the movement of the calibrated camera.

[0036] The target product refers to an item used as a reference standard during the bonding alignment guiding process, and can be used to guide the bonding process of the product to be bonded.

[0037] The first image data can be understood as a set composed of digital information. The first image data can be obtained by quantifying the image of the target product captured by the calibrated camera in digital form. Among them, the target product image can be understood as an image captured by the calibrated camera when photographing product marks on the target product. For example, the first image data at least includes the position information of the product marks on the target product.

[0038] The product to be bonded refers to another item that needs to be bonded to the target product. For example, during the bonding process with the target product, the product to be bonded can be adjusted in position and angle to fit the target product.

[0039] The second image data is similar to the first image data and can be understood as another set composed of digital information. The second image data can be obtained by quantifying the image of the product to be bonded captured by the calibrated camera in digital form. Among them, the product to be bonded image can be understood as an image captured by the calibrated camera when photographing product marks on the product to be bonded. For example, the second image data at least includes the position information of the product marks on the product to be bonded.

[0040] Specifically, the calibrated camera can be used to photograph the target product and the product to be bonded. The image of the target product and the image of the product to be bonded captured by the calibrated camera can be obtained from the calibrated camera, and the obtained images of the target product and the product to be bonded can be processed to obtain the first image data of the target product and the second image data of the product to be bonded.

[0041] Exemplarily, when using a calibration camera to photograph a target product and a product to be attached, the number of calibration cameras is at least one. That is, one calibration camera can be used to photograph multiple product marking positions on the target product and the product to be attached, or the same number of calibration cameras as the product markings can be used to photograph the product markings.

[0042] Exemplarily, before using the calibration camera to obtain the first image data of the target product and the second image data of the product to be attached, the following steps may be included: The operator can use the motion control unit to debug the calibration camera and adjust the focal length of the calibration camera to ensure the clarity of the image data obtained by the calibration camera.

[0043] S102. Obtain the position offset error of the calibration camera moving from the acquisition position of the first image data to the acquisition position of the second image data, and adjust the second image data according to the position offset error.

[0044] Among them, the position offset error refers to the difference value between the observation position and the expected position of the calibration camera after it moves to the acquisition position of the second image data. Among them, the observation position can refer to the specific actual position where the calibration camera is located when photographing the product to be attached, and the expected position can refer to the ideal position where the calibration camera should be theoretically located when photographing the product to be attached. Exemplarily, the position offset error can include: the horizontal axis direction offset error and the vertical axis direction offset error, and the position offset error can be a positive number or a negative number.

[0045] Specifically, the calibration camera can be moved to photograph the product to be attached to obtain the second image data. When the calibration camera moves to the acquisition position of the second image data, the position where the calibration camera is located may have a position offset error due to various factors. The position offset error of the calibration camera when it moves to the acquisition position of the second image data can be obtained, and the second image data collected by the calibration camera can be adjusted according to the obtained position offset error to make the product to be attached fit the target product precisely.

[0046] Exemplarily, various factors that cause the calibration camera to have a position offset error can include: factors such as the motion axis of the calibration camera not being ideally vertical or environmental temperature changes.

[0047] S103. Obtain the first coordinate information and the second coordinate information of the product markings in the first image data and the adjusted second image data, and determine the adjustment offset angle and the adjustment offset amount according to the first coordinate information and the second coordinate information.

[0048] Among them, the product mark refers to a mark used to identify a specific position on the product to be bonded and the target product. For example, the product mark may be a symbolic pattern or a symbolic symbol, etc. It is understandable that the adjustment offset angle and the adjustment offset amount of the product to be bonded relative to the target product can be determined based on the coordinate information of the product mark. To determine the adjustment offset angle of the product to be bonded relative to the target product based on the coordinate information of the product mark, at least two product marks need to be set on the product to be bonded and the target product respectively. The positions of the product marks on the product to be bonded and the target product are not limited.

[0049] The first coordinate information can be understood as a set of values used to determine the position of the product mark in the target product. For example, the first coordinate information includes at least two position coordinates of the product marks, so as to determine together with the second coordinate information the adjustment offset angle of the product to be bonded relative to the target product.

[0050] The second coordinate information is similar to the first coordinate information, and can be understood as a set of numerical values used to determine the position of the product mark in the product to be bonded. For example, the second coordinate information includes at least two position coordinates of the product marks, so as to determine together with the first coordinate information the adjustment offset angle of the product to be bonded relative to the target product.

[0051] Adjusting the offset angle refers to the angular difference between the product to be bonded and the target product in the rotation direction during the bonding process. The adjusting offset angle can be used to guide the rotation of the product to be bonded so that the product to be bonded is parallel to the target product. For example, the adjusting offset angle can include: 10 degrees or 15 radians. The unit of the adjusting offset angle is not limited here.

[0052] The adjustment offset refers to the distance difference between the product to be bonded and the target product in the translation direction during the bonding process, which can be used to guide the translation of the product to be bonded so that the product to be bonded overlaps with the target product. By way of example, the adjustment offset at least includes: a horizontal axis adjustment offset and a vertical axis adjustment offset, wherein the units of the adjustment offset include but are not limited to: millimeters, centimeters and / or meters.

[0053] Specifically, the first image data and the adjusted second image data are obtained, and the first coordinate information for determining the product mark position in the target product can be extracted from the first image data, and the second coordinate information for determining the product mark position in the product to be bonded can be extracted from the adjusted second image data; the angle difference in the rotation direction and the distance difference in the translation direction of the product to be bonded relative to the target product during the bonding process can be determined based on the extracted first coordinate information and the second coordinate information, that is, the adjusted offset angle and the adjusted offset amount of the product to be bonded relative to the target product can be determined based on the extracted first coordinate information and the second coordinate information. It can be understood that the first coordinate information and the second coordinate information need to be in the same coordinate system.

[0054] Exemplarily, the steps of determining the adjustment offset angle according to the first coordinate information and the second coordinate information may include: determining a reference straight line of the target product based on the first coordinate information, determining an offset straight line of the product to be fitted based on the second coordinate information, and taking the included angle between the reference straight line and the offset straight line as the adjustment offset angle.

[0055] Exemplarily, the steps of determining the adjustment offset amount according to the first coordinate information and the second coordinate information may include: using the adjusted second image data of the product to be fitted as the third image data, adjusting the third image data according to the adjustment offset angle, and obtaining the third coordinate information of the product mark in the third image data; taking the difference between the third coordinate information and the first coordinate information as the adjustment offset amount.

[0056] S104. Control the product to be fitted to fit the target product according to the adjustment offset angle and the adjustment offset amount.

[0057] Specifically, obtaining the adjustment offset angle and the adjustment offset amount of the product to be fitted relative to the target product, and adjusting the product to be fitted based on the obtained adjustment offset angle and adjustment offset amount, so that the product to be fitted is fitted to the target product.

[0058] In the embodiment of the present invention, a calibration camera can be used to photograph a target product and a product to be attached. The target product image and the product-to-be-attached product image captured by the calibration camera can be obtained from the calibration camera. Image processing can be performed on the obtained target product image and the product-to-be-attached product image to obtain the first image data of the target product and the second image data of the product to be attached. The position offset error when the calibration camera moves to the acquisition position of the second image data can be obtained. The second image data collected by the calibration camera can be adjusted according to the obtained position offset error. The first coordinate information for determining the product mark position in the target product can be extracted from the first image data, and the second coordinate information for determining the product mark position in the product to be attached can be extracted from the adjusted second image data. The angle difference in the rotation direction and the distance difference in the translation direction of the product to be attached relative to the target product during the attachment process, that is, the adjustment offset angle and the adjustment offset amount of the product to be attached relative to the target product, can be determined according to the extracted first coordinate information and the second coordinate information. The product to be attached can be adjusted based on the obtained adjustment offset angle and adjustment offset amount so that the product to be attached is attached to the target product. In the embodiment of the present invention, a calibration camera with a small depth of field characteristic can be used to photograph the target product and the product to be attached, and a product image with clear image edges can be obtained, ensuring that the position of the product mark can be determined in the product image data and improving the accuracy of product attachment. By obtaining the position offset error of the calibration camera when moving from the acquisition position of the first image data to the acquisition position of the second image data, the deviation of the product mark position information in the second image data caused by factors such as the movement of the calibration camera can be effectively eliminated, further improving the product attachment accuracy. In the process of subsequent multi-layer products to be attached to the target product, the shooting position of the calibration camera can be automatically adjusted according to the determined position offset error, simplifying the debugging process of the calibration camera and reducing the difficulty of product attachment guided by the calibration camera. Through the calculated adjustment offset angle and adjustment offset amount, automatic attachment of the product to be attached to the target product can be achieved, reducing manual intervention and ensuring the stability of the attachment.

[0059] Based on the above embodiments, before obtaining the position offset error of the calibration camera when moving from the acquisition position of the first image data to the acquisition position of the second image data in the embodiment of the present invention, it includes: obtaining different image data corresponding to different product marks on the product to be attached according to the calibration camera; extracting the product mark coordinates of each product mark in each image data, and obtaining the horizontal coordinate movement error relationship and the vertical coordinate movement error relationship of the position offset error of the corresponding calibration camera respectively fitted based on the horizontal coordinates and the vertical coordinates in each product mark coordinate as the camera movement error mapping relationship.

[0060] Among them, the image data can be understood as a collection of a series of digital information. For example, the image data at least includes the coordinate information of different product marks on the product to be attached. It can be understood that all the image data corresponding to different product marks on the target product obtained according to the calibrated camera can be used as the first image data of the target product. All the image data corresponding to different product marks on the product to be attached obtained according to the calibrated camera can be used as the second image data of the product to be attached.

[0061] The product mark coordinates can be understood as a series of numerical values representing the positions of the product marks, which can be used to guide the determination of the camera movement error mapping relationship of the calibrated camera when shooting the product to be attached. It can be understood that all the product mark coordinates extracted from the target product image data can be used as the first coordinate information. All the product mark coordinates extracted from the product to be attached image data can be used as the second coordinate information.

[0062] The camera movement error mapping relationship can be understood as a mathematical expression, which can be used to guide the correction of the position of the calibrated camera when shooting different product marks on the product to be attached, and can be used to describe the position deviation of the calibrated camera when shooting different product marks on the product to be attached. For example, the camera movement error mapping relationship can include: the horizontal coordinate movement error relationship and the vertical coordinate movement error relationship. Among them, the horizontal coordinate movement error relationship can be understood as a mathematical expression describing the position deviation of the calibrated camera in the X direction when shooting different product marks on the product to be attached, and the vertical coordinate movement error relationship can be understood as a mathematical expression describing the position deviation of the calibrated camera in the Y direction when shooting different product marks on the product to be attached.

[0063] Specifically, before obtaining the position offset error of the calibrated camera moving from the acquisition position of the first image data to the acquisition position of the second image data, it may further include the step of obtaining the camera movement error mapping relationship of the calibrated camera: at least one product mark can be set on the product to be attached, the calibrated camera can be used to shoot each product mark, the image data corresponding to each product mark can be obtained, and all the product mark coordinates corresponding to each product mark can be extracted from each image data. Among them, the product mark coordinates at least include the horizontal coordinate and the vertical coordinate of the product mark. The horizontal coordinate movement error relationship used to describe the position deviation of the calibrated camera in the X direction when shooting different product marks on the product to be attached can be fitted according to the obtained horizontal coordinate distribution, and the vertical coordinate movement error relationship used to describe the position deviation of the calibrated camera in the Y direction when shooting different product marks on the product to be attached can be fitted according to the obtained vertical coordinate distribution. The horizontal coordinate movement error relationship and the vertical coordinate movement error relationship can be integrated into the camera movement error mapping relationship of the calibrated camera.

[0064] Based on the above embodiments, the embodiments of the present invention further include: converting the coordinate information of each image data into the same coordinate system.

[0065] Specifically, different image data corresponding to different product marks on the product to be bonded can be obtained, different coordinate information corresponding to different product marks can be extracted from different data, and different coordinate information can be converted into the same coordinate system, so as to unify the measurement standard and improve the calculation efficiency of adjusting the offset angle and the offset amount.

[0066] Embodiment 2

[0067] Figure 2 The following is a flowchart of another fitting alignment guiding method provided by the embodiments of the present invention. Based on the above embodiments, the embodiments of the present invention provide another fitting alignment guiding method. As Figure 2 shown, the method includes:

[0068] S201. Obtain the first image data of the target product and the second image data of the product to be bonded according to the calibrated camera.

[0069] It can be understood that the positions of the calibrated camera when shooting the target product and the product to be bonded can vary according to actual needs. For example, the shooting position of the calibrated camera can be changed according to the size and / or dimensions of the target product and the product to be bonded, which can enhance the flexibility and adaptability of the fitting alignment guiding method.

[0070] S202. Obtain the camera movement error mapping relationship of the calibrated camera.

[0071] Specifically, different image data of the calibrated camera when shooting different product marks of the product to be bonded can be obtained, the coordinate information of different product marks can be extracted from different image data, and a camera movement error mapping relationship for describing the position deviation of the calibrated camera when shooting different product marks on the product to be bonded can be fitted according to the extracted coordinate information.

[0072] S203. Extract the product mark coordinates of the product marks in the second image data.

[0073] Specifically, the second image data of the product to be bonded can be obtained according to the calibrated camera, and the product mark coordinates of the product marks on the product to be bonded can be extracted from the second image data.

[0074] For example, the method of extracting the product mark coordinates of the product marks on the product to be bonded in the second image data can include: extracting by using image processing technology or extracting by using optical character recognition technology, etc.

[0075] S204. Call the horizontal movement error relationship and the vertical movement error relationship in the camera movement error mapping relationship to determine the adjusted horizontal coordinate and the adjusted vertical coordinate of the product mark coordinates.

[0076] Among them, the adjusted horizontal coordinate can be understood as a value describing the position in the X direction, which refers to the position coordinate after correcting the horizontal coordinate in the product mark coordinates of the product mark in the second image data based on the horizontal movement error relationship.

[0077] The adjusted vertical coordinate can be understood as a value describing the position in the Y direction, which refers to the position coordinate after correcting the vertical coordinate in the product mark coordinates of the product mark in the second image data based on the vertical movement error relationship.

[0078] Specifically, the camera movement error mapping relationship of the calibrated camera and the second image data of the product to be fitted can be obtained. The product mark coordinates of the product mark on the product to be fitted can be extracted from the second image data. The horizontal coordinate in the product mark coordinates can be adjusted according to the horizontal movement error relationship in the camera movement error mapping relationship. The adjusted horizontal coordinate in the product mark coordinates can be used as the adjusted horizontal coordinate of the product mark coordinates. The vertical coordinate in the product mark coordinates can be adjusted according to the vertical movement error relationship in the camera movement error mapping relationship. The adjusted vertical coordinate in the product mark coordinates can be used as the adjusted vertical coordinate of the product mark coordinates.

[0079] S205. Obtain the first coordinate information and the second coordinate information of the product mark in the first image data and the adjusted second image data.

[0080] S206. Determine the reference line of the target product based on the first coordinate information, determine the offset line of the product to be fitted based on the second coordinate information, and use the angle between the reference line and the offset line as the adjusted offset angle.

[0081] Among them, the reference line is a line determined according to the first coordinate information of the product mark on the target product. The reference line can be used as a reference line for the offset line to determine the adjusted offset angle of the product to be fitted. It can be understood that at least two product mark coordinates are required to determine a line, that is, at least two product mark coordinates of the product mark are included in the first coordinate information.

[0082] The offset line is a line determined according to the second coordinate information of the product mark on the product to be fitted, and can be used together with the reference line to determine the adjusted offset angle of the product to be fitted. It can be understood that at least two product mark coordinates are required to determine a line, that is, at least two product mark coordinates of the product mark are also included in the second coordinate information.

[0083] Specifically, the first image data of the target product and the second image data of the product to be fitted after being adjusted according to the camera movement error mapping relationship can be obtained. The first coordinate information can be extracted from the first image data, and the second coordinate information can be extracted from the second image data. Among them, at least two product marker coordinates of the product markers are included in both the first coordinate information and the second coordinate information. Based on the extracted first coordinate information and the determined reference line of the target product, the offset line of the product to be fitted can be determined according to the second coordinate information. The angle between the reference line and the offset line can be determined based on the reference line and the offset line, and the angle between the reference line and the offset line can be used as the adjustment offset angle of the product to be fitted.

[0084] S207. Take the adjusted second image data of the product to be fitted as the third image data, adjust the third image data according to the adjustment offset angle, and obtain the third coordinate information of the product markers in the third image data.

[0085] Among them, the third image data is similar to the first image data and the second image data, and can also be understood as a set composed of digital information. It can be understood that the third image data refers to the second image data adjusted according to the camera movement error mapping relationship.

[0086] The third coordinate information is similar to the first coordinate information and the second coordinate information, and can be understood as a set of values used to determine the positions of the product markers in the product to be fitted. It can be understood that the third coordinate information refers to the second coordinate information adjusted according to the camera movement error mapping relationship.

[0087] Specifically, the second image data adjusted according to the camera movement error mapping relationship can be taken as the third image data, and the third coordinate information of each product marker on the product to be fitted can be extracted from the third image data. It can be understood that the third coordinate information refers to the second coordinate information adjusted according to the camera movement error mapping relationship.

[0088] S208. Take the difference between the third coordinate information and the first coordinate information as the adjustment offset amount, where the adjustment offset amount at least includes the horizontal axis adjustment offset amount and the vertical axis adjustment offset amount.

[0089] Specifically, the difference between the third coordinate information and the first coordinate information can be taken as the adjustment offset amount of the product to be fitted. Among them, the difference between the horizontal axis coordinates in the third coordinate information and the horizontal axis coordinates in the first coordinate information can be taken as the horizontal axis adjustment offset amount of the product to be fitted, and the difference between the vertical axis coordinates in the third coordinate information and the vertical axis coordinates in the first coordinate information can be taken as the vertical axis adjustment offset amount of the product to be fitted.

[0090] S209. Control the product to be fitted to fit the target product according to the adjustment offset angle and the adjustment offset amount.

[0091] In an embodiment of the present invention, the first image data of the target product and the second image data of the product to be attached can be obtained according to the calibration camera. Different image data of different product markings of the product to be attached can be obtained by the calibration camera. The coordinate information of different product markings can be extracted from different image data. A camera movement error mapping relationship for describing the position deviation of the calibration camera when photographing different product markings on the product to be attached can be fitted according to the extracted coordinate information. The second image data of the product to be attached can be obtained according to the calibration camera. The product marking coordinates of the product markings on the product to be attached can be extracted from the second image data. The camera movement error mapping relationship of the calibration camera and the second image data of the product to be attached can be obtained. The product marking coordinates of the product markings on the product to be attached can be extracted from the second image data. The abscissa in the product marking coordinates can be adjusted according to the abscissa movement error relationship in the camera movement error mapping relationship. The adjusted abscissa in the product marking coordinates can be used as the adjusted abscissa of the product marking coordinates. The ordinate in the product marking coordinates can be adjusted according to the ordinate movement error relationship in the camera movement error mapping relationship. The adjusted ordinate in the product marking coordinates can be used as the adjusted ordinate of the product marking coordinates. The first coordinate information and the second coordinate information of the product markings in the first image data and the adjusted second image data can be obtained. The first coordinate information can be extracted from the first image data, and the second coordinate information can be extracted from the second image data. Among them, both the first coordinate information and the second coordinate information contain at least the product marking coordinates of two product markings. The reference line of the target product can be determined according to the extracted first coordinate information. The offset line of the product to be attached can be determined according to the second coordinate information. The included angle between the reference line and the offset line can be determined according to the reference line and the offset line. The included angle between the reference line and the offset line can be used as the adjusted offset angle of the product to be attached. The second image data adjusted according to the camera movement error mapping relationship can be used as the third image data. The third coordinate information of each product marking on the product to be attached can be extracted from the third image data. It can be understood that the third coordinate information refers to the second coordinate information adjusted according to the camera movement error mapping relationship. The difference between the third coordinate information and the first coordinate information can be used as the adjusted offset amount of the product to be attached. Among them, the difference between the abscissa coordinates in the third coordinate information and the abscissa coordinates in the first coordinate information can be used as the horizontal axis adjusted offset amount of the product to be attached. The difference between the ordinate coordinates in the third coordinate information and the ordinate coordinates in the first coordinate information can be used as the vertical axis adjusted offset amount of the product to be attached. The product to be attached can be controlled to be attached to the target product according to the adjusted offset angle and the adjusted offset amount.In the embodiments of the present invention, by invoking the horizontal movement error relationship and the vertical movement error relationship in the camera movement error mapping relationship to adjust the product marking coordinates of the product, the position deviation caused by factors such as the movement of the calibrated camera can be effectively eliminated, ensuring the fitting accuracy of the product to be fitted and the target product; by using the angle between the reference line and the offset line as the adjustment offset angle, the deviation of the product to be fitted relative to the target product in terms of angle can be accurately measured, improving the accuracy and stability of product fitting; based on the calculated adjustment offset angle and adjustment offset amount, the automatic fitting of the product to be fitted and the target product can be realized, reducing manual intervention and ensuring the stability of fitting.

[0092] Embodiment III

[0093] On the basis of the above embodiments, the embodiments of the present invention provide a method for guiding the fitting of two layers of transparent products by using four calibrated cameras. The method includes: four positions can be selected on the target product and the product to be fitted respectively for product marking, and the four calibrated cameras will respectively photograph the four product markings. When using the calibrated camera to guide the product to be fitted to fit the target product, the calibrated camera needs to be moved and calibrated, that is, when using the calibrated camera to photograph the product to be fitted, the calibrated camera needs to move up or down along the movement axis. However, in actual operation, the movement axis where the calibrated camera is located is not very ideally perpendicular to the target product. Therefore, when the calibrated camera moves up or down, deviations will occur in the horizontal axis direction and the vertical axis direction, resulting in the product to be fitted not being able to accurately fit the target product. To eliminate the error generated when the calibrated camera performs movement calibration, the calibrated camera can be used to calibrate the product markings of the product to be fitted at different heights, as Figure 3 shown, and record the position coordinates of each calibrated camera projected in the coordinate system where the product to be fitted is located. The horizontal movement error relationship x’ = k1X + b1 and the vertical movement error relationship y’ = k2Y + b2 can be calculated based on the above-recorded position coordinates. Among them, x’ and y’ are the accurate coordinates of the product markings in the product to be fitted, X and Y are the initial position coordinates of the product markings of the product to be fitted, k1 and k2 are the slopes in the movement error relationship, k1 and k2 are the intercepts in the movement error relationship, and k1, k2, k1, and k2 can be calculated according to the above different position coordinates. The product to be fitted can be adjusted according to the horizontal movement error relationship and the vertical movement error relationship. In the embodiments of the present invention, only one determination of the movement error relationship is required, and subsequently, the thickness of the product can be adapted directly according to the determined movement error relationship.

[0094] Use the manipulator to debug the shooting position of each calibrated camera for shooting the target product, and debug the focal length of each calibrated camera, as Figure 4As shown in the figure. In the embodiment of the present invention, a calibration camera is set to perform non-concurrent calibration, that is, there is no common camera field of view among the four calibration cameras.

[0095] The initial calibration positions of each calibration camera can be recorded. It can be understood that if different coordinate systems are used when recording the initial calibration positions of each calibration camera, the checkerboard calibration board can be placed on the target product, such as Figure 4 As shown in the figure, the initial calibration positions in different coordinate systems are unified in one coordinate system by using the checkerboard calibration board. The unified coordinate system can be denoted as the world coordinate system.

[0096] The calibration camera is moved up or down along the motion axis to capture the product marks of the product to be attached. Since the calibration camera will produce an offset when moving up or down along the motion axis, therefore, the position of the product marks of the product to be attached can be corrected according to the previously calculated moving error relationship in the horizontal axis direction and the moving error relationship in the vertical axis direction to obtain the accurate coordinate position.

[0097] Any two product marks in the target product can be connected to obtain a reference line. Any two product marks in the product to be attached can be connected to obtain an offset line. The angular offset R between the target product and the product to be attached can be obtained according to the angle between the reference line and the offset line. The product to be attached can be rotated based on the center of the product to be attached, and the rotation angle is R. At this time, the rotation position of the product marks of the product to be attached can be calculated according to a = (a1 - a2) × cos(R) - (b1 - b2) × sin(R), b = (a1 - a2) × sin(R) - (b1 - b2) × cos(R), where a and b are the rotation positions of the product marks in the product to be attached after rotating by the angle R, a1 and b1 are the coordinate positions of the product marks in the product to be attached before rotation, and a2 and b2 are the coordinate positions of the rotation point. The horizontal axis adjustment offset and the vertical axis adjustment offset of the product to be attached are obtained according to the rotation position and the above-obtained accurate coordinate position. Finally, the attachment of the product to be attached and the target product is guided according to the rotation angle R, the horizontal axis adjustment offset, and the vertical axis adjustment offset.

[0098] Embodiment Four

[0099] Figure 5 A fitting alignment guiding device provided in Embodiment Four of the present invention is shown in Figure 5 the figure. The device includes:

[0100] A data acquisition module, configured to acquire first image data of a target product and second image data of a product to be attached according to a calibration camera;

[0101] An error acquisition module, configured to acquire a position offset error when a calibration camera moves from a first image data acquisition position to a second image data acquisition position, and adjust the second image data according to the position offset error;

[0102] An offset determination module, configured to acquire first coordinate information and second coordinate information of a product mark in the first image data and the adjusted second image data, and determine an adjustment offset angle and an adjustment offset amount according to the first coordinate information and the second coordinate information;

[0103] A fitting control module, configured to control a product to be fitted to fit a target product according to the adjustment offset angle and the adjustment offset amount.

[0104] In an embodiment of the present invention, a calibration camera can be used to photograph a target product and a product to be fitted, a target product image and a product to be fitted image photographed by the calibration camera can be obtained from the calibration camera, image processing can be performed on the obtained target product image and product to be fitted image to obtain first image data of the target product and second image data of the product to be fitted; a position offset error when the calibration camera moves to the second image data acquisition position can be acquired, the second image data acquired by the calibration camera can be adjusted according to the acquired position offset error, first coordinate information for determining the position of the product mark in the target product can be extracted from the first image data, and second coordinate information for determining the position of the product mark in the product to be fitted can be extracted from the adjusted second image data; according to the extracted first coordinate information and second coordinate information, an angle difference in the rotation direction and a distance difference in the translation direction of the product to be fitted relative to the target product during the fitting process, that is, an adjustment offset angle and an adjustment offset amount of the product to be fitted relative to the target product, can be determined; the product to be fitted can be adjusted based on the acquired adjustment offset angle and adjustment offset amount so that the product to be fitted fits the target product. In the embodiment of the present invention, a calibration camera with a small depth of field characteristic can be used to photograph the target product and the product to be fitted, a product image with clear edges can be obtained, ensuring that the position of the product mark can be determined in the product image data and improving the accuracy of product fitting; by acquiring the position offset error when the calibration camera moves from the first image data acquisition position to the second image data acquisition position, the deviation of the product mark position information in the second image data caused by factors such as the movement of the calibration camera can be effectively eliminated, further improving the product fitting accuracy; in the subsequent process of fitting multiple layers of products to be fitted to the target product, the photographing position of the calibration camera can be automatically adjusted according to the already determined position offset error, simplifying the debugging process of the calibration camera and reducing the difficulty of product fitting guided by the calibration camera; through the calculated adjustment offset angle and adjustment offset amount, automatic fitting of the product to be fitted and the target product can be realized, reducing manual intervention and ensuring the stability of fitting.

[0105] Based on the above embodiments, in the embodiments of the present invention, the error acquisition module is specifically configured to obtain different image data corresponding to different product marks on the product to be fitted according to the calibration camera; extract the product mark coordinates of each product mark in each image data, and obtain the horizontal movement error relationship and the vertical movement error relationship of the position offset error of the corresponding calibration camera respectively fitted based on the abscissa and the ordinate in each product mark coordinate as the camera movement error mapping relationship.

[0106] Based on the above embodiments, the embodiments of the present invention further include: converting the coordinate information of each image data into the same coordinate system.

[0107] Based on the above embodiments, in the embodiments of the present invention, the error acquisition module is further specifically configured to obtain the camera movement error mapping relationship of the calibration camera; extract the product mark coordinates of the product mark in the second image data; and call the horizontal movement error relationship and the vertical movement error relationship in the camera movement error mapping relationship to determine the adjusted abscissa and the adjusted ordinate of the product mark coordinates.

[0108] Based on the above embodiments, in the embodiments of the present invention, the offset amount determination module is specifically configured to determine the reference straight line of the target product based on the first coordinate information, determine the offset straight line of the product to be fitted based on the second coordinate information, and use the included angle between the reference straight line and the offset straight line as the overall offset angle.

[0109] Based on the above embodiments, in the embodiments of the present invention, the offset amount determination module is further specifically configured to use the adjusted second image data of the product to be fitted as the third image data, adjust the third image data according to the adjusted offset angle, and obtain the third coordinate information of the product mark in the third image data; use the difference between the third coordinate information and the first coordinate information as the adjusted offset amount, where the adjusted offset amount at least includes the horizontal axis adjusted offset amount and the vertical axis adjusted offset amount.

[0110] Embodiment Five

[0111] The embodiments of the present invention provide a device for executing a fitting alignment guiding method, a computer-readable storage medium, and a computer program product.

[0112] Figure 6The structural schematic diagram of a device that can be used to implement the method of the embodiments of the present invention is shown. The device is intended to represent various forms of digital computers, such as, laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The device can also represent various forms of mobile devices, such as, personal digital processors, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown in the embodiments of the present invention, their connections and relationships, and their functions are only examples and are not intended to limit the implementation of the embodiments of the present invention described and / or claimed herein.

[0113] As Figure 6 shown, the device includes at least one processor 11, and a memory communicatively connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc. The memory stores a computer program executable by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the ROM 12 or the computer program loaded from the storage unit 18 into the RAM 13. In the RAM 13, various programs and data required for the operation of the device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0114] Multiple components in the device are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disc, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the device to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0115] The processor 11 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit, a graphics processing unit, various dedicated artificial intelligence computing chips, various processors running machine learning model algorithms, a digital signal processor, and any suitable processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the fitting alignment guiding method.

[0116] In some embodiments, the fitting alignment guiding method may be implemented as a computer program tangibly embodied in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed onto the device via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the fitting alignment guiding method may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the fitting alignment guiding method by any other suitable means (e.g., by means of firmware).

[0117] The various implementations of the systems and techniques described above in the embodiments of the present invention may be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays, application specific integrated circuits, application specific standard products, systems on a chip, programmable logic devices loaded with a program, computer hardware, firmware, software, and / or combinations thereof. These various implementations may include: being implemented in one or more computer programs that may be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, and may receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit the data and instructions to the storage system, the at least one input device, and the at least one output device.

[0118] The computer programs for implementing the methods of the embodiments of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a dedicated computer, or other programmable data processing device, such that when the computer programs are executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer programs may be executed entirely on the machine, partially on the machine, as a stand-alone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0119] In the context of embodiments of the present invention, a computer-readable storage medium may be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. The computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, the computer-readable storage medium may be a machine-readable signal medium. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0120] In order to provide interaction with a user, the systems and techniques described herein can be implemented on a device having: a display device (e.g., a cathode ray tube or a liquid crystal display monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the device. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including: acoustic input, voice input, or tactile input).

[0121] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer having a graphical user interface or a web browser through which the user can interact with an implementation of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected to each other by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local area networks, wide area networks, blockchain networks, and the Internet.

[0122] A computing system may include a client and a server. The client and the server are generally far from each other and usually interact via a communication network. The relationship between the client and the server is created by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system, and solves the defects of difficult management and weak business scalability existing in traditional physical hosts and virtual private server services.

[0123] It should be understood that various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps recited in the present invention can be executed in parallel, sequentially, or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and no limitations are imposed herein.

[0124] The above specific embodiments do not constitute a limitation on the protection scope of the embodiments of the present invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims

1. A fitting alignment guiding method, characterized in that, The method includes: Obtaining first image data of a target product and second image data of a product to be fitted according to a calibrated camera; Obtaining a position offset error of the calibrated camera moving from the acquisition position of the first image data to the acquisition position of the second image data, and adjusting the second image data according to the position offset error; Obtaining first coordinate information and second coordinate information of product marks in the first image data and the adjusted second image data, and determining an adjustment offset angle and an adjustment offset amount according to the first coordinate information and the second coordinate information; Controlling the product to be fitted to fit the target product according to the adjustment offset angle and the adjustment offset amount.

2. The method according to claim 1, wherein Before obtaining the position offset error of the calibrated camera moving from the acquisition position of the first image data to the acquisition position of the second image data, it includes: Obtaining different image data corresponding to different product marks on the product to be fitted according to the calibrated camera; Extracting product mark coordinates of each product mark in each image data, and obtaining a horizontal coordinate movement error relationship and a vertical coordinate movement error relationship of the position offset error corresponding to the calibrated camera respectively fitted based on the horizontal coordinates and vertical coordinates in each product mark coordinate as a camera movement error mapping relationship.

3. The method according to claim 2, wherein It further includes: Converting the coordinate information of each image data to the same coordinate system.

4. The method according to claim 2, wherein Obtaining the position offset error of the calibrated camera moving from the acquisition position of the first image data to the acquisition position of the second image data, and adjusting the second image data according to the position offset error, includes: Obtaining the camera movement error mapping relationship of the calibrated camera; Extracting product mark coordinates of the product mark in the second image data; Invoking the horizontal coordinate movement error relationship and the vertical coordinate movement error relationship in the camera movement error mapping relationship to determine an adjusted horizontal coordinate and an adjusted vertical coordinate of the product mark coordinate.

5. The method according to claim 1, wherein Determining the adjustment offset angle according to the first coordinate information and the second coordinate information, includes: Determining a reference line of the target product based on the first coordinate information, determining an offset line of the product to be fitted based on the second coordinate information, and taking the angle between the reference line and the offset line as the adjustment offset angle.

6. The method according to claim 1 or 5, characterized in that Determining the adjustment offset amount according to the first coordinate information and the second coordinate information, includes: Taking the adjusted second image data of the product to be fitted as third image data, adjusting the third image data according to the adjustment offset angle, and obtaining third coordinate information of the product mark in the third image data; Taking the difference between the third coordinate information and the first coordinate information as the adjustment offset amount, where the adjustment offset amount at least includes a horizontal axis adjustment offset amount and a vertical axis adjustment offset amount.

7. A fitting alignment guide, characterized in that, The device includes: A data acquisition module, configured to obtain first image data of a target product and second image data of a product to be fitted according to a calibrated camera; An error acquisition module, configured to acquire a position offset error of the calibration camera moving from the acquisition position of the first image data to the acquisition position of the second image data, and adjust the second image data according to the position offset error; An offset determination module, configured to acquire first coordinate information and second coordinate information of product marks in the first image data and the adjusted second image data, and determine an adjustment offset angle and an adjustment offset amount according to the first coordinate information and the second coordinate information; A fitting control module, configured to control the product to be fitted to fit the target product according to the adjustment offset angle and the adjustment offset amount.

8. A device, characterized in that, The device includes: At least one processor; And a memory communicatively connected to the at least one processor; Wherein, the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the fitting and alignment guiding method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores: Computer instructions, which are used to implement the fitting and alignment guiding method according to any one of claims 1-6 when executed by a processor.

10. A computer program product, characterized in that, The computer program product includes: A computer program, which implements the fitting and alignment guiding method according to any one of claims 1-6 when executed by a processor.

Images