Pin shaft and pin hole alignment method, system and equipment based on visual identification

Abstract

The invention relates to the technical field of image processing, in particular to a pin shaft and pin hole alignment method, system and equipment based on visual identification, and the method comprises the steps: processing a collected target image containing a lock pin hole connecting plate and a target label through employing a target detection segmentation network, obtaining a mask image containing the first pixel region and the second pixel region; determining a first reference line of the pin hole and a second reference line of the pin shaft based on the mask image; if the first reference line is parallel to the second reference line, a pixel distance value between the first reference line and the second reference line is determined, and after it is determined that the pixel distance value exceeds a preset range, the pixel distance value is converted into a target physical distance value; and the dismounting device is indicated to move according to the target physical distance value between the first reference line and the second reference line, so that automatic alignment of the pin hole and the pin shaft is realized, and efficient execution of the alignment process is ensured.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a pin shaft and pin hole alignment method, system and device based on visual recognition. Background Art

[0002] The pin is one of the key components used to connect two adjacent standard sections of the tower crane, and plays an important role in the safety and stability of the overall structure of the tower crane. After the tower crane is lifted and the upper and lower standard sections are aligned and installed, the pin needs to be inserted into the pin hole to ensure the stability of the connection between the adjacent standard sections; and when lowering or removing the standard section, the pin needs to be knocked out of the pin hole.

[0003] The traditional way to disassemble and assemble the pin is to use a hammer to knock the pin into or out of the pin hole manually. This method is time-consuming and labor-intensive, and there is a safety hazard that the hammer may slip due to improper operation. The latest method is to manually operate special equipment for disassembling and assembling the pin. The equipment uses a hydraulic or electric push rod to push the pin into or out of the pin hole. Using special disassembly and assembly equipment can save manpower and time when disassembling and assembling the pin, and improve the efficiency and safety of the operation.

[0004] However, before using the special equipment for pin disassembly and assembly to push the pin into or out of the pin hole, it is necessary to manually align the pin or push rod with the pin hole, that is, the pin / push rod and the pin hole are coaxially aligned. For installers with low skills, this installation method will consume a lot of time and the installation quality is difficult to meet the requirements. Summary of the invention

[0005] The embodiments of the present application provide a pin shaft and pin hole alignment method, system and equipment based on visual recognition, which realize the automatic alignment of the pin hole and the pin shaft and ensure the efficiency and accuracy of the alignment.

[0006] In a first aspect, an embodiment of the present application provides a pin shaft and pin hole alignment method based on visual recognition, the method comprising:

[0007] Acquire a target image of a locking pin hole connecting plate and a target label, and obtain a mask image including a first pixel area and a second pixel area, wherein the first pixel area corresponds to the locking pin hole connecting plate, and the second pixel area corresponds to the target label, and the target label is located on the outer surface of the pin-carrying cylinder of the disassembly and assembly device, and is used to mark the pin shaft in the pin-carrying cylinder;

[0008] Determine a first reference line of the pin hole based on first position information of the first pixel area in the mask image, and determine a second reference line of the pin shaft based on second position information of the second pixel area in the mask image;

[0009] If the first reference line is parallel to the second reference line, a pixel distance value between the first reference line and the second reference line is determined. After determining that the pixel distance value exceeds a preset range, the pixel distance value is converted into a target physical distance value, and the disassembly and assembly device is instructed to move according to the target physical distance value between the first reference line and the second reference line so that the first reference line and the second reference line are located on the same straight line.

[0010] Optionally, determining a first reference line of the pin hole based on first position information of the first pixel area in the mask image includes:

[0011] Based on the first position information, determining a first minimum circumscribed rectangle corresponding to the locking pin hole connecting plate;

[0012] Based on the coordinates of each vertex of the first minimum circumscribed rectangle in the image coordinate system of the mask image, a reference line of the locking pin hole connecting plate is determined, and the reference line of the locking pin hole connecting plate is used as the first reference line.

[0013] Optionally, determining the second reference line of the pin based on second position information of the second pixel area in the mask image includes:

[0014] Based on the second position information, determining a second minimum bounding rectangle corresponding to the target tag;

[0015] Based on the coordinates of each vertex of the second minimum circumscribed rectangle in the image coordinate system of the mask image, a reference line of the target label is determined, and the reference line of the target label is used as the second reference line.

[0016] Optionally, the first reference line and the second reference line are determined to be parallel in the following manner:

[0017] Determine a first inclination angle of the first reference line relative to a horizontal direction, determine a second inclination angle of the second reference line relative to the horizontal direction, and determine a difference between the first inclination angle and the second inclination angle;

[0018] If the difference is within a preset difference range, it is determined that the first reference line is parallel to the second reference line.

[0019] Optionally, after determining the difference between the first inclination angle and the second inclination angle, the method further includes:

[0020] If the difference exceeds the preset difference range, an alarm message is generated, and the alarm message is used to prompt the staff to check the installation status of the disassembly and assembly device.

[0021] Optionally, the target image is acquired by an acquisition device;

[0022] The instructing disassembly and assembly device to move according to the target physical distance value between the first reference line and the second reference line, comprising:

[0023] adjusting a position of the second reference line in the mask image based on a first physical distance between the locking pin hole connecting plate and the acquisition device, a second physical distance between the target tag and the acquisition device, and a parameter of the acquisition device;

[0024] The disassembly and assembly device is instructed to move according to a target physical distance value between the first reference line and the adjusted second reference line.

[0025] Optionally, the first reference line and the second reference line are located in the mask image;

[0026] Before instructing the disassembly and assembly device to move according to the target physical distance value between the first reference line and the adjusted second reference line, the method further includes:

[0027] Determining a pixel distance value between the first reference line and the adjusted second reference line in the mask image;

[0028] According to the comparison result of the absolute value of the pixel distance value and the preset value, it is determined whether the first reference line and the adjusted second reference line are aligned. If not, the pixel distance value is converted into the target physical distance value based on the parameters of the acquisition device.

[0029] In a second aspect, an embodiment of the present application provides a pin shaft and pin hole alignment system based on visual recognition, the system comprising:

[0030] An acquisition device, used to acquire target images of the locking pin hole connecting plate and the target label;

[0031] A processing device, used for obtaining a mask image including a first pixel area and a second pixel area, wherein the first pixel area corresponds to the locking pin hole connecting plate, and the second pixel area corresponds to the target label, wherein the target label is located on the outer surface of the pin-carrying cylinder of the disassembling and assembling device and is used for marking the pin shaft in the pin-carrying cylinder, determining a first reference line of the pin hole based on first position information of the first pixel area in the mask image, and determining a second reference line of the pin shaft based on second position information of the second pixel area in the mask image, and if the first reference line and the second reference line are parallel, determining a pixel distance value between the first reference line and the second reference line, and after determining that the pixel distance value exceeds a preset range, converting the pixel distance value into a target physical distance value, and instructing the disassembling and assembling device to move according to the target physical distance value between the first reference line and the second reference line, so that the first reference line and the second reference line are located on the same straight line;

[0032] The disassembly and assembly device is used to move according to the target physical distance value between the first reference line and the second reference line.

[0033] In a third aspect, an embodiment of the present application provides an electronic device, the electronic device comprising:

[0034] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor executes the method as described in the first aspect above.

[0035] In a fourth aspect, an embodiment of the present application provides a computer storage medium, wherein the computer storage medium stores a computer program, and the computer program is used to enable a computer to execute the method in the first aspect as described above.

[0036] In a fifth aspect, an embodiment of the present application provides a computer program product, which implements the method in the first aspect above when the computer program is executed by a processor.

[0037] The embodiment of the present application can determine the angle between the reference line of the pin shaft and the reference line of the pin hole and the target physical distance value by analyzing the image containing the locking pin hole connecting plate and the target label collected by the collection device, and can instruct the disassembly and assembly device to move according to the target physical distance value so that the reference line of the pin shaft and the reference line of the pin hole are on the same straight line, thereby automating the alignment process without manual intervention, thereby ensuring the accuracy and timeliness of the alignment. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1It is a schematic diagram of the structure of a disassembly and assembly device according to an exemplary embodiment of the present invention;

[0039] Figure 2 A schematic top view of a pin-shaft and pin-hole alignment system structure based on visual recognition according to an exemplary embodiment of the present invention;

[0040] Figure 3 A schematic flow chart of a pin-shaft pin-hole alignment method based on visual recognition according to an exemplary embodiment of the present invention;

[0041] Figure 4 It is a schematic front view of a pin-shaft and pin-hole alignment system structure based on visual recognition according to an exemplary embodiment of the present invention;

[0042] Figure 5 A schematic diagram of a mask image according to an exemplary embodiment of the present invention;

[0043] Figure 6 A schematic diagram of a distance between the camera and the camera according to an exemplary embodiment of the present invention;

[0044] Figure 7 A schematic diagram of translation according to an exemplary embodiment of the present invention;

[0045] Figure 8 A schematic diagram of a specific process of a pin-shaft pin-hole alignment method based on visual recognition according to an exemplary embodiment of the present invention;

[0046] Fig. 9 A schematic diagram of a pin-shaft and pin-hole alignment system based on visual recognition according to an exemplary embodiment of the present invention;

[0047] Fig.10 The figure is a schematic diagram of an electronic device according to an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical scheme and advantages of the present application clearer, the technical scheme in the embodiment of the present application will be clearly and completely described below in conjunction with the drawings in the embodiment of the present application. Obviously, the described embodiment is only a part of the embodiment of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application. In the absence of conflict, the embodiments in the present application and the features in the embodiments can be arbitrarily combined with each other. In addition, although the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in an order different from that here.

[0049] The acquisition, storage, use, and processing of data in the embodiments of this application comply with the relevant provisions of national laws and regulations.

[0050] First, some concepts involved in the embodiments of the present application are introduced.

[0051] Tower crane: Also known as tower crane, it is mainly used for vertical and horizontal transportation of materials and installation of building components in house construction. It consists of three parts: metal structure, working mechanism and electrical system. The metal structure includes the tower body, boom and base; the working mechanism has four parts: lifting, luffing, rotation and travel; the electrical system includes motors, controllers, distribution cabinets, connecting lines, signals and lighting devices, etc.

[0052] Pin: One of the key components that connects the two standard main chords of the tower crane, and plays a very important role in the safety and stability of the overall structure of the tower crane.

[0053] Pin hole: Located on the main chord of the standard section. When the tower crane is in the jacking operation scenario, after the upper and lower standard sections are aligned and installed, the pin shaft needs to be inserted into the pin hole; when lowering the section or dismantling the standard section, the pin shaft needs to be knocked out of the pin hole.

[0054] Disassembly and assembly device: used to push the pin into or out of the pin hole, such as Figure 1 The figure shows the top view of the relative position of the disassembly and assembly device and the standard chord rod. The disassembly and assembly device includes a pin-carrying cylinder, a hydraulic push rod and a tensioning wheel. The pin is loaded in the pin-carrying cylinder. When the pin is installed, the push rod behind the pin-carrying cylinder pushes the single pin at the bottom of the cylinder into the pin hole. When the pin is removed, the pin-carrying cylinder is in an empty state, and the hydraulic push rod passes through the pin-carrying cylinder cavity to push the pin in the standard chord hole out and drop it into the U-shaped groove below.

[0055] Acquisition device: used to acquire target images including the locking pin hole connecting plate and the target label, and provide data support for the subsequent execution of the pin shaft pin hole alignment method based on visual recognition.

[0056] Locking pin hole connecting plate: It is a component welded to the specified position of the main chord of the tower crane standard section. The installation position is as follows: Figure 1 As shown, a locking pin hole is reserved on it. When the pin shaft is pushed into the pin hole, the locking pin rod passes through the locking pin hole and the pin shaft to prevent the pin shaft from being displaced.

[0057] The pin shaft and pin hole alignment method based on visual recognition in the embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0058] like Figure 2 The figure shows an application scenario of a pin shaft and pin hole alignment method based on visual recognition provided by an embodiment of the present application, including a disassembly and assembly device, a collection device, a processor and a standard section main chord.

[0059] The disassembly and assembly device can be installed on the outer angle steel of the tower crane climbing frame and can be raised and lowered together with the climbing frame. The disassembly and assembly device is used to push the pin shaft into or out of the pin hole. The disassembly and assembly device includes a pin-carrying cylinder, a hydraulic push rod and a tensioning wheel. The pin-carrying cylinder is relatively fixed in the z direction of the camera coordinate system through the tensioning force of the tensioning wheel and the main chord of the standard section, ensuring that the horizontal reference line of the pin shaft in the pin-carrying cylinder and the horizontal reference line of the pin hole are aligned in the z direction of the camera; the hydraulic push rod is used to push the pin shaft in or out; after the pin shaft is pushed into the pin hole, it can ensure the stability of the connection between the main chords of the adjacent standard sections.

[0060] A locking pin hole connecting plate is installed on the main chord of the standard section, on which a locking pin hole is reserved. When the pin shaft is pushed into the pin hole, the locking pin rod passes through the locking pin hole and the pin shaft to prevent the pin shaft from being displaced.

[0061] The acquisition device can be an industrial camera or other types of image or video stream acquisition devices, which are not specifically limited in the embodiments of the present application. The acquisition device can be fixed to the climbing frame by a bracket, and rise and fall with the climbing frame, and the relative positions of the acquisition device and the disassembly and assembly device remain fixed. The acquisition device is used to acquire images including the locking pin hole connecting plate and the disassembly and assembly device. At the same time, when the tower crane is being raised or lowered, the acquisition device can transmit the automatic disassembly and assembly process to the ground display terminal in real time, so that the staff can pay attention to the disassembly and assembly of the pin shaft in real time.

[0062] The acquisition device uploads the acquired images to the processor for analysis, obtains the target physical distance value, and then instructs the disassembly and assembly device to move according to the target physical distance value. The processor can be a single chip microcomputer or an independent server, which is not specifically limited in the embodiment of the present application.

[0063] Based on the accuracy and timeliness problems in the prior art in the process of aligning the pin shaft and the pin hole, the embodiment of the present application provides a method for aligning the pin shaft and the pin hole based on visual recognition. The specific process is as follows: Figure 3 shown.

[0064] S301: Acquire a target image of a locking pin hole connecting plate and a target label, and obtain a mask image including a first pixel region and a second pixel region.

[0065] When pushing the pin into the pin hole, the reference line of the pin hole needs to be aligned with the reference line of the pin. When pushing the pin out of the pin hole, the reference line of the pin hole and the pin outlet (located on the pin carrier cylinder, such as Figure 4 However, the pin is located inside the pin carrier cylinder, and it is not easy for the acquisition device to capture the image containing the pin, and the pin outlet is not easy to be captured by the acquisition device. Therefore, the pin is marked on the outer surface of the pin carrier cylinder, such as Figure 4For the target label in the image, the embodiment of the present application does not limit the shape and color of the target label. It is only necessary to ensure that the reference line of the target label coincides with the reference line of the pin shaft. Similarly, the pin hole is not easily captured by the acquisition device, so the reference line of the locking pin hole connecting plate is used as the reference line of the pin hole. When the acquisition device acquires an image, the locking pin hole connecting plate and the target label are simultaneously acquired to execute the subsequent pin shaft and pin hole alignment method based on visual recognition. In addition, since the hydraulic push rod is aligned with the pin shaft and the pin outlet during installation of the disassembly and assembly device, the reference line of the hydraulic push rod can also be used as the reference line of the pin shaft. In addition, the reference line can be the central axis, which is not specifically limited in the embodiment of the present application.

[0066] In an embodiment of the present application, a target detection and segmentation network can be used to perform target detection and segmentation on a target image to obtain a mask image. The target detection and segmentation network can be obtained by annotating and training image samples based on a convolutional neural network. The network model performs target detection and segmentation on the image, that is, image detection and segmentation is performed on the locking pin hole connecting plate and the target label to obtain a mask image including a first pixel area and a second pixel area. The size of the mask image is consistent with the target image. The first pixel area in the mask image corresponds to the locking pin hole connecting plate, and the second pixel area corresponds to the target label. The pixel values ​​of the first pixel area and the second pixel area can both be 255, and the pixel values ​​of the remaining pixels are 0.

[0067] S302: Determine a first reference line of the pin hole based on first position information of the first pixel region in the mask image, and determine a second reference line of the pin shaft based on second position information of the second pixel region in the mask image.

[0068] The first position information is the position of the locking pin hole connecting plate in the mask image, and can be specifically identified by words such as "locking pin hole connecting plate". The implementation method for determining the first reference line provided in the embodiment of the present application is as follows:

[0069] Based on the first position information, determining a first minimum circumscribed rectangle corresponding to the locking pin hole connecting plate;

[0070] Based on the coordinates of each vertex of the first minimum circumscribed rectangle in the image coordinate system of the mask image, a reference line of the locking pin hole connecting plate is determined, and the reference line of the locking pin hole connecting plate is used as the first reference line.

[0071] like Figure 5As shown, calculate the contour of the non-zero pixel area in the mask image and find the minimum enclosing rectangle of the contour. Obtain the image coordinates of the four vertices of the minimum enclosing rectangle P1, P2, P3, P4 and the coordinates of the center point PC1 of the rectangle. Sort the four vertices by the x coordinate (in the image coordinate system) value from small to large, take the first two points and calculate their midpoints, and get the image coordinates of the midpoint P5 on the left edge of the rectangle. Then take the last two points to get the coordinates of the midpoint P6 on the right edge of the rectangle. Point PC1 is the midpoint of points P5 and P6. The straight line passing through points P5 and P6 is the horizontal reference line of the rectangle and also the horizontal reference line of the locking pin hole connecting plate. Let the reference line be L1. The linear equation of L1 in the image coordinate system is: A1×x+B1×y+C1=0. The slope of L1 is -A1 / B1, where B1 is not 0. Let the image coordinates of point P5 be (x1, y1) and the image coordinates of point P6 be (x2, y2). The calculation formula for the coefficients A1, B1, and C1 of the L1 straight line equation is the following formula 1:

[0072]

[0073] The above-mentioned second position information is the position of the target label in the mask image. Specifically, it can be marked with text similar to "target label". The implementation method of determining the second reference line is similar to the step of determining the first reference line, that is, based on the second position information, the second minimum enclosing rectangle corresponding to the target label is determined; based on the coordinates of each vertex of the second minimum enclosing rectangle in the image coordinate system of the mask image, the reference line of the target label is determined, and the reference line of the target label is used as the second reference line L2. The linear equation of L2 in the image coordinate system: A2×x+B2×y+C2=0, A2, B2 and C2 are the coefficients of L2, and the slope of L2 is -A2 / B2, where B2 is not 0. A2, B2 and C2 can be determined using the above formula 1, which will not be repeated here.

[0074] S303: If the first reference line is parallel to the second reference line, determine the pixel distance value between the first reference line and the second reference line. After determining that the pixel distance value exceeds a preset range, convert the pixel distance value into a target physical distance value, and instruct the disassembly and assembly device to move according to the target physical distance value between the first reference line and the second reference line, so that the first reference line and the second reference line are located on the same straight line.

[0075] Whether the horizontal reference line L1 of the lock pin hole connecting plate and the horizontal reference line L2 of the target tag are parallel is the premise for the alignment detection of the pin shaft and the pin hole and the distance calculation. If they are not parallel, it means that the pin shaft cannot be smoothly pushed into or out of the pin hole. Therefore, after obtaining L1 and L2, it is necessary to first determine whether L1 is parallel to L2. The specific judgment method can be as follows:

[0076] Determine a first inclination angle of a first reference line (i.e., L1) relative to a horizontal direction, determine a second inclination angle of a second reference line (i.e., L2) relative to the horizontal direction, and determine a difference between the first inclination angle and the second inclination angle; if the difference is within a preset difference range, determine that the first reference line and the second reference line are parallel.

[0077] For example, the first tilt angle (angle value) of the first reference line is Tilt1 = arctan (K1), K1 is the slope of L1. The second tilt angle of the second reference line is Tilt2 = arctan (K2), K2 is the slope of L2, and the angle difference threshold Thr_tilt is set. The value range of Thr_tilt is [0.0, 0.7]. If |Tilt1-Tilt2|≤Thr_tilt, the first reference line is considered to be parallel to the second reference line.

[0078] Optionally, if the angle difference exceeds the preset angle difference threshold, it indicates that the first reference line and the second reference line are not parallel, and an alarm message is generated, which is used to prompt the staff to check the installation of the disassembly and assembly device. For example, the staff is required to adjust the position of the climbing frame or the installation position of the pin cylinder so that the horizontal reference line of the target tag is perpendicular to the vertical reference line of the main chord of the standard section, so that the straight lines L1 and L2 are parallel. Among them, the alarm message can remind the staff through the display interface or indicator light, or it can inform the staff through voice broadcast or prompt sound.

[0079] In the camera coordinate system, the locking pin hole connecting plate and the target tag may be located on different planes due to the different distances between the locking pin hole connecting plate and the target tag and the optical center of the acquisition device. According to the camera pinhole imaging principle, even if the horizontal reference lines L1 and L2 are aligned in the y direction of the camera coordinate system, they will not be completely aligned in the image due to the different distances from the camera. In order to improve the accuracy of alignment detection, the reference line L2 needs to be translated to the plane where the reference line L1 is located (the locking pin hole connecting plate plane) in the camera coordinate system, and the new straight line equation L3 in the image coordinate system is obtained. The specific implementation method is as follows:

[0080] When the first physical distance between the locking pin hole connecting plate and the collection device is different from the second physical distance between the target tag and the collection device, it is determined that the reference line of the target tag (i.e., L2) needs to be translated to the plane of the locking pin hole connecting plate. When the first physical distance and the second physical distance are the same, L2 and L3 are the same straight line.

[0081] First, take a point P on L2, whose coordinate value in the image coordinate system is (x ori ,y ori), the distance between the plane where L2 is located (the target label plane) and the camera in the Z-axis direction is Z2, and the coordinate value of P in the camera coordinate system is (X, Y, Z2). Suppose P is translated along the camera z direction to the plane of the locking pin hole connecting plate to obtain a new point P', and the coordinate value of P' in the image coordinate system is (x new ,y new ), the distance between the plane of the locking pin hole connecting plate and the camera in the Z-axis direction is Z1, and the coordinate value of P' in the camera coordinate system is (X, Y, Z1). According to the pinhole imaging principle, the following formula 2 holds true.

[0082]

[0083] Simplifying formula 2, we get formula 3:

[0084]

[0085] When the acquisition device is an industrial camera, fx is the pixel focal length of the camera in the x-axis direction, fy is the pixel focal length of the camera in the y-axis direction, Cx represents the horizontal coordinate of the center point of the imaging plane in the pixel coordinate system, and Cy represents the vertical coordinate of the center point of the imaging plane in the pixel coordinate system. fx, fy, Cx, and Cy can be obtained after calibrating the camera, such as Figure 6 As shown, Z1 is the physical distance from the plane of the locking pin hole connecting plate to the optical center of the camera, and Z2 is the physical distance from the plane where the target tag is located to the optical center of the camera, in millimeters. The values ​​of Z1 and Z2 can be obtained by measurement after the camera is installed and then stored in the storage medium for subsequent use.

[0086] Take two points on L2 and calculate the coordinates of the new corresponding points after translation. Based on the principle that a straight line can be determined by two points, the calculation formula for the linear equation coefficient of L3 is further derived as shown in Formula 4:

[0087]

[0088] Among them, A3, B3 and C3 are the coefficients of the linear equation of the adjusted second reference line (herein referred to as L3), and A2, B2 and C2 are the coefficients of L2. Finally, the adjusted second reference line L3 is A3×x+B3×y+C3=0. In addition, L1 can also be translated to the plane where L2 is located, which is not specifically limited in the embodiments of the present application.

[0089] After obtaining the first reference line L1 and the adjusted second reference line L3, the alignment detection of the horizontal reference line of the pin shaft and the horizontal reference line of the pin hole in the y direction of the camera coordinate system can be converted into the calculation of the pixel distance value between the reference line L1 and the reference line L3. Since the center point PC1 of the minimum circumscribed rectangle of the locking pin hole connecting plate is located on L1, it is sufficient to calculate the pixel distance d from the point PC1 to the straight line L3. The pixel distance d is as follows: Figure 7 As shown. Let the coordinate value of point PC1 be (x c ,y c ). The calculation formula of pixel distance d is shown in Formula 5:

[0090]

[0091] Among them, A3, B3 and C3 are the coefficients of L3.

[0092] A pixel distance threshold Thr_d=5 is set. If the absolute value of the pixel distance d |d|≤Thr_d, the pin shaft and the pin hole are considered to be aligned in the y direction in the camera coordinate system, otherwise they are considered to be not aligned. In the case of misalignment, the pixel distance d needs to be converted into the actual physical distance D in the y direction in the camera coordinate system, and then the disassembly and assembly device is instructed to move according to the target physical distance value between the first reference line and the adjusted second reference line; if aligned, the pin shaft can be directly pushed in / out of the pin hole.

[0093] Combined with the pinhole imaging principle, the conversion formula is D = Z1 × d / fy, where Z1 is the distance between the locking pin hole connecting plate plane and the camera, such as Figure 6 As shown; D is the target physical distance value between the first reference line and the adjusted second reference line, in millimeters.

[0094] When the D value is negative, it means that the pin is below the pin hole, and it is necessary to indicate the physical distance value of the disassembly and assembly device to move upward; when the D value is positive, it means that the pin is above the pin hole, and it is necessary to indicate the physical distance value of the disassembly and assembly device to move downward. Since the disassembly and assembly device is installed on the climbing frame, the climbing frame can be controlled to move the disassembly and assembly device to the target physical distance value.

[0095] The following is based on Figure 8 A pin shaft and pin hole alignment method based on visual recognition provided in an embodiment of the present application is described in detail.

[0096] S801: using a target detection and segmentation network to perform target detection, segmentation, and minimum bounding rectangle calculation on the collected image containing the locking pin hole connecting plate and the target label; the minimum bounding rectangle includes a first minimum bounding rectangle corresponding to the locking pin hole connecting plate and a second minimum bounding rectangle corresponding to the target label;

[0097] S802: Based on the first minimum circumscribed rectangle, locate the reference line L1 of the locking pin hole connecting plate, and based on the second minimum circumscribed rectangle, locate the reference line L2 of the target label;

[0098] S803: Determine whether L1 and L2 are parallel. If so, execute S805, otherwise execute S804. The specific method of determining parallelism is the same as S303 above, which will not be repeated here.

[0099] S804: Generate an alarm message, which is used to prompt the staff to check the installation status of the disassembly and assembly device;

[0100] S805: Adjust L2 based on the first physical distance, the second physical distance and the parameters of the acquisition device to obtain L3; wherein the first physical distance is the physical distance between the locking pin hole connecting plate and the acquisition device, and the second physical distance is the physical distance between the target tag and the acquisition device;

[0101] S806: Calculate the pixel distance value between L1 and L3, and then execute S807;

[0102] S807: Determine whether the pixel distance value is within a preset distance range, if so, execute S808, otherwise execute S809;

[0103] S808: instructing the disassembly and assembly device to push the pin into or out of the pin hole, and then ending;

[0104] S809: Convert the pixel distance value into a target physical distance value, and then execute S810;

[0105] S810: Instruct the assembly and disassembly device to move according to the target physical distance value, and then end.

[0106] During the automatic disassembly and assembly of the pin shaft in the embodiment of the present application, no manual intervention is required to perform the alignment operation between the pin shaft and the pin hole, which saves manpower expenses, improves the level of automation and work efficiency, and truly realizes unmanned automated operation of the pin shaft disassembly and assembly; in addition, the embodiment of the present application also improves the accuracy of alignment detection by calculating the pixel distance between the transformed reference line and the horizontal reference line of the locking pin hole connecting plate after translation transformation of the horizontal reference line of the target label in the image; at the same time, when the tower crane is being raised or lowered, the acquisition device can transmit the automatic disassembly and assembly process to the ground display terminal in real time, so that the staff can pay attention to the disassembly and assembly of the pin shaft in real time.

[0107] Based on the same inventive concept, the embodiment of the present application provides a pin-shaft pin-hole alignment system based on visual recognition, such as Fig. 9 As shown, the system includes:

[0108] The acquisition device 901 is used to acquire the target image of the locking pin hole connecting plate and the target label;

[0109] The processing device 902 is used to obtain a mask image including a first pixel area and a second pixel area, wherein the first pixel area corresponds to the locking pin hole connecting plate, and the second pixel area corresponds to a target label, wherein the target label is located on the outer surface of the pin-carrying cylinder of the disassembling device and is used to mark the pin shaft in the pin-carrying cylinder, and to determine a first reference line of the pin hole based on first position information of the first pixel area in the mask image, and to determine a second reference line of the pin shaft based on second position information of the second pixel area in the mask image, and if the first reference line and the second reference line are parallel, instruct the disassembling device to move according to a target physical distance value between the first reference line and the second reference line, so that the first reference line and the second reference line are located on the same straight line;

[0110] The disassembly and assembly device 903 is used to move according to the target physical distance value between the first reference line and the second reference line.

[0111] Optionally, the processing device 902 is used to determine a first minimum circumscribed rectangle corresponding to the locking pin hole connecting plate based on the first position information;

[0112] Based on the coordinates of each vertex of the first minimum circumscribed rectangle in the image coordinate system of the mask image, a reference line of the locking pin hole connecting plate is determined, and the reference line of the locking pin hole connecting plate is used as the first reference line.

[0113] Optionally, the processing device 902 is used to determine a second minimum bounding rectangle corresponding to the target tag based on the second position information;

[0114] Based on the coordinates of each vertex of the second minimum circumscribed rectangle in the image coordinate system of the mask image, a reference line of the target label is determined, and the reference line of the target label is used as the second reference line.

[0115] Optionally, the processing device 902 is configured to determine that the first reference line is parallel to the second reference line in the following manner:

[0116] Determine a first inclination angle of the first reference line relative to the horizontal direction, determine a second inclination angle of the second reference line relative to the horizontal direction, and determine a difference between the first inclination angle and the second inclination angle;

[0117] If the difference is within the preset difference range, it is determined that the first reference line is parallel to the second reference line.

[0118] Optionally, the processing device 902 is used for:

[0119] When the difference exceeds the preset difference range, an alarm message is generated, and the alarm message is used to prompt the staff to check the installation status of the disassembly and assembly device.

[0120] Optionally, acquiring the target image by an acquisition device;

[0121] The processing device 902 is used to adjust the position of the second reference line in the mask image based on the first physical distance between the locking pin hole connecting plate and the acquisition device, the second physical distance between the target tag and the acquisition device, and the parameters of the acquisition device;

[0122] The disassembly and assembly device is instructed to move according to the target physical distance value between the first reference line and the adjusted second reference line.

[0123] Optionally, the first reference line and the second reference line are located in the mask image;

[0124] The processing device 902 is used to determine a pixel distance value between the first reference line and the adjusted second reference line in the mask image;

[0125] According to the comparison result of the absolute value of the pixel distance value and the preset value, it is determined whether the first reference line and the adjusted second reference line are aligned. If not, the pixel distance value is converted into the target physical distance value based on the parameters of the acquisition device.

[0126] Based on the same inventive concept, an embodiment of the present application provides an electronic device, the device comprising:

[0127] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the pin shaft and pin hole alignment method based on visual recognition as described above.

[0128] Refer to the following Fig.10 The electronic device 100 according to this embodiment of the present application is described. Fig.10 The electronic device 100 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0129] like Fig.10 As shown, the electronic device 100 is in the form of a general electronic device. The components of the electronic device 100 may include but are not limited to: at least one processor 101, at least one memory 102, and a bus 103 connecting different system components (including the memory 102 and the processor 101).

[0130] The processor 101 is used to read and execute instructions in the memory 102, so that at least one processor can execute a pin shaft and pin hole alignment method based on visual recognition provided in the above embodiment.

[0131] Bus 103 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, and a processor or local bus using any of a variety of bus architectures.

[0132] The memory 102 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 1021 and / or a cache memory 1022 , and may further include a read-only memory (ROM) 1023 .

[0133] Memory 102 may also include a program / utility 1025 having a set (at least one) of program modules 1024, such program modules 1024 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0134] The electronic device 100 may also communicate with one or more external devices 104 (e.g., keyboards, pointing devices, etc.), may also communicate with one or more devices that enable an administrator to interact with the electronic device 100, and / or communicate with any device that enables the electronic device 100 to communicate with one or more other electronic devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 105. Furthermore, the electronic device 100 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 106. As shown, the network adapter 106 communicates with other modules for the electronic device 100 via a bus 103. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 100, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0135] In some possible embodiments, various aspects of a pin shaft and pin hole alignment method based on visual recognition provided by the present application can also be implemented in the form of a program product, which includes program code. When the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of a pin shaft and pin hole alignment method based on visual recognition according to various exemplary embodiments of the present application described above in this specification.

[0136] In addition, the present application also provides a computer-readable storage medium, which stores a computer program, and the computer program is used to enable a computer to execute any method in the above embodiments.

[0137] These computer program instructions may be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0138] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0139] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0140] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A pin shaft and pin hole alignment method based on visual recognition, characterized in that: The method comprises: Acquire a target image of a locking pin hole connecting plate and a target label, and obtain a mask image including a first pixel area and a second pixel area, wherein the first pixel area corresponds to the locking pin hole connecting plate, and the second pixel area corresponds to the target label, and the target label is located on the outer surface of the pin-carrying cylinder of the disassembly and assembly device, and is used to mark the pin shaft in the pin-carrying cylinder; Determine a first reference line of the pin hole based on first position information of the first pixel area in the mask image, and determine a second reference line of the pin shaft based on second position information of the second pixel area in the mask image; If the first reference line is parallel to the second reference line, a pixel distance value between the first reference line and the second reference line is determined. After determining that the pixel distance value exceeds a preset range, the pixel distance value is converted into a target physical distance value, and the disassembly and assembly device is instructed to move according to the target physical distance value between the first reference line and the second reference line so that the first reference line and the second reference line are located on the same straight line.

2. The method according to claim 1, characterized in that The determining a first reference line of the pin hole based on first position information of the first pixel area in the mask image includes: Based on the first position information, determining a first minimum circumscribed rectangle corresponding to the locking pin hole connecting plate; Based on the coordinates of each vertex of the first minimum circumscribed rectangle in the image coordinate system of the mask image, a reference line of the locking pin hole connecting plate is determined, and the reference line of the locking pin hole connecting plate is used as the first reference line.

3. The method according to claim 1, characterized in that The determining the second reference line of the pin shaft based on the second position information of the second pixel area in the mask image includes: Based on the second position information, determining a second minimum bounding rectangle corresponding to the target tag; Based on the coordinates of each vertex of the second minimum circumscribed rectangle in the image coordinate system of the mask image, a reference line of the target label is determined, and the reference line of the target label is used as the second reference line.

4. The method according to claim 1, characterized in that: The first reference line and the second reference line are determined to be parallel by: Determine a first inclination angle of the first reference line relative to a horizontal direction, determine a second inclination angle of the second reference line relative to the horizontal direction, and determine a difference between the first inclination angle and the second inclination angle; If the difference is within a preset difference range, it is determined that the first reference line is parallel to the second reference line.

5. The method according to claim 4, characterized in that After determining the difference between the first tilt angle and the second tilt angle, the method further includes: If the difference exceeds the preset difference range, an alarm message is generated, and the alarm message is used to prompt the staff to check the installation status of the disassembly and assembly device.

6. The method according to claim 1, characterized in that Acquire the target image by an acquisition device; The instructing disassembly and assembly device to move according to the target physical distance value between the first reference line and the second reference line, comprising: adjusting the position of the second reference line in the mask image based on a first physical distance between the locking pin hole connecting plate and the acquisition device, a second physical distance between the target tag and the acquisition device, and a parameter of the acquisition device; The disassembly and assembly device is instructed to move according to a target physical distance value between the first reference line and the adjusted second reference line.

7. The method according to claim 6, characterized in that The converting the pixel distance value into a target physical distance value comprises: Based on the parameters of the acquisition device, the pixel distance value is converted into the target physical distance value.

8. A pin-shaft and pin-hole alignment system based on visual recognition, characterized in that: The system comprises: An acquisition device, used to acquire target images of the locking pin hole connecting plate and the target label; A processing device, used for obtaining a mask image including a first pixel area and a second pixel area, wherein the first pixel area corresponds to the locking pin hole connecting plate, and the second pixel area corresponds to the target label, wherein the target label is located on the outer surface of the pin-carrying cylinder of the disassembling and assembling device and is used for marking the pin shaft in the pin-carrying cylinder, determining a first reference line of the pin hole based on first position information of the first pixel area in the mask image, and determining a second reference line of the pin shaft based on second position information of the second pixel area in the mask image, and if the first reference line and the second reference line are parallel, determining a pixel distance value between the first reference line and the second reference line, and after determining that the pixel distance value exceeds a preset range, converting the pixel distance value into a target physical distance value, and instructing the disassembling and assembling device to move according to the target physical distance value between the first reference line and the second reference line, so that the first reference line and the second reference line are located on the same straight line; The disassembly and assembly device is used to move according to the target physical distance value between the first reference line and the second reference line.

9. An electronic device, characterized in that: The electronic device comprises: At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor so that the at least one processor executes a method as claimed in any one of claims 1 to 7.

10. A computer storage medium, characterized in that: The computer storage medium stores a computer program, and the computer program is used to make a computer execute any one of the methods according to claims 1-7.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images