A bolt pose calculation method, device, equipment and storage medium

Abstract

The present application relates to new energy equipment manufacturing technical field, disclose a kind of bolt pose calculation method, device, equipment and storage medium, the bolt pose calculation method provided by the present application includes: the calibration of each system of vision module is carried out, complete binocular vision system internal and external parameter calibration, while obtaining the left image and right image of the bolt end face of target area by binocular camera;Ellipse detection algorithm is used to respectively extract the features of the ellipse projection of bolt end face on left image and right image, to obtain the parameter matrix of projection ellipse;According to the parameter matrix of the ellipse, the shadow line of the plane where the corresponding bolt end face is on the image is calculated;According to the calibration parameters of binocular camera and the shadow line, the center position coordinates and end face normal vector of bolt end face in binocular camera coordinate system are calculated, and the pose of bolt end face is obtained. Precise positioning in the bolt fastening process is realized, and the product quality of wind turbine, the automation and intelligent degree of bolt fastening process are effectively improved.

Description

Technical Field

[0001] This invention relates to the field of new energy equipment manufacturing technology, specifically to a bolt position calculation method, device, equipment, and storage medium. Background Technology

[0002] With the rapid development of offshore wind power technology, the increasing size of wind turbine units has become an important development trend. Simultaneously, the arrival of the grid parity era has placed higher demands on the quality, reliability, and safety of wind turbines. As a core component of wind turbine units, the pitch bearing is responsible for connecting the hub and blades, bearing blade loads, and transmitting blade torque. Facing the harsh marine environment, if the pitch bearing malfunctions due to improper installation or insufficient pre-tightening of the bolts, replacement will require lifting the entire turbine, including the blades and hub, to the sea surface. This offshore lifting is extremely difficult and costly, posing significant challenges to the rapid and economical maintenance of the wind turbine. The performance and lifespan of the pitch bearing are not only related to machining precision, but also crucially dependent on proper installation. Correct installation is a powerful guarantee for extending the bearing's service life and ensuring the long-term stable operation of the wind turbine.

[0003] Currently, bolt tightening for pitch bearings is primarily done manually, with humans as the main operator. This manual operation, influenced by various factors, makes it difficult to guarantee consistent bolt tightening quality due to random and uncontrollable elements throughout the process. Meanwhile, with the development of computer technology, vision-guided robotic bolt tightening has attracted increasing attention in recent years, and some companies are already using robots to replace manual bolt tightening operations.

[0004] Existing robots lack the ability to identify and locate bolt positions, relying solely on the positioning accuracy of wind turbine hubs. Furthermore, the robot's position and posture at each bolt fastening point must be pre-set, which is complex and cumbersome. For traditional manual tightening and robot tightening operations without visual recognition and positioning capabilities, bolts suffer from low positioning accuracy during the tightening process. Summary of the Invention

[0005] In view of this, the present invention provides a bolt position calculation method, device, equipment and storage medium to solve the problem of low positioning accuracy during the fastening process.

[0006] In a first aspect, the present invention provides a bolt pose calculation method, the method comprising:

[0007] The various systems of the vision module are calibrated to complete the calibration of the intrinsic and extrinsic parameters of the binocular vision system. At the same time, the left and right images of the bolt end face of the target area are acquired through the binocular camera.

[0008] The ellipse detection algorithm is used to extract features from the elliptical projections of the bolt end faces in the left and right images to obtain the parameter matrix of the projected ellipse.

[0009] Based on the parameter matrix of the ellipse, calculate the shadow line on the image corresponding to the plane where the bolt end face is located;

[0010] Based on the calibration parameters and anechoic lines of the binocular camera, the center position coordinates and end face normal vector of the bolt end face in the binocular camera coordinate system are calculated to obtain the pose of the bolt end face.

[0011] The bolt position calculation method provided by this invention enables precise positioning during the bolt tightening process, effectively improving the product quality of wind turbine units and the automation and intelligence of the bolt tightening process.

[0012] In one optional implementation, the calibration of the various systems of the vision module includes:

[0013] Calibration of the binocular vision system and depth camera system, and hand-eye calibration between the two camera systems and the matching sleeves of the bolts to be tightened installed at the front end of the industrial robotic arm.

[0014] The method provided by this invention requires calibration of each system of the vision module to achieve precise positioning of the bolt end face position during the tightening process of the high-strength bolts of the wind turbine pitch bearing.

[0015] In one optional implementation, the calibration of the intrinsic and extrinsic parameters of the binocular vision system includes:

[0016] The internal and external parameters of the binocular vision system installed on the industrial robotic arm are calibrated to obtain the internal visual model parameters of the left and right cameras of the binocular vision system and the external rotation and translation parameters between the left and right cameras.

[0017] The method provided by this invention achieves precise positioning of the bolt end face during the tightening process of high-strength bolts in wind turbine pitch bearings by calibrating the internal and external parameters of a binocular vision system.

[0018] In one optional implementation, RGB and DEPTH images of the working surface of the bolt to be tightened on the wheel hub are acquired by a depth camera. The RGB images are input into a trained convolutional neural network model for bolt end face target segmentation, and the output image is a mask of the bolt end face. The DEPTH and RGB images are aligned, and the depth distance information of the target area of ​​the bolt end face mask in the RGB image in the depth camera coordinate system is calculated. The industrial robotic arm is moved using the distance information and hand-eye calibration parameters, so that the left and right cameras of the binocular vision system capture the left and right images of the bolt end face of the target area, respectively.

[0019] The method provided by the present invention performs coordinate transformation on the transformation parameters obtained by the calibration of the depth camera and the hand-eye calibration, thereby controlling the robotic arm to approach a bolt to be tightened on a certain side and stop in front of it, so that both the left and right cameras of the binocular vision system can capture the bolt end face of the target area.

[0020] In one alternative implementation, the left and right images acquired by the binocular camera are subjected to distortion correction processing based on the distortion coefficients calibrated by the binocular camera.

[0021] The method provided by this invention performs distortion correction processing on the left and right acquired images based on the calibrated distortion coefficients, thereby reducing the impact of stereo camera distortion.

[0022] In one alternative implementation, the parameter matrix of the projected ellipse is calculated using the following formula:

[0023]

[0024] A, B, C, D, E, and F are calculated using the following method:

[0025] The fitted ellipses in the left and right images are combined and matched to ensure that the matched ellipses are projections of the same bolt end face onto their respective images. The fitted and matched ellipses are then rewritten into the following algebraic equation form.

[0026] Ax 2 +Bxy+Cy 2 +Dx+Ey+F=0,

[0027] Where x and y are the horizontal and vertical coordinates of the elliptical contour points in the pixel coordinate system, and the parameters A, B, C, D, E, and F are obtained by solving and rewritten in the form of the ellipse parameter matrix.

[0028] The method provided by this invention calculates the parameter matrix of the projected ellipse to accurately locate the position of each bolt during the tightening of high-strength bolts in the pitch bearing of a wind turbine, thereby increasing the precision control during the bolt tightening process.

[0029] In one optional implementation, the calculation step of the disappearance line includes:

[0030] Select two adjacent elliptical contours, calculate the common external tangent of the two ellipses, and obtain the four common tangent points p1, p2, p3, and p4 where the two common external tangents are tangent to the ellipse;

[0031] Calculate the intersection point t1 of the external common tangents on the image;

[0032] Connect p1p2 and p3p4 respectively to obtain two lines, and calculate the intersection point t2 of these two lines;

[0033] By performing a vector cross product on t1 and t2, the disappearance line l of the plane containing the bolt end face on the image is calculated. vanish = t1 × t2.

[0034] The method provided by this invention calculates the disappearance line to accurately locate the position of each bolt during the tightening of the high-strength bolts of the wind turbine pitch bearing, thereby increasing the precision control during the bolt tightening process.

[0035] In one optional implementation, based on the principle of binocular vision triangulation, and combining the intrinsic and extrinsic parameters of the left and right cameras with the data information of the actual projected image point of the bolt end face center point, the center position coordinates of the bolt end face center in the binocular camera coordinate system are calculated.

[0036] The method provided by this invention calculates the center position coordinates of the bolt end face in the binocular camera coordinate system to accurately locate the position of each bolt during the tightening of high-strength bolts of wind turbine pitch bearings, thereby increasing the precision control during the bolt tightening process.

[0037] In one alternative implementation, the true projected image point of the bolt end face center point on the image is calculated using the following formula:

[0038] O=ω -1 l vanish ,

[0039] Where, ω -1 Let l be the inverse of the elliptic parameter matrix. vanish Let O be the equation of the shadow line of the plane containing the bolt end face on the image, and O be the true projected image point.

[0040] The method provided by this invention calculates the actual projection image point of the center point of the bolt end face on the image, and accurately locates the position of each bolt during the tightening of high-strength bolts of wind turbine pitch bearings, thereby increasing the precision control during the bolt tightening process.

[0041] In one alternative implementation, the normal vector orientation of the plane containing the bolt end face is calculated using the following formula:

[0042] n = K T l vanish ,

[0043] Among them, K T This is the transpose of the intrinsic parameter matrix of the corresponding stereo camera obtained after calibration.

[0044] The method provided by this invention calculates the normal vector of the plane containing the bolt end face to accurately locate the orientation of each bolt during the tightening of high-strength bolts in the pitch bearing of a wind turbine, thereby increasing the precision control during the bolt tightening process.

[0045] In a second aspect, the present invention provides a bolt pose calculation device, the device comprising:

[0046] The image acquisition module is used to calibrate the various systems of the vision module, complete the calibration of the intrinsic and extrinsic parameters of the binocular vision system, and simultaneously acquire the left and right images of the bolt end face of the target area through the binocular camera.

[0047] The feature extraction module is used to extract features from the elliptical projection of the bolt end face on the left and right images respectively using the ellipse detection algorithm, so as to obtain the parameter matrix of the projected ellipse.

[0048] The first calculation module is used to calculate the shadow line on the image corresponding to the plane where the bolt end face is located, based on the parameter matrix of the ellipse.

[0049] The second calculation module is used to calculate the center position coordinates and end face normal vector of the bolt end face in the binocular camera coordinate system based on the calibration parameters and anechoic lines of the binocular camera, so as to obtain the pose of the bolt end face.

[0050] In one optional implementation, the image acquisition module includes:

[0051] The calibration unit is used for calibrating the binocular vision system, the depth camera system, and for hand-eye calibration of the two camera systems with the sleeves that match the bolts to be tightened, which are mounted at the front end of the industrial robotic arm.

[0052] In one optional implementation, the image acquisition module further includes:

[0053] The preprocessing unit is used to perform distortion correction on the acquired left and right images based on the distortion coefficients calibrated by the stereo camera.

[0054] Thirdly, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to perform the bolt pose calculation method of the first aspect or any corresponding embodiment described above.

[0055] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to execute the bolt pose calculation method of the first aspect or any corresponding embodiment described above. Attached Figure Description

[0056] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0057] Figure 1 This is a flowchart of a bolt pose calculation method according to an embodiment of the present invention;

[0058] Figure 2 This is a schematic diagram of calculating the disappearance line at the point of infinity of two ellipses according to an embodiment of the present invention;

[0059] Figure 3 This is a structural block diagram of a bolt pose calculation device according to an embodiment of the present invention;

[0060] Figure 4 This is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0062] The bolt pose calculation method provided in this invention is applied to the field of new energy equipment manufacturing technology. It relates to a visual calculation method for bolt pose of high-strength bolts for wind turbine pitch bearings. This method can accurately locate the position and posture of each bolt during the high-strength bolt tightening process of wind turbine pitch bearings, thereby increasing the precision control during the bolt tightening process.

[0063] According to an embodiment of the present invention, a bolt pose calculation method embodiment is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0064] This embodiment provides a bolt pose calculation method, which can be used in the aforementioned mobile terminals, such as mobile phones, tablets, etc. (the executing entity is described in conjunction with the actual situation). Figure 1 This is a flowchart of a bolt pose calculation method according to an embodiment of the present invention, such as... Figure 1As shown, the process includes the following steps:

[0065] Step S101: Calibrate each system of the vision module to complete the calibration of the intrinsic and extrinsic parameters of the binocular vision system. At the same time, acquire the left and right images of the bolt end face of the target area through the binocular camera.

[0066] In this embodiment of the invention, the vision module consists of a depth camera system and a binocular vision system. First, the parameters of each part of the vision module must be calibrated. The binocular vision system mounted on the industrial robotic arm is first calibrated internally and externally. This involves obtaining the internal visual model parameters of the left and right cameras of the binocular vision system, as well as the external rotation and translation parameters between the cameras. Simultaneously, the nine-point calibration method is used to complete the hand-eye calibration between the binocular vision system, the depth camera system, and the sleeve mounted at the front end of the industrial robotic arm that matches the bolt to be tightened. This determines the transformation matrix between the coordinate systems of these two cameras and the working coordinate system established on the sleeve.

[0067] This invention employs a RealSense D435i depth camera system. Facing a wind turbine hub to be fastened, the RealSense D435i acquires RGB and DEPTH images of the working surface of the bolts to be fastened. The RGB images are input into a trained convolutional neural network model for bolt end-face target segmentation. The model outputs a mask of the bolt end-face in the image. The DEPTH and RGB images are aligned, and the depth distance information of the target area of ​​the bolt end-face mask in the RGB image is calculated in the depth camera coordinate system. After a series of coordinate transformations from the depth camera coordinate system to the industrial robotic arm's base coordinate system, the robotic arm is controlled to approach and stop directly in front of a bolt on one side. This allows both the left and right cameras of the binocular vision system to capture the bolt end-face in that area. The left and right images of the bolt end-face in the target area are acquired through the binocular cameras. Distortion correction processing is performed on the left and right images acquired by the binocular cameras based on the distortion coefficients calibrated by the binocular cameras to reduce the influence of camera distortion.

[0068] Step S102: Use the ellipse detection algorithm to extract features from the elliptical projections of the bolt end faces in the left and right images respectively, so as to obtain the parameter matrix of the projected ellipse.

[0069] In this embodiment of the invention, for the elliptical imaging projection of the bolt end face on the image, an ellipse detection algorithm is used to extract elliptical features from the left and right images. Based on the positional distribution of the bolt end face, the fitted ellipses in the left and right images are combined and matched to ensure that the ellipses matched on the left and right are projections of the same bolt end face on their respective images.

[0070] The general form of the equation of an ellipse is:

[0071] Ax2 +Bxy+Cy 2 +Dx+Ey+F=0,

[0072] The fitted and matched ellipse is rewritten in the above algebraic equation form, where x and y are the horizontal and vertical coordinates of the ellipse contour points in the pixel coordinate system. Based on the obtained parameters A, B, C, D, E, and F, the parametric matrix form of the ellipse is rewritten as follows:

[0073]

[0074] Step S103: Calculate the shadow line on the image corresponding to the plane where the bolt end face is located based on the parameter matrix of the ellipse.

[0075] In this embodiment of the invention, when the bolt end face is not perfectly parallel to the camera's imaging plane, due to "asymmetric projection," the circular feature will be projected as an ellipse on the image. Furthermore, the actual projected image point of the circle's center will not completely coincide with the center of the fitted ellipse. To eliminate the influence of this deviation, it is necessary to first calculate the cancellation line l of the plane containing the bolt end face on the image. vanish .

[0076] The calculation steps for the disappearance line are as follows:

[0077] Step a1: Select two adjacent elliptical contours, calculate the common external tangent of the two ellipses, and obtain the four common external tangent points p1, p2, p3, and p4 where the two common external tangents are tangent to the ellipse;

[0078] Step a2: Calculate the intersection point of the external common tangents on the image, denoted as t1;

[0079] Step a3: Connect p1p2 and p3p4 respectively, and calculate the intersection point of these two lines, denoted as t2.

[0080] Step a4: According to projective geometry, these two points are the projection points of the point at infinity onto the image, and the disappearance line l of the plane containing the bolt end face onto the image. vanish The equation of the line connecting the projection points of two points at infinity (e.g.) Figure 2 As shown, by performing a vector cross product on t1 and t2, the disappearance line l of the plane containing the bolt end face on the image is calculated. vanish = t1 × t2.

[0081] Step S104: Based on the calibration parameters and shadow removal lines of the binocular camera, calculate the center position coordinates and end face normal vector of the bolt end face in the binocular camera coordinate system to obtain the pose of the bolt end face.

[0082] In this embodiment of the invention, the coordinates of the center position of the bolt end face and the normal vector of the end face in the binocular camera coordinate system are calculated to determine the precise pose of the bolt end face. After a series of coordinate changes from the camera coordinate system to the industrial robot arm base coordinate system, the sleeve installed at the front end of the robot arm is controlled to move precisely to the position of the bolt to be tightened and to wrap around the bolt to be tightened.

[0083] The bolt end face pose is calculated and generated using the following steps:

[0084] Step b1: According to the theory of algebraic projective geometry, the true projected image point of the circular feature center is the pole of the center of its projected ellipse about the vanishing line. The specific solution formula is as follows:

[0085] O=ω -1 l vanish ,

[0086] Where, ω -1 Let l be the inverse of the elliptic parameter matrix. vsnish Let O be the equation of the shadow line of the plane containing the bolt end face on the image, and O be the true projected image point.

[0087] The true projected image point of the center point of the bolt end face corresponding to the fitted ellipse in the left and right images is calculated using this formula.

[0088] Step b2: Based on the principle of binocular vision triangulation, and combined with the internal and external parameters of the left and right cameras, the true projected image point of the bolt end face center point, etc., calculate the center position coordinates of the bolt end face center in the camera coordinate system.

[0089] Step b3: Based on the relationship between the annihilation line and the plane normal vector in projective geometry, the orientation of the normal vector of the bolt end face can be obtained as follows:

[0090] n = K T l vanish ,

[0091] Among them, K T This is the transpose of the intrinsic parameter matrix of the corresponding camera obtained after calibration. The pose of the bolt end face is obtained based on the center position coordinates of the bolt end face in the camera coordinate system and the normal vector orientation of the bolt end face.

[0092] The bolt position calculation method provided in this embodiment achieves precise positioning during the bolt tightening process, effectively improving the product quality of wind turbine units and the automation and intelligence of the bolt tightening process.

[0093] This embodiment also provides a bolt pose calculation device, which is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0094] This embodiment provides a bolt pose calculation device, such as Figure 3 As shown, it includes:

[0095] The image acquisition module 301 is used to calibrate the various systems of the vision module, complete the calibration of the internal and external parameters of the binocular vision system, and acquire the left and right images of the bolt end face of the target area through the binocular camera.

[0096] The feature extraction module 302 is used to extract features from the elliptical projection of the bolt end face on the left and right images respectively using an ellipse detection algorithm, so as to obtain the parameter matrix of the projected ellipse.

[0097] The first calculation module 303 is used to calculate the shadow line on the image corresponding to the plane where the bolt end face is located based on the parameter matrix of the ellipse.

[0098] The second calculation module 304 is used to calculate the center position coordinates and end face normal vector of the bolt end face in the binocular camera coordinate system according to the calibration parameters and shadow removal lines of the binocular camera, so as to obtain the pose of the bolt end face.

[0099] In some alternative implementations, the image acquisition module 301 includes:

[0100] The calibration unit is used for calibrating the binocular vision system, the depth camera system, and for hand-eye calibration of the two camera systems with the sleeves that match the bolts to be tightened, which are mounted at the front end of the industrial robotic arm.

[0101] In some alternative implementations, the image acquisition module 301 further includes:

[0102] The preprocessing unit is used to perform distortion correction on the acquired left and right images based on the distortion coefficients calibrated by the stereo camera.

[0103] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0104] In this embodiment, the bolt pose calculation device is presented in the form of a functional unit. Here, a unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0105] This invention also provides a computer device having the above-described features. Figure 3 The bolt pose calculation device shown.

[0106] Please see Figure 4 , Figure 4 This is a schematic diagram of the structure of a computer device provided in an optional embodiment of the present invention, such as... Figure 4 As shown, the computer device includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 4 Take a processor 10 as an example.

[0107] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GDA), or any combination thereof.

[0108] The memory 20 stores instructions executable by at least one processor 10 to cause the at least one processor 10 to perform the method shown in the above embodiments.

[0109] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the computer device. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0110] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0111] The computer device also includes a communication interface 30 for communicating with other devices or communication networks.

[0112] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods shown in the above embodiments.

[0113] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A method for calculating bolt pose, characterized in that, The method includes: The various systems of the vision module are calibrated to complete the intrinsic and extrinsic parameter calibration of the binocular vision system. Simultaneously, left and right images of the bolt end face in the target area are acquired using the binocular cameras. The calibration of the various systems of the vision module includes the calibration of the binocular vision system, the depth camera system, and the hand-eye calibration between the two camera systems and the matching sleeves of the bolts to be tightened, mounted on the front end of the industrial robotic arm. The intrinsic and extrinsic parameter calibration of the binocular vision system includes calibrating the intrinsic and extrinsic parameters of the binocular vision system mounted on the industrial robotic arm, acquiring the visual model parameters inside the left and right cameras of the binocular vision system, and the left and right phase... External rotation and translation parameters between machines; RGB and DEPTH images of the working surface of the bolt to be fastened on the hub are acquired by a depth camera. The RGB images are input into a trained convolutional neural network model for bolt end face target segmentation. The output image is a mask of the bolt end face. The DEPTH and RGB images are aligned. The depth distance information of the target area of ​​the bolt end face mask in the RGB image in the depth camera coordinate system is calculated. The industrial robotic arm is moved according to the distance information and the parameters calibrated by hand and eye, so that the left and right cameras of the binocular vision system capture the left and right images of the bolt end face of the target area, respectively. The ellipse detection algorithm is used to extract features from the elliptical projections of the bolt end faces in the left and right images to obtain the parameter matrix of the projected ellipse. Based on the parameter matrix of the ellipse, calculate the shadow line on the image corresponding to the plane where the bolt end face is located; Based on the calibration parameters and anechoic lines of the binocular camera, the center position coordinates and end face normal vector of the bolt end face in the binocular camera coordinate system are calculated to obtain the pose of the bolt end face.

2. The method according to claim 1, characterized in that, The left and right images acquired by the stereo camera are subjected to distortion correction processing based on the distortion coefficients calibrated by the stereo camera.

3. The method according to claim 1, characterized in that, The parameter matrix of the projected ellipse is calculated using the following formula: ， A, B, C, D, E, and F are calculated using the following method: The fitted ellipses in the left and right images are combined and matched to ensure that the matched ellipses are projections of the same bolt end face onto their respective images. The fitted and matched ellipses are then rewritten into the following algebraic equation form. ， Where x and y are the horizontal and vertical coordinates of the elliptical contour points in the pixel coordinate system, and the parameters A, B, C, D, E, and F are obtained by solving and rewritten in the form of the ellipse parameter matrix.

4. The method according to claim 3, characterized in that, The calculation steps for the disappearance line include: Select two adjacent elliptical contours, calculate the common external tangent of the two ellipses, and obtain the four common tangent points p1, p2, p3, and p4 where the two common external tangents are tangent to the ellipse; Calculate the intersection of the external common tangents on the image. ; Connect p1p2 and p3p4 respectively to obtain two lines, and calculate the intersection of these two lines. ; Through the and Perform a vector cross product to calculate the disappearance line of the plane containing the bolt end face on the image. .

5. The method according to claim 4, characterized in that, Based on the principle of binocular vision triangulation, and combining the intrinsic and extrinsic parameters of the left and right cameras with the data information of the real projected image point of the bolt end face center point, the center position coordinates of the bolt end face center in the binocular camera coordinate system are calculated.

6. The method according to claim 5, characterized in that, The true projected image point of the bolt end face center point on the image is calculated using the following formula: ， in, It is the inverse of the elliptic parameter matrix. The equation of the disappearance line of the plane containing the bolt end face on the image. These are the actual projected image points.

7. The method according to claim 6, characterized in that, The orientation of the normal vector of the plane containing the bolt end face is calculated using the following formula: ， in, This is the transpose of the intrinsic parameter matrix of the corresponding stereo camera obtained after calibration.

8. A bolt position calculation device, characterized in that, The device includes: The image acquisition module is used to calibrate the various systems of the vision module, complete the calibration of the intrinsic and extrinsic parameters of the binocular vision system, and simultaneously acquire left and right images of the bolt end face of the target area through the binocular camera. The calibration of the various systems of the vision module includes the calibration of the binocular vision system, the depth camera system, and the hand-eye calibration between the two camera systems and the matching sleeves of the bolts to be tightened mounted on the front end of the industrial robotic arm. The calibration of the intrinsic and extrinsic parameters of the binocular vision system includes calibrating the intrinsic and extrinsic parameters of the binocular vision system mounted on the industrial robotic arm and acquiring the visual model parameters inside the left and right cameras of the binocular vision system. External rotation and translation parameters between the left and right cameras; RGB and DEPTH images of the working surface of the bolt to be fastened on the wheel hub are acquired by the depth camera. The RGB images are input into the trained bolt end face target segmentation convolutional neural network model, and the output image is the bolt end face mask. The DEPTH image and RGB image are aligned, and the depth distance information of the bolt end face mask target area in the RGB image in the depth camera coordinate system is calculated. The industrial robotic arm is moved according to the distance information and the parameters calibrated by hand and eye, so that the left and right cameras of the binocular vision system can capture the left and right images of the bolt end face of the target area respectively. The feature extraction module is used to extract features from the elliptical projection of the bolt end face on the left and right images respectively using the ellipse detection algorithm, so as to obtain the parameter matrix of the projected ellipse. The first calculation module is used to calculate the shadow line on the image corresponding to the plane where the bolt end face is located, based on the parameter matrix of the ellipse. The second calculation module is used to calculate the center position coordinates and end face normal vector of the bolt end face in the binocular camera coordinate system based on the calibration parameters and anechoic lines of the binocular camera, so as to obtain the pose of the bolt end face.

9. The apparatus according to claim 8, characterized in that, The image acquisition module also includes: The preprocessing unit is used to perform distortion correction on the acquired left and right images based on the distortion coefficients calibrated by the stereo camera.

10. A computer device, characterized in that, include: The system includes a memory and a processor, which are interconnected. The memory stores computer instructions, and the processor executes the computer instructions to perform the bolt pose calculation method according to any one of claims 1 to 7.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to execute the bolt pose calculation method according to any one of claims 1 to 7.

Images