A bolt loosening angle detection method in a complex scene based on a monocular camera

Abstract

The application relates to a bolt loosening angle detection method based on a monocular camera in a complex scene, which comprises the following steps: (1) obtaining a two-dimensional image of a bolt by using a monocular camera; (2) performing ellipse detection on the two-dimensional image obtained by adopting an arc support line segment method; (3) combining the detected ellipse and a standard bolt size to perform space circle re-projection calculation on the bolt head feature circle in the image to obtain the pose of the space circle relative to a camera coordinate system; (4) combining the characteristics that only axial displacement exists before and after bolt loosening and the normal vector does not change to determine the bolt pose and solve the bolt displacement; and (5) using the quantitative relationship between the displacement of a standard bolt relative to a base and the actual rotation amount to solve the bolt loosening angle; the method is far higher than a traditional contact type detection method in terms of detection efficiency, measurement speed and measurement range.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of intelligent detection, and particularly relates to a bolt loosening angle detection method in a complex scene based on a monocular camera. BACKGROUND

[0002] As a typical modern industrial standard fastener, the bolt plays a vital role in the fields of marine vessels, combat tanks, aerospace, etc. However, due to mechanical vibration or casting material defects during the operation of the fastened structure, the bolt is inevitably loose, which causes immeasurable losses. In order to ensure the safe and effective service of the bolt, it is particularly important to detect the loosening of the bolt to achieve fastening with sufficient pre-tightening force. Compared with the traditional manual or contact method of inspection and detection, the method of quantitatively detecting the loosening of the bolt by image processing and analysis of the image of the bolt structure has the advantages of non-contact, traceability, intelligent development, etc., and meets the development strategy of intelligent manufacturing of the country. In the actual service scene of the bolt, due to the different sizes of the bolt target on the fastened structure, the fastening direction is difficult to unify with the direction of the force to be applied, and the detection scene is complex, and there may also be unavoidable obstacles, cluttered background, light noise interference and other defects, which inevitably cause the difficulty of visual recognition and maintenance to rise sharply. The existing technology usually uses deep learning and a series of iterative algorithms in the algorithm, which has the disadvantages of high calculation cost and complex program design; the existing bolt positioning method is limited to two-dimensional plane, and there is little research on the pose recognition of the three-dimensional space of the bolt; and the hardware threshold usually uses binocular or depth camera devices, which are also too high in cost to match the low-cost industry advantage of intelligent industrial detection.

[0003] Among them, CN111932516A discloses an image processing-based identification-free double-nut hexagonal bolt loosening detection system, but the system needs to install an image collector above the bolt group to be monitored, and corresponds to the recognition of the circular feature in S6, the detection scene is single and the requirement is strict, the bolt direction in the actual detection scene is not considered, and the universality is lacking. CN110910350A discloses a nut loosening detection method for a wind power tower, which does not fully utilize the geometric features of the bolt shape, needs a large amount of model training process, and has high recognition complexity.

[0004] How to fully utilize the geometric features of the bolt shape, simplify the recognition algorithm complexity, and increase the diversity of the detection scene to realize the bolt loosening angle detection in a complex scene has become a problem to be solved at present. SUMMARY

[0005] The present application is directed to the deficiency of the existing research on bolt loosening angle detection, and proposes a bolt loosening angle detection method based on monocular camera in complex detection scene, which can realize the quantitative determination of the three-dimensional position and attitude of the bolt head under the two-dimensional image obtained by the monocular camera, so as to obtain the loosening angle of the bolt. The present application can be used for quantitative or qualitative detection of the bolt loosening angle under a series of complex environmental disturbances such as unobstructed or obstructed bolt in bolt service conditions.

[0006] In order to solve the above technical problems, the present application discloses a bolt loosening angle detection method based on monocular camera in complex detection scene, which comprises the following steps:

[0007] Step (1), using a monocular camera to obtain two-dimensional images of the bolt head before and after loosening, in which the circular features are unobstructed or obstructed;

[0008] Step (2), using the arc support line segment method to detect the ellipse of the two-dimensional image obtained in step (1);

[0009] Step (3), combining the ellipse detected in step (2) and the standard bolt size, performing spatial circle re-projection calculation on the bolt head feature circle in the image to obtain the pose of the spatial circle relative to the camera coordinate system;

[0010] Step (4), combining the characteristics that only the axial displacement exists before and after the bolt loosening and the normal vector is unchanged, determining the bolt pose and solving the bolt displacement to solve the problem of the ambiguity of the spatial circle calculated in step (3);

[0011] Step (5), combining the pose and center point position of the bolt head feature circle relative to the camera coordinate system before and after the bolt loosening obtained in step (4), and using the quantitative relationship between the displacement and the actual rotation of the standard bolt relative to the base, to solve the bolt loosening angle.

[0012] Further, the bolt pose determination and bolt displacement solving in step (4) comprises the following steps:

[0013] (4.1) Let the bolt coordinate system before loosening be O b -X b Y b Z b , and the bolt coordinate system after loosening be O' b -X' b Y' b Z' b , and let the spatial circle re-projection calculation of the bolt head before loosening be O c -X c Y c Z cPosition and pose are (x fi ,y fi ,z fi ),(n xfi ,n yfi ,n zfi ),

[0014] The relative camera coordinate system O c -X c Y c Z c Position and pose are (x li ,y li ,z li ),(n xli ,n xli ,n zli ), wherein i takes the value of 1 or 2; i takes the value of 1 or 2 corresponds to two sets of solutions, only one of the two sets of solutions is a true solution, and the other set of solutions is a false solution;

[0015] (4.2) Before the bolt loosens, the coordinate system O b -X b Y b Z b and the coordinate system O′ b -X′ b Y′ b Z′ b There is only displacement along Z b , denoted as The normal vector relationship of the space circle before and after the bolt loosens relative to the camera coordinate system O c -X c Y c Z c is expressed as

[0016] (n xfi ,n yfi ,n zfi )=k*(n xli ,n yli ,n zli ),k≠0

[0017] Verify whether the normal vectors in the position and pose of O c -X c Y c Z c and the position and pose of O c -X c Y c Z c meet the O c -X c Y c Z cThe constraint conditions for the normal vector relationship are defined. If these conditions are met, the solution is considered the true solution; otherwise, it is discarded as a false solution. In the true solution, the value of i is denoted as t. Similarly, the positions of the coordinate system relative to the camera coordinate system before and after the bolt loosening are denoted as […]. and

[0018]

[0019] The attitude angle corresponding to the normal vector of the true solution is denoted as Equation (18):

[0020]

[0021] (4.3) The pose transformation of the coordinate system relative to the camera coordinate system before the bolt is loosened is expressed as equation (19):

[0022]

[0023] in This indicates the position of the coordinate system in the camera coordinate system before the bolt loosened. Since the bolt only exists along the Z-axis before and after loosening,... b The displacement in the positive direction of the axis, i.e., the movement relative to the base, is denoted as Δz. Since Δz > 0, the origin O′ of the coordinate system is zero after the bolt loosens. b Coordinate system O before the bolt loosened b -X b Y b Z b The coordinates below are given by equation (20):

[0024]

[0025] The relationship between the position vector of the coordinate system relative to the camera coordinate system before and after the bolt loosening and the position vector of the coordinate system relative to the coordinate system before the bolt loosening is denoted by equation (21):

[0026]

[0027] Before the bolt was loosened, coordinate system O b -X b Y b Z b Central O b Substituting the pose transformation equation of the coordinate system relative to the camera coordinate system before the bolt loosens, we obtain equation (22).

[0028]

[0029] After the bolt is loosened, the origin O′ in the coordinate system b Substituting the pose transformation equation of the coordinate system relative to the camera coordinate system before the bolt loosens, we obtain equation (23):

[0030]

[0031] In combination with formula (21), formula (22) is subtracted from formula (23) to obtain formula (24):

[0032]

[0033] In combination with the pose conversion relationship of the bolt coordinate system relative to the camera coordinate system and formula (18), formula (24) is specifically recorded as formula (25):

[0034]

[0035] Formula (25) is transformed to obtain the out-momentum of the bolt relative to the base, recorded as formula (26):

[0036]

[0037] As a further limited scheme of the present application, the specific steps for quantitatively obtaining the loosening angle of the bolt in step (5) are:

[0038] (5.1) The bolt to be detected is a standard part and is regarded as a rigid body, and the out-momentum Δz of the bolt relative to the base and the loosening momentum Δθ of the bolt around its own axis have a quantitative relationship, recorded as formula (27):

[0039]

[0040] Wherein d is the pitch, which can be obtained from the standard model of the detected bolt;

[0041] (5.2) In combination with formula (26) and formula (27), the loosening momentum of the bolt is formula (28):

[0042]

[0043] The present application has the following beneficial effects: compared with the traditional contact method for measuring the bolt loosening angle, the method proposed in the present application can determine the bolt pose and the angle change before and after loosening by only using a monocular camera to collect the circular arc feature image of the bolt head. Compared with the related visual detection method for detecting the bolt loosening angle, the present application not only retains the advantages of non-contact detection, but also converts the loosening momentum detection problem into the out-momentum detection problem by combining the standard physical size of the bolt, reduces the complexity of the detection calculation program, and only uses the two-dimensional image collected by the monocular camera combined with the corresponding algorithm to complete the detection, greatly reducing the cost threshold of the equipment. Therefore, the detection efficiency, measurement speed and measurement range of the present application are much higher than those of the traditional contact detection method, and the calculation cost, environmental resistance and software and hardware requirements are lower than those of the remaining visual detection methods, which are extremely critical improvements for the detection closed loop of the safe and stable service of the bolt. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 is the overall flowchart of the bolt loosening angle detection method in a complex scene based on a monocular camera according to the present application;

[0045] Figure 2 is the image acquisition schematic diagram of the bolt loosening angle detection method in a complex scene based on a monocular camera according to the present application;

[0046] Figure 3 is the ellipse detection flowchart of the bolt loosening angle detection method in a complex scene based on a monocular camera according to the present application;

[0047] Figure 4 is the ellipse detection result schematic diagram of the bolt loosening angle detection method in a complex scene based on a monocular camera according to the present application;

[0048] Figure 5 is the conic section perspective projection model diagram of the bolt loosening angle detection method in a complex scene based on a monocular camera according to the present application;

[0049] Figure 6 is the coordinate system conversion relationship schematic diagram of the bolt loosening angle detection method in a complex scene based on a monocular camera according to the present application;

[0050] Figure 7 is the bolt loosening moment and rotation amount schematic diagram of the bolt loosening angle detection method in a complex scene based on a monocular camera according to the present application; DETAILED DESCRIPTION

[0051] The present application will be further described in detail by the following examples, so that those skilled in the art can implement the present application according to the description herein.

[0052] It should be understood that the terms such as "have", "contain" and "include" used herein do not exclude the presence or addition of one or more other elements or combinations thereof.

[0053] Example 1

[0054] The present application will be further described in detail by the following examples, so that those skilled in the art can implement the present application according to the description herein. Figures 1-7 The main steps of the bolt loosening angle detection method in a complex scene based on a monocular camera according to the present application are as follows:

[0055] Step 1, please refer to Figure 2 , use a monocular camera to obtain a two-dimensional image group Image_former of the head circular feature before bolt loosening which is not blocked or blocked, and a two-dimensional image group Image_latter of the head circular feature after bolt loosening which is not blocked or blocked.

[0056] Step 2: Use the Arc-Support Line Segments (ASLS) method to perform ellipse detection on the image group from Step 1.

[0057] Step 2-1: Since the principle involves identifying and detecting ellipses in images, we can first detect ellipses in a single image, and then apply the same detection method to the remaining images. The circular features of the bolt head obtained in Step 1 are processed using the Arc-Support Line Segments (ASLS) method. The algorithm flow is as follows: Figure 3 As shown (for detailed implementation steps, see: Arc-support Line Segments Revisited: An Efficient and High-quality Ellipse Detection). Please refer to the ellipse detection results. Figure 4 .

[0058] Step 2-2: Obtain the ellipse E identified after processing in Step 2-1, mark its center and major and minor axes, and obtain its equation relative to the pixel coordinate system.

[0059] g(u,v)=au 2 +bv 2 +cuv+du+ev+f=0 (1)

[0060] Step 3: Combining the ellipse detected in Step 2 and the standard bolt size, perform spatial circle reprojection calculation on the circular features of the bolt head in the image to obtain the position and orientation of the spatial circle relative to the camera coordinate system.

[0061] Step 3-1, for ease of research, please refer to... Figure 5 Establish a perspective projection model of a conic section and establish a bolt coordinate system O. b -X b Y b Z b Let the normal vector of the bolt head circular feature (geometrically, the bolt's axis) be the bolt coordinate system O. b -X b Y b Z b Z in b Axis, camera coordinate system O c -X c Y c Z c Image coordinate system O-XY, pixel coordinate system O p -uv, the camera imaging plane M.

[0062] Step 3-2, please refer to Figure 5, the parameter equation of the ellipse E in the pixel coordinate system is

[0063] g(u,v) = au 2 +bv 2 +cuv+du+ev+f = 0 (2)

[0064] The ellipse E in the pixel coordinate system is combined with the camera coordinate system and substituted into the following projection transformation

[0065]

[0066] where (u0,v0) is the center position of the pixel coordinate system, and (f x ,f y ) is the focal length of the image coordinate system X and Y direction. The circle cone surface C which passes through the center O c of the camera coordinate system and the ellipse E is constructed, and the parameter equation is

[0067]

[0068] where

[0069]

[0070] Step 3-2, the equation (4) is expressed in the form of quadratic form to obtain

[0071]

[0072] Since Q is a symmetric matrix, it can be decomposed as follows

[0073]

[0074] where P is a standard orthogonal matrix, and the column vectors {e1 e2 e3} corresponding to Q form its characteristic matrix; Λ is a diagonal matrix, and the elements on the diagonal correspond to the eigenvalues of Q.

[0075] Step 3-3, let [X′ c Y′ c Z′ c ] = [X c Y c Z c ]P, which is equivalent to rotating the camera coordinate system around the camera coordinate system origin to a new coordinate system O c -X c Y c Z c , in this coordinate system, the expression of the circle cone surface is:

[0076] C(X′ c 2Y' c 2 Z' c 2 ) = λ1X' c 2 + λ2Y' c 2 + λ3Z' c 2 = 0 (8)

[0077] Since the conic surface C in the new coordinate system and the standard equation of the elliptic cone

[0078]

[0079] Similar, combined with the properties of the standard elliptic cone and the true radius R of the standard bolt head space circle, the relative O' c -X' c Y' c Z' c The center and normal vector of the space circle corresponding to the ellipse E in the O'

[0080]

[0081] Step 3-4, the relative O' c -X' c Y' c Z' c The center and normal vector of the space circle corresponding to the ellipse E in the O' c -X c Y c Z c , the conversion relationship is as follows:

[0082]

[0083] Step 3-5, since the normal vector coincides with the Z b in the bolt coordinate system, combined with the normal vector in equation (11), and the pose conversion relationship of the bolt coordinate system relative to the camera coordinate system (due to the symmetry of the space circle, the yaw angle of the symmetry axis is ignored)

[0084]

[0085] Where, is the pose conversion matrix of the bolt coordinate system to the camera coordinate system, θ is the pitch angle, is the roll angle. The pitch angle θ and the roll angle to describe the attitude angle of the space circle in step 3-4 is

[0086]

[0087] Step 4, the characteristics of only axial displacement and unchanged normal vector exist before and after the loosening of the bolt, and the problem of ambiguity of the spatial circle calculated in step 3 is removed.

[0088] Step 4-1, please refer to Figure 6 , and the coordinate system of the bolt before loosening is O b -X b Y b Z b , and the coordinate system of the bolt after loosening is O b ′-X b ′Y b ′Z b ′, and the relative camera coordinate system O c -X c Y c Z c position and attitude of the bolt head space circle before loosening is

[0089]

[0090] , and there are two sets of solutions. The relative camera coordinate system O c -X c Y c Z c position and attitude of the bolt head space circle after loosening is

[0091]

[0092] It should be noted that the solutions in formula (14) and formula (15) correspond one by one, that is, solution1 in formula (14) corresponds to solution1 in formula (15), which can be used as a set of solutions of the pose before and after the bolt loosening. Only one of the two sets of solutions is the true solution, and the other is the false solution. The solution in the following is also the same.

[0093] Step 4-2, since only the displacement relative to the axis of the bolt exists before and after the loosening of the bolt, in the camera coordinate system O c -X c Y c Z c , that is, the spatial circle only has displacement relative to its normal vector, and the coordinate system O b -X b Y b Z b before the loosening of the bolt and the coordinate system O b ′-X b ′Y b ′Z b ′ after the loosening of the bolt only have displacement along Z b , denoted as , and the spatial circle relative to the camera coordinate system Oc -X c Y c Z c unchanged, i.e. the normal vectors before and after the change are parallel, it is permissible to take the space circle on the bolt before and after the loosening as the space circle relative to the camera coordinate system O c -X c Y c Z c The normal vector relationship is expressed as

[0094] (n xf1 ,n yf1 ,n zf1 )=k*(n xl1 ,n yl1 ,n zl1 ),k≠0 (16)

[0095] Verify whether the normal vectors in equation (14) and equation (15) meet the constraint condition of equation (16). If yes, solution 1 is the real solution and solution 2 is eliminated as a false solution; if not, solution 1 is eliminated as a false solution and solution 2 is the real solution. For the convenience of subsequent description, it is permissible to take solution 1 as the real solution. Similarly, the position of the coordinate system before and after the loosening of the bolt relative to the camera coordinate system is

[0096]

[0097] Combine equation (13) to record the attitude angle corresponding to the real normal vector as

[0098]

[0099] Step 4-3, since the coordinate system before the loosening of the bolt relative to the camera coordinate system not only has an attitude transformation, but also has a translation transformation, the pose transformation of the coordinate system before the loosening of the bolt relative to the camera coordinate system can be expressed as

[0100]

[0101] wherein represents the position of the coordinate system before the loosening of the bolt in the camera coordinate system. Since only displacement along the Z b axis in the positive and negative directions, i.e. the movement of the base, is changed before and after the loosening of the bolt, and is recorded as Δz>0, the coordinates of the origin O b ' of the coordinate system after the loosening of the bolt in the coordinate system O b -X b Y b Z b before the loosening of the bolt can be recorded as

[0102]

[0103] Please refer to Figure 4 , the position vector of the coordinate system before and after the bolt loosening relative to the camera coordinate system, and the relationship between the position vector of the coordinate system after the bolt loosening relative to the coordinate system before the bolt loosening is

[0104]

[0105] The coordinate system before the bolt loosening O b -X b Y b Z b The origin O b Substituted into equation (19), we have

[0106]

[0107] The origin O b ′ of the coordinate system after the bolt loosening is substituted into equation (19), we have

[0108]

[0109] Combined with equation (21), equation (22) is subtracted from equation (23) to obtain

[0110]

[0111] Combined with equation (12) and equation (18), equation (24) is specifically written as

[0112]

[0113] Transforming equation (25) can obtain the bolt relative to the base of the momentum as

[0114]

[0115] Step 5, combined with the bolt loosening before and after the relative camera coordinate system of the head of the circular feature attitude and center point position obtained in step 4, and using the quantitative relationship between the standard bolt relative to the base of the momentum and the actual rotation, the bolt loosening angle is obtained.

[0116] Step 5-1, please refer to Figure 7 First, since the detected bolt is a standard part and can be regarded as a rigid body, the quantitative relationship between the bolt relative to the base of the momentum Δz and the loosening amount Δθ around its own axis is recorded as

[0117]

[0118] Where d is the pitch. The pitch can be obtained from the standard model of the detected bolt. Therefore, the bolt loosening detection problem can be converted into the bolt momentum detection problem, combined with the standard size of the bolt, and the detection process is simplified.

[0119] Step 5-2, combining equation (26) and equation (27) gives the looseness of the bolt as

[0120]

[0121] While embodiments of the application have been disclosed in connection with the preferred embodiments of the application as described and illustrated herein, it will be understood that still other modifications can be made without departing from the spirit and scope of the application as disclosed herein, and, accordingly, other modifications and changes in the application can be undertaken by those skilled in the art.

Claims

1.A method for detecting bolt loosening angle in a complex scene based on a monocular camera, characterized in that, The method comprises: Step (1), obtaining two-dimensional images of the head circular feature of the bolt before and after loosening, which are not blocked or blocked by a monocular camera; Step (2), using the arc support line segment method to detect the ellipse of the two-dimensional images obtained in step (1); Step (3), combining the ellipses detected in step (2) and the standard bolt size, performing spatial circle re-projection calculation on the bolt head feature circle in the image to obtain the pose of the spatial circle relative to the camera coordinate system; Step (4), combining the characteristics that only axial displacement exists before and after bolt loosening and the normal vector does not change, determining the bolt pose and solving the bolt displacement to solve the problem of the ambiguity of the spatial circle calculated in step (3); Step (5), combining the head feature circle pose and center point position of the bolt before and after loosening relative to the camera coordinate system obtained in step (4), and using the quantitative relationship between the actual rotation amount and the displacement of the standard bolt relative to the base, the bolt loosening angle is calculated; The bolt pose determination and bolt displacement solving in step (4) comprises the following steps: (4.1) Let the coordinate system of the bolt before loosening be , the coordinate system of the bolt after loosening be , the relative camera coordinates of the spatial circle of the bolt head calculated by re-projection before loosening be , and the position and pose of the relative camera coordinates be , respectively, Relative camera coordinate system calculated by re-projecting the space circle of the bolt head after loosening The position and the posture are respectively Wherein, i is 1 or 2; i is 1 or 2, corresponding to two groups of solutions, only one of the two groups of solutions is a true solution, and the other is a false solution; (4.2) Coordinate system before bolt loosening Coordinate system after bolt loosening There is only displacement along denoted as The normal vector relationship between the space circles before and after bolt loosening and the camera coordinate system is expressed as ， Verification Position and pose and Whether the normal vector in the position and pose meets The constraint condition of the normal vector relationship, meet is recorded as true solution, not meet is eliminated as false solution; The value of i in the true solution is recorded as t, and the positions of the same bolt before and after loosening are recorded as : ， The pose angle corresponding to the normal vector corresponding to the real solution is denoted as formula (18): ， (4.3) The pose conversion formula of the bolt loosening before coordinate system relative to the camera coordinate system is denoted as formula (19): ， wherein, is a pose transformation matrix from the bolt coordinate system to the camera coordinate system, represents the position of the bolt coordinate system in the camera coordinate system before the bolt loosening, since only the displacement along the positive and negative directions of the axis, i.e. the out-of-plane movement relative to the base, occurs before and after the bolt loosening, denoted as , the origin of the bolt coordinate system after the bolt loosening is , the coordinates of the bolt coordinate system before the bolt loosening in the camera coordinate system are denoted as , and the coordinates of the bolt coordinate system after the bolt loosening in the camera coordinate system are denoted as ， The relationship between the position vector of the bolt loosening before and after coordinate system relative to the camera coordinate system and the position vector of the bolt loosening after coordinate system relative to the bolt loosening before coordinate system is denoted as formula (21): ， Pre-bolt loosening coordinate system Center of the coordinate system Substituting the pose conversion formula of the pre-bolt loosening coordinate system relative to the camera coordinate system, formula (22) is obtained, ， Substitute the pose conversion formula of the coordinate system before the bolt loosening into the pose conversion formula of the coordinate system after the bolt loosening, to obtain formula (23): Substitute the pose conversion formula of the coordinate system before the bolt loosening into the pose conversion formula of the coordinate system after the bolt loosening, to obtain formula (23): ， Combined with formula (21), formula (22) and formula (23) are subtracted to obtain formula (24): ， Combined with the pose conversion relationship of the bolt coordinate system relative to the camera coordinate system and formula (18), formula (24) is specifically denoted as formula (25) ， By transforming formula (25), the bolt displacement relative to the base is denoted as formula (26): 。 2. The bolt loosening angle detection method based on monocular camera in complex scene according to claim 1, characterized in that, The specific steps of step (5) for quantitatively obtaining the loosening angle of the bolt are: (5.1) The bolt to be detected is a standard piece, regarded as a rigid body, and the bolt's momentum relative to the base and the loosening momentum around its own axis There is a quantitative relationship, denoted as equation (27): ， wherein is the pitch, which can correspond to be taken from a standard model of the bolt being detected; (5.2) Combined with formula (26) and formula (27), the loosening amount of the bolt is formula (28): 。

Images