Method and device for determining position of bolt on test piece, equipment and medium

Abstract

The embodiment of the invention provides a method and device for determining the position of a bolt on a test piece, equipment and a medium, and relates to the technical field of intelligent manufacturing. The method comprises the following steps: acquiring a first image of a tracking target and a second image of a calibration target, and determining a coordinate conversion relation based on the first image and the second image; acquiring a third image that a tightening gun head of the tracking target is placed on at least one bolt of the tightening component, and determining a first position of at least one bolt on the teaching template based on the coordinate conversion relation and the real-time position of the tracking target in the third image; and determining a second position of the at least one bolt on the test piece according to the conversion relationship between the teaching template and the test piece and the first position of the at least one bolt on the teaching template. By adopting the technical scheme, the pose information of the tracking target can be accurately positioned, so that high-precision tightening operation can be more accurately realized.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure relate to the technical field of intelligent manufacturing, and in particular, to a method, device, equipment and medium for determining the position of a bolt on a test piece. BACKGROUND

[0002] At present, the tightening assembly process in intelligent manufacturing often requires tightening operation according to a specified order. Direct detection of all bolts on a component faces the problems of missed detection and wrong detection. The distinction between a black bolt and a black part background is low, and the problem of missed detection and wrong detection of direct positioning of the bolt is widespread. Based on general image matching methods, the positioning of all bolts can also be achieved, but image matching is difficult to achieve high-precision positioning in 3D space due to the existence of height difference, and the precision is low.

[0003] Therefore, there is an urgent need for a method for determining the position of a bolt on a test piece, which can accurately locate the position information of the bolt, and thus more accurately implement high-precision tightening operation. SUMMARY

[0004] In order to solve the above technical problems or at least partially solve the above technical problems, embodiments of the present disclosure provide a method, device, equipment and medium for determining the position of a bolt on a test piece.

[0005] A first aspect of the embodiments of the present disclosure provides a method for determining the position of a bolt on a test piece, the method comprising:

[0006] obtaining a first image of a tracking target and a second image of a calibration target, and determining a coordinate conversion relationship based on the first image and the second image; wherein the coordinate conversion relationship is a spatial coordinate conversion relationship from the tracking target to a tightening gun head;

[0007] obtaining a third image of the tightening gun head placed on at least one bolt of a tightening component of the tracking target, and determining a first position of at least one bolt on a teaching template based on the coordinate conversion relationship and a real-time position of the tracking target in the third image;

[0008] determining a second position of at least one bolt on the test piece according to a conversion relationship between the teaching template and the test piece, and the first position of at least one bolt on the teaching template.

[0009] In one example, the determination of the first position of at least one bolt on the teaching template based on the coordinate conversion relationship and the real-time position of the tracking target in the third image comprises:

[0010] calculating a feature contour of at least one feature point of the real-time position of the tracking target in the third image;

[0011] According to the feature profile of the at least one feature point, fitting calculation center coordinate information, and correcting the center coordinate information based on a multi-circle cross ratio, target center coordinate information is obtained;

[0012] According to the target center coordinate information and the three-dimensional coordinate information of the calibration target model feature points, tracking target pose information is determined; wherein the tracking target pose information is used to represent the coordinate information of the tracking target in the camera coordinate system;

[0013] Based on the coordinate conversion relationship and the tracking target pose information, the first position of at least one bolt on the teaching template is determined.

[0014] In one example, the determination of the tracking target pose information according to the target center coordinate information and the three-dimensional coordinate information of the calibration target model feature points comprises:

[0015] Based on the target center coordinate information, the three-dimensional coordinate information of the calibration target model feature points, and a preset mapping relationship, the tracking target pose information is determined; wherein the preset mapping relationship is the relationship between the target center coordinate information, the three-dimensional coordinate information of the calibration target model feature points, the camera intrinsic parameter matrix, and the distance value between the target center coordinate information and the camera.

[0016] In one example, the determination of the conversion relationship of the teaching template and the test piece comprises:

[0017] The position of the test piece is detected by the tightening gun head, and a point detection position is obtained.

[0018] The point detection position is matched with the teaching marker point on the teaching template in space pose, and the conversion relationship of the teaching template and the test piece is obtained.

[0019] In one example, after the conversion relationship of the teaching template and the test piece is obtained, the method further comprises:

[0020] The other marker points are iteratively looped until the error value of the conversion relationship of the teaching template and the test piece is less than a threshold value, and then a target conversion relationship is output.

[0021] In one example, after the target conversion relationship is output, the method further comprises:

[0022] Based on the target conversion relationship and the first position of at least one bolt on the teaching template, the third position of at least one bolt on the test piece is determined.

[0023] A second aspect of the embodiments of the present disclosure provides a device for determining the position of a bolt on a test piece, the device comprising:

[0024] The first acquisition module is configured to acquire a first image of a tracking target and a second image of a calibration target, and determine a coordinate conversion relationship based on the first image and the second image; wherein the coordinate conversion relationship is a spatial coordinate conversion relationship from the tracking target to a tightening gun head;

[0025] The second acquisition module is configured to acquire a third image in which the tightening gun head of the tracking target is placed on at least one bolt of a tightening component, and determine a first position of at least one bolt on the teaching template based on the coordinate conversion relationship and a real-time position of the tracking target in the third image;

[0026] The first determination module is configured to determine a second position of at least one bolt on the test piece according to the conversion relationship of the teaching template and the test piece, and the first position of at least one bolt on the teaching template.

[0027] In one example, the second acquisition module is configured to:

[0028] Calculate a feature contour of at least one feature point of the real-time position of the tracking target in the third image;

[0029] Fit and calculate a circle center coordinate information according to the feature contour of the at least one feature point, and correct the circle center coordinate information based on a multiple circle cross ratio to obtain target circle center coordinate information;

[0030] Determine tracking target pose information according to the target circle center coordinate information and three-dimensional coordinate information of calibration target model feature points; wherein the tracking target pose information is used to represent coordinate information of the tracking target in a camera coordinate system;

[0031] Determine the first position of at least one bolt on the teaching template based on the coordinate conversion relationship and the tracking target pose information.

[0032] In one example, the second acquisition module is configured to:

[0033] Determine the tracking target pose information based on the target circle center coordinate information, the three-dimensional coordinate information of calibration target model feature points, and a preset mapping relationship; wherein the preset mapping relationship is a relationship between the target circle center coordinate information, the three-dimensional coordinate information of calibration target model feature points, a camera intrinsic parameter matrix, and a distance value between the target circle center coordinate information and the camera.

[0034] In one example, the determination of the conversion relationship of the teaching template and the test piece includes:

[0035] Position point inspection is performed on a preset number of marking points on the test piece by the tightening gun head to obtain a point inspection position;

[0036] spatial pose matching the point inspection position and a teaching mark point on the teaching template, to obtain a conversion relationship of the teaching template and the test piece.

[0037] In one example, after the conversion relationship of the teaching template and the test piece is obtained, the device further includes:

[0038] an output module configured to: iteratively loop other mark points until an error value of the conversion relationship of the teaching template and the test piece is less than a threshold value, and then output a target conversion relationship.

[0039] In one example, after the target conversion relationship is output, the device further includes:

[0040] a second determination module configured to determine a third position of at least one bolt on the test piece based on the target conversion relationship and the first position of at least one bolt on the teaching template.

[0041] A third aspect of the embodiments of the present disclosure provides an electronic device, which includes a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the method of the first aspect.

[0042] A fourth aspect of the embodiments of the present disclosure provides a computer readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the method of the first aspect can be implemented.

[0043] The embodiments of the present disclosure provide a method, device, equipment and medium for determining a bolt position on a test piece, which includes: obtaining a first image of a tracking target and a second image of a calibration target, determining a coordinate conversion relationship based on the first image and the second image; obtaining a third image of the tracking target with a tightening gun head placed on at least one bolt of a tightening component, determining a first position of at least one bolt on a teaching template based on the coordinate conversion relationship and a real-time position of the tracking target in the third image; and determining a second position of at least one bolt on the test piece according to a conversion relationship of the teaching template and the test piece and the first position of at least one bolt on the teaching template. By using the technical solution, the pose information of the tracking target can be accurately positioned, and then a high-precision tightening operation can be more accurately implemented. BRIEF DESCRIPTION OF DRAWINGS

[0044] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate embodiments consistent with the present disclosure and, together with the description, further serve to explain the principles of the present disclosure.

[0045] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 The diagram shown illustrates the hardware configuration of a process.

[0047] Figure 2 This is a flowchart illustrating a method for determining the position of bolts on a test piece according to an embodiment of this disclosure;

[0048] Figure 3 This is a flowchart illustrating a method for determining the position of bolts on a test piece according to an embodiment of this disclosure;

[0049] Figure 4 This is a schematic diagram of the structure of a device for determining the position of bolts on a test piece according to an embodiment of this disclosure;

[0050] Figure 5 This is a schematic diagram of the structure of an electronic device according to an embodiment of this disclosure. Detailed Implementation

[0051] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0052] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0053] In this embodiment, please refer to Figure 1 The diagram shows a schematic of the hardware configuration for a process. This hardware configuration includes a vision sensor, an industrial computer, an assembly control system, a tracking target, a tightening gun, a workpiece, and bolts.

[0054] Figure 2 This is a flowchart illustrating a method for determining the position of bolts on a test piece according to an embodiment of this disclosure. This method can be executed by an electronic device. The electronic device can be exemplarily understood as a device such as a mobile phone, tablet computer, laptop computer, desktop computer, or smart TV. Figure 2 As shown, the method provided in this embodiment includes the following steps:

[0055] S201, acquire a first image of the tracking target and a second image of the calibration target, and determine a coordinate conversion relationship based on the first image and the second image; wherein the coordinate conversion relationship is a spatial coordinate conversion relationship from the tracking target to the tightening gun head.

[0056] In one example, the tracking target is fixedly installed around the rear handle of the tightening gun, the tightening gun head with the tracking target is placed on the calibration target hole, and the tightening gun and the calibration target are placed in the camera field of view. The first image is obtained by shooting the tracking target with the camera, and the second image is obtained by shooting the calibration target with the camera. The first image is used to track the position of the tightening gun head in real time, and the second image is used for initial calibration. Then, 5 mark point coordinates of the first image are determined in the first image, and 5 mark point coordinates of the second image are determined in the second image. The 5 mark point coordinates of the first image and the digital model of the tracking target are combined, and the combined result is used to determine the pose information of the tracking target by using the PNP algorithm. The 5 mark point coordinates of the second image and the digital model of the calibration target are combined, and the combined result is used to determine the pose information of the calibration target by using the PNP algorithm. Finally, the coordinate conversion relationship is determined according to the pose information of the tracking target and the pose information of the calibration target.

[0057] S202, acquire a third image of the tightening gun head placed on at least one bolt of the tightening component, and determine the first position of at least one bolt on the teaching template based on the coordinate conversion relationship and the real-time position of the tracking target in the third image.

[0058] In one example, the teaching template is placed at the workpiece detection position, and the tightening gun head is placed on the at least one bolt in sequence. The third image is acquired by the camera. Since the coordinate conversion relationship from the tracking target to the tightening gun head is known, and the tightening gun head is placed on the at least one bolt in sequence, the first positions of all the bolts on the teaching template can be known.

[0059] S203, determine the second position of at least one bolt on the test piece according to the conversion relationship between the teaching template and the test piece, and the first position of at least one bolt on the teaching template.

[0060] In one example, the conversion relationship between the teaching template and the test piece is obtained by testing. After the first positions of all the bolts on the teaching template are determined, and the conversion relationship between the teaching template and the test piece is obtained, the second positions of at least one bolt on the test piece can be obtained through the conversion relationship.

[0061] The embodiment of the present disclosure provides a method for determining the position of a bolt on a test piece, which comprises the following steps: acquiring a first image of a tracking target and a second image of a calibration target, determining a coordinate conversion relationship based on the first image and the second image; acquiring a third image of the tracking target with a tightening gun head placed on at least one bolt of a tightening component, determining a first position of at least one bolt on a teaching template based on the real-time position of the tracking target in the third image and the coordinate conversion relationship; and determining a second position of at least one bolt on the test piece based on the conversion relationship between the teaching template and the test piece and the first position of at least one bolt on the teaching template. By using the technical scheme, the pose information of the tracking target can be accurately positioned, and then the high-precision tightening operation can be more accurately realized.

[0062] Figure 3 A flowchart of a method for determining the position of a bolt on a test piece is shown. The embodiment of the present disclosure is optimized on the basis of the above-mentioned embodiment, and can be combined with each optional scheme in one or more of the above-mentioned embodiments.

[0063] As shown in Figure 3 , the method for determining the position of a bolt on a test piece can comprise the following steps:

[0064] S301, acquiring a first image of a tracking target and a second image of a calibration target, and determining a coordinate conversion relationship based on the first image and the second image; wherein the coordinate conversion relationship is a spatial coordinate conversion relationship from the tracking target to a tightening gun head.

[0065] In one example, this step can refer to the content of step S201, which will not be repeated here.

[0066] S302, acquiring a third image of the tracking target with a tightening gun head placed on at least one bolt of a tightening component.

[0067] In one example, this step can refer to the content of step S202, which will not be repeated here.

[0068] S303, calculating the feature contour of at least one feature point of the real-time position of the tracking target in the third image.

[0069] In one example, the at least one feature point can be a feature point sub-pixel coordinate position in the third image, the feature shape can be a circle, and the number of feature shapes is at least 4. The contours of all circles are extracted.

[0070] S304, fitting and calculating the center coordinate information according to the feature contour of the at least one feature point, and correcting the center coordinate information based on the multiple circle cross ratio to obtain target center coordinate information.

[0071] In one example, the process of fitting the circle center coordinate information is as follows: the general equation of the circle is x2+y2+Dx+Ey+F=0, where D and E are the coefficients of the first-order terms of the circle equation, and F is the constant term.

[0072] Suppose the point set (xi, yi), i=1, 2…n, is solved by minimizing ∑(xi2+yi2+Dxi+Eyi+F)2. Take the partial derivative of D, E, and F and set it to zero, and solve the equation set to get the parameters. The final circle center coordinate information is (-2D, -2E), and the radius r=(2D)2+(2E)2-F.

[0073] In one example, the circle center coordinate information is corrected based on the cross ratio of multiple circles, and the calculation process of the target circle center coordinate information is as follows:

[0074] For the collinear four points G, H, I, and M, the cross ratio is defined as (G, H, I, M)=(G-M)(H-I)(G-I)(H-M). The cross ratio is used to correct the geometric relationship of multiple circles based on the invariance of the cross ratio under the projective transformation. Specifically, the discrete point set of the circle edge in the image is extracted, the ellipse is fitted, and the initial circle center is calculated. A plurality of secant lines are selected on the circle, the midpoints of the secant lines are calculated, the cross ratio constraint equation set is established, the circle center coordinate information is optimized, and the target circle center coordinate information (u, v) is obtained.

[0075] S305, according to the target circle center coordinate information and the three-dimensional coordinate information of the calibration target number module feature points, determine the tracking target pose information; wherein, the tracking target pose information is used to represent the coordinate information of the tracking target in the camera coordinate system.

[0076] In one example, according to the target circle center coordinate information and the three-dimensional coordinate information of the calibration target number module feature points, the tracking target pose information is determined, including:

[0077] Based on the target circle center coordinate information, the three-dimensional coordinate information of the calibration target number module feature points, and the preset mapping relationship, the tracking target pose information is determined; wherein, the preset mapping relationship is the relationship between the target circle center coordinate information, the three-dimensional coordinate information of the calibration target number module feature points, the camera intrinsic parameter matrix, and the distance value of the target circle center coordinate information and the camera.

[0078] In one example, the distance value of the target circle center coordinate information and the camera can be Zc, and specifically, the process of determining the tracking target pose information is as follows:

[0079]

[0080] Wherein, the target circle center coordinate information (u, v), k is the camera intrinsic parameter matrix, R and T are the tracking target pose information, X w , Y w , and Zw To calibrate the three-dimensional coordinate information of the target number model feature points.

[0081] S306, based on the coordinate conversion relationship and the tracking target pose information, determine the first position of at least one bolt on the teaching template.

[0082] In one example, in combination with the calibration pose of the tightening gun head, the position coordinates of the bolt in the three-dimensional space can be calculated, and the calculated position of each bolt is stored in the corresponding product model teaching template.

[0083] The calculation formula of the first position of at least one bolt on the teaching template is P_bolt = H·P_gunCab, wherein P_gunCab is the three-dimensional coordinate information of the tightening gun head under the tracking target calculated in the last step of the calibration process, and H is R and T.

[0084] S307, according to the conversion relationship between the teaching template and the test piece, and the first position of at least one bolt on the teaching template, determine the second position of at least one bolt on the test piece.

[0085] In one example, the determination of the conversion relationship between the teaching template and the test piece includes:

[0086] By the tightening gun head, the position of the preset number of mark points on the test piece is detected, and the detection position is obtained.

[0087] The detection position is matched with the teaching mark point on the teaching template in space pose to obtain the conversion relationship between the teaching template and the test piece.

[0088] In one example, the preset number can be 4. The process of matching the detection position with the teaching mark point on the teaching template in space pose is as follows:

[0089] Any three of the four mark points are determined to form a plane, one point in the plane is taken as the origin, and the direction of the origin and another point is selected as the X direction. The normal of the plane is determined as the Z straight line. The other point outside the plane and the Z straight line are determined as the Z axis direction, the Y direction is determined according to the right-hand rule, and finally the space coordinate system of the test piece and the template piece is determined, and the conversion relationship between the teaching template and the test piece is determined.

[0090] In one example, after obtaining the conversion relationship between the teaching template and the test piece, the method further includes:

[0091] Loop iteration of other mark points until the error value of the conversion relationship between the teaching template and the test piece is less than the threshold value, and then output the target conversion relationship.

[0092] In one example, the threshold value can be a preset value, and the threshold value has different values in different scenarios. By iteratively converting the error value of the relationship until the error value is less than the threshold value, the target conversion relationship is output, which is the optimal conversion relationship of the teaching template and the test piece.

[0093] In one example, after the target conversion relationship is output, the method further includes:

[0094] Based on the target conversion relationship and the first position of the at least one bolt on the teaching template, the third position of the at least one bolt on the test piece is determined.

[0095] In one example, the first position of the at least one bolt on the teaching template is known, and the target conversion relationship is used to convert the bolt position on the teaching template to the coordinate system of the test piece battery pack, thereby obtaining the third position of the at least one bolt on the test piece, which is very important for subsequent assembly and detection work.

[0096] In one example, during the actual tightening operation, the position of the located tightening gun head is calculated with the third position of the at least one bolt on the test piece by Euclidean distance, and if the distance value is less than the set distance threshold value, the current tightening bolt ID number is determined as the ID number recorded by the matching bolt on the teaching template, the actual operation sequence of the multiple tightening guns is controlled according to the process sequence requirement, and finally the recognized tightening bolt ID number is compared with the tightening ID number sequence specified by the business, and the tightening sequence result and the missed tightening prompt are given in real time.

[0097] The embodiment of the present disclosure provides a method for determining the position of a bolt on a test piece, which comprises: calculating the feature contour of at least one feature point of the real-time position of a tracking target in a third image, fitting and calculating the center coordinate information according to the feature contour of the at least one feature point, and correcting the center coordinate information based on the multiple circle intersection ratio to obtain target center coordinate information. According to the target center coordinate information and the three-dimensional coordinate information of the calibration target feature point, the tracking target pose information is determined, the first position of the at least one bolt on the teaching template is determined based on the coordinate conversion relationship and the tracking target pose information, the conversion relationship of the teaching template and the test piece, and the first position of the at least one bolt on the teaching template. The second position of the at least one bolt on the test piece is determined. By using the technical solution, the second position of the at least one bolt on the test piece can be accurately determined, the algorithm detection accuracy is high, the stability is strong, the positioning accuracy is high, and the detection result can be counted and traced.

[0098] Figure 4 is a structural schematic diagram of a test piece bolt position determination device provided by the embodiment of the present disclosure, and the test piece bolt position determination device can be understood as the electronic device described above or part of the functional modules in the electronic device. As shown inFigure 4 As shown, the bolt position determination device 40 on the test piece includes:

[0099] The first acquisition module 401 is configured to acquire a first image of a tracking target and a second image of a calibration target, and determine a coordinate conversion relationship based on the first image and the second image; wherein the coordinate conversion relationship is a spatial coordinate conversion relationship from the tracking target to the tightening gun head.

[0100] The second acquisition module 402 is configured to acquire a third image of the tightening gun head placed on at least one bolt of the tightening component, and determine a first position of at least one bolt on the teaching template based on the coordinate conversion relationship and a real-time position of the tracking target in the third image.

[0101] The first determination module 403 is configured to determine a second position of at least one bolt on the test piece according to the conversion relationship of the teaching template and the test piece, and the first position of at least one bolt on the teaching template.

[0102] In one example, the second acquisition module 402 is configured to:

[0103] Calculate a feature contour of at least one feature point of the real-time position of the tracking target in the third image;

[0104] Fit and calculate the center coordinate information according to the feature contour of the at least one feature point, and correct the center coordinate information based on the multiple circle cross ratio to obtain target center coordinate information;

[0105] Determine tracking target pose information according to the target center coordinate information and three-dimensional coordinate information of the calibration target model feature point; wherein the tracking target pose information is used to represent the coordinate information of the tracking target in the camera coordinate system;

[0106] Determine the first position of at least one bolt on the teaching template based on the coordinate conversion relationship and the tracking target pose information.

[0107] In one example, the second acquisition module 402 is configured to:

[0108] Determine the tracking target pose information based on the target center coordinate information, the three-dimensional coordinate information of the calibration target model feature point, and a preset mapping relationship; wherein the preset mapping relationship is a relationship between the target center coordinate information, the three-dimensional coordinate information of the calibration target model feature point, an internal parameter matrix of the camera, and a distance value between the target center coordinate information and the camera.

[0109] In one example, the determination of the conversion relationship of the teaching template and the test piece includes:

[0110] Position point inspection is performed on a preset number of marker points on the test piece by the tightening gun head to obtain a point inspection position;

[0111] The point inspection position is matched with the teaching mark point on the teaching template in space pose to obtain a conversion relationship of the teaching template and the test piece.

[0112] In one example, after obtaining the conversion relationship of the teaching template and the test piece, the bolt position determination apparatus 40 further includes:

[0113] The output module 404 is configured to: iteratively loop other mark points until the error value of the conversion relationship of the teaching template and the test piece is less than the threshold value, and output the target conversion relationship.

[0114] In one example, after outputting the target conversion relationship, the bolt position determination apparatus 40 further includes:

[0115] The second determination module 405 is configured to determine the third position of the at least one bolt on the test piece based on the target conversion relationship and the first position of the at least one bolt on the teaching template.

[0116] The apparatus provided in the embodiment can execute the method of any of the above embodiments, and has similar execution modes and beneficial effects, which will not be described here again.

[0117] The embodiment of the present disclosure further provides an electronic device, which includes a memory, and the memory stores a computer program; and a processor, which is configured to execute the computer program, and when the computer program is executed by the processor, the method of any of the above embodiments can be implemented.

[0118] The example, Figure 5 is a structural schematic diagram of an electronic device in the embodiment of the present disclosure. The following specifically refers to Figure 5 which shows a structural schematic diagram suitable for implementing the electronic device 1000 in the embodiment of the present disclosure. The electronic device 1000 in the embodiment of the present disclosure can include but is not limited to a mobile terminal such as a mobile phone, a notebook computer, a digital broadcast receiver, a PDA (Personal Digital Assistant), a PAD (Tablet Personal Computer), a PMP (Portable Multimedia Player), a vehicle terminal (such as a vehicle navigation terminal), and the like, and a fixed terminal such as a digital TV, a desktop computer, and the like. Figure 5 The electronic device shown is only an example, and should not bring any limitation to the functions and use range of the embodiment of the present disclosure.

[0119] As Figure 5As shown, the electronic device 1000 can include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 1001 that can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 1002 or loaded into a random access memory (RAM) 1003 from a storage device 1008. Various programs and data required for the operation of the electronic device 1000 are also stored in the RAM 1003. The processing device 1001, the ROM 1002, and the RAM 1003 are connected to each other through a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0120] In general, the following devices can be connected to the I / O interface 1005: input devices 1006 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 1007 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1008 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1009. The communication devices 1009 can allow the electronic device 1000 to communicate wirelessly or wired with other devices to exchange data. Although Figure 5 The electronic device 1000 is shown with various devices, but it should be understood that not all of the shown devices are required to be implemented or present. More or fewer devices can alternatively be implemented or present.

[0121] In particular, the processes described above with reference to the flowcharts can be implemented as a computer software program according to embodiments of the present disclosure. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods illustrated by the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network through the communication devices 1009, or installed from the storage devices 1008, or installed from the ROM 1002. When the computer program is executed by the processing device 1001, the above-described functions defined in the methods of embodiments of the present disclosure are performed.

[0122] It should be noted that the computer-readable medium described above can be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any suitable combination of the above. More specific examples of the computer-readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program used by or in connection with an instruction execution system, apparatus or device. In the present disclosure, the computer-readable signal medium can include a data signal propagated in baseband or propagated as a carrier wave in a propagated data signal, in which the computer-readable program code is contained. Such a propagated data signal can take many forms, including but not limited to, an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer-readable signal medium can also be any computer-readable medium that can send, propagate or transfer the program for use by or in connection with the instruction execution system, apparatus or device. The program code contained in the computer-readable medium can be transmitted by any suitable medium, including but not limited to, wire, cable, RF (radio frequency), etc., or any suitable combination of the above.

[0123] In some embodiments, the client, server, or both can communicate using any current known or future developed network protocol, such as HTTP (HyperText Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks ("LAN"), wide area networks ("WAN"), the Internet, and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any current known or future developed networks.

[0124] The computer-readable medium described above can be included in the electronic device described above; or can exist separately from the electronic device, and is not assembled into the electronic device.

[0125] The computer readable medium described above carries one or more programs, when the one or more programs are executed by the electronic device, cause the electronic device to: acquire a first image of a tracking target and a second image of a calibration target, determine a coordinate conversion relationship based on the first image and the second image; wherein the coordinate conversion relationship is a spatial coordinate conversion relationship of the tracking target to a tightening gun head; acquire a third image of the tracking target with the tightening gun head placed on at least one bolt of a tightening component, determine a first position of the at least one bolt on a teaching template based on the coordinate conversion relationship and a real-time position of the tracking target in the third image; and determine a second position of the at least one bolt on a test piece according to a conversion relationship of the teaching template and the test piece and the first position of the at least one bolt on the teaching template.

[0126] Computer program code for carrying out operations of the present disclosure can be written in any of one or more programming languages, including object oriented programming languages such as Java, Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0127] The flow diagrams and the block diagrams in the drawings are illustrations of architectures, functionalities, and operations of possible implementations of systems, methods, and computer program products according to various embodiments of present disclosure. In this regard, each block in the flow diagrams or block diagrams can represent a module, a procedure, or a portion of code, which comprises one or more executable instructions for implementing the specific logical functions specified for the block. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may

[0128] The units described in the embodiments of the present disclosure can be implemented by means of software, or by hardware. In some cases, the names of the units do not constitute a limitation on the units themselves.

[0129] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include Field-programmable Gate Arrays (FPGAs), Application-specific Integrated Circuits (ASICs), Application-specific Standard Products (ASSPs), System-on-a-chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), etc.

[0130] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0131] The embodiments of the present disclosure also provide a computer-readable storage medium, the storage medium stores a computer program, when the computer program is executed by a processor, the method of any of the above-mentioned embodiments can be implemented, and the execution manner and beneficial effects are similar, which will not be described here.

[0132] It should be noted that, in this document, relational terms such as“first” and“second”, and the like, are used solely to distinguish one entity or action from another entity or action, without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms“comprises”,“comprising”, or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without more limitations, an element defined by the phrase“comprising a” does not exclude the existence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0133] The foregoing is merely illustrative of the various implementations of the present disclosure and the general principles thereof. Numerous modifications can be made to these illustrations, and equivalents can be substituted therefor, without departing from the scope of the present disclosure. The specific embodiments commensurate with the specific application are intended to be illustrative only and not limiting of the scope of the application as set forth in the following claims.

Claims

1. A method of determining the position of a bolt on a test piece, characterized by, The method comprises: acquiring a first image of a tracking target and a second image of a calibration target, determining a coordinate conversion relationship based on the first image and the second image; wherein the coordinate conversion relationship is a spatial coordinate conversion relationship of the tracking target to a tightening gun head; acquiring a third image of the tracking target with the tightening gun head placed on at least one bolt of a tightening component, determining a first position of at least one bolt on a teaching template based on the coordinate conversion relationship and a real-time position of the tracking target in the third image; determining a second position of at least one bolt on a test piece according to a conversion relationship of the teaching template and the test piece and the first position of at least one bolt on the teaching template.

2. The method of claim 1, wherein, The determination of the first position of at least one bolt on the teaching template based on the coordinate conversion relationship and the real-time position of the tracking target in the third image comprises: calculating a feature contour of at least one feature point of the real-time position of the tracking target in the third image; fitting and calculating a circle center coordinate information based on the feature contour of the at least one feature point, and correcting the circle center coordinate information based on a multi-circle cross ratio to obtain target circle center coordinate information; determining tracking target pose information according to the target circle center coordinate information and three-dimensional coordinate information of calibration target model feature points; wherein the tracking target pose information is used to represent coordinate information of the tracking target in a camera coordinate system; determining the first position of at least one bolt on the teaching template based on the coordinate conversion relationship and the tracking target pose information.

3. The method of claim 2, wherein, The determination of the tracking target pose information according to the target circle center coordinate information and the three-dimensional coordinate information of calibration target model feature points comprises: determining the tracking target pose information based on the target circle center coordinate information, the three-dimensional coordinate information of calibration target model feature points and a preset mapping relationship; wherein the preset mapping relationship is a relationship between the target circle center coordinate information, the three-dimensional coordinate information of calibration target model feature points, a camera intrinsic parameter matrix and a distance value of the target circle center coordinate information from the camera.

4. The method of claim 1, wherein, The determination of the conversion relationship of the teaching template and the test piece comprises: performing position point inspection on a preset number of marking points on the test piece by the tightening gun head to obtain a point inspection position; performing spatial pose matching on the point inspection position and teaching marking points on the teaching template to obtain the conversion relationship of the teaching template and the test piece.

5. The method of claim 4, wherein, After obtaining the conversion relationship of the teaching template and the test piece, the method further comprises: iterating other marking points until an error value of the conversion relationship of the teaching template and the test piece is less than a threshold value, and then outputting a target conversion relationship.

6. The method of claim 5, wherein, After outputting the target conversion relationship, the method further comprises: determining a third position of at least one bolt on the test piece based on the target conversion relationship and the first position of at least one bolt on the teaching template.

7. An apparatus for determining the position of a bolt on a test piece, characterized by The device comprises: The first acquisition module is configured to acquire a first image of a tracking target and a second image of a calibration target, and determine a coordinate conversion relationship based on the first image and the second image; wherein the coordinate conversion relationship is a spatial coordinate conversion relationship from the tracking target to a tightening gun head; The second acquisition module is configured to acquire a third image in which the tightening gun head of the tracking target is placed on at least one bolt of a tightening component, and determine a first position of at least one bolt on the teaching template based on the coordinate conversion relationship and a real-time position of the tracking target in the third image. The first determination module is configured to determine a second position of at least one bolt on the test piece according to a conversion relationship between the teaching template and the test piece and the first position of at least one bolt on the teaching template.

8. The apparatus of claim 7, wherein, The second acquisition module is configured to: Calculate a feature contour of at least one feature point of the real-time position of the tracking target in the third image; Fit and calculate a circle center coordinate information according to the feature contour of the at least one feature point, and correct the circle center coordinate information based on a multiple circle cross ratio to obtain target circle center coordinate information; Determine tracking target pose information according to the target circle center coordinate information and three-dimensional coordinate information of a calibration target model feature point; wherein the tracking target pose information is used to represent coordinate information of the tracking target in a camera coordinate system; Determine the first position of at least one bolt on the teaching template based on the coordinate conversion relationship and the tracking target pose information.

9. An electronic device, comprising: It includes: A processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the method of any one of claims 1-6.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, and when the computer program is executed by the processor, the method of any one of claims 1-6 is implemented.

Images