Multi-sensor assembly line-based global camera extrinsic parameter coordinate marking method

Abstract

The invention discloses a multi-sensor assembly line-based global camera external parameter coordinate marking method, which comprises the following steps of: firstly, generating a square marker by using an augmented reality marker generation algorithm; secondly, building an assembly line, recording sampling of motor operation time and motor operation distance by using a motor feedback signal, shooting a square calibration body by using a camera and storing the square calibration body as a camera frame, and recording time of the camera frame; then, on the basis of sampling of multiple camera frames, corresponding time, motor operation time and operation distance, the relative pose of the camera relative to the square calibration body is solved by utilizing a joint optimization algorithm based on image and motor feedback, and a coordinate time relation graph is generated; and finally, calculating the external parameters of the camera by using the relative pose, the coordinate-time relation graph and the recording time of the camera frame. The problem that traditional camera calibration lacks enough calibration information is avoided, and the calibration of the external parameters of the camera is effectively improved in combination with an optimization algorithm.

Description

Technical Field

[0001] The present invention relates to the field of multi-view camera marking and measurement technology, and in particular to a global camera extrinsic coordinate marking method based on a multi-sensor pipeline. Background Art

[0002] In modern industry, the requirements for digital data collection and status detection of individual products have increased, and vision provides rich information about the status of products in the production line. The detection of products with different postures is conducive to the detection of the spatial status of products, which also facilitates the operation of the robotic arms in the assembly line for products with different postures. Before determining the posture of the target object, the absolute posture of the camera needs to be determined. This is conducive to directly calculating the absolute posture of the product in the production line.

[0003] A multi-view camera refers to a camera that collects images in the same production line. The functions of the camera include image acquisition, quality inspection, process control, and improvement of image acquisition accuracy. The existing multi-camera tagging method uses single-camera tagging, tag field establishment, data acquisition, feature matching, and parameter optimization.

[0004] At present, the external parameter marking is usually determined by the position of the reference point or the target of the specified object relative to the world coordinate system, which makes the multi-station camera marking world more complicated, because each time the marking position is collected, the distance relative to the world coordinate system must be recalculated. Therefore, this method integrates the distance feedback mechanism of the conveyor belt into the camera marking, and can systematically solve the unified external parameters of the camera group. Summary of the invention

[0005] In view of the above problems, the present invention provides a global camera extrinsic coordinate marking method based on a multi-sensor assembly line, which is used to unify the global coordinate marking of multiple sets of camera extrinsic coordinates in the production line. The cameras at multiple stations in the production line are transmitted once through the marking and motion module to determine the relationship between their rotation and displacement, synchronize the feedback information of the synchronous motor with the timestamp of the captured image, square marker detection and relative posture recovery. The multi-sensor of the present invention uses camera and motor feedback to jointly optimize multiple positions.

[0006] A method for marking global camera extrinsic coordinates based on a multi-sensor pipeline comprises the following steps:

[0007] Step 1: Use the augmented reality marker Aruco Marker generation algorithm to generate 5 recognizable binary square markers, and splice the markers into a cube, namely the square marker body.

[0008] The square marker body is a solid model based on a cube. The augmented reality markers Aruco Marker are located on the five faces of the cube, and each marker occupies one face alone. The extra face without the augmented reality marker is the placement face. The augmented reality marker includes an n×n dimensional matrix.

[0009] Step 2: Build an assembly line, use the motor feedback signal to record the sampling of the motor running time and the motor running distance, use the camera to shoot the square marker and store it as a camera frame, and record the time of the camera frame.

[0010] Step 3: Based on multiple camera frames, the time corresponding to multiple camera frames, and the sampling of motor running time and running distance, the relative position of the camera relative to the square marker is solved using a joint optimization algorithm based on image and motor feedback; the coordinate-time relationship diagram is obtained using the sampling of motor running time and running distance.

[0011] Step 4: Using the relative pose, coordinate-time relationship diagram, and the recording time of the camera frame, calculate the pose of the camera relative to the pipeline coordinates, i.e., the camera's external parameters.

[0012] The following is a brief description of the above steps:

[0013] The square marker body is a solid model based on a cube. Aruco Marker (augmented reality marker) is located on the five faces of the cube, and each marker occupies one face alone, and the extra face without Aruco Marker (augmented reality marker) is the placement face; the features of Aruco Marker (augmented reality marker) include that it is an n×n dimensional matrix (where n represents the dimension of the matrix, and different n can be selected to control the fault tolerance of generating Aruco Marker); the main algorithm of Aruco Marker (augmented reality marker) generation algorithm is as follows:

[0014] Select the Aruco dictionary. The Aruco dictionary value consists of a unique binary code, each code corresponds to a specific Aruco Marker.

[0015] Using the given Aruco (augmented reality) dictionary, select the ID (label) of the marker, and the marker generated by the sequence is unique.

[0016] Using the ID (serial number) of the tag, the corresponding binary coding matrix is ​​generated.

[0017] A border is added to the binary coding matrix to add an outline to generate a mark recognized by the auxiliary detection algorithm to form a mark with an outline.

[0018] Using the generated marker with outline, an image of the marker is rendered through an image generation algorithm, namely the Aruco Marker (augmented reality marker) as described in step 3.

[0019] Further, the step 2 is specifically as follows:

[0020] Build an assembly line. The assembly line drive motor has a motor feedback function. Build a multi-station camera on the assembly line. Adjust the aperture and focal length of the camera so that the camera can capture clear images of the assembly line.

[0021] The starting point of the assembly line is the starting point of the global external parameter of the required mark, and the spatial position of the conveyor belt is not limited; the square marker is placed in such a way that the side without the Aruco Marker (augmented reality marker) is placed on the conveyor belt, and the other side faces the running direction of the conveyor belt. The assembly line starts slowly, reaches a certain speed with a low acceleration, and then maintains a constant speed. The motor feedback signal records the motor running time and the corresponding motor running distance. When the conveyor belt is running, the feedback with the rotary transformer records the running time of the motor and the running distance of the motor; the camera shoots the square marker. When the marker on the conveyor belt reaches the appropriate position, the camera shoots the transmission data and the host computer records the shooting time.

[0022] The assembly line drive motor has a motor feedback function, specifically: the assembly line drive motor has a position feedback function, which is completed by using a rotary transformer, and uses the electromagnetic induction principle to convert the rotation position into an electrical signal to obtain the motor position signal.

[0023] Beneficial effects of the present invention:

[0024] High-precision extrinsic parameter marking: It avoids the problem of insufficient marking information in traditional camera marking and combines it with optimization algorithms to effectively improve the marking of camera extrinsic parameters.

[0025] New marker: The use of a new type of square marker can effectively increase the amount of information on the marker, as the conventional marker plate has unclear imaging and causes marking errors.

[0026] Fast marking: The algorithm does not need to manually mark square markers and can quickly implement camera extrinsic marking.

[0027] Wide application scenarios: It is applicable to various types of production lines or the placement of cameras at various workstations, and the marking of external parameters can be completed without adjusting unnecessary parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0029] Figure 1 A flowchart of an implementation method of a multi-sensor camera extrinsic parameter marking method provided in an embodiment of the present disclosure;

[0030] Figure 2 A schematic diagram of a scene in which multiple cameras of a production line shoot a square marker according to an embodiment of the present disclosure;

[0031] Figure 3 A schematic diagram of a square marker style of an example provided in an embodiment of the present disclosure;

[0032] Figure 4 A schematic diagram of relative coordinates of a final marking result of a square marking body provided in an embodiment of the present disclosure

[0033] Figure 5 A schematic diagram of the generated augmented reality marker Aruco Marker;

[0034] Figure 6 Schematic diagram of detecting augmented reality marker Aruco Marker at different angles. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solutions and advantages of the present disclosure clearer, the present disclosure will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0036] In the embodiments of the present disclosure, the term "and / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent three situations: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0037] The application scenarios described in the embodiments of the present disclosure are intended to more clearly illustrate the technical solutions of the embodiments of the present disclosure, and do not constitute a limitation on the technical solutions provided by the embodiments of the present disclosure. It is known to those skilled in the art that with the emergence of new application scenarios, the technical solutions provided by the embodiments of the present disclosure are also applicable to similar technical problems. In the description of the present disclosure, unless otherwise specified, the meaning of "multiple" is two or more.

[0038] Before introducing the multi-camera extrinsic parameter labeling method provided by the embodiment of the present disclosure, in order to facilitate understanding, the technical background of the embodiment of the present disclosure is first introduced in detail below.

[0039] In industrial robots, vision systems are often used to guide robots to perform complex tasks such as part assembly, welding, and handling. To ensure that the robot can accurately identify and locate the work object, the camera needs to be extrinsically marked to determine the relative position and orientation of the camera and the robot. This process ensures that the robot can accurately obtain the spatial position of the target object when operating, thereby achieving high-precision operations. Global camera extrinsic marking also plays a key role in automated quality inspection. Cameras are usually installed in fixed positions to capture images of products and perform defect detection. With accurate extrinsic marking, the position and orientation of the camera on the production line can be determined, ensuring that the image processing algorithm can correctly analyze various parts of the product. This is especially important in industries with high precision requirements such as electronic products and automotive parts. In some complex industrial applications, a single camera may not provide sufficient field of view and accuracy, so multiple cameras need to work together. Global camera extrinsic marking can determine the relative position and orientation between each camera, so that multi-camera systems can work together. This application is very common in 3D measurement, object reconstruction, and complex assembly processes. In automated production lines, cameras are used to monitor and control the production process. Through global camera extrinsic markers, it can be ensured that the camera's field of view covers the entire production area and can accurately capture the movement and changes of products. This is essential for achieving efficient operation of automated production lines and stable control of product quality. AR and VR systems are becoming more and more popular in industrial training, maintenance and operation guidance. Global camera extrinsic markers can help these systems accurately overlay virtual information onto the real world, provide workers with more intuitive and efficient operation guidance, reduce error rates, and improve production efficiency. Global camera extrinsic markers are an indispensable part of industrial automation and intelligent manufacturing. It provides a precise spatial positioning basis for various visual systems, ensuring efficient, accurate and reliable production processes.

[0040] The following is a detailed description of the principles of the steps in the invention content and the corresponding theoretical support:

[0041] The driving motor of the assembly line is characterized by having a position feedback function, and a rotary transformer is used to obtain the position signal of the motor. The rotary transformer is a high-precision sensor used to measure the rotation angle and speed. It is an analog angle encoding device that converts the rotation position into an electrical signal using the principle of electromagnetic induction.

[0042] The characteristic of the multi-station camera is that the camera is placed at any position of the observable production line, such as Figure 2 As shown in the figure, two cameras (the two planes are the imaging planes of the cameras, and the cameras are not drawn in the figure) are used to shoot a schematic diagram of the same conveyor belt, and the direction of the conveyor belt is specified. The square on the conveyor belt is the square marker body described in step 2. The schematic diagram of the square marker body is as follows Figure 3 shown.

[0043] like Figure 3 As shown in the figure, the square marker body is a solid model based on a cube. Aruco Markers are located on the five faces of the cube, and each Aruco Marker occupies one face alone. The extra face without Aruco Marker is Figure 2 The placement surface shown is placed on the conveyor belt; the characteristics of ArucoMarker (augmented reality marker) include that it is an n×n dimensional matrix (where n represents the dimension of the matrix, and different n can be selected to control the fault tolerance of generating Aruco Marker).

[0044] like Figure 1 As shown, the square marker is aligned with the desired coordinates and fixed on the conveyor belt by a fixture. The zero point is the geometric center of the square marker, and the square marker is placed at the original coordinate point of the global coordinates, which is also the origin position of the final global external parameter obtained by subsequent marking. The square marker is placed by placing the side without the Aruco Marker (augmented reality marker) on the conveyor belt and facing the direction of the conveyor belt.

[0045] The characteristic of the assembly line operation is that the assembly line starts slowly, reaches a certain speed with a low acceleration, and then maintains a constant speed. Through program control, the conveyor belt and the camera group whose posture needs to be detected are started at the same time, and the collected data information is superimposed in the form of timestamps. In order to optimize the storage space of the equipment, the camera can only record and save the pictures of the complete square marker body and the corresponding timestamps, and collect 30 frames per second for storage. The expression P is used for the storage of pictures. A (t) is recorded, where t∈T camera_record , T camera_record is the timestamp recorded by the current camera, and the obtained P A(t) is the image information of the current timestamp, A represents the camera at position A. For the motor feedback signal, it is only necessary to calculate the odometer of the conveyor belt, that is, to record the total distance transported by the conveyor belt at different times as X(t), where t is the relative time sampling, t∈[0,t max ], t max The final time when the conveyor belt transmits the square marker, the collected image is combined with the motor feedback signal and stored for a time t m , obtain the image frame P of the corresponding camera A at the current moment A (t m ) can obtain the distance odometer X(t m ).

[0046] For image frames that are different from the motor recording time, the time that is not sampled by the motor can be approximated by interpolation. Since the signal frequency of the motor feedback is high, the accuracy of the result is very accurate. If you need to obtain the distance at a specific time, define the time as t sp , we can find that there is a time less than t in the sampling time sp Maximum time sampling t sp_lm Corresponding distance X sp_lm , and there is a time greater than t sp Minimum time sampling t sp_ml Corresponding distance X sp_ml , we can calculate the specific time t sp Corresponding distance X sp :

[0047]

[0048] The above can obtain the sampling of the motor running time and the corresponding motor running distance, as well as the camera frame taken by the camera, and record the time of the camera frame. In practice, the storage method is to record the better camera frame taken, and directly obtain the corresponding distance of the shooting time through the above formula and save them in combination.

[0049] The characteristic of the coordinate time relationship diagram is the sampling relationship between the distance transmitted in the coordinate direction corresponding to the running direction of the transmission axis and time. It is a series of sampling points, and then the distance of the unsampled time point is calculated by interpolation.

[0050] Based on multiple camera frames, the time corresponding to multiple camera frames, and the sampling of motor running time and running distance, the relative position of the camera relative to the square marker is solved using a joint optimization algorithm based on image and motor feedback; the coordinate-time relationship diagram is obtained by sampling the motor running time and running distance.

[0051] The joint optimization algorithm based on image and motor feedback is described as follows:

[0052] 1. For multiple camera frames, use the Aruco Marker detection algorithm and the PnP algorithm to find the displacement vector of the center point on the square marker surface in the camera coordinate system, recorded as t1, t2, t3, the projection vector of the normal vector of each face of the square marker in the camera frame, the scale information on the marker, the marker surface coordinate axis relative to the x-axis, y-axis and z-axis of the camera coordinate system, and the translation vector of the marker surface center relative to the camera optical center. The translation vectors represent the three faces of the square marker respectively.

[0053] 2. Using the translation vector of the center of the marker surface relative to the optical center of the camera, calculate the displacement t of the geometric center of the square marker relative to the optical center of the camera n .

[0054] Specifically, we use the translation vectors t1, t2, t3 of the center of the marker surface relative to the optical center of the camera to sum the averages, and then use similar triangles to find out: (The vector of the extended mean sum is the squared marker body displacement).

[0055] 3. Determine the center position: Use the projection vector corresponding to the center point of the marker surface in the camera frame to calculate the projection coordinates of the geometric center of the square marker body on the camera frame.

[0056] Specifically, the geometric center O is formed by the reverse extension line of the normal vector corresponding to the axial center point. Due to measurement errors, the normal vector is translated to the center point on the surface of the square marker, and the normal vector is extended in the reverse direction to form a closed triangle. The geometric center of the triangle is calculated, and the projection of the geometric center of the square marker on the camera frame is obtained.

[0057] 4. Calculate the direction of a single axis on the image: For multiple camera frames, use the projection coordinates of the geometric center of the square marker on the camera frame. There will be multiple points on the image. Use RANSAC (random sampling consensus) to obtain a fitted projection line and extract its relative displacement direction under the camera perspective.

[0058] Specifically: calculate the direction of the x-axis in the image (RANSAC method), for multiple sampling images, calculate the position of the geometric center of each square marker projected on the camera frame, and use the RANSAC (random sampling consensus) method to fit a line, which is the projection line represented by the projection of the x-axis line in the image; through the direction of the single axis on the image, obtain the relative displacement direction under the camera perspective.

[0059] 5. Calculate the x-axis position of the square marker in the camera coordinate system (operating displacement):

[0060] According to the sampling time of the camera frame, the coordinate time relationship diagram is used to calculate the displacement distance of the square marker in three-dimensional space (the direction of the conveyor belt is the x-axis of the marker).

[0061] The position of the geometric center of the marker on different camera frames is used to calculate the travel distance of the motor sampling at the corresponding time through interpolation, and the pixel distance translated on the camera frame is calculated.

[0062] The pixel distance and the scale information on the square marker are used to calculate the displacement of the restored square marker in the camera frame plane.

[0063] The displacement distance of the square marker in the three-dimensional space is used to restore the displacement of the square marker in the image plane, and the angle between the x-axis of the square marker and the z-axis of the camera is calculated.

[0064] The position of the square marker on the x-axis in the camera coordinate system is calculated using the angle of the z-axis and the relative displacement direction under the image viewing angle.

[0065] Specific process: Using the pixel distance in the image and the scale information on the square marker, different camera frames have different scale information. The scale information not obtained by the camera frame in the middle needs to be obtained through interpolation. Because the displacement of the square marker in the image changes linearly with respect to the scale information of the camera, the displacement in the image is a scalar, so the change of scale information is linear. The displacement of the square marker in the camera frame plane can be calculated. The displacement of the restored square marker is different from the displacement of the conveyor belt because the displacement in the scale direction is lost due to the projection transformation and is not calculated.

[0066] Using the displacement distance of the square marker in three-dimensional space and restoring the displacement of the square marker in the camera frame plane, a right triangle model is constructed with the displacement distance as the hypotenuse and the restored displacement of the square marker as an adjacent side of the right triangle. The corresponding angle of this adjacent side is the angle of the square marker relative to the camera z-axis (the direction perpendicular to the camera imaging plane).

[0067] The z-axis angle corresponds to the projection line of the x-axis on the image. The angles of the projection line relative to the horizontal and vertical axes of the image correspond to the x-axis and y-axis of the camera coordinate system respectively. The position of the square marker on the x-axis in the camera coordinate system is calculated. The x-axis is the displacement direction of the square marker in three-dimensional space.

[0068] 6. Solve the rotation matrix R of the square marker relative to the camera n : Calculate the rotation matrix of the square marker relative to the camera coordinate system based on the x-axis position of the existing square marker in the camera coordinate system and the rotation axis of the square marker surface of multiple camera frames relative to the camera coordinate system.

[0069] Specifically, the constraint optimization algorithm is performed on the existing x-axis direction. The coordinate of the existing direction x in the camera coordinate system is x. The goal is to determine the new y-axis and z-axis directions so that they are as close as possible to the directions of multiple given axes. This can be achieved by minimizing the angles or other error metrics between all given axes and the new axes. The calculated y-axis and z-axis vectors in the camera coordinate system are recorded as y1, y2, …, y n and z1,z2,…,z n The orientation of the y-axis and z-axis of the square marker is represented by y and z, then the cost function of the objective function is to minimize the dot product difference between all given axes and the new axis, that is:

[0070]

[0071] The following conditions are met:

[0072] y·x=0

[0073] z·x=0

[0074] y·z=0

[0075] ‖y‖=1

[0076] ‖z‖=1

[0077] This objective function jointly optimizes the axes in each direction. i To maximize its projection on the y-axis and minimize its projection on the z-axis, we can optimize and solve the optimal y and z-axis vectors. Similarly, the coordinate system has certain orthogonality and normalization conditions, so we need to have the constraints of the above formula to satisfy them. Through the numerical optimization method, we can solve the y-axis and z-axis to get the orientation y′ and z′ of the square marker in the camera coordinate system, then we can get the rotation matrix R of the square marker relative to the camera. n =[x,y′,z′].

[0078] According to the above solved square marker body relative to the camera pose matrix, we can get the position matrix of the square marker body at a certain time t s The relative position of the camera relative to the square marker

[0079] Because the square marker does not rotate in the camera coordinate system (the square marker does not rotate in the scene, and the camera does not rotate), the same rotation matrix can be used at each time, but different displacements are used for different positions. For the joint optimization method based on image and motor feedback, if the camera can only capture two ArucoMarkers on the surface of the marker board, it also has a certain marking ability, but its fusion information is less and there will be large errors. Ignoring the case where the camera only captures one Aruco Marker on the surface of the marker board, the marking result will degenerate into a planar mark, which is inconsistent with the innovation direction of this patent.

[0080] The square markers at different positions on the conveyor belt have different translation vectors. They have the same y-axis and z-axis, but the detection results at different angles will have errors. The algorithm is to reduce this error. If the scale information is not obtained through the interpolation method, its accuracy will be greatly lost. After interpolation, the result will be more accurate. The time relationship diagram obtained by comparing the time corresponding to the camera frame is used to calculate the position of the square marker at the time recorded by the corresponding camera frame. Combined with the relative posture of the camera relative to the square marker obtained by describing the description, the global external parameters of a single camera are calculated using the joint optimization method based on image and motor feedback.

[0081] By combining the image with the position signal fed back by the motor, the pose of the camera relative to the square marker is optimized and calculated, and the pose relationship of the square marker relative to the camera can be directly calculated.

[0082] The algorithm based on the joint optimization method of image and motor feedback obtains the relative position T of the camera relative to the square marker c_to_g ; For a certain moment t s , use the coordinate time relationship diagram to find the transmission distance t of the square marker block relative to the pipeline g_to_w ; Since the direction of the x-axis of the square marker is the same as the direction of the conveyor belt and does not rotate, the rotation matrix of the x-axis of the square marker relative to the world coordinate system is I (unit matrix, indicating no rotation); then the transformation matrix of the square marker relative to the world coordinate system can be obtained: Then we can calculate the pose matrix of the camera relative to the world coordinate system, that is, the external parameters of the camera:

[0083]

[0084] It is the camera external parameter.

[0085] The present invention does not involve a transmission method with a rotating nature, because the odometer results are different at different lateral positions of the conveyor belt (the position perpendicular to the transmission direction), which brings uncertainty to the measurement. Only considering one-way straight-line transmission, and the conveyor belt can be transmitted at a non-uniform speed, considering the use of odometers for measurement, and considering that the world coordinates here use the position of the marker zero initialization, the displacement vector tvec of the transmission relative to the global coordinates can be obtained g_to_w , the rotation matrix is ​​the identity matrix, such as Figure 4 shown.

[0086] Experimental proof:

[0087] like Figure 5 The following is the generated Aruco Marker (augmented reality marker). Figure 6 As shown, it can detect Aruco Marker at different angles, identify Aruco Marker with different IDs, and mark the position coordinates of Aruco Marker in space (actually the coordinates in camera space). The blocks made by it can prove that it has the ability to mark in camera space.

[0088] Obviously, those skilled in the art can make various changes and modifications to the present disclosure without departing from the spirit and scope of the present disclosure. Thus, if these modifications and variations of the present disclosure fall within the scope of the claims of the present disclosure and their equivalents, the present disclosure is also intended to include these modifications and variations.

[0089] Matters not covered by the present invention are known technologies.

[0090] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable people familiar with the technology to understand the content of the present invention and implement it accordingly, and they cannot be used to limit the protection scope of the present invention. Any equivalent changes or modifications made according to the spirit of the present invention should be included in the protection scope of the present invention.

Claims

1. A global camera extrinsic coordinate marking method based on a multi-sensor pipeline, characterized in that: The following steps are involved: Step 1: Use the augmented reality marker Aruco Marker generation algorithm to generate 5 recognizable binary square markers, and splice the markers into a cube, namely the square marker body; Step 2: Build an assembly line, use the motor feedback signal to record the sampling of the motor running time and the motor running distance, use the camera to shoot the square calibration volume and store it as a camera frame, and record the time of the camera frame; Step 3: Based on the sampling of multiple camera frames, the time corresponding to the multiple camera frames, the motor running time and the running distance, the relative position and posture of the camera relative to the square calibration body are solved by using the joint optimization algorithm based on image and motor feedback; By sampling the motor running time and running distance, a coordinate-time relationship diagram is obtained; Step 4: Using the relative pose, coordinate-time relationship diagram, and the recording time of the camera frame, calculate the pose of the camera relative to the pipeline coordinates, i.e., the camera's external parameters.

2. The method for marking global camera extrinsic coordinates based on a multi-sensor pipeline according to claim 1, characterized in that: The square marker body is a solid model based on a cube. The augmented reality markers Aruco Marker are located on the five faces of the cube, and each marker occupies one face alone. The extra face without the augmented reality marker is the placement face. The augmented reality marker includes an n×n dimensional matrix.

3. The method for marking global camera extrinsic coordinates based on a multi-sensor pipeline according to claim 2, characterized in that: The specific implementation process of the augmented reality marker Aruco Marker generation algorithm is as follows: S1.

1. Select the augmented reality Aruco dictionary. The Aruco dictionary value consists of a unique binary code, and each code corresponds to an augmented reality marker Aruco Marker; S1.2, using the given Aruco dictionary, select the ID of the marker, the marker generated by the serial number is unique; S1.3, using the marked ID, generating the corresponding binary coding matrix; S1.4, adding a border to the binary coding matrix, adding an outline to generate a mark recognized by the auxiliary detection algorithm, and forming a mark with an outline; S1.

5. Using the generated marker with contour, render an image of the marker through an image generation algorithm.

4. The method for marking global camera extrinsic coordinates based on a multi-sensor pipeline according to claim 3, characterized in that: The specific implementation process of step 2 is as follows: Build an assembly line. The assembly line drive motor has a motor feedback function. Build a multi-station camera on the assembly line. Adjust the aperture and focal length of the camera so that the camera can capture clear images of the assembly line. The starting point of the assembly line is the starting point of the calibrated global external parameter, and the spatial position of the conveyor belt is not restricted; the placement method of the square marker is to place the side without the augmented reality marker Aruco Marker on the conveyor belt, and one side faces the running direction of the conveyor belt; The motor feedback signal records the motor running time and the corresponding motor running distance. When the sampling conveyor belt is running, the feedback with a rotary transformer records the motor running time and the motor running distance; after the camera shoots the square marker on the conveyor belt and the marker reaches the position, the camera captures the camera frame and the host computer records the shooting time.

5. The method for marking global camera extrinsic coordinates based on a multi-sensor pipeline according to claim 4, characterized in that: The assembly line drive motor has a motor feedback function, specifically: the assembly line drive motor has a position feedback function, which is completed by using a rotary transformer, and uses the electromagnetic induction principle to convert the rotation position into an electrical signal to obtain the motor position signal.

6. The method for marking external coordinates of a global camera based on a multi-sensor pipeline according to claim 5, characterized in that: The specific implementation process of the joint optimization algorithm based on image and motor feedback is as follows: S3.

1. For multiple camera frames, use the Aruco Marker detection algorithm and the PnP algorithm to find the displacement vector of the center point on the surface of the square marker in the camera coordinate system, recorded as t1, t2, t3, the projection vector of the normal vector of each surface of the square marker in the camera frame, the scale information on the calibration body, the coordinate axis of the marker surface relative to the x-axis, y-axis and z-axis of the camera coordinate system, and the translation vector of the center of the marker surface relative to the optical center of the camera. The translation vectors represent the three surfaces of the square marker respectively. S3.

2. Use the translation vectors t1, t2, t3 of the center of the marker surface relative to the optical center of the camera to sum the average, and then use similar triangles to calculate the displacement of the geometric center of the square marker relative to the optical center of the camera. S3.3, using the projection vector corresponding to the center point of the marker surface in the camera frame, calculate the projection coordinates of the geometric center of the square marker body on the camera frame; S3.

4. For multiple camera frames, the projection coordinates of the geometric center of the square marker on the camera frame are used. There are multiple points on the image. A fitting projection line is obtained using random sampling consistency RANSAC, which is the projection line represented by the projection of the x-axis line in the image. The relative displacement direction under the camera perspective is obtained through the direction of the single axis on the image. S3.5, calculating the x-axis position of the square marker in the camera coordinate system; S6.

3. Calculate the rotation matrix R of the square calibration body relative to the camera coordinate system for the x-axis position of the existing square marker body in the camera coordinate system and the rotation axis of the square marker surface of multiple camera frames relative to the camera coordinate system. n , and obtain the relative pose of the square calibration body relative to the camera.

7. The method for marking global camera extrinsic coordinates based on a multi-sensor pipeline according to claim 6, characterized in that: The specific implementation of step S3.3 is as follows: the geometric center O formed by the reverse extension line of the normal vector corresponding to the axial center point is translated to the center point on the surface of the square marker body, and the normal vector is extended in the reverse direction to form a closed triangle. The geometric center of the triangle is calculated, and the projection of the geometric center of the square marker body on the camera frame is obtained.

8. The method for marking global camera extrinsic coordinates based on a multi-sensor pipeline according to claim 7, characterized in that: The specific implementation process of step S3.5 is as follows: According to the sampling time of the camera frame, the displacement distance of the square calibration body in the three-dimensional space is calculated using the coordinate time relationship diagram; Using the position of the geometric center of the calibration body on different camera frames, the travel distance of the motor sampling at the corresponding time is calculated by interpolation, and the pixel distance translated on the camera frame is calculated; Using the pixel distance and the scale information on the square marker, the displacement of the square marker is calculated to restore the square marker in the camera frame plane; Using the displacement distance of the square marker in the three-dimensional space, the displacement of the square marker is restored in the image plane, and the angle between the x-axis of the square marker and the z-axis of the camera is calculated; The position of the square marker on the x-axis in the camera coordinate system is calculated using the angle of the z-axis and the relative displacement direction under the image viewing angle.

9. The method for marking external coordinates of a global camera based on a multi-sensor pipeline according to claim 8, characterized in that: The calculation of the angle between the x-axis of the square marker and the z-axis of the camera is specifically as follows: using the displacement distance of the square marker in the three-dimensional space and restoring the displacement of the square marker in the camera frame plane, a right triangle model is constructed based on the displacement distance as the hypotenuse and the restored displacement of the square marker as an adjacent side of the right triangle, and the corresponding angle of the adjacent side is the angle between the square marker and the z-axis of the camera; The method of calculating the x-axis position of the square marker in the camera coordinate system is specifically as follows: using the z-axis angle corresponding to the projection line of the x-axis on the image in the image, the angles of the projection line relative to the horizontal and vertical axes of the image correspond to the x-axis and y-axis of the camera coordinate system respectively, and the x-axis position of the square marker in the camera coordinate system is calculated, and the x-axis is the displacement direction of the square marker in the three-dimensional space.

10. The method for marking global camera extrinsic coordinates based on a multi-sensor pipeline according to claim 9, characterized in that: The specific implementation process of step S3.6 is as follows: The constraint optimization algorithm is performed on the x-axis direction. For the existing direction x, the coordinate of the camera coordinate system is x, and the calculated y-axis and z-axis vectors in the camera coordinate system are recorded as y1, y2, ..., y n and z1,z2,…,z n ; The orientation of the y-axis and z-axis of the square calibration volume is represented by y and z. The cost function of the objective function is to minimize the dot product difference between all given axes and the new axis, that is: The following conditions are met: y·x=0; z·x=0; y·z=0 ‖y‖=1;‖z‖=1 This objective function jointly optimizes the axes in each direction. i To maximize its projection on the y-axis and minimize its projection on the z-axis, the optimal y- and z-axis vectors are optimized and solved. Through the numerical optimization method, the y-axis and z-axis obtained are the directions y′ and z′ of the square calibration body in the camera coordinate system, and the rotation matrix R of the square calibration body relative to the camera is obtained. n =[x,y′,z′]; According to the above solved square calibration body relative to the camera pose matrix, we can get s Below, the relative position of the camera relative to the square calibration volume: Represents R n The inverse matrix of T represents a matrix of all 0s and 1 represents a scalar.

11. The method for marking external coordinates of a global camera based on a multi-sensor pipeline according to claim 10, characterized in that: The specific implementation process of step 4 is as follows: The algorithm based on the joint optimization method of image and motor feedback obtains the relative pose T of the camera relative to the square calibration body. c_to_g ; For a certain moment t s , use the coordinate time relationship diagram to find the transmission distance t of the square calibration block relative to the pipeline g_to_w ; The rotation matrix of the x-axis of the square calibration body relative to the world coordinates is the unit matrix I; Get the transformation matrix of the square calibration volume relative to the world coordinate system: Calculate the camera's pose matrix relative to the world coordinate system, that is, the camera's external parameter T c_to_w :

Images