A multi-sensor pipeline-based global camera parameter coordinate marking method

Abstract

The application discloses a kind of based on multi-sensor pipeline global camera external parameter coordinate marking method, this method first utilizes augmented reality mark generation algorithm, generates square mark body.Secondly, pipeline is built, and the sampling of motor running time and the running distance of motor is recorded using motor feedback signal, square calibration body is photographed using camera and is stored as camera frame, the time of camera frame is recorded.Then based on multiple camera frames and corresponding time, motor running time and the sampling of running distance, the relative pose of camera relative to square calibration body is solved using based on image and motor feedback joint optimization algorithm, and coordinate time relationship diagram is generated.Lastly, using relative pose, coordinate time relationship diagram and the recording time of camera frame, the external parameter of camera is calculated.The application avoids the problem that traditional camera calibration lacks enough calibration information, and effectively improves the calibration of camera external parameter in combination with optimization algorithm.

Description

Technical Field

[0001] The present invention relates to the field of multi-view camera marking and measurement technology, and in particular to a global camera extrinsic coordinate marking method based on a multi-sensor pipeline. Background Art

[0002] In modern industry, the demand for digital data collection and individual product status detection is increasing. Vision provides rich information about product status within the production line. Detecting products in different postures facilitates spatial position detection, which in turn facilitates robotic manipulation of products in varying postures within the assembly line. Before determining the target object's posture, the camera's absolute posture must be determined. This facilitates direct calculation of the product's absolute position within the production line.

[0003] Multi-view cameras are used to capture images from multiple cameras within a single production line. The cameras' functions include image acquisition, quality inspection, process control, and improved image accuracy. Existing multi-camera tagging methods utilize single-camera tagging, tag field establishment, data acquisition, feature matching, and parameter optimization.

[0004] Currently, extrinsic parameters are typically determined by using a reference point or the position of a specified object relative to the world coordinate system. This complicates the process of labeling multiple cameras, as the distance relative to the world coordinate system must be recalculated each time a marker position is acquired. Therefore, this method incorporates the distance feedback mechanism of a conveyor belt into camera labeling, systematically solving for unified extrinsic parameters for a camera group. Summary of the Invention

[0005] In response to the above problems, the present invention provides a global camera extrinsic coordinate marking method based on a multi-sensor pipeline, which is used to unify the global coordinate marking of multiple sets of camera extrinsic coordinates in the production line. The relationship between the rotation and displacement of the cameras at multiple stations in the production line is determined by the marking and motion module transmission, the synchronous motor feedback information is synchronized with the captured image timestamp, the square marker detection and the relative posture recovery are carried out. The multi-sensor of the present invention uses camera and motor feedback to perform joint optimization of multiple positions,

[0006] A method for marking global camera extrinsic coordinates based on a multi-sensor pipeline includes the following steps:

[0007] Step 1: Use the augmented reality marker Aruco Marker generation algorithm to generate 5 recognizable binary square markers and splice the markers into a cube, namely the square marker body.

[0008] The square marker body is a solid model based on a cube. The augmented reality markers Aruco Marker are located on the five faces of the cube, and each marker occupies one face alone. The extra face without the augmented reality marker is the placement face. The augmented reality marker includes an n×n dimensional matrix.

[0009] Step 2: Build an assembly line, use the motor feedback signal to record the sampling of the motor running time and the motor running distance, use the camera to shoot the square marker and store it as a camera frame, and record the time of the camera frame.

[0010] Step 3: Based on multiple camera frames, the time corresponding to the multiple camera frames, and the sampling of the motor running time and running distance, the relative position of the camera relative to the square marker is solved using a joint optimization algorithm based on image and motor feedback; the coordinate-time relationship diagram is obtained using the sampling of the motor running time and running distance.

[0011] Step 4: Using the relative pose, coordinate-time relationship graph, and the recording time of the camera frame, calculate the pose of the camera relative to the pipeline coordinates, that is, the camera's external parameters.

[0012] The following is a brief description of the above steps:

[0013] The square marker body is a solid model based on a cube. Aruco Markers (augmented reality markers) are located on the five faces of the cube, and each marker occupies one face. The extra face without an Aruco Marker (augmented reality marker) is the placement face. The characteristics of the Aruco Marker (augmented reality marker) include that it is an n×n matrix (where n represents the dimension of the matrix, and different n can be selected to control the fault tolerance of the generated Aruco Marker). The main algorithm of the Aruco Marker (augmented reality marker) generation algorithm is as follows:

[0014] Select the Aruco (augmented reality) dictionary. The Aruco (augmented reality) dictionary value consists of a unique binary code, and each code corresponds to a specific Aruco Marker.

[0015] Using the given Aruco (augmented reality) dictionary, select the ID (label) of the marker, and the marker generated by the sequence is unique.

[0016] Using the ID (serial number) of the tag, the corresponding binary coding matrix is ​​generated.

[0017] A border is added to the binary coding matrix to add an outline to generate a marker recognized by the auxiliary detection algorithm to form a marker with an outline.

[0018] The generated marker with the outline is used to render the image of the marker through the image generation algorithm, that is, the Aruco Marker (augmented reality marker) as described in step 3.

[0019] Further, the step 2 is specifically as follows:

[0020] Build an assembly line, the assembly line drive motor has a motor feedback function, build a multi-station camera on the assembly line, adjust the camera's aperture and focal length, so that the camera can capture clear images of the assembly line.

[0021] The starting point of the assembly line is the starting point of the global external parameter of the required mark, and the spatial position of the conveyor belt is not restricted. The square marker is placed on the conveyor belt with the side without the Aruco Marker (augmented reality marker) facing the direction of the conveyor belt. The assembly line starts slowly, reaches a certain speed with a low acceleration, and then maintains a constant speed. The motor feedback signal records the motor running time and the corresponding motor running distance. When the conveyor belt is running, the feedback with the rotary transformer records the motor running time and the distance the motor runs. The camera shoots the square marker. When the marker on the conveyor belt reaches the appropriate position, the camera shoots the transmission data and the host computer records the shooting time.

[0022] The pipeline drive motor has a motor feedback function, specifically: the pipeline drive motor has a position feedback function, which is completed by using a rotary transformer, and uses the electromagnetic induction principle to convert the rotation position into an electrical signal to obtain the motor position signal.

[0023] Beneficial effects of the present invention:

[0024] High-precision extrinsic parameter marking: This avoids the problem of insufficient marking information in traditional camera marking and combines it with optimization algorithms to effectively improve the marking of camera extrinsic parameters.

[0025] New marker: Using a new type of square marker, the conventional marking plate has unclear imaging and causes marking errors, effectively increasing the information content of the marker.

[0026] Fast marking: The algorithm does not require manual identification of square markers and can quickly implement camera extrinsic marking.

[0027] Wide range of applicable scenarios: It is applicable to various production line types or various camera placement methods, and the marking of external parameters can be completed without adjusting unnecessary parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0029] Figure 1 A flowchart of an implementation method for marking extrinsic parameters of a multi-sensor camera provided in an embodiment of the present disclosure;

[0030] Figure 2 A schematic diagram of a scene in which multiple cameras on a production line capture a square marker according to an embodiment of the present disclosure;

[0031] Figure 3 A schematic diagram of a square marker style as an example provided in an embodiment of the present disclosure;

[0032] Figure 4 A schematic diagram of the relative coordinates of the final marking result of a square marker provided in an embodiment of the present disclosure

[0033] Figure 5 Schematic diagram of the generated augmented reality marker Aruco Marker;

[0034] Figure 6 Schematic diagram of detecting augmented reality marker Aruco Marker at different angles. DETAILED DESCRIPTION

[0035] To make the objectives, technical solutions, and advantages of the present disclosure more clear, the present disclosure will be further described in detail below with reference to the accompanying drawings. It is apparent that the embodiments described are only a portion of the embodiments of the present disclosure, rather than all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present disclosure without creative effort are intended to fall within the scope of protection of the present disclosure.

[0036] In the embodiments of the present disclosure, the term "and / or" describes the association relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0037] The application scenarios described in the embodiments of the present disclosure are intended to more clearly illustrate the technical solutions of the embodiments of the present disclosure and do not constitute a limitation on the technical solutions provided by the embodiments of the present disclosure. Persons skilled in the art will appreciate that, as new application scenarios emerge, the technical solutions provided by the embodiments of the present disclosure will also be applicable to similar technical problems. In the description of the present disclosure, unless otherwise specified, "multiple" means two or more.

[0038] Before introducing the multi-camera extrinsic parameter labeling method provided by the embodiment of the present disclosure, for ease of understanding, the technical background of the embodiment of the present disclosure is first introduced in detail below.

[0039] In industrial robots, vision systems are often used to guide robots in complex tasks such as part assembly, welding, and handling. To ensure that robots can accurately identify and locate work objects, cameras require extrinsic marking to determine the relative position and orientation of the camera and robot. This process ensures that the robot can accurately determine the spatial position of the target object during operation, achieving high-precision work. Global camera extrinsic marking also plays a key role in automated quality inspection. Cameras are typically installed in fixed locations to capture images of products and detect defects. Precise extrinsic marking can determine the camera's position and orientation on the production line, ensuring that image processing algorithms can correctly analyze every part of the product. This is particularly important in industries with high precision requirements, such as electronics and automotive parts. In some complex industrial applications, a single camera may not provide sufficient field of view and accuracy, so multiple cameras must work together. Global camera extrinsic marking can determine the relative position and orientation of each camera, enabling the coordinated operation of multi-camera systems. This application is very common in 3D measurement, object reconstruction, and complex assembly processes. In automated production lines, cameras are used to monitor and control the production process. Global camera extrinsic markers ensure that the camera's field of view covers the entire production area and accurately captures product movement and changes. This is crucial for achieving efficient operation of automated production lines and stable control of product quality. AR and VR systems are becoming increasingly popular in industrial training, maintenance, and operational guidance. Global camera extrinsic markers can help these systems accurately overlay virtual information onto the real world, providing workers with more intuitive and efficient operational guidance, reducing errors and improving production efficiency. Global camera extrinsic markers are an indispensable component of industrial automation and intelligent manufacturing. They provide a precise spatial positioning foundation for various vision systems, ensuring efficient, accurate, and reliable production processes.

[0040] The following describes in detail the principles and corresponding theoretical support of the steps in the invention content:

[0041] The assembly line's drive motor features position feedback, using a resolver to obtain the motor's position signal. A resolver is a high-precision sensor used to measure rotational angle and speed. It's an analog angle encoder that converts rotational position into an electrical signal using the principle of electromagnetic induction.

[0042] The characteristic of the multi-station camera is that the camera is placed at any position of the observable production line, such as Figure 2 As shown in the figure, two cameras (the two planes are the imaging planes of the cameras, and the cameras are not drawn in the figure) are used to shoot a schematic diagram of the same conveyor belt, and the direction of the conveyor belt is specified. The square on the conveyor belt is the square marker described in step 2. The schematic diagram of the square marker is as follows Figure 3 shown.

[0043] like Figure 3 As shown, the square marker body is a solid model based on a cube. Aruco Markers are located on the five faces of the cube, and each Aruco Marker occupies one face alone. The extra face without Aruco Marker is Figure 2 The placement surface shown is placed on the conveyor belt; the characteristics of ArucoMarker (augmented reality marker) include that it is an n×n dimensional matrix (where n represents the dimension of the matrix, and different n can be selected to control the fault tolerance of generating Aruco Marker).

[0044] like Figure 1 As shown, the square marker is aligned with the desired coordinates and fixed to the conveyor belt using a clamp. The zero point is the geometric center of the square marker, and the square marker is placed at the original coordinate point of the global coordinates. This is also the origin position of the final global external reference obtained by subsequent marking. The square marker is placed on the conveyor belt with the side without the Aruco Marker (augmented reality marker) facing the direction of the conveyor belt.

[0045] The characteristic of the assembly line operation is that the assembly line starts slowly, reaches a certain speed with a low acceleration, and then maintains a constant speed. Through program control, the conveyor belt and the camera group whose posture needs to be detected are started at the same time, and the collected data information is superimposed in the form of a timestamp. In order to optimize the storage space of the equipment, the camera can only record and save the pictures of the complete square marker body and the corresponding timestamp, and collect 30 frames per second for storage. The expression P is used for the storage of pictures. A (t) is recorded, where t∈T camera_record , T camera_record The timestamp recorded by the current camera, the obtained P A(t) is the image information of the current timestamp, A represents the camera at position A. For the motor feedback signal, it is only necessary to calculate the odometer of the conveyor belt, that is, the total distance transported by the conveyor belt at different times is recorded as X(t), where t is the relative time sampling, t∈[0,t max ], t max The final time when the conveyor belt transmits the square marker, the collected image is combined with the motor feedback signal and stored. m , obtain the image frame P of the corresponding camera A at the current moment A (t m ) can obtain the distance odometer X(t m ).

[0046] For image frames that are different from the motor recording time, the time not sampled by the motor can be approximated by interpolation. Since the signal frequency of the motor feedback is high, the accuracy of the obtained result is very accurate. If you need to obtain the distance at a specific time, define the time as t sp , we can find that there is a time less than t in the sampling time sp Maximum time sampling t sp_lm Corresponding distance X sp_lm , and the sampling time is greater than t sp Minimum time sampling t sp_ml Corresponding distance X sp_ml , we can calculate the specific time t sp Corresponding distance X sp :

[0047]

[0048] The above can obtain the sampling of the motor running time and the corresponding motor running distance, as well as the camera frame taken by the camera, and record the time of the camera frame. In practice, the storage method is to record the better camera frame taken, and directly obtain the corresponding distance of the shooting time through the above formula and save them in combination.

[0049] The characteristic of the coordinate time relationship diagram is the sampling relationship between the distance transmitted in the coordinate direction corresponding to the running direction of the transmission axis and time. It is a series of sampling points, and then the distance of the unsampled time point is calculated by interpolation.

[0050] Based on multiple camera frames, the time corresponding to multiple camera frames, and the sampling of motor running time and running distance, the relative position of the camera relative to the square marker is solved using a joint optimization algorithm based on image and motor feedback; the coordinate-time relationship diagram is obtained by sampling the motor running time and running distance.

[0051] The joint optimization algorithm based on image and motor feedback is described as follows:

[0052] 1. For multiple camera frames, use the Aruco Marker detection algorithm and the PnP algorithm to calculate the displacement vector of the center point on the square marker surface in the camera coordinate system, denoted as t1, t2, t3, the projection vector of the normal vector of each face of the square marker in the camera frame, the scale information on the marker, the marker surface coordinate axis relative to the x-axis, y-axis and z-axis of the camera coordinate system, and the translation vector of the marker surface center relative to the camera optical center. The translation vectors represent the three faces of the square marker respectively.

[0053] 2. Using the translation vector of the marker surface center relative to the camera optical center, calculate the displacement t of the geometric center of the square marker body relative to the camera optical center n .

[0054] Specifically, we use the translation vectors t1, t2, and t3 of the center of the marker surface relative to the optical center of the camera to calculate the mean, and then use similar triangles to find the value. (The vector of the extended mean sum is the squared marker body displacement).

[0055] 3. Determine the center position: Use the projection vector corresponding to the center point of the marker surface in the camera frame to calculate the projection coordinates of the geometric center of the square marker body on the camera frame.

[0056] Specifically, the geometric center O is formed by the reverse extension line of the normal vector corresponding to the axial center point. Due to measurement errors, the normal vector is translated to the center point on the surface of the square marker and extended in the reverse direction to form a closed triangle. The geometric center of the triangle is calculated, and the projection of the geometric center of the square marker on the camera frame is obtained.

[0057] 4. Calculate the direction of a single axis on the image: For multiple camera frames, use the projection coordinates of the geometric center of the square marker on the camera frame. There will be multiple points on the image. Use RANSAC (random sampling consensus) to obtain a fitted projection line and extract its relative displacement direction under the camera perspective.

[0058] Specifically, the direction of the x-axis in the image is calculated (RANSAC method). For multiple sampling images, the position of the geometric center of each square marker projected on the camera frame is calculated, and a line is fitted using the RANSAC (random sampling consensus) method, which is the projection line represented by the projection of the x-axis in the image. The relative displacement direction under the camera perspective is obtained through the direction of the single axis on the image.

[0059] 5. Calculate the x-axis position of the square marker in the camera coordinate system (running displacement):

[0060] Based on the sampling time of the camera frame, the coordinate time relationship diagram is used to calculate the displacement distance of the square marker in three-dimensional space (the direction of the conveyor belt is the x-axis of the marker).

[0061] The position of the geometric center of the marker on different camera frames is used to calculate the travel distance of the motor sampling at the corresponding time through interpolation, and the pixel distance translated on the camera frame is calculated.

[0062] The pixel distance and the scale information on the square marker are used to calculate the displacement of the restored square marker in the camera frame plane.

[0063] The displacement distance of the square marker in three-dimensional space is used to restore the displacement of the square marker in the image plane, and the angle between the x-axis of the square marker and the z-axis of the camera is calculated.

[0064] The position of the square marker on the x-axis in the camera coordinate system is calculated using the angle of the z-axis and the relative displacement direction under the image viewing angle.

[0065] Specific process: Using the pixel distance in the image and the scale information on the square marker, different camera frames have different scale information. The scale information not obtained by the camera frame in the middle needs to be obtained through interpolation. Because the displacement of the square marker in the image changes linearly with respect to the scale information of the camera, the displacement in the image is a scalar, so the change of scale information is linear. The displacement of the restored square marker in the camera frame plane can be calculated. The difference between the displacement of the restored square marker and the displacement of the conveyor belt is because the displacement in the scale direction is lost due to the projection transformation and is not calculated.

[0066] Using the displacement distance of the square marker in three-dimensional space and restoring the displacement of the square marker in the camera frame plane, a right triangle model is constructed based on the displacement distance as the hypotenuse and the restored displacement of the square marker as one of the adjacent sides of the right triangle. The corresponding angle of this adjacent side is the angle of the square marker relative to the camera z-axis (the direction perpendicular to the camera imaging plane).

[0067] The z-axis angle corresponds to the projection line of the x-axis on the image. The angles of the projection line relative to the horizontal and vertical axes of the image correspond to the x-axis and y-axis of the camera coordinate system respectively. The position of the square marker on the x-axis in the camera coordinate system is calculated. The x-axis is the displacement direction of the square marker in three-dimensional space.

[0068] 6. Solve the rotation matrix R of the square marker relative to the camera n : Calculate the rotation matrix of the square marker relative to the camera coordinate system based on the x-axis position of the existing square marker in the camera coordinate system and the rotation axis of the square marker surface relative to the camera coordinate system of multiple camera frames.

[0069] Specifically, a constraint optimization algorithm is performed on the existing x-axis direction. For the existing direction x, whose coordinate in the camera coordinate system is x, the goal is to determine the new y-axis and z-axis directions so that they are as close as possible to the directions of multiple given axes. This can be achieved by minimizing the angles or other error metrics between all given axes and the new axes. The calculated y-axis and z-axis vectors in the camera coordinate system are denoted as y1, y2, ..., y n and z1,z2,…,z n The orientation of the y-axis and z-axis of the square marker is represented by y and z, so the cost function of the objective function is to minimize the dot product difference between all given axes and the new axis, that is:

[0070]

[0071] The following conditions are met:

[0072] y·x=0

[0073] z·x=0

[0074] y·z=0

[0075] ‖y‖=1

[0076] ‖z‖=1

[0077] This objective function jointly optimizes the axis in each direction. i To maximize its projection on the y-axis and minimize its projection on the z-axis, we can optimize the solution to the optimal y- and z-axis vectors. Similarly, the coordinate system has certain orthogonality and normalization conditions, so it is necessary to have the constraints of the above formula to satisfy them. Through the numerical optimization method, the y-axis and z-axis can be solved to be the orientation y′ and z′ of the square marker in the camera coordinate system, then the rotation matrix R of the square marker relative to the camera can be obtained. n =[x,y′,z′].

[0078] According to the above solved square marker body relative to the camera pose matrix, we can get the pose matrix of the square marker body at a certain time t s The relative position of the camera relative to the square marker

[0079] Because the square marker does not rotate in the camera coordinate system (the square marker does not rotate in the scene, and the camera does not rotate), the same rotation matrix can be used at each time, but different displacements are used for different positions. For the joint optimization method based on image and motor feedback, if the camera can only capture two ArucoMarkers on the surface of the marker plate, it also has a certain marking ability, but its fusion information is less and there will be large errors. Ignoring the fact that the camera only captures one Aruco Marker on the surface of the marker plate, the marking result will degenerate into a planar mark, which is inconsistent with the innovative direction of this patent.

[0080] The square markers at different positions on the conveyor belt have different translation vectors. They have the same y-axis and z-axis, but the detection results at different angles will have errors. The algorithm is to reduce this error. If the scale information is not obtained through the interpolation method, its accuracy will be greatly lost. After interpolation, the result will be more accurate. The time relationship diagram obtained by comparing the time corresponding to the camera frame is used to calculate the position of the square marker at the time recorded by the corresponding camera frame. Combined with the relative posture of the camera relative to the square marker obtained by describing the description, the global external parameters of a single camera are calculated using the joint optimization method based on image and motor feedback.

[0081] By combining the image with the position signal fed back by the motor, the pose of the camera relative to the square marker is optimized and calculated, and the pose relationship of the square marker relative to the camera can be directly calculated.

[0082] The algorithm based on the joint optimization method of image and motor feedback will obtain the relative pose T of the camera relative to the square marker c_to_g ; for a certain moment t s , use the coordinate time relationship diagram to calculate the transmission distance t of the square marker block relative to the pipeline g_to_w Since the x-axis of the square marker is oriented in the same direction as the conveyor belt and does not rotate, the x-axis rotation matrix of the square marker relative to the world coordinate system is I (unit matrix, indicating no rotation). Then, the transformation matrix of the square marker relative to the world coordinate system can be obtained: Then we can calculate the pose matrix of the camera relative to the world coordinate system, that is, the external parameters of the camera:

[0083]

[0084] This is the camera extrinsic parameter.

[0085] The present invention does not involve a conveying method with a rotating nature, because the odometer results are different at different lateral positions of the conveyor belt (the position perpendicular to the conveying direction), which brings uncertainty to the measurement. Only one-way straight-line transmission is considered, and the conveyor belt can be transmitted at a non-uniform speed. Since the odometer is used for measurement, and the world coordinates here are initialized with the marker zero position, the displacement vector tvec of the transmission relative to the global coordinates can be obtained. g_to_w , the rotation matrix is ​​the identity matrix, such as Figure 4 shown.

[0086] Experiments have shown that:

[0087] like Figure 5 The following shows the generated Aruco Marker (augmented reality marker). Figure 6 As shown in the figure, it can detect Aruco Markers at different angles, identify Aruco Markers with different IDs, and mark the position coordinates of the Aruco Markers in space (actually the coordinates in camera space). The blocks made from it can be confirmed to have the ability to mark in camera space.

[0088] Obviously, those skilled in the art may make various changes and modifications to the present disclosure without departing from the spirit and scope of the present disclosure. Thus, if these modifications and variations of the present disclosure fall within the scope of the claims of the present disclosure and their equivalents, the present disclosure is intended to include these modifications and variations.

[0089] Matters not covered by the present invention are known technologies.

[0090] The above embodiments are intended only to illustrate the technical concepts and features of the present invention. Their purpose is to enable those skilled in the art to understand the contents of the present invention and implement them accordingly. They are not intended to limit the scope of protection of the present invention. Any equivalent changes or modifications made in accordance with the spirit of the present invention are intended to be covered by the scope of protection of the present invention.

Claims

1. A global camera extrinsic coordinate marking method based on a multi-sensor pipeline, characterized in that: The following steps are involved: Step 1: Use the augmented reality marker Aruco Marker generation algorithm to generate five recognizable binary square markers, and splice the markers into a cube, i.e., a square marker body; Step 2: Build an assembly line, use the motor feedback signal to record the motor running time and the motor running distance sampling, use the camera to shoot the square calibration volume and store it as a camera frame, and record the camera frame time; Step 3: Based on the sampling of multiple camera frames, the time corresponding to the multiple camera frames, the motor running time and running distance, the relative position of the camera relative to the square calibration volume is solved using a joint optimization algorithm based on image and motor feedback; The coordinate-time relationship diagram is obtained by sampling the motor running time and running distance; Step 4: Using the relative pose, coordinate-time relationship graph, and camera frame time, calculate the pose of the camera relative to the pipeline coordinates, i.e., the camera's external parameters.

2. The method for marking global camera extrinsic coordinates based on a multi-sensor pipeline according to claim 1, wherein: The square marker body is a solid model based on a cube. The augmented reality markers Aruco Marker are located on the five faces of the cube, and each marker occupies one face alone. The extra face without the augmented reality marker is the placement face. The augmented reality marker includes an n×n dimensional matrix.

3. The method for marking global camera extrinsic coordinates based on a multi-sensor pipeline according to claim 2, wherein: The specific implementation process of the augmented reality marker Aruco Marker generation algorithm is as follows: S1.

1. Select the augmented reality Aruco dictionary. The Aruco dictionary value consists of a unique binary code, and each code corresponds to an augmented reality marker Aruco Marker. S1.

2. Using the given Aruco dictionary, select the ID of the marker. The marker generated by the serial number is unique. S1.

3. Generate a corresponding binary coding matrix using the marked ID; S1.4, adding a border to the binary coding matrix to add an outline to generate a mark recognized by the auxiliary detection algorithm, thereby forming a mark with an outline; S1.

5. Using the generated marker with the outline, render an image of the marker through an image generation algorithm.

4. The method for marking global camera extrinsic coordinates based on a multi-sensor pipeline according to claim 3, wherein: The specific implementation process of step 2 is as follows: Build an assembly line. The assembly line drive motor has a motor feedback function. Build a multi-station camera on the assembly line and adjust the camera's aperture and focal length to allow the camera to capture clear images of the assembly line. The starting point of the assembly line is the starting point of the calibrated global external parameters, and the spatial position of the conveyor belt is not restricted. The square marker is placed in such a way that the side without the augmented reality marker Aruco Marker is placed on the conveyor belt, with one side facing the direction of the conveyor belt. The motor feedback signal records the motor running time and the corresponding motor running distance. When the sampling conveyor belt is running, the feedback with a rotary transformer records the motor running time and the motor running distance; the camera shoots the square marker on the conveyor belt. After the marker reaches the position, the camera captures the camera frame and the host computer records the shooting time.

5. The method for marking global camera extrinsic coordinates based on a multi-sensor pipeline according to claim 4, characterized in that: The pipeline drive motor has a motor feedback function, specifically: the pipeline drive motor has a position feedback function, which is completed by using a rotary transformer, and uses the electromagnetic induction principle to convert the rotation position into an electrical signal to obtain the motor position signal.

6. The method for marking global camera extrinsic coordinates based on a multi-sensor pipeline according to claim 5, characterized in that: The specific implementation process of the joint optimization algorithm based on image and motor feedback is as follows: S3.

1. For multiple camera frames, use the Aruco Marker detection algorithm and the PnP algorithm to determine the displacement vector of the center point on the square marker surface in the camera coordinate system, denoted as t1, t2, t3, the projection vector of the normal vector of each face of the square marker in the camera frame, the scale information on the calibration body, the coordinate axis of the marker surface relative to the x-axis, y-axis, and z-axis of the camera coordinate system, and the translation vector of the center of the marker surface relative to the camera optical center. The translation vectors represent the three faces of the square marker respectively. S3.

2. Use the translation vectors t1, t2, and t3 of the center of the marker surface relative to the camera optical center to perform mean summation and then use similar triangles to calculate the displacement of the geometric center of the square marker relative to the camera optical center. S3.

3. Using the projection vector corresponding to the center point of the marker surface in the camera frame, calculate the projection coordinates of the geometric center of the square marker body on the camera frame; S3.

4. For multiple camera frames, use the projection coordinates of the geometric center of the square marker on the camera frame. There are multiple points on the image. Use RANSAC to obtain a fitting projection line, which is the projection line represented by the x-axis line on the image. The relative displacement direction under the camera perspective is obtained by the direction of the single axis on the image. S3.

5. Calculate the x-axis position of the square marker in the camera coordinate system; S6.

3. Calculate the rotation matrix R of the square calibration body relative to the camera coordinate system based on the x-axis position of the existing square marker body in the camera coordinate system and the rotation axis of the square marker surface of multiple camera frames relative to the camera coordinate system. n , and obtain the relative pose of the square calibration body relative to the camera.

7. The method for marking global camera extrinsic coordinates based on a multi-sensor pipeline according to claim 6, wherein: The specific implementation of step S3.3 is as follows: the geometric center O formed by the reverse extension line of the normal vector corresponding to the axial center point is translated to the center point on the surface of the square marker body, and the normal vector is extended in the reverse direction to form a closed triangle. The geometric center of the triangle is calculated, and the projection of the geometric center of the square marker body on the camera frame is obtained.

8. The method for marking global camera extrinsic coordinates based on a multi-sensor pipeline according to claim 7, wherein: The specific implementation process of step S3.5 is as follows: Based on the sampling time of the camera frame, the coordinate time relationship diagram is used to calculate the displacement distance of the square calibration body in three-dimensional space; Using the position of the geometric center of the calibration body on different camera frames, the travel distance of the motor sampling at the corresponding time is calculated by interpolation, and the pixel distance translated on the camera frame is calculated; Using the pixel distance and the scale information on the square marker, the displacement of the restored square marker in the camera frame plane is calculated; Using the displacement distance of the square marker in three-dimensional space, the displacement of the square marker is restored in the image plane, and the angle between the x-axis of the square marker and the z-axis of the camera is calculated; The position of the square marker on the x-axis in the camera coordinate system is calculated using the angle of the z-axis and the relative displacement direction under the image viewing angle.

9. The method for marking global camera extrinsic coordinates based on a multi-sensor pipeline according to claim 8, characterized in that: The calculation of the angle between the x-axis of the square marker and the z-axis of the camera is specifically as follows: using the displacement distance of the square marker in three-dimensional space and restoring the displacement of the square marker in the camera frame plane, a right triangle model is constructed based on the displacement distance as the hypotenuse and the restored displacement of the square marker as an adjacent side of the right triangle, and the angle corresponding to the adjacent side is the angle between the square marker and the z-axis of the camera; The method of calculating the x-axis position of the square marker in the camera coordinate system is specifically as follows: using the z-axis angle corresponding to the projection line of the x-axis on the image in the image, the angles of the projection line relative to the horizontal and vertical axes of the image correspond to the x-axis and y-axis of the camera coordinate system respectively, and finding the x-axis position of the square marker in the camera coordinate system, where the x-axis is the displacement direction of the square marker in three-dimensional space.

10. The method for marking global camera extrinsic coordinates based on a multi-sensor pipeline according to claim 9, wherein: The specific implementation process of step S3.6 is as follows: The constraint optimization algorithm is performed on the x-axis direction. For the existing direction x, the coordinate in the camera coordinate system is x, and the calculated y-axis and z-axis vectors in the camera coordinate system are recorded as y1, y2, ..., y n and z1,z2,…,z n ; The orientation of the y-axis and z-axis of the square calibration volume is represented by y and z. The cost function of the objective function is to minimize the dot product difference between all given axes and the new axis, that is: The following conditions are met: y·x=0; z·x=0; y·z=0 ‖y‖=1;‖z‖=1 This objective function jointly optimizes the axis in each direction. i To maximize its projection on the y-axis and minimize its projection on the z-axis, the optimal y- and z-axis vectors are optimized and solved. The y- and z-axis obtained by numerical optimization method are the orientations y′ and z′ of the square calibration body in the camera coordinate system, and the rotation matrix R of the square calibration body relative to the camera is obtained. n =[x,y′,z′]; According to the above solved square calibration body relative to the camera pose matrix, we can get the pose matrix of the calibration body at a certain time t s Below, the relative position of the camera relative to the square calibration volume: Represents R n The inverse matrix of T represents a matrix of all 0s, and 1 represents a scalar.

11. The method for marking global camera extrinsic coordinates based on a multi-sensor pipeline according to claim 10, wherein: The specific implementation process of step 4 is as follows: The algorithm based on the joint optimization method of image and motor feedback obtains the relative pose T of the camera relative to the square calibration body. c_to_g ; for a certain moment t s , use the coordinate time relationship diagram to calculate the transmission distance t of the square calibration block relative to the pipeline g_to_w ; The rotation matrix of the x-axis of the square calibration body relative to the world coordinate is the unit matrix I; Get the transformation matrix of the square calibration volume relative to the world coordinate system: Calculate the camera's pose matrix relative to the world coordinate system, that is, the camera's external parameter T c_to_w :

Images