Cross-view joint calibration method, device and equipment

Abstract

The application belongs to the technical field of multi-sensor calibration, and discloses a cross-view joint calibration method, device and equipment. The method comprises the following steps: acquiring rotation image data of a calibration kit, the calibration kit comprising a calibration board and an encoding board; encoding the calibration board according to the two-dimensional code information of the encoding board in the rotation image data to obtain encoding information; determining a first relative position relationship between the calibration board and a starting calibration board according to the rotation image data and the encoding information; determining a second relative position relationship between cameras in a to-be-calibrated whole machine according to the first relative position relationship; and calibrating the cameras of the to-be-calibrated whole machine according to the second relative position relationship. Through the above method, the calibration targets are flexible and have encoding between each other, and can be flexibly adjusted under different application scenarios, so that the requirement of joint calibration for calculation accuracy under different machine types and different camera conditions can be met, and the cost is reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-sensor calibration, in particular to a cross-view joint calibration method, device and equipment. BACKGROUND

[0002] The current common multi-sensor joint calibration scheme needs the help of site prior information. The walls of the site are usually pasted with targets for sensor calibration. The size of the target pattern and the relative position relationship between the targets are strictly determined based on the process level of the site construction party. Finally, all target patterns have an accurate coordinate in the same physical world. Finally, based on the target image coordinates detected by the sensor acquired image, the correct sensor calibration parameters can be calculated by the visual pose estimation algorithm. However, due to the high cost of the construction of site prior information and poor flexibility, once the construction is completed, it is difficult to change. When the model or camera is changed to a large extent, it is easy to cause the result of pose estimation calculation to be inaccurate. Therefore, the original site cannot be reused, and the construction cost is high, which will cause a great waste of resources.

[0003] The above content is only used to assist in understanding the technical solutions of the present application and does not represent the acknowledgement of the above content as prior art. SUMMARY

[0004] The main purpose of the present application is to provide a cross-view joint calibration method, device and equipment, which aims to solve the technical problem that the prior information of the traditional calibration site in the prior art is usually fixed, the flexibility is poor, after changing the model or camera, it is difficult to ensure the accuracy of calibration, and it is necessary to re-construct the prior information of the calibration site, which has high cost.

[0005] To achieve the above purpose, the present application provides a cross-view joint calibration device, which comprises a calibration kit, a rotating part and a calibration equipment, the rotating part is arranged in the center of the calibration site, the calibration kit is arranged around the rotating part, the calibration kit comprises a calibration board and an encoding board, and the encoding board is provided with two-dimensional code information;

[0006] The rotating part is used for acquiring the rotating image data of the calibration kit;

[0007] The calibration equipment is used for encoding the calibration board according to the two-dimensional code information of the encoding board in the rotating image data, and obtaining the encoding information;

[0008] The calibration equipment is also used for determining the first relative position relationship between the calibration board and the starting calibration board according to the rotating image data and the encoding information;

[0009] The calibration device is further configured to determine a second relative position relationship between cameras in the whole machine to be calibrated according to the first relative position relationship.

[0010] The calibration device is further configured to calibrate the cameras in the whole machine to be calibrated according to the second relative position relationship.

[0011] Optionally, the rotating component is a lifting and rotating platform, which can cover the whole machine to be calibrated.

[0012] The lifting and rotating platform is configured to park the whole machine to be calibrated and adjust a shooting height of the whole machine to be calibrated, so that a field of view of a camera in the whole machine to be calibrated at least includes two calibration kits.

[0013] The lifting and rotating platform is further configured to rotate the whole machine to be calibrated, so that the camera in the whole machine to be calibrated collects rotation image data of the calibration kits.

[0014] The calibration device is further configured to determine static image data of the calibration kits according to the rotation image data, and determine the second relative position relationship according to the static image data and the first relative position relationship information.

[0015] Optionally, the rotating component includes a rotating manipulator and a collection camera, the rotating manipulator holds the collection camera, and the rotating manipulator is arranged on a ceiling in the center of the calibration site.

[0016] The rotating manipulator is configured to adjust a height and a pitch angle of the collection camera, so that a field of view of the collection camera at least includes two calibration kits.

[0017] The rotating manipulator is further configured to rotate the collection camera, so that the collection camera acquires rotation image data of the calibration kits in the rotating process.

[0018] Optionally, the rotating component further includes a lifting platform, the lifting platform can cover the whole machine to be calibrated, and the calibration kits are located at a periphery of the lifting platform.

[0019] The lifting platform is configured to park the whole machine to be calibrated and adjust a shooting height of the whole machine to be calibrated, so that a camera in the whole machine to be calibrated collects static image data of the calibration kits, and a field of view of the camera in the whole machine to be calibrated at least includes two calibration kits.

[0020] The calibration device is further configured to determine the second relative position relationship according to the static image data and the first relative position relationship information.

[0021] In addition, to achieve the above object, the application further provides a cross-view joint calibration method, steps of the cross-view joint calibration method comprising:

[0022] acquiring rotation image data of a calibration kit, the calibration kit comprising a calibration board and an encoding board;

[0023] encoding the calibration board according to two-dimensional code information of the encoding board in the rotation image data to obtain encoding information;

[0024] determining a first relative position relationship between the calibration board and a starting calibration board according to the rotation image data and the encoding information;

[0025] determining a second relative position relationship between cameras in a to-be-calibrated whole machine according to the first relative position relationship;

[0026] calibrating the cameras of the to-be-calibrated whole machine according to the second relative position relationship.

[0027] Optionally, the determining of the first relative position relationship between the calibration board and the starting calibration board according to the rotation image data and the encoding information comprises:

[0028] determining a third relative position relationship between adjacent calibration boards according to the rotation image data and the encoding information;

[0029] determining the first relative position relationship according to the third relative position relationship.

[0030] Optionally, the determining of the third relative position relationship between adjacent calibration boards according to the rotation image data and the encoding information comprises:

[0031] determining a relative transformation between adjacent calibration boards according to the rotation image data and the encoding information;

[0032] determining observation constraint information of the calibration board according to the rotation image data and the encoding information;

[0033] determining an iterative solution corresponding to the third relative position relationship based on the relative transformation between the adjacent calibration boards and the observation constraint information of the calibration board;

[0034] optimizing and decomposing the iterative solution corresponding to the third relative position relationship to obtain the third relative position relationship.

[0035] Optionally, the determining of the first relative position relationship according to the third relative position relationship comprises:

[0036] determining a fourth relative position relationship between calibration boards according to the third relative position relationship;

[0037] According to the preset number and the encoding information, a starting calibration board is determined;

[0038] Based on the chain strategy and the fourth relative position relationship, a first relative position relationship between the calibration board and the starting calibration board is determined.

[0039] Optionally, the determining the second relative position relationship between the cameras in the whole machine to be calibrated according to the first relative position relationship comprises:

[0040] Obtaining static image data of the calibration kit, extracting the calibration board in the static image data to obtain calibration extraction information;

[0041] According to the calibration extraction information and the first relative position relationship, a fifth relative position relationship between the camera and the calibration board is determined;

[0042] Based on the fifth relative position relationship, the second relative position relationship is determined.

[0043] In addition, in order to achieve the above-mentioned purpose, the application further provides a cross-view joint calibration device, which comprises the cross-view joint calibration apparatus and applies the steps of the cross-view joint calibration method.

[0044] In the application, by obtaining the rotation image data of the calibration kit, the calibration board is encoded according to the two-dimensional code information of the encoding board in the rotation image data to obtain the encoding information, the first relative position relationship between the calibration board and the starting calibration board is determined according to the rotation image data and the encoding information, the second relative position relationship between the cameras in the whole machine to be calibrated is determined according to the first relative position relationship, and the cameras of the whole machine to be calibrated are calibrated according to the second relative position relationship. The prior information of the traditional calibration site is usually fixed, and the flexibility is poor. After replacing the model or the camera, it is difficult to ensure the accuracy of the calibration, and the prior information of the calibration site needs to be re-constructed, which has high cost. The calibration target used in the application is flexible and has encoding between targets. In different application scenarios, it can be flexibly adjusted to meet the requirements of joint calibration for calculation accuracy under different models and different cameras, reduce the cost, and the application adopts the rotation image acquisition mode. Any form and position of target information can be acquired, the global optimal solution can be obtained based on the constraint of multiple frame information, and high-precision calibration calculation is realized. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 is a structural schematic diagram of the first embodiment of the cross-view joint calibration apparatus of the application;

[0046] Figure 2 is a calibration kit schematic diagram of the first embodiment of the cross-view joint calibration apparatus of the application;

[0047] Figure 3 is a structural schematic view of a second embodiment of the cross-view joint calibration device of the present application;

[0048] Figure 4 is a structural schematic view of a third embodiment of the cross-view joint calibration device of the present application;

[0049] Figure 5 is a flow schematic view of a first embodiment of the cross-view joint calibration method of the present application;

[0050] Figure 6 is a flow schematic view of a first embodiment of the cross-view joint calibration method of the present application;

[0051] Figure 7 is a flow schematic view of a second embodiment of the cross-view joint calibration method of the present application.

[0052] BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Reference Name Reference Name 10 Calibration kit 201 Lifting and rotating platform 20 Rotating part 202 Rotating robot 30 Calibration device 203 Acquisition camera 101 Calibration plate 204 Lifting platform 102 Code plate

[0054] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0055] It should be understood that the specific embodiments described herein are intended to be illustrative only and not limiting of the present application.

[0056] Reference Figure 1 , Figure 1 is a structural schematic view of a first embodiment of the cross-view joint calibration device of the present application.

[0057] As shown in Figure 1 , the cross-view joint calibration device comprises a calibration kit 10, a rotating component 20 and a calibration device 30, the rotating component 20 is arranged at the center of a calibration site, the calibration kit 10 is arranged around the rotating component 20, and the calibration kit 10 comprises a calibration board 101 and an encoding board 102, and the encoding board 102 is provided with two-dimensional code information.

[0058] It should be noted that the calibration kit 10 is usually placed around the rotating component 20 at a corresponding position for one circle, and the sensor layout and characteristics of the specific model need to be considered when placing. The calibration kit 10 is composed of two parts of the calibration board 101 and the encoding board 102, the pattern on the calibration board 101 can be set as a black and white checkerboard pattern, or other patterns can be used according to actual needs, and the present embodiment does not limit this, and the pattern on the encoding board 102 is usually a two-dimensional code. Figure 2As shown, the encoding plate 102 is usually arranged above / below the calibration plate 101 to constitute the calibration kit 10, and the information provided by the two-dimensional code of the encoding plate 102 can be used to encode the calibration plate 101.

[0059] It can be understood that the placement position and number of the calibration kit 10 in the embodiment are not fixed, and can be flexibly adjusted according to actual needs to adapt to different models and cameras, and therefore, the relative position relationship between the calibration kits 10 is unknown. The model of the whole machine to be calibrated and the cameras in the whole machine to be calibrated are not limited in the embodiment.

[0060] It should be understood that the rotating component 20 is used to obtain the rotation image data of the calibration kit 10. The rotating component 20 usually has a rotating function, and can collect the image data of the calibration kit 10 by rotating the rotating component 20 itself or rotating the whole machine to be calibrated by the rotating component 20, and the collected data is the rotation image data, i.e., the image data of one whole rotation. When collecting the rotation image data, it is necessary to ensure that the field of view can cover at least two calibration kits 10. In addition, the rotating component 20 usually has a height adjustment function, so that the collection of the rotation image data can cover all the calibration kits 10.

[0061] It should be understood that the rotating component 20 is used to obtain the rotation image data of the calibration kit 10. The rotating component 20 usually has a rotating function, and can collect the image data of the calibration kit 10 by rotating the rotating component 20 itself or rotating the whole machine to be calibrated by the rotating component 20, and the collected data is the rotation image data, i.e., the image data of one whole rotation. When collecting the rotation image data, it is necessary to ensure that the field of view can cover at least two calibration kits 10. In addition, the rotating component 20 usually has a height adjustment function, so that the collection of the rotation image data can cover all the calibration kits 10.

[0062] It can be understood that the two-dimensional code information is the two-dimensional code pattern on the encoding plate 102, and the encoding information is the encoding of each calibration plate 101. The first relative position relationship refers to the relative position relationship between the calibration plates 101 unified to the same coordinate system, i.e., all the calibration plates 101 are unified to the starting calibration plate. The starting calibration plate usually refers to the first calibration plate, and can be flexibly adjusted according to actual needs, which is not limited in the embodiment. The first relative position relationship can be considered as the prior information required for the current joint calibration. The second relative position relationship refers to the relative position relationship between all the cameras in the whole machine to be calibrated.

[0063] In a specific implementation, the rotation component 20 obtains rotation image data, determines the relative position relationship between the calibration boards 101 based on the obtained rotation image data and the codes of the respective calibration boards 101, converts to the same coordinate system, and further calculates the relative position relationship between all the cameras to realize the calibration of all the cameras.

[0064] Further, the calibration device 30 is further configured to determine a third relative position relationship between adjacent calibration boards according to the rotation image data and the code information, and determine the first relative position relationship according to the third relative position relationship.

[0065] It should be understood that the adjacent calibration boards refer to two adjacent calibration boards 101, and the third relative position relationship refers to the relative position relationship between the two adjacent calibration boards 101. The third relative position relationship is converted to the coordinate system corresponding to the starting calibration board to obtain the first relative position relationship relative to the starting calibration board.

[0066] Further, the calibration device 30 is further configured to determine a relative transformation between adjacent calibration boards according to the rotation image data and the code information, determine observation constraint information of the calibration board 101 according to the rotation image data and the code information, determine an iterative solution corresponding to the third relative position relationship based on the relative transformation between the adjacent calibration boards and the observation constraint information of the calibration board, and obtain the third relative position relationship by optimizing and decomposing the iterative solution corresponding to the third relative position relationship.

[0067] It should be noted that the relative position relationship between the two adjacent calibration boards 101 is calculated according to the pose estimation and multi-frame superposition based on the recorded whole circle image data, and the calibration board 101 is encoded by the two-dimensional code above / below the calibration board 101, and finally the relative position relationship relative to the starting calibration board is obtained to realize the calibration of the prior information.

[0068] Further, the calibration device 30 is further configured to determine a fourth relative position relationship between the calibration boards according to the third relative position relationship, determine a starting calibration board according to a preset number and the code information, and determine the first relative position relationship between the calibration board and the starting calibration board based on a chain strategy and the fourth relative position relationship.

[0069] It should be understood that in order to obtain the first relative position relationship, the fourth relative position relationship needs to be determined according to the third relative position relationship. The fourth relative position relationship is the relative position relationship between all the calibration boards 101. After the fourth relative position relationship is determined, the relative position relationship between all the calibration boards 101 relative to the starting calibration board, i.e., the first relative position relationship, can be obtained according to the chain rule.

[0070] Further, the calibration device 30 is further configured to acquire static image data of the calibration kit 10, extract the calibration board 101 in the static image data to obtain calibration extraction information, determine a fifth relative position relationship between the camera and the calibration board according to the calibration extraction information and the first relative position relationship, and determine the second relative position relationship based on the fifth relative position relationship.

[0071] It should be understood that the static image data refers to a static image frame captured by the camera in the whole machine to be calibrated, the calibration extraction information refers to the calibration board information extracted from the image, and the fifth relative position relationship refers to the relative position relationship between the camera and the calibration board in the whole machine to be calibrated, i.e., the relative position relationship between the camera and the prior information coordinate system.

[0072] In a specific implementation, the data recorded after one rotation is used for calculation to determine the prior information of the calibration site. At this time, the calibration board 101 captured by any frame of the camera under the static state can be extracted, and the relative position relationship between all cameras and the prior information coordinate system can be calculated to obtain the relative position relationship between the cameras and the cameras. Meanwhile, if multiple frames are superimposed, a more robust global optimal solution can be obtained. Whether it is the prior relative position relationship between the calibration boards or the relative position relationship between the cameras, a global optimal solution can be obtained based on the constraint of multiple frame information. Compared with the prior art which uses a single frame for calculation, the present embodiment is less susceptible to light and imaging stability, etc. (for example, when recognizing a two-dimensional code, if the two-dimensional code cannot be recognized, pose estimation calculation cannot be performed), and has higher robustness.

[0073] It should be noted that in the prior art, when the model is replaced for calibration, the height, orientation and distance from the calibration board will change, causing the calibration board to not be able to well cover the field of view, and thus the accuracy of the multi-camera joint calibration calculation is reduced. In the case of replacing the camera specifications at the same position, the camera specifications such as resolution, field of view, etc. change, and then the target that is originally clearly imaged may not be within the depth of field of the new camera, causing the extraction accuracy of the target in the image to be reduced, and thus accurate calibration parameters cannot be calculated. In the present embodiment, the position of the calibration kit 10 can be flexibly moved, and has coding support, has the characteristics of being flexibly adjusted and the accuracy advantage under different application scenarios, and can meet the requirements of joint calibration calculation accuracy under different models and different camera conditions. In addition, in the prior art, the prior information target is fixed, so the camera in different depth ranges can only face the target array of the same depth for detection and calculation of the calibration parameters, which will reduce the calibration accuracy. In the present embodiment, the targets can be placed at different heights, distances and positions, and due to the advantage of rotating image acquisition, any form and position of target information can be acquired, and different specifications of cameras can acquire appropriate target images at corresponding depth positions, and the high-precision calibration calculation is realized.

[0074] In the present embodiment, the cross-view joint calibration device comprises a calibration kit 10, a rotating component 20 and a calibration device 30. The rotating component 20 is arranged in the center of the calibration site, and the calibration kit 10 is arranged around the rotating component 20. The calibration kit 10 comprises a calibration board 101 and a coding board 102. The coding board 102 is provided with two-dimensional code information. The rotating component 20 is used to acquire rotating image data of the calibration kit 10. The calibration device 30 is used to encode the calibration board 101 according to the two-dimensional code information of the coding board 102 in the rotating image data to obtain coding information. The first relative position relationship between the calibration board and the starting calibration board is determined according to the rotating image data and the coding information. The second relative position relationship between the cameras in the machine to be calibrated is determined according to the first relative position relationship. The cameras in the machine to be calibrated are calibrated according to the second relative position relationship. The calibration targets used in the present embodiment are flexible and have coding support between each target. The targets can be flexibly adjusted under different application scenarios, can meet the requirements of joint calibration calculation accuracy under different models and different camera conditions, reduce costs, and the rotating image acquisition method is adopted in the present embodiment. Any form and position of target information can be acquired, the global optimal solution can be obtained based on the constraint of multiple frames of information, and high-precision calibration calculation is realized.

[0075] Reference Figure 3 , Figure 3 FIG. 1 is a structural schematic diagram of a cross-view joint calibration device according to an embodiment of the present application.

[0076] As Figure 3As shown, the rotating component 20 is a lifting rotating platform 201, which can cover the whole machine to be calibrated.

[0077] It should be noted that the lifting rotating platform 201 is a heavy lifting and rotating platform, which is arranged in the center of the calibration site, and the size of the platform can basically cover all machine types. A plurality of calibration kits 10 are placed around the lifting rotating platform 201 at corresponding positions.

[0078] It can be understood that the lifting rotating platform 201 is used to park the machine to be calibrated and adjust the shooting height of the machine to be calibrated, so that the field of view of the camera in the machine to be calibrated at least includes two calibration kits 10. The lifting rotating platform 201 is also used to rotate the machine to be calibrated, so that the camera in the machine to be calibrated collects the rotating image data of the calibration kit 10. The calibration device 30 is also used to determine the static image data of the calibration kit 10 according to the rotating image data, and determine the second relative position relationship according to the static image data and the first relative position relationship information.

[0079] It should be understood that when calibrating, the machine to be calibrated is opened into the lifting rotating platform 201, and the lifting rotating platform 201 is started to rotate one circle to collect image data to obtain rotating image data. In this process, the lifting rotating platform 201 can be adjusted to an appropriate height so that the camera in the machine to be calibrated can take pictures. The field of view of the camera during shooting at least includes two calibration kits 10. The specific calibration process can refer to the first embodiment described above, and will not be described here.

[0080] It should be noted that compared with the prior art solution which needs to create a high-precision calibration site or use a high-precision movable mechanical hand to realize high-precision prior information, the development cost is higher. The embodiment only needs to customize high-precision calibration plates and print two-dimensional codes to make calibration kits, and develop a heavy rotating lifting platform, which can meet the requirements of joint calibration for calculation accuracy under different machine types and different camera conditions, realize the reuse of the calibration site, and has lower development cost.

[0081] In the embodiment, the rotating component 20 is a lifting rotating platform 201, the lifting rotating platform 201 can cover the whole machine to be calibrated, the lifting rotating platform 201 is used for parking the whole machine to be calibrated and adjusting the shooting height of the whole machine to be calibrated, so that at least two calibration kits 1 are included in the field of view of the camera in the whole machine to be calibrated, the whole machine to be calibrated is rotated, so that the camera in the whole machine to be calibrated collects the rotating image data of the calibration kit 10, and the calibration device 30 is also used for determining the static image data of the calibration kit 10 according to the rotating image data, and determining the second relative position relationship according to the static image data and the first relative position relationship information. The calibration targets used in the embodiment are flexible and have coding between each other, can be flexibly adjusted under different application scenarios, can meet the requirement of joint calibration on calculation precision under different camera conditions of different models, reduce the cost, and the embodiment adopts the rotating image collection mode, target information of any form and position can be collected, the global optimal solution can be obtained based on the constraint of multiple frame information, and high-precision calibration calculation is realized.

[0082] Referring to Figure 4 , Figure 4 FIG. 3 is a structural schematic diagram of a third embodiment of a cross-view joint calibration device.

[0083] As Figure 4 shown, the rotating component 20 includes a rotating manipulator 202 and an acquisition camera 203, the rotating manipulator 202 holds the acquisition camera 203, and the rotating manipulator 202 is arranged on the ceiling in the center of the calibration site. The rotating component 20 further includes a lifting platform 204, the lifting platform 204 can cover the whole machine to be calibrated, and the calibration kit 10 is located at the periphery of the lifting platform 204.

[0084] It should be noted that the rotating manipulator 202 is used for adjusting the height and the pitch angle of the acquisition camera 203, so that at least two calibration kits 10 are included in the field of view of the acquisition camera 203; and the rotating manipulator 202 is also used for rotating the acquisition camera 203, so that the acquisition camera 203 acquires the rotating image data of the calibration kit 10 in the rotating process.

[0085] It can be understood that the rotating manipulator 202 is usually suspended on the ceiling in the center of the calibration site, and the acquisition camera 203 is usually a high-resolution high-quality camera, the acquisition camera 203 is held by the rotating manipulator 202 and rotated to collect the rotating image data, in this process, the height and the pitch angle of the acquisition camera 203 are adjusted by the rotating manipulator 202, so that at least two calibration kits 10 are included in the field of view at the same time.

[0086] It should be understood that the lifting platform 204 in the embodiment is arranged in the center of the calibration site, and the size of the platform can basically cover all machine types. A plurality of calibration kits 10 are placed around the lifting platform 204 at corresponding positions. The lifting platform 204 generally has a lifting function, and the lifting platform 204 can be adjusted to a suitable height so that the camera in the machine to be calibrated can take a picture, and the field of view of the camera during the picture taking at least includes two calibration kits 10. The lifting platform 204 generally does not need to have a rotating function.

[0087] It should be noted that the lifting platform 204 is used to park the machine to be calibrated and adjust the shooting height of the machine to be calibrated, so that the camera of the machine to be calibrated collects static image data of the calibration kit 10, and the field of view of the camera in the machine to be calibrated at least includes two calibration kits 10. The calibration device 30 is also used to determine the second relative position relationship according to the static image data and the first relative position relationship information.

[0088] It can be understood that during calibration, the machine to be calibrated is opened into the lifting platform 204 for calibration, a group of images of multiple targets are recorded statically, and then the platform is left, without the need for rotation.

[0089] In a specific implementation, the relative position relationship between all calibration boards is obtained by using visual pose calculation and multi-frame superposition. Since the specification of the acquisition camera 203 is high, the relative position relationship of the prior information calculated is theoretically more reliable, and such prior information calibration only needs to be performed once. The specific calibration process can refer to the first embodiment, and details are not described herein.

[0090] In the embodiment, the rotating component 20 comprises a rotating manipulator 202, a collection camera 203 and a lifting platform 204, the rotating manipulator 202 clamps the collection camera 203, the rotating manipulator 202 is arranged on the ceiling in the center of the calibration site, the lifting platform 204 can cover the whole machine to be calibrated, the calibration kit 10 is located at the periphery of the lifting platform 204, the rotating manipulator 202 is used for adjusting the height and the pitch angle of the collection camera 203, so that the field of view of the collection camera 203 at least includes two calibration kits 10, the collection camera 203 is rotated, so that the rotating image data of the calibration kit 10 is acquired in the rotating process, the lifting platform 204 is used for parking the whole machine to be calibrated, and the shooting height of the whole machine to be calibrated is adjusted, so that the camera in the whole machine to be calibrated collects the static image data of the calibration kit 10, the field of view of the camera in the whole machine to be calibrated at least includes two calibration kits 10, and the calibration device 30 is also used for determining the second relative position relationship according to the static image data and the first relative position relationship information. The calibration targets used in the embodiment are flexible and have coding support between the targets, can be flexibly adjusted under different application scenarios, can meet the requirements of joint calibration on calculation precision under different models and different cameras, reduce the cost, and the rotating collection image mode is adopted in the embodiment, so that target information of any form and position can be collected, the global optimal solution can be obtained based on the constraint of multiple frames of information, and high-precision calibration calculation is realized.

[0091] The embodiment of the application provides a cross-view joint calibration method, referring to Figure 5 , Figure 5 FIG. 1 is a flowchart of a cross-view joint calibration method according to a first embodiment of the application.

[0092] In the embodiment, the cross-view joint calibration method comprises the following steps:

[0093] Step S10: acquiring rotating image data of a calibration kit, the calibration kit comprising a calibration board and a coding board.

[0094] It should be noted that the embodiment is applied to the cross-view joint calibration device described above, the cross-view joint calibration device comprising a calibration kit, a rotating component and a calibration device, the execution subject of the embodiment is the calibration device, the calibration device is provided with a cross-view joint calibration program, and the calibration of all cameras in the whole machine to be calibrated is realized by running the cross-view joint calibration program. The model of the whole machine to be calibrated and the cameras in the whole machine to be calibrated are not limited in the embodiment.

[0095] It can be understood that the rotating image data is image data obtained by rotating, and contains the position information of all calibration kits. The calibration kit comprises a calibration board and a coding board, and the coding board has a two-dimensional code pattern.

[0096] Step S20: encode the calibration board according to the two-dimensional code information of the encoding board in the rotation image data to obtain encoding information.

[0097] It should be understood that the two-dimensional code information is the two-dimensional code pattern on the encoding board, and the encoding information is the encoding of each calibration board. In this embodiment, the two-dimensional code pattern on the encoding board is used to encode all the calibration boards.

[0098] Step S30: determine the first relative position relationship between the calibration board and the starting calibration board according to the rotation image data and the encoding information.

[0099] It should be noted that the first relative position relationship refers to the relative position relationship between the calibration boards unified to the same coordinate system, that is, the relative position relationship of all calibration boards relative to the starting calibration board after being unified to the starting calibration board. The starting calibration board usually refers to the first calibration board, which can be flexibly adjusted according to actual needs, and this embodiment does not limit it. The first relative position relationship can be considered as the prior information required for current joint calibration.

[0100] In specific implementation, based on the obtained rotation image data and the encoding of each calibration board, the relative position relationship between two adjacent calibration boards is calculated according to pose estimation and multi-frame superposition, and then the relative position relationship between the calibration boards is determined and converted to the same coordinate system to obtain the relative position relationship relative to the starting calibration board, thereby realizing the calibration of the prior information.

[0101] Step S40: determine the second relative position relationship between the cameras in the whole machine to be calibrated according to the first relative position relationship.

[0102] It can be understood that the second relative position relationship refers to the relative position relationship between all cameras in the whole machine to be calibrated.

[0103] Further, the step S40 includes: obtaining static image data of the calibration kit, extracting the calibration board in the static image data to obtain calibration extraction information; determining the fifth relative position relationship between the camera and the calibration board according to the calibration extraction information and the first relative position relationship; and determining the second relative position relationship based on the fifth relative position relationship.

[0104] It should be understood that the static image data refers to the static image frame collected by the camera in the whole machine to be calibrated, the calibration extraction information refers to the calibration board information extracted from the image, and the fifth relative position relationship refers to the relative position relationship between the camera and the calibration board in the whole machine to be calibrated, that is, the relative position relationship between the camera and the prior information coordinate system.

[0105] In a specific implementation, according to the rotating image data, the prior information of the calibration site is determined, at this time, the calibration board captured by any frame of the camera under the static state can be extracted, the relative position relationship between all cameras relative to the prior information coordinate system is calculated, and then the relative position relationship between the cameras is obtained, and if a plurality of frames are superimposed, a more robust global optimal solution can also be obtained.

[0106] Further, the second relative position relationship is determined based on the fifth relative position relationship, including: determining a sixth relative position relationship between the camera and the starting camera according to the fifth relative position relationship; and obtaining the second relative position relationship based on a chain strategy and the sixth relative position relationship.

[0107] It should be noted that the sixth relative position relationship refers to the relative position relationship obtained by unifying the fifth relative position relationship to the same coordinate system, that is, the relative position relationship of all cameras relative to the starting camera after unifying all cameras to the starting camera. The starting camera usually refers to the first camera / front-view camera, which can also be flexibly adjusted according to actual needs, and the embodiment does not limit this.

[0108] In a specific implementation, in order to obtain the second relative position relationship, the fifth relative position relationship is unified to the coordinate system corresponding to the starting camera to obtain the sixth relative position relationship, and then the relative position relationship between all cameras, that is, the second relative position relationship, is obtained according to the chain rule (chain strategy).

[0109] It can be understood that both the prior relative position relationship between the calibration boards and the relative position relationship between the cameras can obtain a global optimal solution based on the constraint of multiple frames of information. Compared with the existing technical solution which adopts single frame calculation, it is easy to be affected by light and imaging stability, etc. The embodiment can obtain higher robustness.

[0110] Step S50: calibrating the cameras of the to-be-calibrated whole machine according to the second relative position relationship.

[0111] In a specific implementation, the coordinates of the cameras are calculated according to the relative position relationship between all cameras, and the calibration of the cameras in the to-be-calibrated whole machine is realized.

[0112] As shown in the overall flowchart, first, the rotating lifting platform is arranged and adjusted, the calibration kit is arranged and adjusted, then the rotating image data is recorded, and finally the multi-camera coordinates are calculated by using pose estimation. Figure 6

[0113] ​It should be understood that in the prior art, when the machine type is replaced for calibration, the height, orientation and distance from the calibration board will change, causing the calibration board to not be well covered in the field of view, and thus the accuracy of the multi-camera joint calibration calculation to decrease. In the case of replacing the camera specifications at the same position, the camera specifications such as resolution, field of view and the like change, and then the originally clear imaging target can not be within the depth of field of the new camera, causing the extraction accuracy of the target in the image to decrease, and thus the accurate calibration parameters cannot be calculated. In the present embodiment, the position of the calibration kit 10 can be flexibly moved, and has coding support, has the characteristics of being flexibly adjusted and the accuracy advantage under different application scenarios, and can meet the requirements of joint calibration calculation accuracy under different machine types and different camera conditions. In addition, the prior art fixes the prior information target, so that the camera in different depth ranges can only face the target array of the same depth for detection and calculation of the calibration parameters, which will cause the calibration accuracy to decrease. In the present embodiment, the targets can be placed at different heights, distances and positions, and due to the advantage of rotating collection, any form and position of target information can be collected, and different specifications of cameras can collect appropriate target images at corresponding depth positions, and thus high-precision calibration calculation can be achieved.

[0114] In the present embodiment, by acquiring the rotating image data of the calibration kit, the calibration board is coded according to the two-dimensional code information of the coding board in the rotating image data to obtain coding information, the first relative position relationship between the calibration board and the starting calibration board is determined according to the rotating image data and the coding information, and the second relative position relationship between the cameras in the whole machine to be calibrated is determined according to the first relative position relationship; and the cameras of the whole machine to be calibrated are calibrated according to the second relative position relationship. The prior information of the traditional calibration site is usually fixed, and the flexibility is poor. After the machine type or the camera is replaced, it is difficult to ensure the accuracy of the calibration, and the prior information of the calibration site needs to be re-constructed, which has a high cost. The calibration targets used in the present embodiment are flexible and have coding support between each other, and can be flexibly adjusted under different application scenarios, and can meet the requirements of joint calibration calculation accuracy under different machine types and different camera conditions, reduce the cost, and the present embodiment adopts the rotating image collection mode, any form and position of target information can be collected, the global optimal solution can be obtained based on the constraint of multiple frames of information, and high-precision calibration calculation can be achieved.

[0115] Reference Figure 7 , Figure 7 is a flowchart of a second embodiment of a cross-view joint calibration method of the present application.

[0116] Based on the above embodiment, the step S30 comprises:

[0117] Step S301: determining a third relative position relationship between adjacent calibration boards according to the rotation image data and the encoding information.

[0118] Further, the step S301 comprises:

[0119] determining a relative transformation between adjacent calibration boards according to the rotation image data and the encoding information.

[0120] It should be noted that the relative transformation between adjacent calibration boards is also the relative transformation between two adjacent calibration boards, for example, the relative transformation between the calibration board 0 and the calibration board 1 is [R, t] 0,1 By analogy, the relative transformation between all adjacent calibration boards can be obtained as follows:

[0121]

[0122] In the formula, 0-n are the encodings of the calibration boards respectively, and [R, t] represents the relative transformation.

[0123] According to the rotation image data and the encoding information, the observation constraint information of the calibration board is determined.

[0124] It can be understood that the observation constraint information refers to the constraint information collected by each calibration board. Based on the constraint of multiple sets of observation data, the m sets of constraints collected by the calibration board with the encoding i-1 are respectively:

[0125]

[0126] In the formula, represents the mth constraint of the calibration board with the encoding i-1. Similarly, the m sets of constraints collected by the calibration board with the encoding i are respectively:

[0127]

[0128] Based on the relative transformation between the adjacent calibration boards and the observation constraint information of the calibration board, an iterative solution corresponding to the third relative position relationship is determined.

[0129] It should be understood that the relative position relationship [R, t] i-1,i between the i-1th calibration board and the i-th calibration board can be obtained by least squares method when loop optimization is performed as follows:

[0130]

[0131] In the formula, if represents the coordinates of the i-th calibration board in the camera coordinate system at time j, then:

[0132]

[0133] Optimizing and decomposing the iterative solution corresponding to the third relative position relationship to obtain the third relative position relationship.

[0134] It should be noted that the solving problem of the relative position relationship [R, t] i-1,i between the i-1th calibration plate and the ith calibration plate is converted into an optimization problem, and is calculated by solving the centroid and SVD (Singular Value Decomposition) decomposition:

[0135]

[0136] It can be understood that the third relative position relationship between all adjacent calibration plates can be obtained according to the above formula.

[0137] Step S302: determining the first relative position relationship according to the third relative position relationship.

[0138] Further, the step S302 comprises:

[0139] determining a fourth relative position relationship between calibration plates according to the third relative position relationship.

[0140] It should be understood that according to the obtained third relative position relationship, the fourth relative position relationship between all calibration plates and calibration plates can be calculated, so as to obtain:

[0141]

[0142] According to the preset number and the encoding information, a starting calibration plate is determined, and the encoding of the starting calibration plate is usually 0.

[0143] It should be noted that the initial calibration plate is selected according to actual needs.

[0144] Based on the chain strategy and the fourth relative position relationship, the first relative position relationship between the calibration plate and the starting calibration plate is determined.

[0145] It can be understood that the first relative position relationship is finally calculated by the chain rule, as follows:

[0146] [R,t] 0,k = [R,t] k-1,k … [R,t] 2,3 [R,t] 1,2 [R,t] 0,1

[0147] In the formula, [R,t] 0,kindicates a first relative position relationship between the calibration board coded as k and the start calibration board coded as 0.

[0148] In the embodiment, the relative transformation between adjacent calibration boards is determined according to the rotation image data and the coding information, the observation constraint information of the calibration board is determined according to the rotation image data and the coding information, the iterative solution corresponding to the third relative position relationship is determined based on the relative transformation between adjacent calibration boards and the observation constraint information of the calibration board, the iterative solution corresponding to the third relative position relationship is optimized and decomposed to obtain the third relative position relationship, the fourth relative position relationship between the calibration boards is determined according to the third relative position relationship, the start calibration board is determined according to the preset number and the coding information, and the first relative position relationship between the calibration board and the start calibration board is determined based on the chain strategy and the fourth relative position relationship. The calibration target used in the embodiment is flexible and has coding between targets. The calibration target can be flexibly adjusted under different application scenarios, can meet the requirements of joint calibration on calculation precision under different models and different cameras, reduces the cost, and the embodiment adopts the mode of rotating and collecting images. Target information in any form and position can be collected, the global optimal solution can be obtained based on the constraint of multiple frames of information, high-precision calibration calculation is realized.

[0149] In addition, the embodiment of the present application further provides a cross-view joint calibration device, which comprises the cross-view joint calibration apparatus and applies the steps of the cross-view joint calibration method.

[0150] It should be understood that the above is only an example, and does not constitute any limitation on the technical solutions of the present application. In specific applications, those skilled in the art can set it according to the needs, and the present application does not limit it.

[0151] It should be noted that the above-described workflow is only illustrative and does not limit the scope of protection of the present application. In actual application, those skilled in the art can select part or all of them to achieve the purpose of the embodiment according to actual needs, which is not limited here.

[0152] In addition, technical details not described in detail in the embodiment can be referred to the cross-view joint calibration method provided by any embodiment of the present application, which will not be described here.

[0153] Moreover, it is to be understood that the phraseology and terminology used herein is for the purpose of description and should not be regarded as limiting. The use of "including" "comprising" or "having" and variations thereof herein is intended to encompass the presence of one or more recited elements or steps and not the exclusion of any other integers or steps. The use of "including", "comprising", "having" and "with" and variations thereof herein is intended to encompass the presence of one or more recited elements or steps and not the exclusion of any other integers or steps.

[0154] The above-mentioned embodiment numbers of the present application are only for description, and do not represent the advantages or disadvantages of the embodiments.

[0155] Those skilled in the art can clearly understand the above-mentioned embodiment methods by means of software and the necessary general hardware platform, of course, they can also be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as a read only memory (ROM) / RAM, a magnetic disk, an optical disk), and includes a plurality of instructions for making a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) execute the methods described in the embodiments of the present application.

[0156] The above is only the preferred embodiment of the present application, and does not limit the patent scope of the present application, and any equivalent structure or equivalent process transformation, or direct or indirect application in other related technical fields, which is made by using the content of the specification and drawings of the present application, is also included in the patent protection scope of the present application.

Claims

1. A cross-view joint calibration device, characterized in that, The cross-view joint calibration device includes a calibration kit, a rotating component, and a calibration device. The rotating component is located in the center of the calibration site, and the calibration kit is arranged around the rotating component. The calibration kit includes a calibration plate and an encoding plate, and the encoding plate is provided with QR code information. The rotating component is used to acquire the rotational image data of the calibration kit; The calibration device is used to encode the calibration board according to the QR code information of the encoding board in the rotated image data to obtain the encoding information; The calibration device is further configured to determine a first relative positional relationship between the calibration plate and the starting calibration plate based on the rotated image data and the encoded information; The calibration device is further configured to determine a second relative positional relationship between cameras in the calibrated unit based on the first relative positional relationship; The calibration device is also used to calibrate the camera in the machine to be calibrated according to the second relative positional relationship; The calibration device is further configured to acquire static image data of the calibration kit, extract the calibration board from the static image data to obtain calibration extraction information, determine a fifth relative positional relationship between the camera and the calibration board based on the calibration extraction information and the first relative positional relationship, and determine a second relative positional relationship based on the fifth relative positional relationship.

2. The cross-view joint calibration device as described in claim 1, characterized in that, The rotating component is a lifting and rotating platform, which can cover the machine to be calibrated. The lifting and rotating platform is used to park the camera to be calibrated and adjust the shooting height of the camera to be calibrated so that the field of view of the camera in the camera to be calibrated includes at least two calibration kits. The lifting and rotating platform is also used to rotate the machine to be calibrated so that the camera of the machine to be calibrated can acquire rotational image data of the calibration kit; The calibration device is further configured to determine the static image data of the calibration kit based on the rotated image data, and to determine the second relative position relationship based on the static image data and the first relative position relationship information.

3. The cross-view joint calibration device as described in claim 1, characterized in that, The rotating component includes a rotating manipulator and a data acquisition camera. The rotating manipulator holds the data acquisition camera and is mounted on the ceiling in the center of the calibration site. The rotating manipulator is used to adjust the height and pitch angle of the acquisition camera so that at least two calibration kits are included in the field of view of the acquisition camera; The rotating manipulator is also used to rotate the acquisition camera so that the acquisition camera acquires rotational image data of the calibration kit during the rotation process.

4. The cross-view joint calibration device as described in claim 3, characterized in that, The rotating component also includes a lifting platform, which can cover the machine to be calibrated, and the calibration kit is located around the lifting platform. The lifting platform is used to park the device to be calibrated and adjust the shooting height of the device to be calibrated so that the camera of the device to be calibrated can collect static image data of the calibration kit. The field of view of the camera in the device to be calibrated includes at least two calibration kits. The calibration device is further configured to determine the second relative position relationship based on the static image data and the first relative position relationship information.

5. A cross-view joint calibration method, characterized in that, The cross-view joint calibration method is applied to the cross-view joint calibration device as described in any one of claims 1 to 4, wherein the cross-view joint calibration method comprises: Acquire rotational image data of a calibration kit, the calibration kit including a calibration board and an encoding board; The calibration board is encoded based on the QR code information of the encoding board in the rotated image data to obtain the encoded information; Based on the rotated image data and the encoded information, a first relative positional relationship between the calibration board and the starting calibration board is determined; Based on the first relative positional relationship, determine the second relative positional relationship between the cameras in the calibration system; The camera of the device to be calibrated is calibrated according to the second relative position relationship.

6. The method as described in claim 5, characterized in that, Determining the first relative positional relationship between the calibration board and the starting calibration board based on the rotated image data and the encoded information includes: Based on the rotated image data and the encoded information, a third relative positional relationship between adjacent calibration plates is determined; The first relative position relationship is determined based on the third relative position relationship.

7. The method as described in claim 6, characterized in that, Determining the third relative positional relationship between adjacent calibration plates based on the rotated image data and the encoded information includes: Based on the rotated image data and the encoded information, the relative transformation between adjacent calibration plates is determined; Based on the rotated image data and the encoded information, the observation constraint information of the calibration plate is determined; Based on the relative transformation between the adjacent calibration plates and the observation constraint information of the calibration plates, the iterative solution corresponding to the third relative position relationship is determined; The iterative solution corresponding to the third relative position relationship is optimized and decomposed to obtain the third relative position relationship.

8. The method as described in claim 6, characterized in that, Determining the first relative position relationship based on the third relative position relationship includes: Based on the third relative positional relationship, determine the fourth relative positional relationship between the calibration plates; The starting calibration board is determined based on the preset number and the encoding information; Based on the chain strategy and the fourth relative position relationship, the first relative position relationship between the calibration board and the starting calibration board is determined.

9. The method as described in claim 5, characterized in that, The step of determining the second relative positional relationship between cameras in the calibrated system based on the first relative positional relationship includes: Obtain static image data of the calibration kit, extract the calibration board from the static image data, and obtain calibration extraction information; Based on the calibration extraction information and the first relative position relationship, a fifth relative position relationship between the camera and the calibration board is determined; Based on the fifth relative position relationship, the second relative position relationship is determined.

10. A cross-view joint calibration device, characterized in that, The device applies the steps of the cross-view joint calibration method as described in any one of claims 5 to 9.

Images