Apparatus and method for calibrating external parameters of multiple cameras, storage medium, and electronic device

Abstract

The present application discloses an apparatus and method for calibrating extrinsic parameters of multiple cameras, a storage medium, and an electronic device. [Solution] The apparatus includes: a plurality of moving calibration plates provided within a setting space for completing calibration of the extrinsic parameters of all cameras mounted on a moving carrier on which a plurality of cameras are mounted, the moving calibration plates being mounted on slide rails that are respectively mounted on wall surfaces within the setting space; and a control device for controlling each of the plurality of moving calibration plates to slide along the slide rail corresponding to the moving calibration plate. By providing multiple moving calibration plates within the setting space, an embodiment of the present disclosure achieves simultaneous calibration of the extrinsic parameters of the multiple cameras mounted on the moving carrier, improving the noise resistance and interference resistance capabilities of the extrinsic parameter calibration and providing relatively high robustness, effectively improving the accuracy of the calibration results and the calibration efficiency, while saving the cost and time of the calibration.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to a Chinese patent application, application number 202110700660.0, entitled "Multiple camera external parameter calibration apparatus and method, storage medium, and electronic device," filed with the State Intellectual Property Office of China on June 23, 2021, the entire contents of which are incorporated herein by reference.

[0002] The present disclosure relates to the technical field of extrinsic parameter calibration, and in particular to an apparatus and method for calibrating extrinsic parameters of multiple cameras, a storage medium, and an electronic device. [Background technology]

[0003] Traffic safety has always been one of the important issues people are concerned about, and many traffic accidents occur on highways every year, causing serious casualties and huge economic losses, so it is of great significance to develop advanced driving assistance systems. The imaging module acts as the eye of the driving assistance system, and is used to measure the surrounding environment and make driving decisions based on the external surrounding environment. In the process of measuring the camera image, it is necessary to establish a geometric model of the camera image formation to determine the correlation between the three-dimensional geometric position of a point on the surface of a spatial object and its corresponding point in the image, and the model parameters in these geometric models can be camera parameters.

[0004] In most cases, camera parameters must be obtained through experiments and calculations, and the process of obtaining these parameters is called camera calibration. Camera calibration is a very important part in both image measurement and machine vision applications, and the accuracy of the calibration results and the stability of the algorithm directly affect the accuracy of the results obtained by the camera operation. Therefore, a good camera calibration is the prerequisite for the good performance of subsequent operations, and improving the accuracy of the calibration is the focus of scientific research. Summary of the Invention [Problem to be solved by the invention]

[0005] In order to solve the above technical problems, the present disclosure is proposed. The embodiments of the present disclosure provide an apparatus and method for calibrating external parameters of multiple cameras, a storage medium, and an electronic device. [Means for solving the problem]

[0006] According to one aspect of the embodiment of the present disclosure, There is provided a calibration device for external parameters of multiple cameras, the device including: a plurality of moving calibration plates arranged within a setting space for completing calibration of external parameters of all of the cameras on a moving carrier on which the plurality of cameras are mounted, each of the moving calibration plates being arranged on a slide rail which is arranged on a wall surface within the setting space; and a control device for controlling each of the plurality of moving calibration plates to slide along the slide rail corresponding to the moving calibration plate.

[0007] According to another aspect of the present disclosure, A method for calibrating external parameters of multiple cameras is provided, the method including the steps of responding to a moving carrier carrying multiple cameras entering a setting space, controlling multiple moving calibration plates in the setting space to move to positions corresponding to each of the multiple cameras, and realizing calibration of the external parameters of the multiple cameras based on the multiple moving calibration plates.

[0008] According to another aspect of the present disclosure, A method for calibrating external parameters of multiple cameras is provided, the method including: acquiring images on a corresponding calibration plate based on each of a plurality of cameras mounted on a moving carrier to obtain a plurality of images, each of the cameras corresponding to one image; and determining target external parameter information of each of the multiple cameras based on each of the plurality of images and internal parameter information of each of the multiple cameras.

[0009] According to yet another aspect of an embodiment of the present disclosure, there is provided a computer-readable storage medium having stored therein a computer program for executing the method for calibrating external parameters of multiple cameras described in any of the above embodiments.

[0010] According to yet another aspect of the present disclosure, a processor and a memory for storing instructions executable by the processor; An electronic device is provided, in which the processor is adapted to read the executable instructions from the memory and execute the instructions to implement the multi-camera extrinsic parameter calibration method according to any of the above embodiments. Effect of the Invention

[0011] Based on the multiple-camera external parameter calibration apparatus and method, storage medium, and electronic device provided in the above embodiments of the present disclosure, by providing multiple moving calibration plates in the setting space, simultaneous calibration of the external parameters of multiple cameras provided on a moving carrier is realized, the noise resistance and interference resistance ability of the external parameter calibration is improved, and the calibration has relatively high robustness, which effectively improves the accuracy of the calibration result and the calibration efficiency, and saves the calibration cost and time. [Brief description of the drawings]

[0012] [Figure 1a] FIG. 1 illustrates a scene from a mass production calibration site to which the present disclosure is applicable. [Figure 1b] FIG. 1b is a plan view of the production calibration site shown in FIG. [Diagram 2] FIG. 13 is a diagram showing the structure of a mechanically movable position limiting guide rail in a mass production calibration site to which the present disclosure is applied. [Diagram 3] FIG. 2 is a diagram illustrating a structure of a multi-camera external parameter calibration device provided in an exemplary embodiment of the present disclosure. [Figure 4] 1 is a flowchart of a method for calibrating extrinsic parameters of multiple cameras provided in an exemplary embodiment of the present disclosure. [Diagram 5] 11 is a flowchart of a multi-camera extrinsic parameter calibration method provided in another exemplary embodiment of the present disclosure. [Figure 6] 6 is a flow chart of one of step 502 in the embodiment shown in FIG. 5 of the present disclosure. [Figure 7] FIG. 2 is a structural diagram of an electronic device provided in an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] The above and other objects, features and advantages of the present disclosure will become more apparent by describing the embodiments of the present disclosure in more detail with reference to the drawings. The drawings are used to further understand the embodiments of the present disclosure, are constituted as a part of the specification, and are for interpreting the present disclosure together with the embodiments of the present disclosure, and are not intended to limit the present disclosure. In the drawings, similar reference symbols generally represent similar components or steps. Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the drawings. Obviously, it should be understood that the described embodiments are only a part of the embodiments of the present disclosure, but are not all of the embodiments of the present disclosure, and the present disclosure is not limited to the exemplary embodiments described herein.

[0014] Unless otherwise specifically stated, the relative arrangement of components and steps, formulas and numerical values ​​described in these examples do not limit the scope of the present disclosure.

[0015] As can be understood by those skilled in the art, the terms "first", "second", etc. in the embodiments of the present disclosure are merely used to distinguish different steps, devices, modules, etc., and do not represent any particular technical meaning, nor do they represent a necessary logical order between them.

[0016] It should also be understood that in the embodiments of the present disclosure, "plurality" can mean two or more, and "at least one" can mean one, two, or more than two.

[0017] Additionally, any one of the components, data, or structures referred to in the embodiments of the present disclosure should generally be understood to be one or more unless expressly limited or suggested otherwise by the context.

[0018] In addition, the term "and / or" in this disclosure is merely for describing the related relationship of related objects and indicates that three relationships can exist, for example, A and / or B can indicate three cases: only A exists, A and B exist simultaneously, and only B exists. In addition, the character " / " in this disclosure generally indicates that the related objects before and after it are in an "or" relationship.

[0019] It should also be understood that the description of each embodiment in this disclosure will particularly emphasize the differences between the embodiments, and that identical or similar points may be cross-referenced and will not be described again for the sake of brevity.

[0020] It should also be understood that for ease of explanation, the dimensions of each part shown in the drawings are not drawn to scale.

[0021] The following description of at least one illustrative embodiment is merely interpretive in nature and is in no way intended to limit the disclosure and its application or uses.

[0022] Techniques, methods and apparatus known to those skilled in the relevant art need not be discussed in detail, but where appropriate, said techniques, methods and apparatus should be considered part of this specification.

[0023] It should be noted that in the following drawings, like symbols and letters represent like items, and thus, once an item is defined in one drawing, it need not be described further in subsequent drawings.

[0024] Embodiments of the present disclosure may be applied to electronic devices, such as terminals, computer systems, servers, etc., operable with many other general purpose or special purpose computing system environments and configurations. Examples of well-known terminals, computing systems, environments and / or configurations suitable for use with electronic devices, such as terminals, computer systems or servers, include, but are not limited to, personal computer systems, server computer systems, thin clients, fat clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, networked personal computers, small computer systems, large computer systems, and distributed cloud computing technology environments that include any of the above systems.

[0025] Electronic devices such as terminals, computer systems, servers, etc. may be described in the general context of computer system executable instructions (e.g., program modules) executed by a computer system. Generally, program modules may include routines, programs, object programs, components, logic, data structures, etc. that perform particular tasks or implement particular abstract data types. Computer systems / servers may be implemented in distributed cloud computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules may be located in storage media, including storage devices, of local or remote computing systems.

[0026] Overview of the Application In the process of realizing the present disclosure, the applicant has found that the general calibration methods in the prior art have at least the problems of complicated processes, poor noise resistance and interference resistance, and low calibration efficiency.

[0027] Exemplary System The static calibration method requires the use of a calibration object of known size, and obtains the internal and external parameters of the camera model through a certain algorithm by establishing the correspondence between the coordinate points of the calibration object and its image points. The calibration object can be divided into three-dimensional calibration object and planar calibration object. The three-dimensional calibration object can be calibrated by a single image, and the calibration accuracy is relatively high, and it is mostly applied to mass production calibration by camera modules in vehicle factories.

[0028] Among them, the static mass production calibration site can improve the reliability and accuracy of the calibration results, and greatly reduce the time cost of calibration.

[0029] Fig. 1a is a diagram showing the scene of the mass production calibration site to which the present disclosure is applied. In this embodiment, the specifications of the fixed calibration site are calculated according to the mass production vehicle model, and the ground 2D chessboard pattern (fixed calibration plate) 101 and the 3D wall chessboard pattern (moving calibration plate) 102 are respectively arranged, and Fig. 1b is a plan view of the mass production calibration site shown in Fig. 1a, in which the distribution of the 2D chessboard pattern 101 and the 3D wall chessboard pattern 102 in the mass production calibration site can be more intuitively understood.

[0030] The calibration site can be adapted for different vehicle models and calibrate the vehicle-mounted camera module using different camera mounting means.

[0031] The 3D wall chessboard pattern 102 is mounted on a PLC controlled slide rail 103, and the controller can translate the 3D wall chessboard pattern 102 up, down, left, and right as needed to within the visible range of the camera module for calibration.

[0032] The 3D wall chessboard pattern 102 has a relatively high movement accuracy, for example, the movement accuracy is 0.1 mm, and can realize control operations such as inching, long pressing or parameter setting by a controller.

[0033] The 3D wall chessboard pattern 102 can realize a one-click automatic return to origin function, and can read parameters according to the set parameters to move the chessboard pattern 102 to a specified position. The controller at the calibration site can realize functions such as creating, reading, deleting, and saving parameters.

[0034] Each 3D wall chessboard 102 pattern has a power failure self-locking function to prevent the 3D wall chessboard pattern 102 from slipping off.

[0035] A mechanically movable position-limiting guide rail 104 is arranged on the ground in the calibration site, and in one optional embodiment, FIG. 2 is a diagram showing the structure of the mechanically movable position-limiting guide rail in the mass production calibration site to which the present disclosure is applied. As shown in FIG. 2, the mechanically movable position-limiting guide rail can adjust its width according to the tire specifications of the vehicle model, and is used to control the position distance between the camera module installed on the vehicle and the calibration plate by limiting the position of the vehicle in the calibration site. However, the calibration plate may include at least one or more of a 2D chessboard pattern 101 and a 3D wall chessboard pattern 102.

[0036] An embodiment of the present disclosure limits the position of a vehicle within a calibration site based on a mechanically movable position limiting guide rail, and the position of the vehicle within the calibration site can be accurately determined through the position of the mechanically movable position limiting guide rail, thereby obtaining the distance between the target calibration plate and the position of the camera module, improving the accuracy of the calibration result, and saving the cost time of calibration.

[0037] Exemplary Apparatus 3 is a diagram showing the structure of a calibration device for external parameters of multiple cameras provided in an exemplary embodiment of the present disclosure. As shown in FIG. 3, the device provided in this embodiment includes multiple moving calibration plates 301 and a control device 302.

[0038] The multiple moving calibration plates 301 are used to complete the calibration of the external parameters of all the cameras on a moving carrier on which multiple cameras are mounted, which is installed in a set space.

[0039] Each of the movable calibration plates is mounted on a slide rail, and the slide rail is mounted on a wall surface within the setting space.

[0040] Optionally, the moving calibration plate in this embodiment can be understood with reference to the 3D wall chessboard pattern 102 in the embodiment shown in Fig. 1a, and the slide rail in this embodiment can be understood with reference to the PLC controlled slide rail 103 in the embodiment shown in Fig. 1a. Of course, in actual use, information such as the size, shape and internal grid size of the moving calibration plate can be freely set according to specific application scenarios, including, for example, information such as the direction and number of slide rails, and the setting process of these information is not limited to the embodiment provided in Fig. 1a.

[0041] The control device 302 is used to control each of the plurality of moving calibration plates 301 to slide along a slide rail corresponding to the moving calibration plate 301 .

[0042] The control device 302 in this embodiment may be any controller and is used to realize control operations for multiple moving calibration plates 301, such as inching, long pressing, or parameter setting, and this embodiment improves the accuracy of the calibration results when calibrating external parameters for the camera by using high-precision position control of the control device 302.

[0043] This embodiment provides a multi-camera external parameter calibration device, and by providing multiple movable calibration plates in the set space, it realizes simultaneous calibration of the external parameters of multiple cameras provided on a movable carrier. This embodiment completes the calibration of the external parameters in the set space and greatly reduces interference such as external noise, thereby improving the noise resistance and interference resistance capabilities of the external parameter calibration and having relatively high robustness, which effectively improves the accuracy of the calibration result and the calibration efficiency, and saves the cost and time of calibration.

[0044] In some optional embodiments, the control device 302 is further used to control the sliding between a plurality of connected slide rails, thereby controlling the moving calibration plate 301 to slide along the up, down, left, and right directions on a plane corresponding to the wall surface on which it is located.

[0045] In this embodiment, referring to the embodiment shown in Figure 1a, a 3D wall chessboard pattern 102 is mounted on a wall surface via a vertical sliding rail, and the 3D wall chessboard pattern 102 slides on the vertical sliding rail to realize the movement (up and down movement) of the moving calibration plate in the y-axis direction in the coordinate system of the plane on which the wall surface is located.

[0046] Optionally, the vertical slide rail is movably connected to a lateral slide rail arranged laterally, and the vertical guide rail slides on the lateral slide rail to realize the movement (left and right movement) of the movable calibration plate 301 in the x-axis direction in the coordinate system of the plane on which the wall surface is located. In this embodiment, by combining the lateral slide rail and the vertical slide rail, the movable calibration plate 301 can be moved arbitrarily in each direction in the plane corresponding to the wall surface on which the movable calibration plate 301 is located, and the accuracy of the calibration of external parameters is further improved.

[0047] Optionally, under the control of the control device 302, the moving calibration plate 301 can realize sliding in at least one of the following moving modes: the at least one moving mode includes a set unit move, a set length move, and a continuous move.

[0048] The control device 302 is further provided with a return button, and the multiple moving calibration plates 301 can be returned to their initial positions based on the control of the return button.

[0049] In this embodiment, the size of the set unit can be set according to the actual scene, and the smaller the set unit is, the higher the movement accuracy that can be achieved.For example, if the set unit is set to 1 mm, the set unit movement realizes a small distance and high accuracy of movement.However, when a large movement is required, the set unit movement may slow down the speed, therefore, this embodiment further provides a set length movement and continuous movement method.

[0050] Wherein, the set length movement is set by setting the movement distance of the moving calibration plate 301 in any direction directly in the control device, and the moving calibration plate 301 can be directly moved to the corresponding position based on the provided motor, and if the corresponding position is not accurate, it can be further fine-tuned by combining with the set unit movement, and the continuous movement is realized by controlling the designated moving calibration plate to continuously move in the designated direction by an operation such as long pressing of the control device, and stop when it reaches the designated position. If there is a small deviation between the reached position and the target position, it can be similarly fine-tuned by the set unit movement.

[0051] The control device of this embodiment is further provided with a return button for controlling one-click return of all the moving calibration plates 301, so that the next time the moving calibration plates need to be moved again, they can be moved from their initial positions, improving the accuracy and efficiency of the movement.

[0052] Optionally, each moving calibration plate 301 is provided with a locking device to lock the position of the chessboard calibration plate in the event of a power failure.

[0053] The locking device provided in this embodiment realizes a power failure self-locking function for the moving calibration plate, effectively avoiding the problem of the moving calibration plate sliding off due to a sudden power failure while the slide rail is in use, and improves the safety of the device.

[0054] In some optional embodiments, the size of the set space is set according to the size of the moving carrier.

[0055] The configuration space in this embodiment can be understood with reference to the calibration site shown in Fig. 1a, where the configuration space can be used to accommodate a moving carrier, such as a vehicle, and calibrate external parameters for multiple cameras mounted on the moving carrier, where the size of the configuration space matches the size of the moving carrier, such as the matching relationship between the calibration site in the embodiment shown in Fig. 1a and the vehicle in the calibration site.

[0056] In some optional embodiments, the apparatus provided in this embodiment further includes a position limiting guide rail, which may be provided on the ground of the setting space, and the width of the position limiting guide rail is set according to the moving carrier and is used to limit the position of the moving carrier within the setting space.

[0057] In this embodiment, the position of the moving carrier within the set space is limited by the position limiting guide rail.

[0058] Optionally, the action of the mechanically movable position limiting guide rail 104 arranged on the ground in the calibration site is shown, as shown in the embodiment of FIG. 1a, and the structure of the position limiting guide rail may refer to the embodiment of FIG. 2, and the width-adjustable position limiting guide rail realizes fixation suitable for multiple types of moving carriers, and the range of use of the device is increased.

[0059] In some optional embodiments, the apparatus provided in the present embodiment further includes a plurality of fixed calibration plates, which are mounted on the ground of the setting space and are arranged around the position limiting guide rail in the setting space.

[0060] The multiple fixed calibration plates in this embodiment can be understood with reference to the 2D chessboard pattern 101 placed on the ground in the embodiment shown in Figure 1a, and the distribution of the fixed calibration plates can be understood with reference to the plan view shown in Figure 1b. By distributing the fixed calibration plates around the periphery of the position limiting guide rail, an installation that surrounds the moving carrier is realized, which makes it easy to calibrate the external parameters of the camera capable of collecting ground images mounted on the moving carrier, and realizes simultaneous calibration of the external parameters of the multi-directional cameras on the moving carrier.

[0061] Exemplary Methods 4 is a flowchart of a method for calibrating external parameters of multiple cameras provided in an exemplary embodiment of the present disclosure. The method provided in this embodiment can be applied to an electronic device, and as shown in FIG. 4, the method includes the following steps 401 to 403.

[0062] Step 401 responds to a moving carrier carrying multiple cameras entering a setting space.

[0063] In this embodiment, the mobile carrier may be a mobile carrier, such as a vehicle, on which multiple cameras can be mounted, and the setup space may be a calibration site, as provided in the embodiment shown in FIG. 1a.

[0064] In step 402, a plurality of moving calibration plates in a setup space are controlled to move to positions corresponding to each of a plurality of cameras.

[0065] In one embodiment, the position of the camera on the moving carrier is fixed, and when the position of the moving carrier is fixed, the corresponding position in the setting space of each camera is determined, and a control device is used to control each moving calibration plate of the multiple moving calibration plates and correspond each moving calibration plate to one camera.

[0066] In step 403, calibration of extrinsic parameters of multiple cameras is realized based on multiple moving calibration plates.

[0067] Specifically, the external parameters of the camera may include the coordinate position of the camera in space and the attitude information of the camera, where the coordinate position is the coordinate of the x-axis, y-axis, and z-axis in the calibration coordinate system, and the attitude information may include the pitch angle, yaw angle, and roll angle, etc. The calibration coordinate system may be a world coordinate system or a coordinate system with an arbitrary point as the origin, for example, the arbitrary point is a point with the camera as the origin.

[0068] The multi-camera external parameter calibration device provided in this embodiment realizes simultaneous calibration of the external parameters of multiple cameras mounted on a mobile carrier by setting up multiple movable calibration plates in a set space, and completes the calibration of these external parameters within the set space, greatly reducing interference such as external noise, improving the noise resistance and interference resistance capabilities of the external parameter calibration, and having relatively high robustness, effectively improving the accuracy of the calibration results and the calibration efficiency, and saving the cost and time of calibration.

[0069] Optionally, based on the above embodiment, step 402 includes: The method may include controlling each of the plurality of moving calibration plates to slide on at least one slide rail, and moving the moving calibration plate to a position corresponding to each of the plurality of cameras.

[0070] In this embodiment, the relationship between the moving calibration plate and the slide rail can refer to the relationship between the 3D wall chessboard pattern 102 and the PLC controlled slide rail 103 in the embodiment provided in Fig. 1a, where each moving calibration plate is mounted on a slide rail, and the slide rail is mounted on the wall in the set space. The 3D wall chessboard pattern 102 moves on the vertical slide rail to realize the movement of the moving calibration plate in the y-axis direction (up and down movement) in the coordinate system of the plane on which the wall is located.

[0071] Optionally, the vertical slide rail is movably connected to a lateral slide rail provided laterally, and the vertical guide rail slides on the lateral slide rail to realize movement of the moving calibration plate in the x-axis direction (left and right movement) in the coordinate system of the plane on which the wall surface is located. In this embodiment, by installing the lateral slide rail and the vertical slide rail, the moving calibration plate 301 can be arbitrarily slid in each direction up, down, left and right on the plane corresponding to the wall surface on which it is located, thereby improving the accuracy of the calibration of external parameters.

[0072] Optionally, controlling each moving calibration plate of the plurality of moving calibration plates to slide on a plurality of slide rails comprises controlling each moving calibration plate of the plurality of moving calibration plates to slide on a plurality of slide rails in at least one movement mode, where the at least one movement mode includes a set unit movement, a set length movement, and a continuous movement.

[0073] In this embodiment, the size of the set unit can be set according to the actual scene, and the smaller the set unit is, the higher the movement accuracy that can be achieved.For example, if the set unit is set to 1 mm, the set unit movement realizes a small distance and high accuracy of movement.However, when a large movement is required, the set unit movement may slow down the speed, therefore, this embodiment further provides a set length movement and continuous movement method.

[0074] Among them, the set length movement is achieved by setting the movement distance in any direction of the moving calibration plate 301 directly in the control device, and then moving the moving calibration plate 301 directly to the corresponding position based on the provided motor. If the corresponding position is not accurate, it can be further fine-tuned by combining with the set unit movement. The continuous movement is achieved by controlling the moving calibration plate to move continuously in a specified direction by an operation such as a long press on the control device, and stop when it reaches the specified position. If there is a small deviation between the reached position and the target position, fine adjustment can be similarly made by the set unit movement.

[0075] Optionally, the method provided in this embodiment further comprises, in response to completion of the calibration of the extrinsic parameters of the multiple cameras, returning the multiple moving calibration plates to their initial positions along the slide rails.

[0076] Optionally, the process of each moving calibration plate returning to its initial position may include determining a moving distance of the moving calibration plate in at least one direction based on a distance between the coordinates of the current position of the moving calibration plate and the coordinates of the initial position, and returning the moving calibration plate to its initial position based on the moving distance in the at least one direction. In this embodiment, a return button can control one-click return of all moving calibration plates, so that the moving calibration plate can be moved from its initial position next time when it needs to be moved, thereby improving the accuracy and efficiency of the movement.

[0077] 5 is a flowchart of a method for calibrating external parameters of multiple cameras provided in another exemplary embodiment of the present disclosure. The method can be applied to an electronic device, and as shown in FIG. 5, the method includes the following steps 501-502.

[0078] In step 501, image collection is performed on a corresponding calibration plate based on each camera among a plurality of cameras provided on a moving carrier, and a plurality of images are obtained.

[0079] Here, each camera corresponds to one calibration plate, and each calibration plate collects one image, so multiple cameras collect multiple images.

[0080] Optionally, each camera in this embodiment corresponds to at least one calibration plate, which may include a moving calibration plate and / or a fixed calibration plate. The camera may include, but is not limited to, a general camera, a fisheye camera, a 3D camera, etc., and any device capable of realizing an image collection function falls within the scope of the camera described in this disclosure.

[0081] In step 502, target extrinsic parameter information for each camera of the plurality of cameras is determined based on each image of the plurality of images and the intrinsic parameter information for each camera of the plurality of cameras.

[0082] The internal parameter information of the camera includes information such as the distortion value, center point, focal length, etc. of the camera, which is specific to each camera and is fixed at the time of shipment of the device, and is known information of each camera. In this embodiment, the target external parameter information of the camera can be determined according to the correspondence relationship between the image coordinates in the collected image and the coordinates in the real space.

[0083] The external parameter calibration method of multiple cameras provided in the embodiments of the present disclosure uses multiple cameras to simultaneously collect and calibrate images, thereby realizing simultaneous calibration of the external parameters of multiple cameras on a moving carrier, and realizing mass production calibration for the same moving carrier (such as a vehicle, etc.), which has relatively high applicability and can be applied to mass production calibration of multiple different moving carriers, effectively improving the accuracy of the calibration result and improving the calibration efficiency. For example, the external parameter calibration of a panoramic camera module consisting of four fisheye cameras is realized at one time to form a panoramic stitching, and the calibration plate of the vehicle-mounted camera module consisting of four fisheye cameras is located on the ground (2D chessboard pattern shown in FIG. 1a), and the size of the site is determined according to the mass production vehicle model, and the external parameter calibration of a camera module consisting of eight narrow-angle cameras can be realized at one time.

[0084] As shown in FIG. 6, based on the embodiment shown in FIG. 5 above, step 502 may include the following steps 5021-5022.

[0085] In step 5021, for each camera of the multiple cameras, first external parameter information in a first coordinate system of the camera is determined based on an image corresponding to the camera and the internal parameter information of the camera.

[0086] In this embodiment, after obtaining the correspondence between the coordinates of a point in an image in the world coordinate system and the pixel coordinates of the point in the image, as well as the internal parameter information of the camera that collects the image, a first external parameter information in a first coordinate system of the camera can be determined, and the first coordinate system may be a coordinate system with the camera as its origin.

[0087] In step 5022, coordinate system transformation is performed on the first external parameter information of the camera to obtain target external parameter information in the second coordinate system of the camera.

[0088] In this embodiment, when the first coordinate systems are device coordinate systems corresponding to each camera, the coordinate systems of the obtained sets of first external parameter information are different, and therefore it is impossible to realize the determination of the relationship to the entire camera module. To solve this problem, the first external parameter information corresponding to each camera is converted from the corresponding first coordinate system to a second coordinate system to which all cameras uniformly correspond, and the second coordinate system is the camera coordinate system. At this time, the external parameter information of all cameras is in the same coordinate system, and the annotation of the entire external parameter information of the camera module installed on the moving carrier is realized.

[0089] In some alternative embodiments, the plurality of cameras includes a plurality of first cameras and a plurality of second cameras, and the calibration plates include a fixed calibration plate and a moving calibration plate, where the first images collected by each of the first cameras correspond to the plurality of fixed calibration plates, and the second images collected by each of the second cameras correspond to one moving calibration plate.

[0090] Step 5021 in the above embodiment may include a step of determining first external parameter information of the first camera in a first coordinate system based on the first image and the internal parameter information of the first camera, and a step of determining first external parameter information of the second camera in the first coordinate system based on the second image and the internal parameter information of the second camera.

[0091] Optionally, in this embodiment, two kinds of cameras are included, such as a first camera and a second camera. Furthermore, the first camera is a fisheye camera, and the second camera is a general camera. Different cameras can correspond to different types of calibration plates, and the fisheye camera can correspond to a fixed calibration plate, and the general camera can correspond to a moving calibration plate, and the images collected by different types of cameras may be different, for example, the fisheye camera has a large viewing angle range, and the viewing angle can generally reach 220° or 230°, which creates conditions for photographing a wide range of scenery at a close distance; the fisheye camera can bring about a very strong perspective effect when approaching the subject, and by emphasizing the contrast due to the change in size according to the distance of the subject, the captured image has a powerful impact; the fisheye camera has a fairly deep depth of field, which is helpful in expressing the effect of the deep depth of field of the photo.

[0092] This embodiment provides different types of calibration plates for different kinds of cameras to improve the accuracy of the acquired first external parameter information.

[0093] Optionally, in the above embodiment, the step of determining first external parameter information in the first coordinate system of the first camera based on the first image and the internal parameter information of the first camera includes a step of determining a mapping relationship between a first setting point in the world coordinate system and a first setting point in an image coordinate system corresponding to the first image in the fixed calibration plate based on an overlapping area between images corresponding to each pair of first cameras among the multiple first cameras, and the internal parameter information of each of the first cameras.

[0094] Here, the first cameras of each pair are two first cameras adjacent in position.

[0095] For example, when the mobile carrier is a vehicle and the first camera is a fisheye camera, the vehicle may be provided with four fisheye cameras, one at the front of the vehicle, one at the rear of the vehicle, and one on each side of the vehicle body, and a mapping relationship from any one point on the vehicle surface to a 2D point on the ground may be calculated based on the internal parameter information of the four fisheye cameras and the overlapping areas of the images between two of the four fisheye cameras. The advantages of using a fisheye camera as the first camera are as follows: The viewing angle of the fisheye camera is large, and when four fisheye cameras are provided in four directions of the vehicle body (each direction corresponds to 180 degrees), the viewing angle of the fisheye camera can reach 220° or 230°, so that there is always an overlapping area between two images captured by two adjacent fisheye cameras, and further, a mapping relationship for mapping one point on the surface of a spatial solid object to a 2D point on the ground is calculated based on the overlapping areas.

[0096] Based on a mapping relationship between the first set points in the world coordinate system and the first set points in the image coordinate system, external parameter information in the first coordinate system of each first camera of the pair of first cameras is determined.

[0097] In this embodiment, on the premise that the internal parameter information of the fisheye camera is known, including distortion value, center point, focal length information, etc., the correspondence between a point in the world coordinate system space and the image pixel coordinates is found based on the internal parameter information of the camera, i.e., the correspondence between a point in the actual space (in this embodiment, a point on the fixed calibration plate) and the pixel coordinate of the point in the image collected by the camera, and the spatial orientation of the first camera in the world coordinate system is calculated by recognizing the corner points in the fixed calibration plate based on the first camera.

[0098] For example, based on establishing a world calibration coordinate system for the position of the vehicle body where the camera modules each consisting of four cameras are located, obtain the corner point distance information of the upper left corner of each corresponding set of chessboard patterns in the calibration coordinate system, for example, measure the spatial position distance of the upper left inner corner point of each chessboard pattern corresponding to each camera among the multiple cameras in the corresponding calibration coordinate system by using a laser level or a tape measure, and obtain the distance in the world coordinate system between each first camera and the first set point in the corresponding calibration plate, as well as the internal parameter information of the first camera, and the mapping relationship between the first set point in the world coordinate system and the first set point in the image coordinate system, and then obtain the first external parameter information of the first camera, where the first coordinate system is a camera coordinate system with the first camera as the origin. In this embodiment, the simultaneous calibration of four fisheye cameras is realized by a fixed calibration plate installed on the ground, and the calibration speed of the cameras is improved.

[0099] Optionally, in the above embodiment, the step of determining first external parameter information in a first coordinate system of the second camera based on the second image and the internal parameter information of the second camera includes a step of determining a mapping relationship between a second setting point in the world coordinate system and a second setting point in an image coordinate system corresponding to the second image in the moving calibration plate based on the second image and the internal parameter information of the second camera.

[0100] Optionally, the second set point may be a corner point in the moving calibration plate, for example a corner point located in the upper left corner of the moving calibration plate.

[0101] The first external parameter information in the first coordinate system of the camera is determined based on the mapping relationship between the second set point in the world coordinate system and the second set point in the image coordinate system.

[0102] This embodiment provides calibration for at least eight narrow-angle cameras. For example, when the mobile carrier is a vehicle, eight narrow-angle cameras are installed at the front and rear of the vehicle, near the left front door of the vehicle, near the right front door of the vehicle, near the left rear wheel of the vehicle, near the right rear wheel of the vehicle, the left corner of the front and the right corner of the front, respectively, and based on the corresponding wall chessboard pattern within the visible range of the eight narrow-angle cameras, corner point recognition in the chessboard pattern is performed to extract pixel coordinates, and the ideal coordinates of the corner points are calculated based on the pixel coordinates and calculation relationships of the corner points in the collected images.

[0103] Here, the calculation relationship includes the distance between the camera and a corner point on the moving calibration plate corresponding to the image (e.g., determining the distance by actual measurement) and the internal parameter information of the camera, and calculates the ideal coordinates of the corner point, where the ideal coordinates represent the pixel coordinates obtained when there is no deviation in the pose of the camera.

[0104] Specifically, the method may include the steps of determining an actual image height of the image point based on the pixel coordinates of the corner point in the collected image, calculating an ideal image height of the corner point based on the actual image height of the corner point and a calculation relationship, determining ideal coordinates of the corner point based on the ideal image height of the corner point, and determining posture information of the camera based on the relationship between the pixel coordinates of the corner point in the image and the ideal coordinates. This embodiment realizes simultaneous calibration of external parameters for multiple cameras, improves the calibration efficiency of external parameters of multiple common cameras installed on a moving carrier such as a vehicle, and realizes the technical effect of mass production calibration.

[0105] Based on the embodiment shown in FIG. 6 above, step 5022 includes: determining transformation parameters based on a positional relationship between an origin of the first coordinate system and an origin of the second coordinate system; The method may further include a step of performing a translation and a rotation on the first external parameter information based on the transformation parameters to obtain target external parameter information of the camera in the second coordinate system.

[0106] Optionally, for one camera, the first coordinate system may be a shooting coordinate system with the camera as its origin, where in the coordinate system the direction the camera points is the x-axis, the direction perpendicular to the x-axis and pointing left is the y-axis, and the direction perpendicular to the x-axis and pointing up is the z-axis, i.e., when the coordinate system is determined, the origin of the coordinate system is known. If the second coordinate system is a coordinate system with the origin at the foot of the perpendicular line from the camera to the ground, the transformation parameters only include distance information in the z-axis direction in the world coordinate system, and of course, if the origin of the second coordinate system is elsewhere, it may also include distance information on the x-axis and / or y-axis.

[0107] By performing corresponding translation and rotation on the first external parameter information by the transformation parameters, the target external parameter information in the second coordinate system of the camera can be obtained, so that this embodiment not only realizes the quick acquisition of external parameter information, but also unifies the external parameter information of all cameras into the same coordinate system by coordinate system transformation, which makes it easier to process the entire moving carrier, and increases the application range and scenes of this embodiment.

[0108] Any of the multiple camera external parameter calibration methods provided in the embodiments of the present disclosure may be performed by any suitable device having data processing capabilities, including but not limited to a terminal device, a server, etc.

[0109] Exemplary Electronic Devices 7 shows a block diagram of an electronic device according to an embodiment of the present disclosure. As shown in FIG. 7, the electronic device 70 includes at least one processor 71 and at least one memory 72.

[0110] Among them, each processor 71 may be a central processing unit (CPU) or other form of processing device having data processing and / or instruction execution capabilities, and may control other components within electronic device 70 to perform desired functions.

[0111] The memory 72 may include one or more computer program products, which may include various forms of computer readable storage media, such as, for example, volatile memory and / or non-volatile memory. The volatile memory may include, for example, Random Access Memory (RAM) and / or cache memory. The non-volatile memory may include, for example, Read-Only Memory (ROM), a hard disk, flash memory, and the like.

[0112] The computer-readable storage medium may store one or more computer program instructions, and the processor 71 may execute the program instructions to realize the multi-camera extrinsic parameter calibration method according to each embodiment of the present disclosure described above and / or other desired functions. The computer-readable storage medium may further store other information, such as various contents including input signals, signal components, noise components, etc.

[0113] As an example, electronic device 70 may further include input devices 73 and output devices 74, with these components being interconnected via a bus system and / or other form of connection (not shown).

[0114] Optionally, the input device 73 may be the microphone or microphone array and is used to capture the input signal of the sound source. If the electronic device is a standalone device, the input device 73 may be a communication network connector and is used to receive the collected input signal.

[0115] Optionally, the input device 73 may further include, for example, a keyboard, a mouse, etc. The output device 74 can output various information including determined distance information, direction information, etc. to the outside. The output device 74 may include, for example, a display, a speaker, a printer, and a communication network and a remote output device connected thereto.

[0116] Of course, for simplicity, Figure 7 illustrates only some of the components of electronic device 70 that are relevant to the present disclosure, and omits components such as buses, input / output interfaces, etc. In addition, electronic device 70 may further include any other suitable components depending on the particular application.

[0117] Exemplary Computer Program Products and Computer-Readable Storage Media In addition to the above methods and apparatus, an embodiment of the present disclosure may be a computer program product including computer program instructions that, when executed by a processor, cause the processor to perform steps in the multi-camera extrinsic parameter calibration method according to various embodiments of the present disclosure described in the "Exemplary Method" section above of this specification.

[0118] The computer program product may be written with program code for carrying out operations of embodiments of the present disclosure in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, and the like, as well as conventional process-oriented programming languages ​​such as "C" or similar programming languages. The program code may execute entirely on the user's computing device, partially on the user's computing device, as a stand-alone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0119] Further, an embodiment of the present disclosure may be a computer-readable storage medium having stored thereon computer program instructions which, when executed by a processor, cause the processor to perform steps in the method for calibrating extrinsic parameters of a multi-camera according to various embodiments of the present disclosure described in the "Exemplary Method" section above of this specification.

[0120] The computer readable storage medium may be any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable medium may include, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include an electrical connection having one or more leads, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a compact disc read-only memory (CD-ROM), an optical storage element, a magnetic storage element, or any suitable combination of the above.

[0121] Although the basic principles of the present disclosure have been described above with reference to specific examples, the advantages, advantages, effects, etc. mentioned in the present disclosure are not limiting but merely illustrative, and it cannot be considered that each embodiment of the present disclosure necessarily has these advantages, advantages, effects, etc. Furthermore, the specific details disclosed above are not limiting but merely serve to illustrate and facilitate understanding, and the above details do not necessarily limit the realization of the present disclosure by the above specific details.

[0122] Each embodiment in this specification will be described in a cumulative manner, with each embodiment being described focusing on the differences from other embodiments, and the same or similar parts between the embodiments may be referred to each other. System embodiments are described briefly since they essentially correspond to method embodiments, and the relevant parts may be referred to the partial description of the method embodiments.

[0123] Block diagrams of elements, apparatus, devices, and systems according to the present disclosure are merely illustrative examples and do not require or suggest that they be connected, arranged, or configured as shown in the block diagrams. As will be understood by those skilled in the art, these elements, apparatus, devices, and systems can be connected, arranged, or configured in any manner. Terms such as "including," "including," "having," and the like are open-ended terms and can be used interchangeably to mean "including, but not limited to." As used herein, the words "or" and "and" mean and can be used interchangeably to mean the word "and / or," unless the context clearly indicates otherwise. As used herein, the word "for example" means and can be used interchangeably to mean the conjunction "including, but not limited to."

[0124] The method and apparatus of the present disclosure can be realized in many forms. For example, the method and apparatus of the present disclosure can be realized by software, hardware, firmware, or any combination of software, hardware, and firmware. Unless otherwise stated, the above order of steps used in the method is for illustrative purposes only, and the steps of the method of the present disclosure are not limited to the order specifically described above. Furthermore, in some embodiments, the present disclosure can be implemented as a program recorded on a recording medium, which includes machine-readable instructions for implementing the method according to the present disclosure. Thus, the present disclosure also covers a recording medium having a program stored thereon for implementing the method according to the present disclosure. In addition, in the apparatus, device, and method of the present disclosure, each component or each step can be disassembled and / or recombined, and such disassembly and / or recombination should be considered as equivalents of the present disclosure.

[0125] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of the present disclosure. Thus, the present disclosure is not intended to be limited to the aspects shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0126] The above description has been presented for purposes of illustration and description. Moreover, the description is not intended to limit the disclosed embodiments to the disclosed form. Although several exemplary aspects and embodiments have been discussed above, those skilled in the art will appreciate certain variations, modifications, variations, additions, and subcombinations.

Claims

1. a plurality of moving calibration plates provided within a setting space for completing calibration of external parameters of all of the cameras on a moving carrier on which the plurality of cameras are mounted, each of the moving calibration plates being provided on a slide rail, the slide rail being provided on a wall surface within the setting space; a control device for controlling each of the plurality of moving calibration plates to slide along the slide rail corresponding to the moving calibration plate; The control device is further used to realize a calibration device for external parameters of multiple cameras, in which the control device controls the sliding of each of the moving calibration plates between a plurality of the slide rails having a connecting relationship, thereby controlling each of the moving calibration plates to slide up, down, left, and right on a plane corresponding to the wall surface on which it is located.

2. The control device controls the moving calibration plate to slide along the corresponding slide rail to realize at least one of a set unit movement, a set length movement, and a continuous movement mode; The control device is provided with a reset button, The apparatus of claim 1 , wherein the control device is further adapted to control the return button to return the plurality of moving calibration plates to their initial positions.

3. 10. The apparatus of claim 1, wherein each said moving calibration plate is further provided with a locking device for locking the position of said moving calibration plate in the event of a power failure.

4. The apparatus according to any one of claims 1 to 3, wherein the size of the setting space is set according to the size of the moving carrier.

5. The apparatus according to claim 4 , further comprising a position limiting guide rail provided on a ground surface of the setting space for limiting a position of the moving carrier within the setting space.

6. Responsive to a moving carrier carrying a plurality of cameras entering a configuration space, the configuration space including a plurality of moving calibration plates; controlling the plurality of moving calibration plates to move to positions corresponding to each of the plurality of cameras; and realizing calibration of extrinsic parameters of the plurality of cameras based on the plurality of moving calibration plates; The step of controlling the movement of the plurality of moving calibration plates to positions corresponding to each camera among the plurality of cameras includes: A method for calibrating external parameters of multiple cameras, comprising a step of controlling each of the multiple moving calibration plates to slide up, down, left, and right on a plane corresponding to a wall surface located on a plurality of slide rails having a connecting relationship provided on a wall surface within the setting space, and moving each of the moving calibration plates to a position corresponding to each of the multiple cameras.

7. The method of claim 6 , further comprising the step of automatically returning the plurality of moving calibration plates to an initial position in response to completion of calibration of the extrinsic parameters of the plurality of cameras.

8. A computer-readable storage medium having a computer program stored therein, A computer-readable storage medium that, when the computer program is executed, realizes the method for calibrating external parameters of multiple cameras according to claim 6 or 7.

9. A processor; a memory for storing instructions executable by said processor; An electronic device adapted for implementing the method for multi-camera extrinsic parameter calibration according to claim 6 or 7, wherein the processor reads the executable instructions from the memory and executes the instructions.

Images