Calibration parameter generating system and method

Abstract

A calibration parameter generating system and method are provided. The system receives a plurality of calibration image frames from a plurality of image capturing devices, each of the calibration image frames at least includes a feature sub-pattern on at least one curved surface corresponding to a calibration object, and the feature sub-pattern is a part of at least one feature pattern. The system generates a calibration parameter of each of the image capturing devices based on the feature sub-pattern of each of the calibration image frames.

Description

[Technical Field]

[0001] This disclosure relates to a system and method for generating correction parameters. Specifically, this disclosure relates to a system and method for generating correction parameters that can correctly generate a plurality of correction parameters. [Previous Technology]

[0002] Multi-camera systems (e.g., ultra-wide angle FOV camera arrays) are frequently used in computer vision fields such as simultaneous localization and mapping (SLAM) and pose capture. For the operation to be executed accurately, the parameters of each camera in the system need to be correctly calibrated.

[0003] In the prior art, each camera can perform correction by capturing the feature patterns on a conventional planar correction plate. However, since the camera located on the back of the planar correction plate cannot capture the feature patterns on the correction plate, it cannot perform the correction operation correctly. In other words, a conventional planar correction plate can only support camera correction within a 180-degree outward viewing angle of the planar correction plate at most, which makes it impossible for any multi-camera array with a shooting angle greater than 180 degrees to perform panoramic correction.

[0004] In addition, in the prior art, the correction structure of the polyhedron is still prone to errors in calculation because the observation angle of some correction surfaces captured by the camera is too large, resulting in unclear images (e.g., multiple correction surfaces at multiple joints on the polyhedron structure captured by a multi-camera system), which makes it impossible to perform parameter correction correctly.

[0005] In view of this, how to provide a correction parameter generation technology that can correctly generate correction parameters is an urgent goal for the industry. [Summary of the Invention]

[0006] One object of this disclosure is to provide a correction parameter generation system. The correction parameter generation system includes a correction object, a plurality of image capture devices, and a processing device communicatively connected to the image capture devices. The correction object includes at least one curved surface, and each of the at least one curved surface includes at least one feature pattern. The processing device receives a plurality of correction image frames from the image capture devices, wherein each of the correction image frames includes at least one feature sub-pattern corresponding to the at least one curved surface of the correction object, and the feature sub-pattern is a portion of the at least one feature pattern. The processing device generates a correction parameter for each of the image capture devices based on the feature sub-pattern of each of the correction image frames.

[0007] Another object of this disclosure is to provide a method for generating correction parameters, which is used in a correction parameter generation system. The correction parameter generation system includes a correction object, a plurality of image capture devices, and a processing device. The correction object includes at least one curved surface, and each of the at least one curved surface includes at least one feature pattern. The correction parameter generation method includes the following steps: the processing device receives a plurality of correction image frames from the image capture devices, wherein each of the correction image frames includes at least one feature sub-pattern corresponding to the at least one curved surface of the correction object, and the feature sub-pattern is a portion of the at least one feature pattern; and the processing device generates a correction parameter for each of the image capture devices based on the feature sub-pattern of each of the correction image frames.

[0008] The correction parameter generation technology (including at least a system and method) disclosed herein sets a correction object comprising at least one curved surface in the system to provide each image capture device in the system with a recognizable viewing angle without blind spots (i.e., a recognizable viewing angle of 360 degrees). Then, a processing device in the system generates correction parameters for each of the image capture devices based on a plurality of correction image frames containing feature sub-patterns on the at least one curved surface of the correction object. Since the correction parameter generation technology disclosed herein can provide panoramic correction for multi-camera arrays with shooting angles greater than 180 degrees, a full-angle correction effect can be achieved. Furthermore, the new correction object structure solves the blind spot and viewpoint bias that occur when using a flat / polyhedron calibration chart. Accordingly, the correction parameter generation technology disclosed herein can accurately generate correction parameters, improving the accuracy of device operation and enhancing the user experience.

[0009] The following detailed technical and implementation methods of this disclosure are illustrated in conjunction with the figures, so that those skilled in the art to which this disclosure pertains can understand the technical features of the claimed invention.

Implementation Method

[0010] The following will explain the calibration parameter generation system and method provided in this disclosure through embodiments. However, these embodiments are not intended to limit the implementation of this disclosure to any environment, application, or manner described in these embodiments. Therefore, the description of the embodiments is for illustrative purposes only and is not intended to limit the scope of this disclosure. It should be understood that in the following embodiments and drawings, elements not directly related to this disclosure have been omitted and are not shown, and the dimensions of each element and the dimensional ratio between elements are only illustrative and are not intended to limit the scope of this disclosure.

[0011] Firstly, the first embodiment disclosed herein is a calibration parameter generation system 1, the schematic diagram of which is depicted in Figure 1. In this embodiment, the calibration parameter generation system 1 includes a calibration object CO, a processing device PD, and a plurality of image capture devices ICD1, ICD2, ..., ICDn, where n is a positive integer. The processing device PD is communicatively connected to the image capture devices ICD1, ICD2, ..., ICDn.

[0012] As shown in Figure 1, the image capture devices ICD1, ICD2, ..., ICDn can be set in different locations in the environment / electronic device and generate corresponding image frames from different multiple image capture perspectives.

[0013] It should be noted that, in some embodiments, the image capture devices ICD1, ICD2, ..., ICDn may be a multi-camera array disposed on an electronic device (e.g., an inside-out multi-camera array). In some embodiments, the image capture devices ICD1, ICD2, ..., ICDn may be a multi-camera array disposed in the environment (e.g., an outside-in multi-camera array).

[0014] In some embodiments, the architecture of the processing device PD is shown in Figure 2. The processing device PD includes a processor 11 and a memory 13, with the processor 11 electrically connected to the memory 13.

[0015] It should be noted that the processor 11 may be any processing unit, central processing unit (CPU), microprocessor, or other computing device known to those skilled in the art to which this disclosure pertains. The storage device 13 may be memory, a universal serial bus (USB) disk, hard disk, optical disk, flash drive, or any other storage medium or circuit known to those skilled in the art to which this disclosure pertains and having the same function. The image capture devices ICD1, ICD2, ..., ICDn may be image capture devices with an image capture function (e.g., a depth camera lens) for generating real-time images corresponding to a field of view (FOV).

[0016] For example, as shown in Figure 1, image capture device ICD1 has a field of view FOV1, image capture device ICD2 has a field of view FOV2, and image capture device ICDn has a field of view FOVn. In this example, the multi-camera array composed of image capture devices ICD1, ICD2, ..., ICDn has a shooting angle greater than 180 degrees.

[0017] It should be noted that this disclosure does not limit the number or location of the image captures included in the correction parameter generation system 1. Furthermore, in some embodiments, the processing device PD may be integrated into an electronic device that houses the image captures ICD1, ICD2, ..., ICDn. In some embodiments, the electronic device may be a head-mounted display or a tracking device, which generates correction parameters to correct its plurality of image captures.

[0018] Next, the following paragraphs will describe the specific details of how the correction parameter generation system 1 generates the correction parameters for each of the image capture devices in this embodiment.

[0019] First, in this embodiment, the calibration object CO includes at least one curved surface, and each of the at least one curved surface includes at least one feature pattern.

[0020] It should be noted that each feature pattern may have a different geometric shape, each feature pattern has a unique identifiable characteristic, and each feature pattern is used for subsequent feature comparison operations. A feature pattern may contain a plurality of feature sub-patterns (e.g., a portion of a feature pattern), each feature sub-pattern having a unique identifiable characteristic. Furthermore, the feature pattern can be modified according to the user's camera requirements, such as conventional printing, IRLED, or any other optical emitter.

[0021] In some embodiments, the correction object CO corresponds to one or a combination of a cylindrical shape, a curved shape, a spherical shape or an annular shape.

[0022] For ease of understanding, please refer to Figures 3A to 4B, which illustrate different implementation methods of the calibration object CO and corresponding operation diagrams.

[0023] First, as shown in Figure 3A, the calibration object CO can be a cylindrical shape. A multi-camera array (MCA) consisting of a plurality of image capture units is positioned outside the calibration object CO, with these image capture units facing inwards. In this example, a high-dpi printed feature pattern can be flatly applied to the outer wall of a smooth plastic cylinder, and the cylinder size is determined according to the shooting distance. This embodiment can be applied to the calibration of a ring-shaped camera array shooting from the outside in.

[0024] Next, as shown in Figure 3B, the calibration object CO can be a curved shape. A multi-camera array (MCA) consisting of a plurality of image captures is positioned outside the calibration object CO, with the image captures facing inward. In this example, an acrylic plate with a curved shape can be used as a carrier for the feature pattern, and the feature pattern is covered onto the carrier by a film (e.g., feature patterns on both sides). This embodiment can be applied to the calibration of a ring-shaped camera array that shoots from the outside in.

[0025] Next, as shown in Figure 4A, the calibration object CO can be a sphere, and a multi-camera array (MCA) consisting of a plurality of image captures is positioned outside the calibration object CO, with the image captures facing inward. In this example, the matte-finished steel sphere can be dyed with a surface paint to create a characteristic pattern, and the size of the sphere is determined according to the shooting distance. This embodiment can be applied to the calibration of a ring-shaped camera array shooting from the outside in.

[0026] Additionally, as shown in Figure 4B, the calibration object CO can be a ring shape, with a multi-camera array (MCA) consisting of a plurality of image captures positioned outside the calibration object CO, and these image captures facing inward. In this example, a ring-shaped acrylic plate can be used as a feature pattern carrier, and the feature pattern can be coated onto the carrier using a film. This embodiment can be applied to the calibration of a ring-shaped camera array that shoots from the outside in.

[0027] It should be noted that, since the calibration object CO disclosed herein includes at least one curved surface, and each curved surface can provide a wider identifiable field of view to the image capturer. In other words, since each curved surface can provide at least more than 180 degrees of identifiable viewing angle, the image capturer in this system can acquire calibration image frames with characteristic patterns at more locations to perform corresponding calibration operations.

[0028] In some embodiments, the angle at which the correction object CO can be identified is the full angle, so the multi-camera array has no blind spots (i.e., the feature pattern can be captured from any position). In other words, it does not have the 180-degree shooting limitation that traditional planar correction plates have when used for multi-camera array correction. Accordingly, panoramic correction can be performed even if there is no overlap between the fields of view of multiple cameras.

[0029] In some embodiments, the at least one curved surface of the correction object CO provides a 360-degree identifiable viewing angle. For example, the spherical correction object in Figure 4A.

[0030] In some embodiments, the at least one arcuate surface of the correction object CO comprises only one continuous surface, and the continuous surface provides a 360-degree identifiable viewing angle. For example, the annular correction object in Figure 4B has only one continuous surface.

[0031] In some embodiments, the at least one arcuate surface of the correction object CO comprises only two continuous surfaces, and each of the continuous surfaces provides one of the identifiable viewing angles exceeding 180 degrees. For example, the curved shape correction object as shown in Figure 3B has two continuous surfaces (i.e., the front and rear surfaces).

[0032] Next, in this embodiment, the processing device PD receives a plurality of corrected image frames from the image capture devices, wherein each of the corrected image frames includes at least one feature sub-pattern on the at least one arc surface of the corresponding corrected object CO, and the feature sub-pattern is a part of the at least one feature pattern.

[0033] Finally, in this embodiment, the processing device PD generates a correction parameter for each of the image capture devices based on the feature sub-pattern of each of the correction image frames.

[0034] In some embodiments, the calibration parameters include one or a combination of extrinsic calibration parameters and intrinsic calibration parameters.

[0035] For example, external correction parameters may include information such as rotation matrix, translation vector, projection matrix, position, and orientation. Internal correction parameters may include information such as focal length, principal point, distortion coefficients, pixel size, and camera matrix.

[0036] It should be noted that, since the feature pattern is uniquely identifiable, in this disclosure, the processing device PD only needs to identify a portion of the feature pattern to deduce the current position and orientation of the image capture device.

[0037] In some embodiments, the correction parameters generated by the processing device PD describe the position and orientation of the image capture device in the world coordinate system (i.e., the orientation and position of the image capture device). The processing device PD can correct one of the position information and one of the orientation information corresponding to each of the image capture devices based on the correction parameters of each of the image capture devices.

[0038] In some embodiments, as shown in Figure 2, the processing device PD includes a memory 13 electrically connected to the processor 11. The memory 13 is used to store a plurality of view images corresponding to the calibration object CO, a positional relationship corresponding to each of the view images, and a device orientation. The processing device PD can determine the view angle and position of the image capture device by comparing the view images and feature sub-patterns.

[0039] Specifically, the processor 11 compares the feature sub-patterns of the viewpoint images and the corrected image frames to determine the positional relationship of the image captures and the device orientation. Then, the processor 11 generates the correction parameters of the image captures based on the positional relationship of the image captures and the device orientation.

[0040] In some embodiments, the storage 13 may be used to store a three-dimensional model corresponding to the calibration object CO, the three-dimensional model storing a plurality of viewpoints corresponding to the calibration object CO and the device posture corresponding to each of the viewpoints.

[0041] In some embodiments, the correction parameter generation system 1 includes an inside-out multi-camera array composed of image captures located in an internal region of the correction object CO.

[0042] In some embodiments, the corrected image frames correspond to the same first arc surface as one of the at least one arc surface of the corrected object CO.

[0043] For example, as shown in Figure 5A, the calibration object CO can be cylindrical. In this example, a high-dpi printed feature pattern can be flatly applied to the inner wall of a smooth plastic cylinder. When the inside-out multi-camera array is placed in the internal region IR, camera array calibration can be performed from the inside out, and these calibration image frames correspond to the same arc surface of the calibration object CO (i.e., the inner arc surface of the cylinder).

[0044] As another example, as shown in Figure 5B, the calibration object CO can be annular in shape. In this example, an annular acrylic plate can be used as a feature pattern carrier, and the feature pattern can be wrapped onto the carrier through a film. When the inside-out multi-camera array is placed in the internal region IR, camera array calibration can be performed by shooting from the inside out, and the calibration image frames correspond to the same arc surface of the calibration object CO.

[0045] In some embodiments, in order to make the operation of the correction parameters more accurate, the processing device PD can simultaneously consider the position information of different image capture devices to obtain the relative distance relationship more accurately.

[0046] Specifically, the image capture devices include a first image capture device, a second image capture device, and a third image capture device. The processing device PD performs a panoramic correction operation based on the positional relationship between any two of the image capture devices and the feature sub-pattern of each of the corrected image frames to generate the correction parameters of each of the image capture devices.

[0047] As described above, the correction parameter generation system 1 provided by this disclosure includes a correction object comprising at least one curved surface in the system to provide each image capture device in the system with a recognizable viewing angle without blind spots (i.e., a recognizable viewing angle of 360 degrees). Then, the processing device in the system generates correction parameters for each of the image capture devices based on a plurality of correction image frames containing feature sub-patterns on the at least one curved surface of the correction object. Since the correction parameter generation system 1 provided by this disclosure can provide panoramic correction for multi-camera arrays with shooting angles greater than 180 degrees, a full-angle correction effect can be achieved. Furthermore, the new correction object structure solves the blind spots and viewpoint deviations that occur when using planar / polyhedral calibration maps. Accordingly, the correction parameter generation system 1 provided by this disclosure can correctly generate correction parameters, improving the accuracy of device operation and enhancing the user's service experience.

[0048] The second embodiment disclosed herein is a method for generating correction parameters, the flowchart of which is depicted in Figure 6. The correction parameter generation method 600 is applicable to a correction parameter generation system, such as the correction parameter generation system 1 described in the first embodiment. This correction parameter generation system includes a correction object, a plurality of image capture devices, and a processing device, such as the correction object CO, image capture devices ICD1, ICD2, ..., ICDn, and processing device PD described in the first embodiment. The correction object includes at least one curved surface, each of which includes at least one feature pattern. The correction parameter generation method 600 generates correction parameters for each of the image capture devices through steps S601 to S603.

[0049] In step S601, the processing device receives a plurality of corrected image frames from the image capture devices, wherein each of the corrected image frames includes at least one feature sub-pattern corresponding to the at least one arc surface of the object to be corrected, and the feature sub-pattern is a part of the at least one feature pattern.

[0050] Next, in step S603, the processing device generates a correction parameter for each of the image capture devices based on the feature sub-pattern of each of the corrected image frames.

[0051] In some embodiments, the correction parameter generation method 600 further includes the following steps: correcting a position information and a direction information corresponding to each of the image capture devices based on the correction parameters of each of the image capture devices.

[0052] In some embodiments, the correction parameter generation system further includes a storage device for storing a plurality of viewpoint images corresponding to the object to be corrected, a positional relationship of each of the viewpoint images, and a device orientation. The generation of the correction parameters of each of the image capture devices further includes the following steps: the processing device compares the feature sub-patterns of each of the viewpoint images and the correction image frames to determine the positional relationship and the device orientation of each of the image capture devices; and the processing device generates the correction parameters of each of the image capture devices based on the positional relationship and the device orientation of each of the image capture devices.

[0053] In some embodiments, the at least one curved surface of the correction object provides a recognizable viewing angle of 360 degrees.

[0054] In some embodiments, the correction parameter generation system includes an inside-out multi-camera array composed of image captures located in an internal region of the correction object.

[0055] In some embodiments, the corrected image frames correspond to the same first arc surface of one of the at least one arc surface of the object to be corrected.

[0056] In some embodiments, the image capture devices include a first image capture device, a second image capture device and a third image capture device, and the correction parameter generation method 600 includes the following steps: performing a panoramic correction operation based on the positional relationship of any two of the image capture devices and the feature sub-pattern of each of the correction image frames to generate the correction parameters of each of the image capture devices.

[0057] In some embodiments, the correction object corresponds to one or a combination of a cylindrical shape, a curved shape, a spherical shape or an annular shape.

[0058] In some embodiments, the at least one arc surface of the correction object comprises only one continuous surface, and the continuous surface provides one identifiable viewing angle of 360 degrees.

[0059] In some embodiments, the at least one arc surface of the correction object comprises only two continuous surfaces, and each of the continuous surfaces provides one of the identifiable viewing angles exceeding 180 degrees.

[0060] In addition to the above steps, the second embodiment can also perform all the operations and steps of the correction parameter generation system 1 described in the first embodiment, have the same function, and achieve the same technical effect. Those skilled in the art to which this disclosure pertains can directly understand how the second embodiment performs these operations and steps based on the first embodiment described above, has the same function, and achieves the same technical effect, so it will not be described in detail here.

[0061] The calculation method described in the second embodiment can be implemented by a computer program having a plurality of instructions. Each computer program can be a file that can be transmitted over a network, or it can be stored in a non-transitory computer-readable recording medium. For each computer program, after the instructions contained therein are loaded into an electronic device (e.g., calibration parameter generation system 1), the computer program executes the calibration parameter generation method described in the second embodiment. The non-transitory computer-readable storage medium can be an electronic product, such as: a read-only memory (ROM), a flash memory, a floppy disk, a hard disk, a compact disk (CD), a USB flash drive, a database accessible via a network, or any other storage medium known to those skilled in the art to which this disclosure pertains and having the same function.

[0062] It should be noted that in this patent specification and the claims, certain terms (including: curved surface, image capture device) are preceded by "first", "second" or "third". These "first", "second" or "third" are only used to distinguish different terms. For example, "first" and "second" in "first image capture device" and "second image capture device" are only used to indicate different image capture devices in operation.

[0063] The correction parameter generation technology (including at least a system and method) disclosed herein sets a correction object comprising at least one curved surface in the system to provide each image capture device in the system with a recognizable viewing angle without blind spots (i.e., a recognizable viewing angle of 360 degrees). Then, a processing device in the system generates correction parameters for each of the image capture devices based on a plurality of correction image frames containing feature sub-patterns on the at least one curved surface of the correction object. Since the correction parameter generation technology disclosed herein can provide panoramic correction for multi-camera arrays with shooting angles greater than 180 degrees, a full-angle correction effect can be achieved. Furthermore, the new correction object structure solves the blind spots and viewpoint deviations that occur when using planar / polyhedral calibration maps. Accordingly, the correction parameter generation technology disclosed herein can accurately generate correction parameters, improving the accuracy of device operation and enhancing the user's service experience.

[0064] The above embodiments are only used to illustrate some implementations of this disclosure and to explain the technical features of this disclosure, and are not intended to limit the scope and range of protection of this disclosure. Any changes or equivalent arrangements that can be easily made by those skilled in the art to which this disclosure pertains are within the scope of this disclosure, and the scope of protection of this disclosure is determined by the scope of the patent application. [Simplified Explanation of the Diagram]

[0065] Figure 1 is a schematic diagram of the architecture of the calibration parameter generation system of the first embodiment; Figure 2 is a schematic diagram of the architecture of the processing device of some embodiments; Figure 3A is a schematic diagram of the operation of the calibration object of some embodiments; Figure 3B is a schematic diagram of the operation of the calibration object of some embodiments; Figure 4A is a schematic diagram of the operation of the calibration object of some embodiments; Figure 4B is a schematic diagram of the operation of the calibration object of some embodiments; Figure 5A is a schematic diagram of the operation of the calibration object of some embodiments; Figure 5B is a schematic diagram of the operation of the calibration object of some embodiments; and Figure 6 is a partial flowchart of the calibration parameter generation method of the second embodiment. [Biomaterial Storage]

[0067] Domestic storage information (please note in order of storage institution, date, and number): None. International storage information (please note in order of storage country, institution, date, and number): None.

Claims

1. A calibration parameter generation system, comprising: a calibration object, wherein the calibration object includes at least one curved surface, each of the at least one curved surface including at least one feature pattern, and the at least one curved surface of the calibration object provides a recognizable viewing angle of 360 degrees; a plurality of image capture devices; and a processing device communicatively connected to the image capture devices and configured to perform the following operations: receiving a plurality of calibration image frames from the image capture devices, wherein each of the calibration image frames includes at least one feature sub-pattern corresponding to the at least one curved surface of the calibration object, and the feature sub-pattern is a portion of the at least one feature pattern; and generating a calibration parameter for each of the image capture devices based on the feature sub-pattern of each of the calibration image frames.

2. The correction parameter generation system as described in claim 1, wherein the processing device further performs the following operation: based on the correction parameters of each of the image capture devices, correcting position information and orientation information corresponding to each of the image capture devices.

3. The correction parameter generation system as described in claim 1, wherein the processing device further comprises: a storage device for storing a plurality of viewpoint images corresponding to the object to be corrected, a positional relationship between each of the viewpoint images, and a device orientation; wherein, The generation of the correction parameters for each of the image capture devices further includes the following operations: comparing the feature sub-patterns of the viewpoint images and the correction image frames to determine the positional relationship and device orientation of each of the image capture devices; and generating the correction parameters for each of the image capture devices based on the positional relationship and device orientation of each of the image capture devices.

4. The correction parameter generation system as claimed in claim 1, wherein the correction parameter generation system includes an inside-out multi-camera array, the inside-out multi-camera array being composed of image captures, and the image captures being located in one of the internal regions of the correction object.

5. The correction parameter generation system as described in claim 4, wherein the correction image frames correspond to the same first arc surface of one of the at least one arc surface of the object to be corrected.

6. The correction parameter generation system as claimed in claim 1, wherein the image captures include a first image capture, a second image capture, and a third image capture, and the processing apparatus further performs the following operation: performing a panoramic correction operation based on the positional relationship of any two of the image captures and the feature sub-pattern of each of the correction image frames to generate the correction parameters of each of the image captures.

7. The correction parameter generation system as described in claim 1, wherein the correction object corresponds to one or a combination of a cylindrical shape, a curved shape, a spherical shape, or a ring shape.

8. The correction parameter generation system as claimed in claim 1, wherein the at least one arcuate surface of the correction object comprises only one continuous surface, and the recognizable viewing angle provided by the continuous surface is 360 degrees.

9. The correction parameter generation system as described in claim 1, wherein the at least one arcuate surface of the correction object comprises only two continuous surfaces, and each of the continuous surfaces provides a recognizable viewing angle of more than 180 degrees.

10. A method for generating correction parameters, used in a correction parameter generation system, wherein the correction parameter generation system includes a correction object, a plurality of image captures, and a processing device, the correction object including at least one curved surface, each of the at least one curved surface including at least one feature pattern, the at least one curved surface of the correction object providing a recognizable viewing angle of 360 degrees, and the correction parameter generation method comprising the following steps: the processing device receiving a plurality of correction image frames from the image captures, wherein each of the correction image frames includes at least one feature sub-pattern corresponding to the at least one curved surface of the correction object, and the feature sub-pattern is a portion of the at least one feature pattern; and the processing device generating a correction parameter for each of the image captures based on the feature sub-pattern of each of the correction image frames.

Images