Parameter calibration method and device, projection system and storage medium

By projecting the encoded structured light patterns by projecting the projector and shooting and decoding with the camera, online calibration of the internal and external references of the projection system is realized, solving the problem of cumbersome calibration process and not real-time in the existing technology, and is suitable for scenarios such as engineering projection.

CN118967824APending Publication Date: 2024-11-15APPOTRONICS CORP LTD

Patent Information

Application Number
CN202410913587.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-05
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

The calibration process of existing projection systems is cumbersome, offline and not real-time, and is difficult to apply especially in engineering projection scenarios.

Method used

By controlling the projector to project the encoded structured light pattern, the first camera and the second camera respectively capture and decode the structured light images, determine the pixel correspondence relationship, and realize online calibration of the internal and external parameters of the projector and the camera.

Benefits of technology

The online calibration of the internal and external references of the projection system is realized without the need for calibration boards, which improves the real-time and application capabilities of the projection system, and is suitable for engineering projection and other scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118967824A_ABST
    Figure CN118967824A_ABST
Patent Text Reader

Abstract

The invention discloses a parameter calibration method and device, a projection system and a storage medium, and relates to the technical field of projection. The method comprises the following steps: controlling a projector to project a coding structured light pattern; obtaining a first structured light image and a second structured light image; decoding the first structured light image to obtain a first pixel corresponding relation between the first structured light image and the coded structured light pattern, and decoding the second structured light image to obtain a second pixel corresponding relation between the second structured light image and the coded structured light pattern; determining a plurality of matched pixel pairs between the first structured light image and the second structured light image based on the first pixel corresponding relation and the second pixel corresponding relation; and based on the plurality of matched pixel pairs, respectively calibrating internal parameters and external parameters of the first camera, the second camera and the projector. Therefore, external calibration objects such as a calibration plate are not needed, and on-line calibration of internal and external parameters of the placed projector and camera can be realized.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of projection technology, and in particular to a parameter calibration method, device, projection system and storage medium. Background Art

[0002] The projection system generally includes components such as cameras and projectors. The projector is used to project image information, and the camera is used to assist the projector in focusing, side projection, and other functions. In order to ensure the projection effect of the projection system, it is generally necessary to calibrate the projection equipment before using it.

[0003] However, in the related technology, the calibration process is rather cumbersome and requires the use of a professional calibration board for calibration. In addition, the internal and external parameters are generally calibrated offline and are not real-time, especially in some projection scenarios, especially in engineering projection scenarios, and are not applicable. Summary of the invention

[0004] The present application proposes a parameter calibration method, device, projection system and storage medium to realize online calibration of placed projectors and cameras.

[0005] In a first aspect, an embodiment of the present application provides a parameter calibration method, which is applied to a projection system, wherein the projection system includes at least a projector, a first camera, and a second camera, and the method includes: controlling the projector to project a coded structured light pattern; acquiring a first structured light image and a second structured light image, wherein the first structured light image is obtained by taking the coded structured light pattern by the first camera, and the second structured light image is obtained by taking the coded structured light pattern by the second camera; decoding the first structured light image to obtain a first pixel correspondence between the first structured light image and the coded structured light pattern, and decoding the second structured light image to obtain a second pixel correspondence between the second structured light image and the coded structured light pattern; based on the first pixel correspondence and the second pixel correspondence, determining a plurality of matching pixel pairs between the first structured light image and the second structured light image; based on the plurality of matching pixel pairs, calibrating the intrinsic parameters and extrinsic parameters of the first camera, the second camera, and the projector, respectively.

[0006] In a second aspect, an embodiment of the present application provides a parameter calibration device, which is applied to a projection system, wherein the projection system includes at least a projector, a first camera, and a second camera, and the device includes: an image projection module, which is used to control the projector to project a coded structured light pattern; an image capture module, which is used to obtain a first structured light image and a second structured light image, wherein the first structured light image is obtained by capturing the coded structured light pattern by the first camera, and the second structured light image is obtained by capturing the coded structured light pattern by the second camera; a decoding module, which is used to decode the first structured light image to obtain a first pixel correspondence between the first structured light image and the coded structured light pattern, and to decode the second structured light image to obtain a second pixel correspondence between the second structured light image and the coded structured light pattern; a pixel matching module, which is used to determine a plurality of matching pixel pairs between the first structured light image and the second structured light image based on the first pixel correspondence and the second pixel correspondence; and a parameter calibration module, which is used to calibrate the intrinsic parameters and extrinsic parameters of the first camera, the second camera, and the projector respectively based on the plurality of matching pixel pairs.

[0007] In a third aspect, an embodiment of the present application provides a projection system, comprising: one or more processors; a memory; one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the above-mentioned method.

[0008] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a program code is stored. The program code can be called by a processor to execute the above method.

[0009] In the solution provided in the present application, the illumination control of the scene is achieved by controlling the projector to project the coded structured light pattern, thereby providing the necessary conditions for subsequent image acquisition, and the projection of the coded structured light pattern in the scene is photographed by the first camera and the second camera respectively to obtain the corresponding structured light image; and the first and second structured light images are decoded to obtain the correspondence between the pixels in the image and the coded structured light pattern, which provides a basis for subsequent pixel matching, and then based on the decoded pixel correspondence, multiple matching pixel pairs between the first structured light image and the second structured light image are determined, further strengthening the correlation between the images. Finally, the internal parameters (such as focal length, principal point) and external parameters (such as position, orientation) of the first camera, the second camera and the projector are calibrated using multiple matching pixel pairs, thereby realizing the online calibration of the internal and external parameters of the camera and the projector without the need for a calibration plate. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0011] Figure 1 A schematic diagram of the system architecture of a projection system provided in an embodiment of the present application is shown.

[0012] Figure 2 A flow chart of a parameter calibration method provided in an embodiment of the present application is shown.

[0013] Figure 3 A schematic diagram of Gray code stripes provided in an embodiment of the present application is shown.

[0014] Figure 4 A schematic diagram of the stripes of the phase-shift code provided in an embodiment of the present application is shown.

[0015] Figure 5 An image schematic diagram of a structured light image captured by a camera and a decoded image provided in an embodiment of the present application is shown.

[0016] Figure 6 This application shows Figure 2 Schematic diagram of the flow chart of the sub-steps of step S230.

[0017] Figure 7 A schematic diagram of matching pixel pairs in an embodiment of the present application is shown.

[0018] Figure 8 A flow chart of a parameter calibration method provided in another embodiment of the present application is shown.

[0019] Fig. 9 It shows that an embodiment of the present application provides Figure 8 Schematic diagram of the process flow of the steps after step S310.

[0020] Fig.10 It is a block diagram of a parameter calibration device provided according to an embodiment of the present application.

[0021] Fig.11 It is a block diagram of a projection system according to an embodiment of the present application for executing a parameter calibration method according to an embodiment of the present application.

[0022] Fig.12 It is a storage unit of an embodiment of the present application for storing or carrying program codes for implementing the parameter calibration method according to an embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0024] It should be noted that in some processes described in the specification, claims and the above-mentioned drawings of the present application, multiple operations that appear in a specific order are included, and these operations may not be executed or executed in parallel in the order in which they appear in this article. The sequence numbers of operations such as S110, S120, etc. are only used to distinguish different operations, and the sequence numbers themselves do not represent any execution order. In addition, these processes may include more or less operations, and these operations may be executed in sequence or in parallel. And, the terms "first", "second", etc. in the specification, claims and the above-mentioned drawings of the present application are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described here can be implemented in an order other than those illustrated or described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or server that includes a series of steps or sub-modules is not necessarily limited to those steps or sub-modules explicitly listed, but may include other steps or sub-modules that are not explicitly listed or that are inherent to these processes, methods, products, or devices.

[0025] The inventor proposes a parameter calibration method, device, projection system and storage medium. The parameter calibration method provided in the embodiment of the present application is described in detail below.

[0026] See also Figure 1 , Figure 1 1 is a schematic diagram of the system architecture of a projection system 10 provided in an embodiment of the present application. The projection system 10 at least includes: a projector 101, a first camera 102, and a second camera 103, wherein the shooting ranges of the first camera 102 and the second camera 103 both cover the projection range of the projector 101, so that the first camera 102 and the second camera 103 can both capture the pattern projected by the projector 101. The projector 101 can be a household long-focus projector, an engineering projector, or an ultra-short-focus projector, etc.

[0027] In this embodiment, the projection system 10 further includes a processor and a signal transmission and receiving module (not shown). Figure 1), the information transmission between the processor and the projector 101, the first camera 102 and the second camera 103 is realized by the signal transmission receiving module. The processor can control the projector 101 to project the coded structured light pattern; obtain the first structured light image and the second structured light image, the first structured light image is obtained by the first camera 102 shooting the coded structured light pattern, and the second structured light image is obtained by the second camera 103 shooting the coded structured light pattern; decode the first structured light image to obtain the first pixel correspondence between the first structured light image and the coded structured light pattern, and decode the second structured light image to obtain the second pixel correspondence between the second structured light image and the coded structured light pattern; based on the first pixel correspondence and the second pixel correspondence, determine a plurality of matching pixel pairs between the first structured light image and the second structured light image; based on the plurality of matching pixel pairs, calibrate the internal and external parameters of the first camera, the second camera and the projector respectively.

[0028] In some embodiments, the projection system may include more other projectors and more other cameras in addition to the projector 101, the first camera 102, and the second camera 103. It should be noted that the projectors are associated with each other by cameras, so in order to associate the two projectors, there must be a camera that can see both projector 1 and projector 2. In order to ensure the quality of three-dimensional reconstruction, try to ensure that both cameras can see both projector 1 and projector 2. Of course, seeing here does not mean seeing 100% of the pixels, as long as a part of them is seen, the more pixels seen, the more accurate the system. Therefore, when the number of cameras and the number of projectors are greater than or equal to 2, for any two projectors, there are at least two cameras that have a common field of view with the multi-device composed of the two projectors.

[0029] Please refer to Figure 2 , Figure 2 A flow chart of a parameter calibration method provided in an embodiment of the present application is applied to a projection system, wherein the projection system at least includes a projector, a first camera, and a second camera. Figure 2 The parameter calibration method provided in the embodiment of the present application is described in detail. The parameter calibration method may include the following steps:

[0030] Step S210: controlling the projector to project a coded structured light pattern.

[0031] Step S220: acquiring a first structured light image and a second structured light image, wherein the first structured light image is obtained by photographing the coded structured light pattern with the first camera, and the second structured light image is obtained by photographing the coded structured light pattern with the second camera.

[0032] In this embodiment, a series of coded structured light patterns can be generated based on the existing coding principle, and the generated series of coded structured light patterns can be transmitted to the projector, and the projector can be controlled to project the series of coded structured light patterns. Based on this, the first camera and the second camera are controlled to capture each coded structured light image projected by the projector, and a series of first structured light images captured by the first camera and a series of second structured light images captured by the second camera are obtained. Among them, the projector can project the coded structured light group to the area to be projected, and the area to be projected can be any display area that can be used for projection, such as a wall, a projection curtain, a projection screen, or a target object (such as a table, chair, or bench), and this embodiment does not limit this.

[0033] Optionally, the coded structured light pattern may be a pattern generated based on a phase shift code encoding method, wherein a schematic diagram of the phase shift code can be found in Figure 3 Of course, the pattern may be generated based on the encoding method of the three-step phase shift method, the four-step phase shift method or the five-step phase shift method, and this embodiment does not limit this.

[0034] Optionally, the coded structured light pattern may also be a pattern generated based on a Gray code encoding method, wherein a schematic diagram of the Gray code can be found in Figure 4 .

[0035] Step S230: decoding the first structured light image to obtain a first pixel correspondence relationship between the first structured light image and the encoded structured light pattern, and decoding the second structured light image to obtain a second pixel correspondence relationship between the second structured light image and the encoded structured light pattern.

[0036] Further, after acquiring the first structured light image and the second structured light image, the first structured light image and the second structured light image obtained by shooting can be decoded respectively according to the decoding principle corresponding to the encoding principle to obtain a decoding diagram of the first structured light image and a decoding diagram of the second structured light image; illustratively, Figure 5The images and decoded images captured by the first camera and the second camera for the images projected by projector 1 and projector 2 respectively are shown. Furthermore, based on the decoded image of the first structured light image and the decoded image of the second structured light image, the first pixel correspondence between the first structured light image and the coded structured light pattern (i.e., the correspondence between the pixels of the first camera and the pixels of the projector) and the second pixel correspondence between the second structured light image and the coded structured light pattern (i.e., the correspondence between the pixels of the second camera and the pixels of the projector) are obtained. It should be noted that the correspondence between the camera pixels and the projector pixels can be understood as the pixels corresponding to the two different devices are formed by the same spatial point, and can also be considered as the correspondence between the projector pixels projected on the object and captured by the camera to form the camera pixels.

[0037] In some embodiments, the coded structured light pattern is a pattern generated by a coding method based on a Gray code, and the first structured light image can be decoded according to a Gray code decoding algorithm to obtain the correspondence between the integer pixels of the coded structured light pattern and the integer pixels of the first structured light image (i.e., the correspondence between the integer pixels of the projector and the integer pixels of the first camera); and the second structured light image can be decoded according to the Gray code decoding algorithm to obtain the correspondence between the integer pixels of the coded structured light pattern and the integer pixels of the second structured light image (i.e., the correspondence between the integer pixels of the projector and the integer pixels of the second camera). It should be noted that Gray code is a pixel-level decoding accuracy, and therefore, what is obtained by decoding is the correspondence between the integer pixels of the projector and the integer pixels of the camera.

[0038] In some other embodiments, the coded structured light pattern is a pattern generated by a coding method based on a phase shift code, see Figure 6 , step S230 may include the contents of step S231 to step S232:

[0039] Step S231: decoding the first structured light image according to a phase shift code decoding algorithm to obtain a correspondence between pixels of the first structured light image and sub-pixels of the encoded structured light pattern as a first correspondence.

[0040] In this way, compared with the aforementioned Gray code, the phase shift method can be regarded as a continuously changing value. For example, when the phase solution is converted into pixel coordinates, it can usually achieve decimal (sub-pixel) accuracy. Therefore, when the coded structured light pattern is a pattern generated by a coding method based on a phase shift code, the first structured light image is decoded, and the first corresponding relationship obtained can also be understood as the corresponding relationship between the integer pixels of the first camera and the sub-pixels of the projector.

[0041] Step S232: performing interpolation conversion on the first corresponding relationship using a preset interpolation algorithm to obtain a corresponding relationship between pixels of the encoded structured light pattern and sub-pixels of the first structured light image as a first pixel corresponding relationship.

[0042] Since the coded structured light pattern projected by the projector can be captured by multiple cameras (such as the first camera and the second camera), it is necessary to convert the first correspondence in the above decoding result into the correspondence between the projector pixels and the camera sub-pixels, that is, to obtain the correspondence between the camera pixels with the projector pixels as the intermediary, so as to adapt to the subsequent motion reconstruction algorithm (structure from motion, SFM) process. Among them, the core of the conversion is interpolation, and the difference algorithm includes but is not limited to affine interpolation, bilinear interpolation, B-spline interpolation, Poisson interpolation and other difference algorithms.

[0043] Optionally, taking the preset interpolation algorithm as an affine interpolation algorithm as an example, the pixel coordinates corresponding to the sub-pixels of the encoded structured light pattern in the first corresponding relationship are rounded to obtain the corresponding relationship between the pixels of the encoded structured light pattern and the pixels of the first structured light image as the second corresponding relationship. In other words, the sub-pixels (i.e., floating-point pixels) of the first structured light image in the first corresponding relationship are rounded, that is, if the decimal part of the floating-point pixel is less than 0.5, its integer part is taken, otherwise its integer part is taken and 1 is added to obtain the corresponding integer pixel; for example, the pixel coordinates of a sub-pixel in the first structured light image obtained by decoding are (100.1, 99, 9), and after rounding, the integer pixel coordinates of (100, 100) can be obtained.

[0044] Further, a preset area is determined according to the pixels of the coded structured light pattern in each of the second corresponding relationships, and a plurality of preset pixels are taken in the predetermined area, and the plurality of preset pixels form a preset pixel set corresponding to each pixel of the coded light pattern. Wherein, the predetermined area may be a target shape area centered on any integer pixel point in the coded light pattern, and the target shape may be a rectangle or a square, which is not limited in this embodiment. It should be noted that there may be overlapping areas between the plurality of preset pixels. Then, according to the second corresponding relationship, a target pixel set corresponding to each of the preset pixel sets is determined from the first structured light image, and a plurality of the target pixel sets are obtained, and the plurality of target pixel sets correspond one-to-one to the plurality of preset pixel sets, wherein the target pixel sets all contain integer pixels. According to the first corresponding relationship, a sub-pixel set corresponding to each of the target pixel sets is determined from the coded light pattern, and a plurality of the sub-pixel sets are obtained, and each sub-pixel set contains sub-pixels (i.e., floating-point pixels).

[0045] Then, an affine transformation matrix between each sub-pixel set and its corresponding target pixel set is obtained, and the sub-pixels in the first structured light pattern corresponding to each pixel of the coded structured light pattern are obtained by using the affine transformation matrix, and the correspondence between the pixels of the coded structured light pattern and the sub-pixels in the first structured light pattern is used as the first pixel correspondence (i.e., the correspondence between the integer pixels of the projector and the sub-pixels of the first camera). The sub-pixels in the first structured light pattern can be calculated by the following formula:

[0046] b1×A=a1

[0047] Wherein, A is an affine matrix, b1 is an integer pixel in the coded structured light pattern, and a1 is a correspondence between a pixel of the coded structured light pattern calculated based on the affine matrix A and a sub-pixel in the first structured light pattern.

[0048] In this manner, the second structured light image is also decoded according to the phase shift code decoding algorithm to obtain the correspondence between the sub-pixels of the encoded structured light pattern and the integer pixels of the second structured light image as the third correspondence; the third correspondence is interpolated and converted using a preset interpolation algorithm to obtain the correspondence between the integer pixels of the encoded structured light pattern and the sub-pixels of the second structured light image as the second pixel correspondence. Specifically, how to decode the second structured light image and the interpolation conversion after decoding can be referred to the above content, which will not be repeated here.

[0049] It should be noted that if, in addition to the first camera and the second camera, there are other cameras to be calibrated that can also capture the coded structured light pattern projected by the aforementioned projector, the correspondence between the pixels of each other camera to be calibrated and the pixels of the projector can be obtained in a similar manner to the above. The pixels of the first structured light image and the pixels of the coded structured light pattern in steps S231 to S232 can all refer to integer pixels; optionally, considering that the integer pixels of the coded structured light pattern projected by the projector correspond to the sub-pixels in the image captured by the camera are too sparse, the pixels of the first structured light image and the pixels of the coded structured light pattern can also be non-integer pixels, for example, 1.5 pixels or 2.5 pixels.

[0050] Step S240: determining a plurality of matching pixel pairs between the first structured light image and the second structured light image based on the first pixel correspondence relationship and the second pixel correspondence relationship.

[0051] Specifically, based on the first pixel correspondence and the second pixel correspondence, two pixels corresponding to the same pixel in the coded structured light pattern are determined from the first structured light image and the second structured light image, respectively, as the matching pixel pairs, to obtain the multiple matching pixel pairs. Of course, since the aforementioned first pixel correspondence and the second pixel correspondence are both dense pixel-level, based on the requirements for parameter calibration time and parameter calibration accuracy, sparse pixel correspondences can be selected randomly or according to a certain step size in the correspondence according to experience, that is, not all matching pixel pairs are used for calibration calculations, but a part of the matching pixel pairs are selected for calibration calculations, so as to ensure the calibration accuracy while ensuring the parameter calibration time, and avoid the occurrence of problems such as too long parameter calibration time due to the use of too many matching pixel pairs.

[0052] Step S250: based on the multiple matching pixel pairs, calibrate the intrinsic parameters and extrinsic parameters of the first camera, the second camera and the projector respectively.

[0053] The projected image can be regarded as the inverse process of the camera imaging, and the calibration principle is the same, so the camera calibration process can be reused. Based on this, the intrinsic and extrinsic parameters of the first camera, the second camera and the projector can be calibrated based on the SFM principle, such as Figure 7 As shown, the first camera and the second camera have matching pixel pairs P1, matching pixel pairs P2, matching pixel pairs P3 and matching pixel pairs P4. Among them, the internal parameters can be understood as the internal parameters of camera imaging or projector inverse imaging, such as focal length and principal point coordinates; the external parameters can be understood as the rotation matrix and translation matrix between the camera and the projector. Optionally, after calibrating the first camera and the second camera to obtain sufficiently accurate camera parameters and three-dimensional point clouds, the projector can be calibrated, which can effectively ensure accurate projector calibration results. In this way, when the projector is an engineering projector or an ultra-short-throw projector, the principal point or focal length range is large. The camera is calibrated first, and then the projector is calibrated, so that a more accurate projector calibration result can be obtained.

[0054] In this embodiment, the illumination control of the scene is achieved by controlling the projector to project the coded structured light pattern, thereby providing the necessary conditions for subsequent image acquisition, and the projection of the coded structured light pattern in the scene is photographed by the first camera and the second camera respectively to obtain the corresponding structured light image; and the first and second structured light images are decoded to obtain the correspondence between the pixels in the image and the coded structured light pattern, which provides a basis for subsequent pixel matching, and then based on the decoded pixel correspondence, multiple matching pixel pairs between the first structured light image and the second structured light image are determined, further strengthening the correlation between the images. Finally, the internal parameters (such as focal length, principal point) and external parameters (such as position, orientation) of the first camera, the second camera and the projector are calibrated using multiple matching pixel pairs, thereby realizing the online calibration of the internal and external parameters of the camera and the projector without the need for a calibration plate.

[0055] Please refer to Figure 8 , Figure 8 A flow chart of a parameter calibration method provided in another embodiment of the present application is applied to a projection system, wherein the projection system at least includes a projector, a first camera, and a second camera. Figure 8 The parameter calibration method provided in the embodiment of the present application is described in detail. The parameter calibration method may include the following steps:

[0056] Step S301: controlling the projector to project a coded structured light pattern.

[0057] Step S302: acquiring a first structured light image and a second structured light image, wherein the first structured light image is obtained by photographing the coded structured light pattern with the first camera, and the second structured light image is obtained by photographing the coded structured light pattern with the second camera.

[0058] Step S303: decoding the first structured light image to obtain a first pixel correspondence relationship between the first structured light image and the encoded structured light pattern, and decoding the second structured light image to obtain a second pixel correspondence relationship between the second structured light image and the encoded structured light pattern.

[0059] Step S304: determining a plurality of matching pixel pairs between the first structured light image and the second structured light image based on the first pixel correspondence relationship and the second pixel correspondence relationship.

[0060] In this embodiment, the specific implementation of steps S301 to S304 can refer to the contents of the above embodiments, which will not be described in detail here.

[0061] Step S305: Estimate initial extrinsic parameters of the first camera and initial extrinsic parameters of the second camera according to the initial intrinsic parameters of the first camera and the second camera and the plurality of matching pixel pairs.

[0062] Step S306: triangulate the plurality of matching pixel pairs based on the initial external parameters of the first camera and the initial external parameters of the second camera to obtain an initial point cloud.

[0063] Specifically, the PNP algorithm can be used to estimate the initial external parameters of the first camera and the initial external parameters of the second camera based on multiple pixel matching pairs. In other words, the camera poses of the first camera and the second camera are estimated respectively, and multiple pixel matching pairs are triangulated based on the estimated camera poses to obtain three-dimensional points in the world coordinate system, that is, the pixel coordinate information is converted into three-dimensional coordinate information in the world coordinate system. It should be noted that the initial internal parameters of the first camera or the second camera can be obtained based on the camera model, technical documents or calibrated parameters, etc., at least the resolution and rough focal length data are obtained as initial parameters, and when there is no principal point data, the principal point at the center of the imaging sensor is used as the initial principal point.

[0064] Step S307: Based on the initial external parameters and initial internal parameters of the first camera, at least some of the points in the initial point cloud are mapped to pixel coordinate information in the first structured light image to obtain first pixel coordinate information; and based on the initial external parameters and initial internal parameters of the second camera, at least some of the points in the initial point cloud are mapped to pixel coordinate information in the second structured light image to obtain second pixel coordinate information.

[0065] Among them, the first pixel coordinate information can be regarded as the coordinate information in the first pixel coordinate system established based on the first camera, and the second pixel coordinate information can be regarded as the coordinate information in the second pixel coordinate system established based on the second camera. It should be noted that the pixel coordinate system is a two-dimensional coordinate system. Mapping at least part of the points in the initial point cloud to the first pixel coordinate information can be understood as mapping the three-dimensional coordinate information of some three-dimensional points in the world coordinate system to the two-dimensional coordinate information in the first pixel coordinate system based on the imaging principle of the camera and using the initial external parameters and initial internal parameters of the first camera. Mapping at least part of the points in the initial point cloud to the second pixel coordinate information can also be understood as mapping the three-dimensional coordinate information of some three-dimensional points in the world coordinate system to the two-dimensional coordinate information in the second pixel coordinate system based on the imaging principle of the camera and using the initial external parameters and initial internal parameters of the second camera.

[0066] It should be noted that the first pixel coordinate information may be pixel coordinate information obtained by mapping the first partial points in the initial point cloud to the first pixel coordinate system, where the first partial points are partial points in the initial point cloud obtained by triangulating partial pixel points in the second structured light image. The second pixel coordinate information may be pixel coordinate information obtained by mapping the second partial points in the initial point cloud to the second pixel coordinate system, where the second partial points are partial points in the initial point cloud obtained by triangulating partial pixel points in the first structured light image. It should be noted that both the first partial points and the second partial points can be captured by the first camera and the second camera.

[0067] Step S308: determining a first reprojection error value according to a difference between the first pixel coordinate information and the first preset coordinate information, and determining a second reprojection error value according to a difference between the second pixel coordinate information and the second preset coordinate information.

[0068] Among them, the first preset coordinate information can be understood as the actual coordinate information of the first part of the points in the first pixel coordinate system, and the actual coordinate information can be the pixel coordinate information determined by the first camera based on the captured structured light image. The first pixel coordinate information is based on the initial intrinsic parameters of the first camera and the estimated initial extrinsic parameters, and maps the three-dimensional coordinate information of the first part of the points in the world coordinate system to the pixel coordinates in the first pixel coordinate system. Among them, the first reprojection error value can be understood as the difference between the actual coordinate information of the first part of the points in the first pixel coordinate system and the estimated first pixel coordinate system; that is, the more accurate the initial intrinsic parameters and the estimated initial extrinsic parameters of the first camera are, the smaller the aforementioned difference value will be, and therefore, the internal and external parameters of the camera can be optimized based on the difference value.

[0069] Similarly, the second preset coordinate information can be understood as the actual coordinate information of the second part of the points in the second pixel coordinate system, and the actual coordinate information can be the pixel coordinate information determined by the second camera based on the captured structured light image. The second pixel coordinate information is based on the initial intrinsic parameters and the estimated initial extrinsic parameters for the second camera, and maps the three-dimensional coordinate information of the second part of the point cloud from the world coordinate system to the pixel coordinates in the second pixel coordinate system. Among them, the second reprojection error value can be understood as the difference between the actual coordinate information of the second part of the points in the second pixel coordinate system and the estimated second pixel coordinate system.

[0070] Step S309: using the preset optimization algorithm and according to the first reprojection error value and the second reprojection error value, respectively, performing iterative optimization on the initial external parameters and the initial internal parameters of the first camera and the second camera, and performing iterative optimization on the initial point cloud, and using the optimized initial point cloud as the target point cloud.

[0071] Among them, the preset optimization algorithm can be the Levenberg-Marquardt method (LM algorithm) or the bundle adjustment method. Of course, other optimization algorithms that can be used to optimize the calibration parameters are also within the scope of protection of this application, and no examples are given here one by one. It can be understood that the smaller the reprojection error value, the more accurate the parameter value representing the camera calibration parameter. Therefore, after obtaining the first reprojection error value and the second reprojection error value, the preset optimization algorithm can be used to iteratively optimize the initial external parameters and initial internal parameters of the first camera according to the first reprojection error value until the preset optimization conditions are met, thereby obtaining the optimized initial external parameters and initial internal parameters of the first camera; and the preset optimization algorithm can be used to iteratively optimize the initial external parameters and initial internal parameters of the second camera according to the second reprojection error value until the preset optimization conditions are met, thereby obtaining the optimized initial external parameters and initial internal parameters of the second camera. Among them, the preset optimization conditions include at least one of: the reprojection error value is less than the first threshold, the reprojection error value no longer changes, and the number of iterative optimization times reaches a first number threshold. The first threshold and the first number threshold are both values ​​that can be pre-set.

[0072] It should be noted that in the iterative optimization process, in addition to optimizing the initial external parameters of the first and second cameras, the initial point cloud will also be optimized at the same time. It can be understood that the internal and external parameters are optimized using the point cloud, and then the new point cloud is calculated using the internal and external parameters. This cycle continues, making the internal and external parameters of the camera and the point cloud more and more accurate.

[0073] Step S310: Calibrate initial external parameters and initial internal parameters of the projector based on the target point cloud.

[0074] Specifically, since the projected image of the projector can be regarded as the inverse process of camera imaging, the calibration principle is the same, so the camera calibration process can be reused, and the PNP (Perspective-n-Point) algorithm can still be used to estimate the initial external parameters of the projector based on multiple pixel matching pairs. It should be noted that the initial internal parameters of the projector can be obtained based on the projector model, technical documents or calibrated parameters and other data, at least the resolution and rough focal length data as initial parameters, and when there is no principal point data, the principal point at the center of the imaging sensor is used as the initial principal point. Based on this, the initial external parameters of the projector can be estimated based on the three-dimensional coordinates of at least some of the target points in the optimized initial point cloud (i.e., the target point cloud), the two-dimensional coordinates of the corresponding points of at least some of the target points in the coded structured light pattern, and the initial internal parameters of the projector, and based on the PNP algorithm. Optionally, at least some of the target points in the target point cloud can be some of the points that can be photographed by both the first camera and the second camera; it can also be some of the points that can be photographed by the first camera; it can also be some of the points that can be photographed by the second camera, and this embodiment does not limit this.

[0075] Further, based on the initial external parameters and initial internal parameters of the projector, at least part of the target points in the target point cloud are mapped to pixel coordinate information in the coded structured light pattern to obtain fourth pixel coordinate information. Specifically, the third part of the point cloud in the target point cloud is mapped to the projector pixel coordinate system to obtain the fourth pixel coordinate. The projector pixel coordinate is a two-dimensional pixel coordinate system established based on the projector. Then, according to the difference between the fourth pixel coordinate information and the fourth preset coordinate information, the fourth reprojection error value is determined; similarly, the fourth preset coordinate information can be understood as the actual coordinate information of the third part of the point cloud in the projector pixel coordinate system, and the actual coordinate information can be the pixel coordinate information determined by the second target camera based on the captured structured light image. The fourth pixel coordinate information is based on the initial internal parameters for the projector and the estimated initial external parameters, and the third part of the point cloud is mapped from the three-dimensional coordinate information in the world coordinate system to the pixel coordinates in the projector pixel coordinate system. Among them, the fourth reprojection error value can be understood as the difference between the actual coordinate information of the third part of the point cloud in the projector pixel coordinate system and the estimated fourth pixel coordinate system.

[0076] Finally, the preset optimization algorithm is used, and according to the fourth reprojection error value, the initial external parameters and initial internal parameters of the projector are iteratively optimized respectively, and the target point cloud is iteratively optimized again. Among them, the preset optimization algorithm can be an LM algorithm or a bundle adjustment method. Of course, other optimization algorithms that can be used to optimize the calibration parameters are also within the protection scope of this application, and no examples are given here one by one. It can be understood that the smaller the reprojection error value, the more accurate the parameter value of the internal and external parameters of the calibrated projector. Therefore, after obtaining the fourth reprojection error value, the preset optimization algorithm can be used, and according to the fourth reprojection error value, the initial external parameters and initial internal parameters of the projector are iteratively optimized until the preset optimization conditions are met, thereby obtaining the optimized initial external parameters and initial internal parameters of the projector. Among them, the preset optimization conditions include at least one of: the fourth reprojection error value is less than the first threshold, the fourth reprojection error value no longer changes, and the number of iterative optimizations reaches the first number threshold. The first threshold and the first number threshold are both values ​​that can be preset. It should be noted that in the iterative optimization process, in addition to optimizing the initial external parameters and initial internal parameters of the projector, the target point cloud will also be optimized again. It can be understood that the internal and external parameters are optimized using the point cloud, and then the new point cloud is calculated using the internal and external parameters. This cycle continues, making the internal and external parameters and point cloud of the projector more and more accurate.

[0077] In some implementations, the projection system includes a first camera, a second camera, and a plurality of cameras to be calibrated. In a specific embodiment, the first camera and the second camera are the two cameras with the largest number of corresponding pixels among all cameras. Of course, two cameras can also be selected from all cameras in other ways as the first camera and the second camera. The present invention is not limited to the specific selection method. Please refer to Fig. 9 , after step S310, the contents of steps S311 to S316 may also be included:

[0078] Step S311: From the multiple cameras to be calibrated, determine a camera to be calibrated that can capture the largest number of points in the target point cloud as the third camera.

[0079] Step S312: Calculate the initial extrinsic parameters of the third camera according to the three-dimensional coordinates of at least some of the target points in the target point cloud that can be captured by the third camera, the two-dimensional coordinates of the corresponding points of at least some of the target points in the image captured by the third camera, and the initial intrinsic parameters of the third camera.

[0080] It should be noted that, when the projection system includes a plurality of cameras to be calibrated in addition to the first camera and the second camera, the internal and external parameters of all cameras may be calibrated first, and then the internal and external parameters of the projector may be calibrated.

[0081] In this embodiment, the third camera is the next camera to be calibrated after the first camera and the second camera are calibrated. Similarly, the third camera can also be generally understood as the camera that can capture the largest number of point clouds in the optimized initial point cloud among multiple cameras to be calibrated. Obviously, the more point clouds captured by the third camera, the more pixel matching pairs representing the image captured by the third camera and the image projected by the projector, so that the initial external parameters of the third camera can be estimated more accurately.

[0082] Similarly, the PNP (Perspective-n-Point) algorithm can still be used to estimate the initial external parameters of the third camera based on multiple matching pixel pairs. It should be noted that the initial internal parameters of the third camera can be obtained based on the model, technical documents or calibrated parameters of the third camera, at least the resolution and rough focal length data as initial parameters, and when there is no principal point data, the principal point at the center of the imaging sensor is used as the initial principal point. Based on this, the initial external parameters of the third camera can be estimated based on the three-dimensional coordinates of at least some of the target points in the target point cloud that can be captured by the third camera, the two-dimensional coordinates of the corresponding points of at least some of the target points in the image captured by the third camera, and the initial internal parameters of the third camera, and the PNP algorithm.

[0083] Step S313: Based on the initial external parameters and initial internal parameters of the third camera, at least part of the target points in the target point cloud are mapped to pixel coordinate information in the third structured light image to obtain third pixel coordinate information.

[0084] Step S314: determining a third reprojection error value according to a difference between the third pixel coordinate information and the third preset coordinate information.

[0085] Specifically, for ease of description, at least part of the points in the target point cloud in step S313 are named as the fourth part of points; the fourth part of points in the target point cloud are mapped to the third camera pixel coordinate system to obtain fourth pixel coordinates. The fourth part of points can be captured by both the third camera and the first target camera, and the first target camera can be a camera with the largest number of target points in the target point cloud captured simultaneously with the third camera, determined from the first camera and the second camera. In addition, the fourth part of points is a part of the point cloud obtained by triangulating part of the pixel points in the structured light image captured by the first target camera; the third pixel coordinate system is a two-dimensional pixel coordinate system established based on the third camera. Then, the third reprojection error value is determined based on the difference between the third pixel coordinate information and the third preset coordinate information; similarly, the third preset coordinate information can be understood as the actual coordinate information of the fourth part of points in the third camera coordinate system, and the actual coordinate information can be the pixel coordinate information determined by the first target camera based on the captured structured light image. The third pixel coordinate information is based on the initial intrinsic parameters and the estimated initial extrinsic parameters for the third camera, and maps the three-dimensional coordinate information of the fourth part of the point cloud from the world coordinate system to the pixel coordinates in the third camera pixel coordinate system. The third reprojection error value can be understood as the difference between the actual coordinate information of the fourth part of the point cloud in the third camera pixel coordinate system and the estimated third pixel coordinate system.

[0086] Step S315: using the preset optimization algorithm and according to the third reprojection error value, respectively iteratively optimizing the initial external parameters and initial internal parameters of the third camera, and iteratively optimizing the target point cloud again.

[0087] Similarly, the preset optimization algorithm is used, and according to the third reprojection error value, the initial external parameters and initial internal parameters of the third camera are iteratively optimized, and the target point cloud is iteratively optimized again. Among them, the preset optimization algorithm can be an LM algorithm or a bundle adjustment method. Of course, other optimization algorithms that can be used to optimize calibration parameters are also within the protection scope of this application, and no examples are given here. It can be understood that the smaller the third reprojection error value, the more accurate the parameter values ​​of the initial external parameters and initial internal parameters of the calibrated third camera. Therefore, after obtaining the third reprojection error value, the preset optimization algorithm can be used, and according to the third reprojection error value, the initial external parameters and initial internal parameters of the third camera are iteratively optimized until the preset optimization conditions are met, thereby obtaining the optimized initial external parameters and initial internal parameters of the third camera. Among them, the preset optimization conditions include at least one of: the third reprojection error value is less than the first threshold, the third reprojection error value no longer changes, and the number of iterative optimizations reaches the first number threshold. The first threshold and the first number threshold are both values ​​that can be preset. It should be noted that in the iterative optimization process, in addition to optimizing the initial external parameters and initial internal parameters of the camera, the optimized initial point cloud (i.e., target point cloud) will also be optimized again. It can be understood that the internal and external parameters are optimized using the point cloud, and then the new point cloud is calculated using the internal and external parameters. This cycle continues, making the internal and external parameters and point cloud of the camera more and more accurate.

[0088] It can be understood that after the calibration of the third camera is completed, the calibration process of the remaining cameras to be calibrated is similar to the calibration process of the third camera, and reference may be made to the aforementioned content, which will not be described in detail here.

[0089] Optionally, if the projection system includes multiple projectors, after completing the calibration of the internal and external parameters of all cameras, according to the specific implementation method mentioned in the above content, first calibrate one of the multiple projectors; further, from the other multiple projectors, determine the projector that can project the largest number of point clouds in the optimized initial point cloud as the target projector. From the calibrated multiple cameras and calibrated projectors, determine the device (camera or projector) that simultaneously captures the largest number of the same point clouds as the target projector as the target device. Estimate the initial external parameters of the target device based on the three-dimensional coordinates of at least some points in the optimized initial point cloud that can be captured by the target device, the two-dimensional coordinates of the corresponding points of at least some of the target points in the captured image (or encoded structured light pattern) of the target device, and the initial internal parameters of the target device; based on the initial external parameters and initial internal parameters of the target device, map at least some of the points in the optimized initial point cloud to pixel coordinate information in the captured image (or encoded structured light pattern) of the target device to obtain fifth pixel coordinate information; determine a fifth reprojection error value based on the difference between the fifth pixel coordinate information and the fifth preset coordinate information; use the preset optimization algorithm and according to the fifth reprojection error value to iteratively optimize the initial external parameters and initial internal parameters of the target device, and iteratively optimize the optimized initial target point cloud again until the preset optimization conditions are met.

[0090] In this embodiment, the illumination control of the scene is achieved by controlling the projector to project the coded structured light pattern, thereby providing the necessary conditions for subsequent image acquisition, and the projection of the coded structured light pattern in the scene is photographed by the first camera and the second camera respectively to obtain the corresponding structured light images; and the first and second structured light images are decoded to obtain the correspondence between the pixels in the image and the coded structured light pattern, which provides a basis for subsequent pixel matching, and then based on the decoded pixel correspondence, multiple matching pixel pairs between the first structured light image and the second structured light image are determined, further strengthening the correlation between the images. Finally, using multiple matching pixel pairs, with limited and imprecise initial internal parameters known, using multi-view geometry and nonlinear optimization principles, according to the SFM process, the precise internal parameters and precise external parameters of the first camera, the second camera and the projector are calculated, thereby realizing the online calibration of the internal and external parameters of the camera and the projector without the need for a calibration plate. Obviously, the projection system provided in this application is used in engineering, home and business projectors to realize online calibration of multiple projectors and multiple cameras, and can well meet the basic needs of realizing personalized dynamic naked-eye 3D display or polarized 3D display, dynamic virtual-reality fusion display, generative artificial intelligence (Artificial Intelligence Generated Content, AIGC) personalized 3D Mapping display, foreground-background separation display, and real-time interactive display.

[0091] Please refer to Fig.10 , which shows a structural block diagram of a parameter calibration device 400 provided in an embodiment of the present application. The device 400 may include: an image projection module 410, an image capture module 420, a decoding module 430, a pixel matching module 440 and a parameter calibration module 450.

[0092] The image projection module 410 is used to control the projector to project a coded structured light pattern.

[0093] The image capturing module 420 is used to obtain a first structured light image and a second structured light image, wherein the first structured light image is captured by the first camera with respect to the coded structured light pattern, and the second structured light image is captured by the second camera with respect to the coded structured light pattern.

[0094] The decoding module 430 is used to decode the first structured light image to obtain a first pixel correspondence between the first structured light image and the encoded structured light pattern, and to decode the second structured light image to obtain a second pixel correspondence between the second structured light image and the encoded structured light pattern.

[0095] The pixel matching module 440 is configured to determine a plurality of matching pixel pairs between the first structured light image and the second structured light image based on the first pixel corresponding relationship and the second pixel corresponding relationship.

[0096] The parameter calibration module 450 is used to calibrate the intrinsic parameters and extrinsic parameters of the first camera, the second camera and the projector respectively based on the multiple matching pixel pairs.

[0097] In some embodiments, the coded structured light pattern is a pattern generated by a coding method based on a phase shift code. The decoding module 430 may include a decoding unit and an interpolation conversion unit. Among them, the decoding unit can be used to decode the first structured light image according to the phase shift code decoding algorithm, and obtain the correspondence between the pixels of the first structured light image and the sub-pixels of the coded structured light pattern as the first correspondence. The interpolation conversion unit can be used to perform an interpolation conversion on the first correspondence using a preset interpolation algorithm, and obtain the correspondence between the pixels of the coded structured light pattern and the sub-pixels of the first structured light image as the first pixel correspondence.

[0098] In this manner, the interpolation conversion unit can be specifically used to round off the pixel coordinates corresponding to the sub-pixels of the coded structured light pattern in the first corresponding relationship to obtain the corresponding relationship between the pixels of the coded structured light pattern and the integer pixels of the first structured light image as the second corresponding relationship; determine a preset area according to each pixel of the coded structured light pattern in the second corresponding relationship, take a plurality of preset pixels in the predetermined area, and the plurality of preset pixels form a preset pixel set corresponding to each pixel of the coded light pattern; determine a target pixel set corresponding to each preset pixel set from the first structured light image according to the second corresponding relationship to obtain a plurality of target pixel sets; determine a sub-pixel set corresponding to each target pixel set from the coded light pattern according to the first corresponding relationship to obtain a plurality of sub-pixel sets; obtain an affine transformation matrix between each sub-pixel set and its corresponding target pixel set; based on the affine transformation matrix, obtain the sub-pixels in the first structured light pattern corresponding to each pixel of the coded structured light pattern, and use the corresponding relationship between the pixels of the coded structured light pattern and the sub-pixels in the first structured light pattern as the first pixel corresponding relationship.

[0099] In some embodiments, the pixel matching module 440 can be specifically used to determine two pixels corresponding to the same pixel in the encoded structured light pattern from the first structured light image and the second structured light image respectively based on the first pixel correspondence relationship and the second pixel correspondence relationship, as the matching pixel pair, to obtain the multiple matching pixel pairs.

[0100] In some embodiments, the parameter calibration module 450 may include: a first parameter estimation unit, a point cloud generation unit, a reprojection error acquisition unit, an iterative optimization unit, and a second parameter estimation unit. The first parameter estimation unit may be used to estimate the initial external parameters of the first camera and the initial external parameters of the second camera based on the initial internal parameters of the first camera and the second camera and the multiple matching pixel pairs. The point cloud generation unit may be used to triangulate the multiple matching pixel pairs based on the initial external parameters of the first camera and the initial external parameters of the second camera to obtain an initial point cloud. The reprojection error acquisition unit may be used to map at least part of the points in the initial point cloud to pixel coordinate information in the first structured light image based on the initial external parameters and initial internal parameters of the first camera to obtain first pixel coordinate information, and to map at least part of the points in the initial point cloud to pixel coordinate information in the second structured light image based on the initial external parameters and initial internal parameters of the second camera to obtain second pixel coordinate information; determine a first reprojection error value based on the difference between the first pixel coordinate information and the first preset coordinate information, and determine a second reprojection error value based on the difference between the second pixel coordinate information and the second preset coordinate information. The iterative optimization unit can be used to use the preset optimization algorithm and, based on the first reprojection error value and the second reprojection error value, iteratively optimize the initial external parameters and initial internal parameters of the first camera and the second camera, respectively, and iteratively optimize the initial point cloud, and use the optimized initial point cloud as the target point cloud. The second parameter estimation unit can be used to calibrate the initial external parameters and initial internal parameters of the projector based on the target point cloud.

[0101] In this manner, the second parameter estimation unit can be specifically used to: estimate the initial external parameters of the projector based on the three-dimensional coordinates of at least part of the target points in the target point cloud, the two-dimensional coordinates of the corresponding points of at least part of the target points in the coded structured light pattern, and the initial internal parameters of the projector; map at least part of the target points in the target point cloud to pixel coordinate information in the coded structured light pattern based on the initial external parameters and initial internal parameters of the projector to obtain fourth pixel coordinate information; determine a fourth reprojection error value based on the difference between the fourth pixel coordinate information and the fourth preset coordinate information; and iteratively optimize the initial external parameters and initial internal parameters of the projector based on the preset optimization algorithm and the fourth reprojection error value, and iteratively optimize the target point cloud again.

[0102] In this manner, the projection system includes a first camera, a second camera, and a plurality of cameras to be calibrated, and the parameter calibration module 450 can also be specifically used to iteratively optimize the initial external parameters and initial internal parameters of the first camera and the second camera respectively according to the first reprojection error value and the second reprojection error value, and iteratively optimize the initial point cloud, and after using the optimized initial point cloud as the target point cloud, determine, from the plurality of cameras to be calibrated, a camera to be calibrated that can capture the largest number of points in the target point cloud as a third camera; and select, according to the number of at least some target points in the target point cloud that can be captured by the third camera, The method comprises the steps of: calculating the initial extrinsic parameters of the third camera based on the three-dimensional coordinates, the two-dimensional coordinates of the corresponding points of at least part of the target points in the image taken by the third camera, and the initial intrinsic parameters of the third camera; mapping at least part of the target points in the target point cloud to pixel coordinate information in a third structured light image based on the initial extrinsic parameters and the initial intrinsic parameters of the third camera to obtain third pixel coordinate information; determining a third reprojection error value based on the difference between the third pixel coordinate information and the third preset coordinate information; and iteratively optimizing the initial extrinsic parameters and the initial intrinsic parameters of the third camera according to the preset optimization algorithm and the third reprojection error value, and iteratively optimizing the target point cloud again.

[0103] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0104] In several embodiments provided in the present application, the coupling between modules may be electrical, mechanical or other forms of coupling.

[0105] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules.

[0106] To summarize, this embodiment realizes illumination control of the scene by controlling the projector to project the coded structured light pattern, thereby providing necessary conditions for subsequent image acquisition, and uses the first camera and the second camera to respectively shoot the projection of the coded structured light pattern in the scene to obtain the corresponding structured light image; and decodes the first and second structured light images to obtain the correspondence between the pixels in the image and the coded structured light pattern, which provides a basis for subsequent pixel matching, and then determines multiple matching pixel pairs between the first structured light image and the second structured light image based on the decoded pixel correspondence, further strengthening the correlation between the images. Finally, the internal parameters (such as focal length, principal point) and external parameters (such as position, orientation) of the first camera, the second camera and the projector are calibrated using multiple matching pixel pairs, thereby realizing online calibration of the internal and external parameters of the camera and the projector without the need for a calibration plate.

[0107] The following will be combined Fig.11 A projection system provided by the present application is described.

[0108] Reference Fig.11 , Fig.11 A structural block diagram of a projection system 500 provided in an embodiment of the present application is shown. The above method provided in an embodiment of the present application can be executed by the projection system 500.

[0109] The projection system 500 in the embodiment of the present application may include one or more of the following components: a processor 501, a memory 502, and one or more applications, wherein the one or more applications may be stored in the memory 502 and configured to be executed by one or more processors 501, and the one or more programs are configured to execute the method as described in the aforementioned method embodiment.

[0110] The processor 501 may include one or more processing cores. The processor 501 uses various interfaces and lines to connect various parts of the entire projection system 500, and executes various functions of the projection system 400 and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 502, and calling data stored in the memory 502. Optionally, the processor 501 can be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 501 can integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem can also be integrated into the processor 501 and implemented separately through a communication chip.

[0111] The memory 502 may include a random access memory (RAM) or a read-only memory (ROM). The memory 502 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 502 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc. The data storage area may also store data (such as the various corresponding relationships described above) created by the projection system 500 during use.

[0112] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0113] In several embodiments provided in the present application, the coupling or direct coupling or communication connection between the modules shown or discussed may be an indirect coupling or communication connection through some interfaces, devices or modules, which may be electrical, mechanical or other forms.

[0114] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules.

[0115] Please refer to Fig.12 , which shows a structural block diagram of a computer-readable storage medium provided in an embodiment of the present application. The computer-readable medium 600 stores program codes, which can be called by a processor to execute the method described in the above method embodiment.

[0116] The computer readable storage medium 600 may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM. Optionally, the computer readable storage medium 600 includes a non-transitory computer-readable storage medium. The computer readable storage medium 600 has storage space for program code 610 that performs any method step of the above method. These program codes can be read from or written to one or more computer program products. The program code 610 can be compressed, for example, in an appropriate form.

[0117] In some embodiments, a computer program product or computer program is provided, the computer program product or computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a projection system reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the projection system performs the steps in the above-mentioned method embodiments.

[0118] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A parameter calibration method, characterized in that: Applied to a projection system, the projection system at least includes a projector, a first camera and a second camera, the method includes: Controlling the projector to project a coded structured light pattern; Acquire a first structured light image and a second structured light image, wherein the first structured light image is obtained by photographing the coded structured light pattern with the first camera, and the second structured light image is obtained by photographing the coded structured light pattern with the second camera; Decoding the first structured light image to obtain a first pixel correspondence relationship between the first structured light image and the coded structured light pattern, and decoding the second structured light image to obtain a second pixel correspondence relationship between the second structured light image and the coded structured light pattern; Determine a plurality of matching pixel pairs between the first structured light image and the second structured light image based on the first pixel correspondence relationship and the second pixel correspondence relationship; Based on the multiple matching pixel pairs, the intrinsic parameters and extrinsic parameters of the first camera, the second camera, and the projector are calibrated respectively.

2. The method according to claim 1, characterized in that The coded structured light pattern is a pattern generated by a coding method based on a phase shift code; The decoding of the first structured light image to obtain a first pixel correspondence relationship between the first structured light image and the coded structured light pattern includes: Decoding the first structured light image according to a phase shift code decoding algorithm to obtain a correspondence between pixels of the first structured light image and sub-pixels of the encoded structured light pattern as a first correspondence; The first corresponding relationship is interpolated and converted by using a preset interpolation algorithm to obtain a corresponding relationship between pixels of the encoded structured light pattern and sub-pixels of the first structured light image as a first pixel corresponding relationship.

3. The method according to claim 2, characterized in that The interpolation conversion of the first corresponding relationship by using a preset interpolation algorithm to obtain a corresponding relationship between pixels of the coded structured light pattern and sub-pixels of the first structured light image as a first pixel corresponding relationship includes: Rounding off pixel coordinates corresponding to sub-pixels of the coded structured light pattern in the first corresponding relationship to obtain a corresponding relationship between pixels of the coded structured light pattern and pixels of the first structured light image as a second corresponding relationship; Determine a preset area according to the pixels of the coded structured light pattern in each of the second corresponding relationships, take a plurality of preset pixels in the predetermined area, and the plurality of preset pixels form a preset pixel set corresponding to each pixel of the coded light pattern; According to the second corresponding relationship, determining a target pixel set corresponding to each of the preset pixel sets from the first structured light image to obtain a plurality of the target pixel sets; According to the first corresponding relationship, determining a sub-pixel set corresponding to each of the target pixel sets from the coded light pattern to obtain a plurality of the sub-pixel sets; Obtaining an affine transformation matrix from each of the sub-pixel sets to its corresponding target pixel set; Based on the affine transformation matrix, sub-pixels in the first structured light pattern corresponding to each pixel of the coded structured light pattern are obtained, and the correspondence between the pixels of the coded structured light pattern and the sub-pixels in the first structured light pattern is used as the first pixel correspondence.

4. The method according to claim 1, characterized in that: The determining, based on the first pixel correspondence relationship and the second pixel correspondence relationship, a plurality of matching pixel pairs between the first structured light image and the second structured light image includes: Based on the first pixel correspondence relationship and the second pixel correspondence relationship, two pixels corresponding to the same pixel in the coded structured light pattern are determined from the first structured light image and the second structured light image respectively as the matching pixel pair, to obtain the multiple matching pixel pairs.

5. The method according to any one of claims 1 to 4, characterized in that: The calibrating the intrinsic parameters and extrinsic parameters of the first camera, the second camera, and the projector respectively based on the multiple matching pixel pairs includes: Estimate initial extrinsic parameters of the first camera and initial extrinsic parameters of the second camera according to initial intrinsic parameters of the first camera and the second camera and the plurality of matching pixel pairs; triangulate the plurality of matching pixel pairs based on initial extrinsic parameters of the first camera and initial extrinsic parameters of the second camera to obtain an initial point cloud; Based on the initial extrinsic parameters and initial intrinsic parameters of the first camera, mapping at least some of the points in the initial point cloud to pixel coordinate information in the first structured light image to obtain first pixel coordinate information; and based on the initial extrinsic parameters and initial intrinsic parameters of the second camera, mapping at least some of the points in the initial point cloud to pixel coordinate information in the second structured light image to obtain second pixel coordinate information; Determine a first reprojection error value according to a difference between the first pixel coordinate information and the first preset coordinate information, and determine a second reprojection error value according to a difference between the second pixel coordinate information and the second preset coordinate information; Using the preset optimization algorithm and according to the first reprojection error value and the second reprojection error value, respectively iteratively optimize the initial external parameters and initial internal parameters of the first camera and the second camera, and iteratively optimize the initial point cloud, and use the optimized initial point cloud as the target point cloud; Based on the target point cloud, initial external parameters and initial internal parameters of the projector are calibrated.

6. The method according to claim 5, characterized in that The step of calibrating the initial external parameters and the initial internal parameters of the projector based on the target point cloud includes: estimating initial extrinsic parameters of the projector based on the three-dimensional coordinates of at least some of the target points in the target point cloud, the two-dimensional coordinates of corresponding points of at least some of the target points in the coded structured light pattern, and the initial intrinsic parameters of the projector; Based on the initial external parameters and the initial internal parameters of the projector, at least part of the target points in the target point cloud are mapped to pixel coordinate information in the coded structured light pattern to obtain fourth pixel coordinate information; Determining a fourth reprojection error value according to a difference between the fourth pixel coordinate information and the fourth preset coordinate information; Using the preset optimization algorithm and according to the fourth reprojection error value, the initial external parameters and initial internal parameters of the projector are iteratively optimized respectively, and the target point cloud is iteratively optimized again.

7. The method according to claim 5, characterized in that The projection system includes the first camera, the second camera, and a plurality of cameras to be calibrated; After using the preset optimization algorithm and iteratively optimizing the initial external parameters and initial internal parameters of the first camera and the second camera according to the first reprojection error value and the second reprojection error value, and iteratively optimizing the initial point cloud, and using the optimized initial point cloud as the target point cloud, the method further includes: From the multiple cameras to be calibrated, determine a camera to be calibrated that can capture the largest number of points in the target point cloud as a third camera; Calculate the initial extrinsic parameters of the third camera according to the three-dimensional coordinates of at least some of the target points in the target point cloud that can be captured by the third camera, the two-dimensional coordinates of the corresponding points of the at least some of the target points in the image captured by the third camera, and the initial intrinsic parameters of the third camera; Based on the initial external parameters and initial internal parameters of the third camera, mapping at least part of the target points in the target point cloud into pixel coordinate information in the third structured light image to obtain third pixel coordinate information; Determining a third reprojection error value according to a difference between the third pixel coordinate information and the third preset coordinate information; Using the preset optimization algorithm and according to the third reprojection error value, the initial external parameters and the initial internal parameters of the third camera are iteratively optimized respectively, and the target point cloud is iteratively optimized again.

8. A parameter calibration device, characterized in that: It is characterized in that Applied to a projection system, the projection system at least includes a projector, a first camera and a second camera, and the device includes: An image projection module, used to control the projector to project a coded structured light pattern; An image capturing module, used to acquire a first structured light image and a second structured light image, wherein the first structured light image is captured by the first camera with respect to the coded structured light pattern, and the second structured light image is captured by the second camera with respect to the coded structured light pattern; A decoding module, configured to decode the first structured light image to obtain a first pixel correspondence relationship between the first structured light image and the coded structured light pattern, and to decode the second structured light image to obtain a second pixel correspondence relationship between the second structured light image and the coded structured light pattern; a pixel matching module, configured to determine a plurality of matching pixel pairs between the first structured light image and the second structured light image based on the first pixel corresponding relationship and the second pixel corresponding relationship; A parameter calibration module is used to calibrate the intrinsic parameters and extrinsic parameters of the first camera, the second camera and the projector respectively based on the multiple matching pixel pairs.

9. A projection system, characterized in that: The projection system comprises: Memory; One or more processors coupled to the memory; One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program code, and the program code can be called by a processor to execute the method as claimed in any one of claims 1 to 7.

Citation Information

Patent Citations

  • Linear structured light vision sensor calibration method based on projector

    CN112179292A

  • Multi-camera joint calibration method and device, terminal equipment and readable storage medium

    CN112330752A

  • Calibration method and device for distance measurement device and camera fusion system

    CN113538592A

  • Projection equipment and projection equipment parameter calibration method

    CN115953481A

  • Camera intrinsic parameter calibration method and device based on three-dimensional target

    CN116883511A

Cited By

  • Plano-convex lens curvature radius detection method and system based on structured light reconstruction and sub-pixel fitting, and medium

    CN121033013A