Projector calibration method and device, electronic equipment and storage medium

Abstract

The invention provides a projector calibration method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining at least three images projected by a projector to a preset rectangular calibration region, the at least three images being images projected by the projector at at least three positions; for any part in the camera and the ray machine, calibrating parameters of the part according to the corresponding relation between the pixel coordinates and the world coordinates of the at least three images under the part coordinate system; and calibrating external parameters between every two of the camera, the ray machine and the IMU according to the rotation relations between the camera, the ray machine and the IMU and the world coordinate system. By adopting the implementation mode, calibration can be carried out by means of the preset rectangular calibration area, and the equipment cost can be reduced. In addition, in the technical scheme, calibration can be completed by obtaining three images, and the cost of a CPU can be reduced.

Description

Technical Field

[0001] The present application belongs to the field of electronic technology, and specifically relates to a projector calibration method, device, electronic equipment and storage medium. Background Art

[0002] Projector calibration refers to determining the internal parameters of each device in the projector and the external parameters between the devices. After the projector is calibrated, various optimization algorithms of the projector can be called, such as the projector's keystone correction algorithm.

[0003] A conventional projector calibration method requires the use of an industrial camera, a turntable, an IMU, and an AprilTag calibration plate installed on the projector. The projector is moved left, right, up, and down by the turntable, and the camera images of the projector at multiple different positions are obtained by the industrial camera, and the spatial posture of the projector is identified by the Inertial Measurement Unit (IMU). Then, the AprilTag algorithm is collected and the AprilTag calibration plate is used to calibrate the projector.

[0004] It can be seen that this calibration method not only requires the assistance of multiple high-precision devices, which is costly, but also requires the use of a complex AprilTag algorithm for calibration, resulting in a large central processing unit (CPU) overhead. Summary of the invention

[0005] The present application proposes a projector calibration method, device, electronic device and storage medium, which can reduce the calibration cost of the projector and also reduce the CPU overhead.

[0006] The first embodiment of the present application proposes a projector calibration method, wherein the projector includes a built-in camera, an optical machine, and an inertial measurement unit IMU, and the method includes:

[0007] Acquire at least three images projected by the projector onto a preset rectangular calibration area, wherein the at least three images are images projected by the projector at at least three positions, respectively, and projection positions corresponding to any two images are different;

[0008] For any component of the camera and the optical machine, calibrate the parameters of the component according to the correspondence between the pixel coordinates of the at least three images and the world coordinates in the component coordinate system; the reference coordinate system of the world coordinates is a world coordinate system with the upper left corner of the preset rectangular calibration area as the origin;

[0009] According to the rotation relationship between the camera, the optical machine and the IMU and the world coordinate system respectively, the external parameters between the camera, the optical machine and the IMU are calibrated.

[0010] In some embodiments of the present application, for the camera, calibrating the parameters of the camera according to the correspondence between the pixel coordinates of the at least three images in the camera coordinate system and the world coordinates includes:

[0011] For any image of the at least three images, obtain a conversion relationship between pixel coordinates of the image in the coordinate system of the camera and world coordinates in the coordinate system of the camera to obtain at least three conversion relationships;

[0012] The at least three conversion relationships are used as parameters by adopting a preset algorithm to calculate the intrinsic parameters of the camera and the extrinsic parameters between the camera and the preset rectangular calibration area.

[0013] In some embodiments of the present application, the image is an icon array, and acquiring the conversion relationship between the pixel coordinates of the image in the coordinate system of the camera and the world coordinates in the coordinate system of the camera includes:

[0014] Determine icons of the image at four vertices of the preset rectangular calibration area;

[0015] Obtain the conversion relationship between the pixel coordinates of the icons of the four fixed points in the coordinate system of the camera and the world coordinates in the coordinate system of the camera.

[0016] In some embodiments of the present application, for the optical machine, calibrating the parameters of the optical machine according to the correspondence between the pixel coordinates of the at least three images and the world coordinates in the optical machine coordinate system includes:

[0017] For any image of the at least three images, calculate the world coordinates of the image in the coordinate system of the optical machine according to the pixel coordinates and world coordinates of the image in the coordinate system of the camera and the pixel coordinates in the coordinate system of the optical machine, so as to obtain the world coordinates of the at least three images in the coordinate system of the optical machine;

[0018] According to the pixel coordinates of the at least three images in the coordinate system of the optical machine and the world coordinates in the coordinate system of the optical machine, the intrinsic parameters of the optical machine and the extrinsic parameters of the optical machine to the preset rectangular calibration area are calculated.

[0019] In some embodiments of the present application, calculating the world coordinates of the image in the coordinate system of the optical machine according to the pixel coordinates and the world coordinates of the image in the coordinate system of the camera and the pixel coordinates in the coordinate system of the optical machine includes:

[0020] Calculate a conversion parameter matrix between pixel coordinates of the image in the camera's coordinate system and world coordinates in the camera's coordinate system;

[0021] The world coordinates of the image in the coordinate system of the optical machine are calculated according to the pixel coordinates of the image in the coordinate system of the optical machine and the conversion parameter matrix.

[0022] In some embodiments of the present application, calibrating the external parameters between the camera and the optical machine according to the rotation relationship between the camera and the optical machine and the world coordinate system respectively includes:

[0023] Acquire a rotation relationship between the camera coordinate system and the world coordinate system to obtain a first rotation relationship;

[0024] Acquire the rotation relationship between the optical machine coordinate system and the world coordinate system to obtain a second rotation relationship;

[0025] The external parameters between the camera and the optical machine are calibrated according to the first rotation relationship and the second rotation relationship.

[0026] In some embodiments of the present application, the method further includes: reading static IMU data when the position of the projector faces the middle position of the preset rectangular calibration area; calibrating the external parameters between the components and the IMU according to the rotation relationship between any component of the camera and the optical machine and the world coordinate system, including:

[0027] Calculating a rotation relationship between a coordinate system of the IMU aligned in a vertical direction and the world coordinate system to obtain a third rotation relationship, and calculating a rotation relationship between a coordinate system of the component aligned in a horizontal direction and the world coordinate system to obtain a fourth rotation relationship;

[0028] Calculate the external parameters between the component and the IMU according to the third rotation relationship and the fourth rotation relationship;

[0029] Alternatively, calculating the rotation relationship between the coordinate system of the IMU aligned in the horizontal direction and the world coordinate system to obtain a fifth rotation relationship, and calculating the rotation relationship between the coordinate system of the component aligned in the vertical direction and the world coordinate system to obtain a sixth rotation relationship;

[0030] According to the fifth rotation relationship and the sixth rotation relationship, the external parameters between the component and the IMU are calculated.

[0031] An embodiment of a second aspect of the present application provides a projector calibration device, the device comprising:

[0032] An acquisition module, used for acquiring at least three images projected by the projector onto a preset rectangular calibration area, wherein the at least three images are images projected by the projector at at least three positions, and projection positions corresponding to any two images are different;

[0033] A first calibration module is used to calibrate the parameters of any component in the camera and optical machine built into the projector according to the correspondence between the pixel coordinates of the at least three images and the world coordinates in the component coordinate system; the reference coordinate system of the world coordinate is a world coordinate system with the upper left corner of the preset rectangular calibration area as the origin;

[0034] The second calibration module is used to calibrate the external parameters between the camera, the optical machine and the IMU according to the rotation relationship between the camera, the optical machine and the IMU built into the projector and the world coordinate system respectively.

[0035] An embodiment of the third aspect of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect above.

[0036] An embodiment of the fourth aspect of the present application provides a computer-readable storage medium on which a computer program is stored, and the program is executed by a processor to implement the method described in the first aspect above.

[0037] The technical solution provided in the embodiments of the present application has at least the following technical effects or advantages:

[0038] In an embodiment of the present application, a rectangular calibration area is preset, and then, at least three images are acquired by projecting the projector to the preset rectangular calibration area at at least three positions. That is to say, the technical solution uses the preset rectangular calibration area to perform calibration, and does not need to use the high-precision equipment of the peripheral for calibration, so as to reduce the equipment cost. Further, the world coordinate system with the upper left corner of the preset rectangular calibration area as the origin is used as the reference coordinate system for calibration. Afterwards, for any component in the camera and the optical machine, the parameters of the component are calibrated according to the correspondence between the pixel coordinates of the at least three images and the world coordinates in the component coordinate system. And, according to the rotation relationship between the camera, the optical machine and the IMU and the world coordinate system respectively, the external parameters between the camera, the optical machine and the IMU are calibrated. That is, in the technical solution, even if three images are acquired, the calibration can be completed based on the conversion between the coordinate systems, not only the amount of image data processed is reduced, but also the algorithm is relatively simple, which is conducive to reducing the CPU overhead and improving the efficiency of calibration.

[0039] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] By reading the detailed description of the preferred embodiment below, various other advantages and benefits will become clear to those of ordinary skill in the art. The accompanying drawings are only used for the purpose of illustrating the preferred embodiment and are not considered to be limitations of the present application. In addition, the same reference symbols are used to represent the same components throughout the accompanying drawings.

[0041] In the attached picture:

[0042] Figure 1 A schematic diagram of a method flow of a projector calibration method provided by an embodiment of the present application is shown;

[0043] Figure 2 A schematic diagram of a rectangular calibration area provided in an embodiment of the present application is shown;

[0044] Figure 3 A schematic diagram showing a projection image provided by an embodiment of the present application is shown;

[0045] Figure 4A A schematic diagram of a projection scene provided by an embodiment of the present application is shown;

[0046] Figure 4B A schematic diagram of a second projection scene provided by an embodiment of the present application is shown;

[0047] Figure 4C A schematic diagram of a third projection scene provided by an embodiment of the present application is shown;

[0048] Figure 5 A schematic diagram of the structure of a projector calibration device provided in one embodiment of the present application is shown;

[0049] Figure 6 A schematic diagram of the structure of an electronic device provided by an embodiment of the present application is shown;

[0050] Figure 7 A schematic diagram of a storage medium provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0051] The exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0052] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in this application should have the common meanings understood by technicians in the field to which this application belongs.

[0053] The embodiments of the present application relate to the technical scenario of projector calibration. Projector calibration refers to obtaining the internal parameters of the projector's built-in camera, the internal parameters of the projector's built-in optical machine, the internal parameters of the projector's built-in IMU, the external parameters between the projector's built-in camera and the optical machine, the external parameters between the projector's built-in camera and IMU, etc.

[0054] As described in the background technology of this specification, the conventional projector calibration method requires the use of an external high-precision turntable, a high-precision industrial camera and a high-precision AprilTag calibration board, and these devices are relatively expensive, making the calibration cost high. In addition, calibration using the AprilTag calibration board requires at least 5 images and requires the acquisition of the AprilTag algorithm. In this way, more image data needs to be processed, and the algorithm is relatively complex, the computational complexity is high, and the CPU overhead is also large.

[0055] In view of this, an embodiment of the present application provides a projector calibration scheme, which uses a preset rectangular calibration area to obtain images projected by the projector at different positions to the preset rectangular calibration area, and obtains at least three images. Further, the world coordinate system with the upper left corner of the preset rectangular calibration area as the origin is used as the reference coordinate system, and the internal parameters of the corresponding components are obtained through the relationship between the pixel coordinates of the image at each position and the world coordinates under the coordinate system of each component built into the projector. And, through the rotation relationship between the coordinate system of each component and the world coordinate system, the external parameters between the components are obtained. It can be seen that by adopting this implementation method, calibration can be performed with the help of a preset rectangular calibration area, without the need for calibration with the help of external high-precision equipment, thereby reducing equipment costs. In addition, in the technical solution, even if three images are obtained, calibration can be completed based on the conversion between coordinate systems, not only the amount of image data processed is reduced, but also the algorithm is relatively simple, which is conducive to reducing CPU overhead.

[0056] The executor of the technical solution can be the projector body to be calibrated, or any device that supports reading the parameters of the built-in devices of the projector, such as a CPU, a graphics processing unit (GPU), a chip system, etc.

[0057] A projector calibration method, device, electronic device and storage medium proposed according to an embodiment of the present application are described below in conjunction with the accompanying drawings.

[0058] See also Figure 1 , Figure 1 A projector calibration method provided by an embodiment of the present application is shown. The projector to be calibrated involved in the embodiment of the present application includes a built-in camera, an optical machine and an IMU. Figure 1The illustrated exemplary projector calibration method can be explained by calibrating the intrinsic and extrinsic parameters of the camera, the intrinsic and extrinsic parameters of the optomechanical machine, and the extrinsic parameters between the camera, the optomechanical machine, and the IMU.

[0059] like Figure 1 As shown, the projector calibration method may include steps S101-S103.

[0060] Step S101, obtaining at least three images projected by the projector onto a preset rectangular calibration area.

[0061] The preset rectangular calibration area may be a blank screen or a projection area defined in any flat projectable area. In some embodiments, in order to clarify the range of the rectangular calibration area and to determine the size of the rectangular calibration area, the boundary of the preset rectangular calibration area may be marked.

[0062] For example, a preset rectangular calibration area can be Figure 2 As shown, the black rectangular frame is the boundary of the preset rectangular calibration area, and the area enclosed by the black rectangular frame is the preset rectangular calibration area.

[0063] In some embodiments, the at least three images may be images projected by the optical machine of the projector onto the above-mentioned preset rectangular calibration area, and these images may be collected by the camera of the projector. The at least three images may be images projected by the projector at at least three positions, wherein the projection positions corresponding to any two images are different. Exemplarily, the position of the projector may be moved multiple times in at least one of the horizontal and vertical directions to obtain images projected onto the preset rectangular calibration area at each position. For example, FIG. 4A to FIG. 4C The illustrated implementation method is to move the position of the projector to the left and right in the horizontal direction to obtain images at the left, center and right positions.

[0064] Exemplarily, the at least three projection positions may include a projection position facing the middle of a preset rectangular calibration area. When the position of the projector faces the middle of the preset rectangular calibration area, the static IMU data of the projector may be read to use the static IMU data as reference data for calibrating the IMU.

[0065] In some embodiments, the contents of the at least three images may be the same, which is helpful to minimize the information interference caused by image changes, so that during the calibration process, the changes and differences in the position data represented by the images at different positions can be more direct and obvious.

[0066] For example, in order to further reduce the computational complexity during the calibration process, the projected image may be icons evenly arranged in a preset rectangular calibration area. Figure 3As shown in the evenly arranged black circles, the accuracy of the data can be improved by obtaining the pixel coordinates of the center of the circle, and the data matrix can be obtained according to the order of the circular icons, which is not only conducive to improving the data processing efficiency but also can achieve relatively accurate matching algorithms.

[0067] It should be understood that Figure 3 This is only an illustrative example of the present application. The projected image may also be other icons evenly arranged in a preset rectangular calibration area, which will not be listed one by one here.

[0068] It can be seen that, by adopting this implementation, there is no need to use a calibration plate and more than or equal to five images for calibration. Calibration can be performed by using a preset rectangular calibration area and using at least three images, thereby reducing equipment costs and simplifying the algorithm.

[0069] Step S102 , for any component in the camera and the optical machine, calibrate the parameters of the component according to the correspondence between the pixel coordinates of the at least three images and the world coordinates in the component coordinate system.

[0070] Among them, the pixel coordinates can be coordinates in a two-dimensional coordinate system, and the two-dimensional coordinate system can be a Cartesian coordinate system. The world coordinates can be coordinates in a three-dimensional coordinate system. The three-dimensional reference coordinate system in this embodiment can be a world coordinate system with the upper left corner of the preset rectangular calibration area as the origin. The X-axis of the world coordinate system is the direction of the width of the rectangular frame, and the width value of the rectangular frame is used to calibrate the coordinates of the X-axis of the world coordinate system; the Y-axis of the world coordinate system is the direction of the height of the rectangular frame, and the height and width value of the rectangular frame are used to calibrate the coordinates of the Y-axis of the world coordinate system; the Z-axis of the world coordinate system can point to the direction of the projector, and can be used to calibrate the distance between the projector and the rectangular calibration area.

[0071] For example, for a camera or optical machine, the parameters to be calibrated may include the internal parameters of the corresponding component and the external parameters of the corresponding component relative to a preset rectangular calibration area. The external parameters of the corresponding component relative to the preset rectangular calibration area can also be considered as the external parameters of the coordinate system of the corresponding component relative to the world coordinate system.

[0072] In view of the relationship between the camera and the optical machine in the projector, in some embodiments, the parameters of the camera can be calibrated first, and then, based on the data obtained during the camera calibration process, the parameters of the optical machine can be calibrated.

[0073] Exemplarily, for the built-in camera of the projector, the parameters of the camera can be calibrated according to the correspondence between the pixel coordinates of at least three images in the camera coordinate system and the world coordinates. Specifically, for any of the at least three images, the conversion relationship between the pixel coordinates of the image in the camera coordinate system and the world coordinates in the camera coordinate system is obtained to obtain at least three conversion relationships. Then, a preset algorithm can be used to use the at least three conversion relationships as parameters to calculate the intrinsic parameters of the camera and the extrinsic parameters between the camera and the preset rectangular calibration area.

[0074] It should be noted that obtaining the conversion relationship between the pixel coordinates of the image in the coordinate system of the camera and the world coordinates in the coordinate system of the camera may be to obtain the pixel coordinates of each icon in the image and the world coordinates in the coordinate system of the camera. In order to reduce the amount of calculation, the icons of the four vertices of the image in the preset rectangular calibration area may also be determined, that is, Figure 3 Then, the conversion relationship between the pixel coordinates of the icons of the four fixed points in the coordinate system of the camera and the world coordinates in the coordinate system of the camera is obtained.

[0075] Furthermore, in order to calibrate the optical machine, the world coordinates of each image in the coordinate system of the optical machine can be obtained. The world coordinates of each image in the coordinate system of the optical machine can be obtained according to the mapping relationship between the pixel coordinates and the world coordinates in the camera coordinate system.

[0076] Exemplarily, for each image, a conversion parameter matrix of the pixel coordinates of the image in the camera's coordinate system and the world coordinates in the camera's coordinate system can be calculated, and the conversion parameter matrix can be, for example, a homography matrix. The homography matrix corresponding to each image can represent the conversion angle data and translation data of the pixel coordinates converted to the world coordinates at the projection position. Furthermore, the world coordinates of the image in the optical machine's coordinate system can be calculated based on the pixel coordinates of the image in the optical machine's coordinate system and the conversion parameter matrix. The optical machine's coordinate system can be, for example, a checkerboard coordinate system.

[0077] Furthermore, for the optical machine, the intrinsic parameters of the optical machine and the extrinsic parameters of the optical machine to the preset rectangular calibration area can be calculated according to the pixel coordinates of the at least three images in the coordinate system of the optical machine and the world coordinates in the coordinate system of the optical machine.

[0078] For example, for each of the at least three images, the coordinates of each icon (such as Figure 3 The pixel coordinates of each circle center in the diagram, and the pixel coordinates of each icon in the checkerboard coordinate system (such as Figure 3Then, the intrinsic parameters of the optical machine and the extrinsic parameters of the optical machine to the preset rectangular calibration area are calculated according to the transformation relationships corresponding to the at least three images.

[0079] It can be seen that with this implementation, there is no need to use the AprilTag algorithm for calibration, but the calibration is performed using the conversion relationship between the pixel coordinates and the world coordinates in the coordinate system of the component to be calibrated when the image is at different positions. This can reduce the complexity of the algorithm and the CPU overhead, thereby improving the efficiency of calibration.

[0080] Step S103, calibrating the external parameters between the camera, the optical machine and the IMU according to the rotation relationship between the camera, the optical machine and the IMU and the world coordinate system respectively.

[0081] Among them, the external parameters between the camera, the optical machine and the IMU can be the relative posture relationship between the camera, the optical machine and the IMU.

[0082] The external parameters between the camera and the optical machine can be determined based on the rotation relationship between the two and the world coordinate system. Specifically, the rotation relationship between the coordinate system of the camera and the world coordinate system can be obtained to obtain a first rotation relationship; and the rotation relationship between the coordinate system of the optical machine and the world coordinate system can be obtained to obtain a second rotation relationship; according to the first rotation relationship and the second rotation relationship, the external parameters between the camera and the optical machine are calibrated.

[0083] It should be noted that the first rotation relationship may be the rotation relationship between the coordinate system of the camera and the world coordinate system calculated based on the image projected by the projector at the middle position of the preset rectangular calibration area. The second rotation relationship may be the rotation relationship between the coordinate system of the optical machine and the world coordinate system calculated based on the image projected by the projector at the middle position of the preset rectangular calibration area.

[0084] For any component in the camera and optical machine, the external parameters between the component and the IMU can be calibrated according to the rotation relationship between the corresponding component and the world coordinate system. Specifically, the rotation relationship between the coordinate system of the IMU aligned in the vertical direction and the world coordinate system can be calculated to obtain a third rotation relationship, and the rotation relationship between the coordinate system of the component aligned in the horizontal direction and the world coordinate system can be calculated to obtain a fourth rotation relationship; according to the third rotation relationship and the fourth rotation relationship, the external parameters between the component and the IMU are calculated. Alternatively, the rotation relationship between the coordinate system of the IMU aligned in the horizontal direction and the world coordinate system can be calculated to obtain a fifth rotation relationship, and the rotation relationship between the coordinate system of the component aligned in the vertical direction and the world coordinate system can be calculated to obtain a sixth rotation relationship; according to the fifth rotation relationship and the sixth rotation relationship, the external parameters between the component and the IMU are calculated.

[0085] It can be seen that by adopting this implementation method, even three projection images at different positions can be used, and the world coordinate system determined by the preset rectangular calibration area can be used as a reference. The calibration can be completed based on the conversion between coordinate systems. Not only the amount of processed image data is reduced, but also the algorithm is relatively simple, which is conducive to reducing CPU overhead.

[0086] The following introduces the projector calibration method of the embodiment of the present application through an exemplary projection scene and an exemplary algorithm.

[0087] The rectangular calibration area is implemented as Figure 2 As shown, the image projected by the projector onto the rectangular calibration area is as follows Figure 3 The following table shows an example. FIG. 4A to FIG. 4C , Figure 4A is the scene projected by the projector in the middle position, Figure 4B is the scene projected by the projector at the left position, Figure 4C This is the scene projected by the projector at the right position. FIG. 4A to FIG. 4C , take the projection images at the left, center and right positions obtained by the built-in camera of the projector as an example.

[0088] It should be pointed out that in Figure 4A In the scenario, the optical machine of the projector can be perpendicular to the cross-section of the rectangular calibration area, and the projector faces the middle position of the rectangular calibration area in the width direction of the rectangular calibration area.

[0089] For example, in Figure 4A In the scenario, the IMU data of the projector can also be collected when it is stationary.

[0090] Furthermore, the internal parameters of the built-in camera of the projector can be calibrated. For example, the pixel coordinates of the corner points in the rectangular calibration area in the camera coordinate system can be extracted. Then we can use the pixel coordinates of the corner points in the camera coordinate system and the world coordinates of the corner points Construct the camera calibration function:

[0091]

[0092] Among them, K c refers to the camera's intrinsic parameter matrix, It refers to the middle position (i.e. Figure 4A The external parameter matrix between the world coordinate system and the camera coordinate system corresponding to the image collected by the scene, Refers to the left position (i.e. Figure 4B The external parameter matrix between the world coordinate system and the camera coordinate system corresponding to the image collected by the scene, Refers to the right position (i.e. Figure 4C The external parameter matrix between the world coordinate system and the camera coordinate system corresponding to the image collected by the scene, Represents the pixel coordinates extracted in the camera coordinate system, where i refers to the images at the center, left, and right positions, and j represents the number of pixel coordinates extracted in each image. Represents the world coordinates of the rectangular calibration area.

[0093] It should be noted that even though the three images correspond to different placements of the projector, the world coordinates of the rectangular calibration area remain unchanged. There is no difference in the values ​​between the different images.

[0094] Furthermore, the internal parameters of the built-in optical machine of the projector can be calibrated. For example, the pixel coordinates of the corner points in the rectangular edge area in the camera coordinate system can be first calculated. and world coordinates The corresponding relationship between the left, middle and right images is constructed by the homography matrix H i Then, extract the center coordinates of the circular icons in each image Using the homography matrix H i The pixel coordinates of the center of the circle are converted from the camera coordinate system to the world coordinate system using the following conversion algorithm: The conversion algorithm between pixel coordinates and world coordinates is, for example:

[0095]

[0096] After that, extract the pixel coordinates of the center of the circular icon in the optical-mechanical coordinate system The optical-mechanical coordinate system is, for example, in the form of a checkerboard. Then, the world coordinates of the center of the circle obtained by the above conversion are Pixel coordinates in the optical coordinate system Perform the optical machine internal parameter calibration. The calibration function is shown in the following formula:

[0097]

[0098] Among them, K p is the internal parameter matrix of the optical machine, It refers to the middle position (i.e. Figure 4A The external parameter matrix between the world coordinate system and the optomechanical coordinate system corresponding to the image collected by the scene, Refers to the left position (i.e. Figure 4B The external parameter matrix between the world coordinate system and the optomechanical coordinate system corresponding to the image collected by the scene, Refers to the right position (i.e. Figure 4C The extrinsic parameter matrix between the world coordinate system and the optomechanical coordinate system corresponding to the image captured by the scene.

[0099] After that, external parameter calibration can be performed, which can include calibrating the external parameters between the camera and the optical machine, and the external parameters between the camera and the IMU.

[0100] For example, the images at the left, center, and right positions can be used, and the external parameter matrix between the world coordinate system and the camera coordinate system can be And the external parameter matrix between the world coordinate system and the optomechanical coordinate system Calibrate the external parameters between the camera and the optomechanics The external parameter calibration function is, for example, the following formula:

[0101]

[0102] The external parameters between the camera and the IMU can be calibrated based on the static IMU data and the border line of the rectangular calibration area in the image. For example, when the position of the projector in the projection scene is not tilted in the horizontal direction, the static IMU data is used for vertical alignment to calculate the rotation matrix R from the IMU coordinate system to the world coordinate system. wb For the image in the middle (i.e. Figure 4A The image collected in the scene) is calculated to obtain the rotation matrix R between the camera coordinate system and the world coordinate system wc , then, the rotation matrix R between the camera coordinate system and the IMU coordinate system can be obtained by using the conversion relationship between the rotation matrices bc : R bc =(R wb ) T R wc , and then obtain the external parameters between the camera and IMU.

[0103] It should be understood that FIG. 4A to FIG. 4CThe illustrated projection scene is a schematic description and does not limit the projector calibration involved in the embodiments of the present application. In other embodiments, the projection position of the projector can also be other, for example, the projector moves in the vertical direction; the number of images collected can also be greater than three. This is not limited here.

[0104] In an embodiment of the present application, a rectangular calibration area is preset, and then, at least three images are acquired by projecting the projector to the preset rectangular calibration area at at least three positions. That is to say, the technical solution uses the preset rectangular calibration area to perform calibration, and does not need to use the high-precision equipment of the peripheral for calibration, so as to reduce the equipment cost. Further, the world coordinate system with the upper left corner of the preset rectangular calibration area as the origin is used as the reference coordinate system for calibration. Afterwards, for any component in the camera and the optical machine, the parameters of the component are calibrated according to the correspondence between the pixel coordinates of the at least three images and the world coordinates in the component coordinate system. And, according to the rotation relationship between the camera, the optical machine and the IMU and the world coordinate system respectively, the external parameters between the camera, the optical machine and the IMU are calibrated. That is, in the technical solution, even if three images are acquired, the calibration can be completed based on the conversion between the coordinate systems, not only the amount of image data processed is reduced, but also the algorithm is relatively simple, which is conducive to reducing the CPU overhead and improving the efficiency of calibration.

[0105] The present application also provides a projector calibration device, which can be used to execute the projector calibration method provided in any of the above embodiments. Figure 5 As shown, the device comprises:

[0106] The acquisition module 501 is used to acquire at least three images projected by the projector to a preset rectangular calibration area, wherein the at least three images are images projected by the projector at at least three positions, and the projection positions corresponding to any two images are different; the first calibration module 502 is used to calibrate the parameters of any component in the camera and optical machine built into the projector according to the correspondence between the pixel coordinates of the at least three images and the world coordinates in the coordinate system of the component; the reference coordinate system of the world coordinate is the world coordinate system with the upper left corner of the preset rectangular calibration area as the origin; the second calibration module 503 is used to calibrate the external parameters between the camera, the optical machine and the IMU built into the projector according to the rotation relationship between the camera, the optical machine and the IMU built into the projector and the world coordinate system respectively.

[0107] Optionally, the first calibration module 502 is further used to obtain, for any image among the at least three images, a conversion relationship between the pixel coordinates of the image in the coordinate system of the camera and the world coordinates in the coordinate system of the camera, so as to obtain at least three conversion relationships; and to use a preset algorithm to use the at least three conversion relationships as parameters to calculate the intrinsic parameters of the camera and the extrinsic parameters between the camera and the preset rectangular calibration area.

[0108] Optionally, the image is an icon array, and the first calibration module 502 is further used to determine the icons of the four vertices of the image in the preset rectangular calibration area; and obtain the conversion relationship between the pixel coordinates of the icons of the four fixed points in the coordinate system of the camera and the world coordinates in the coordinate system of the camera.

[0109] Optionally, the first calibration module 502 is further used to calculate, for any image of the at least three images, the world coordinates of the image in the coordinate system of the optical machine based on the pixel coordinates and the world coordinates of the image in the coordinate system of the camera and the pixel coordinates in the coordinate system of the optical machine, so as to obtain the world coordinates of the at least three images in the coordinate system of the optical machine; and to calculate the intrinsic parameters of the optical machine and the extrinsic parameters of the optical machine to the preset rectangular calibration area based on the pixel coordinates of the at least three images in the coordinate system of the optical machine and the world coordinates in the coordinate system of the optical machine.

[0110] Optionally, the first calibration module 502 is also used to calculate the transformation parameter matrix of the pixel coordinates of the image in the coordinate system of the camera and the world coordinates in the coordinate system of the camera; based on the pixel coordinates of the image in the coordinate system of the optical machine and the transformation parameter matrix, the world coordinates of the image in the coordinate system of the optical machine are calculated.

[0111] Optionally, the second calibration module 503 is also used to obtain the rotation relationship between the coordinate system of the camera and the world coordinate system to obtain a first rotation relationship; obtain the rotation relationship between the coordinate system of the optical machine and the world coordinate system to obtain a second rotation relationship; and calibrate the external parameters between the camera and the optical machine according to the first rotation relationship and the second rotation relationship.

[0112] Optionally, the second calibration module 503, the acquisition module 501 is also used to read the static IMU data when the position of the projector faces the middle position of the preset rectangular calibration area. The second calibration module 503 is also used to calculate the rotation relationship between the coordinate system of the IMU aligned in the vertical direction and the world coordinate system to obtain a third rotation relationship, and calculate the rotation relationship between the coordinate system of the component aligned in the horizontal direction and the world coordinate system to obtain a fourth rotation relationship; according to the third rotation relationship and the fourth rotation relationship, the external parameters between the component and the IMU are calculated. The second calibration module 503 is also used to calculate the rotation relationship between the coordinate system of the IMU aligned in the horizontal direction and the world coordinate system to obtain a fifth rotation relationship, and calculate the rotation relationship between the coordinate system of the component aligned in the vertical direction and the world coordinate system to obtain a sixth rotation relationship; according to the fifth rotation relationship and the sixth rotation relationship, the external parameters between the component and the IMU are calculated.

[0113] The projector calibration device provided in the embodiment of the present application and the projector calibration method provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, operated or implemented therein.

[0114] The present application also provides an electronic device to perform the above projector calibration method. Figure 6 It shows a schematic diagram of an electronic device provided by some embodiments of the present application. Figure 6 As shown, the electronic device 6 includes: a processor 600, a memory 601, a bus 602 and a communication interface 603, and the processor 600, the communication interface 603 and the memory 601 are connected via the bus 602; the memory 601 stores a computer program that can be run on the processor 600, and when the processor 600 runs the computer program, it executes the projector calibration method provided in any of the aforementioned embodiments of the present application.

[0115] The memory 601 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk storage. The communication connection between the device network element and at least one other network element is realized through at least one communication interface 603 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0116] The bus 602 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The memory 601 is used to store a program, and the processor 600 executes the program after receiving an execution instruction. The projector calibration method disclosed in any implementation of the embodiment of the present application may be applied to the processor 600, or implemented by the processor 600.

[0117] The processor 600 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 600. The above processor 600 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor to be executed, or the hardware and software modules in the decoding processor can be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 601, and the processor 600 reads the information in the memory 601 and completes the steps of the above method in combination with its hardware.

[0118] The electronic device provided in the embodiment of the present application and the projector calibration method provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, operated or implemented therein.

[0119] The present application also provides a computer-readable storage medium corresponding to the projector calibration method provided in the above embodiment. Figure 7 The computer-readable storage medium shown is a CD 30 on which a computer program (ie, a program product) is stored. When the computer program is run by a processor, the projector calibration method provided by any of the aforementioned embodiments will be executed.

[0120] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.

[0121] The computer-readable storage medium provided in the above-mentioned embodiments of the present application and the projector calibration method provided in the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the method adopted, run or implemented by the application program stored therein.

[0122] It should be noted that:

[0123] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known structures and technologies are not shown in detail so as not to obscure the understanding of this description.

[0124] Similarly, it should be understood that in order to streamline the present application and help understand one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present application, the various features of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, the disclosed method should not be interpreted as reflecting the following schematic diagram: the claimed application requires more features than the features clearly stated in each claim. More specifically, as reflected in the claims below, the inventive aspects are less than all the features of the single embodiment disclosed above. Therefore, the claims following the specific embodiment are hereby expressly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the present application.

[0125] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present application and form different embodiments. For example, in the claims below, any one of the claimed embodiments may be used in any combination.

[0126] The above is only a preferred specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A projector calibration method, characterized in that: The projector includes a built-in camera, an optical machine, and an inertial measurement unit (IMU), and the method includes: Acquire at least three images projected by the projector onto a preset rectangular calibration area, wherein the at least three images are images projected by the projector at at least three positions, respectively, and projection positions corresponding to any two images are different; For any component of the camera and the optical machine, calibrate the parameters of the component according to the correspondence between the pixel coordinates of the at least three images and the world coordinates in the component coordinate system; the reference coordinate system of the world coordinates is a world coordinate system with the upper left corner of the preset rectangular calibration area as the origin; According to the rotation relationship between the camera, the optical machine and the IMU and the world coordinate system respectively, the external parameters between the camera, the optical machine and the IMU are calibrated.

2. The method according to claim 1, characterized in that For the camera, calibrating the parameters of the camera according to the correspondence between the pixel coordinates of the at least three images in the camera coordinate system and the world coordinates includes: For any image of the at least three images, obtain a conversion relationship between pixel coordinates of the image in the coordinate system of the camera and world coordinates in the coordinate system of the camera to obtain at least three conversion relationships; The at least three conversion relationships are used as parameters by adopting a preset algorithm to calculate the intrinsic parameters of the camera and the extrinsic parameters between the camera and the preset rectangular calibration area.

3. The method according to claim 2, characterized in that The image is an icon array, and obtaining the conversion relationship between the pixel coordinates of the image in the coordinate system of the camera and the world coordinates in the coordinate system of the camera includes: Determine icons of the image at four vertices of the preset rectangular calibration area; Obtain the conversion relationship between the pixel coordinates of the four fixed-point icons in the camera's coordinate system and the world coordinates in the camera's coordinate system.

4. The method according to claim 1 or 2, characterized in that: For the optical machine, calibrating parameters of the optical machine according to the correspondence between the pixel coordinates of the at least three images and the world coordinates in the coordinate system of the optical machine includes: For any image of the at least three images, calculate the world coordinates of the image in the coordinate system of the optical machine according to the pixel coordinates and world coordinates of the image in the coordinate system of the camera and the pixel coordinates in the coordinate system of the optical machine, so as to obtain the world coordinates of the at least three images in the coordinate system of the optical machine; According to the pixel coordinates of the at least three images in the coordinate system of the optical machine and the world coordinates in the coordinate system of the optical machine, the intrinsic parameters of the optical machine and the extrinsic parameters of the optical machine to the preset rectangular calibration area are calculated.

5. The method according to claim 4, characterized in that The calculating the world coordinates of the image in the coordinate system of the optical machine according to the pixel coordinates and the world coordinates of the image in the coordinate system of the camera and the pixel coordinates in the coordinate system of the optical machine comprises: Calculate a conversion parameter matrix between pixel coordinates of the image in the camera's coordinate system and world coordinates in the camera's coordinate system; The world coordinates of the image in the coordinate system of the optical machine are calculated according to the pixel coordinates of the image in the coordinate system of the optical machine and the conversion parameter matrix.

6. The method according to claim 1, characterized in that According to the rotation relationship between the camera and the optical machine and the world coordinate system, calibrating the external parameters between the camera and the optical machine includes: Acquire a rotation relationship between the camera coordinate system and the world coordinate system to obtain a first rotation relationship; Acquire the rotation relationship between the optical machine coordinate system and the world coordinate system to obtain a second rotation relationship; The external parameters between the camera and the optical machine are calibrated according to the first rotation relationship and the second rotation relationship.

7. The method according to claim 6, characterized in that Also includes: When the position of the projector faces the middle position of the preset rectangular calibration area, reading the static IMU data; The calibrating the external parameters between the camera and any component of the optical machine and the IMU according to the rotation relationship between the world coordinate system includes: Calculating a rotation relationship between a coordinate system of the IMU aligned in a vertical direction and the world coordinate system to obtain a third rotation relationship, and calculating a rotation relationship between a coordinate system of the component aligned in a horizontal direction and the world coordinate system to obtain a fourth rotation relationship; Calculate the external parameters between the component and the IMU according to the third rotation relationship and the fourth rotation relationship; Alternatively, calculating the rotation relationship between the coordinate system of the IMU aligned in the horizontal direction and the world coordinate system to obtain a fifth rotation relationship, and calculating the rotation relationship between the coordinate system of the component aligned in the vertical direction and the world coordinate system to obtain a sixth rotation relationship; According to the fifth rotation relationship and the sixth rotation relationship, the external parameters between the component and the IMU are calculated.

8. A projector calibration device, characterized in that: The device comprises: An acquisition module, used for acquiring at least three images projected by the projector onto a preset rectangular calibration area, wherein the at least three images are images projected by the projector at at least three positions, and projection positions corresponding to any two images are different; A first calibration module is used to calibrate the parameters of any component in the camera and optical machine built into the projector according to the correspondence between the pixel coordinates of the at least three images and the world coordinates in the component coordinate system; the reference coordinate system of the world coordinate is a world coordinate system with the upper left corner of the preset rectangular calibration area as the origin; The second calibration module is used to calibrate the external parameters between the camera, the optical machine and the IMU according to the rotation relationship between the camera, the optical machine and the IMU built into the projector and the world coordinate system respectively.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: The processor runs the computer program to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the method according to any one of claims 1 to 7.

Images