Camera calibration method and device

Abstract

The invention relates to a camera calibration method and device, and the method comprises the steps: obtaining a first image shot by an image collection device for a calibration image; wherein the calibration image comprises at least one feature point; a second image shot by the image acquisition device is acquired, and the brightness corresponding to the second image is smaller than the brightness corresponding to the first image; the second image and the first image shot by the image acquisition device are the same in pose; generating a target image based on the first image and the second image; and feature point extraction is carried out in the target image, and the image acquisition device is calibrated based on the extracted feature points. According to the method and the device, the image acquisition device shoots the first image in a relatively high illumination environment and shoots the second image in a relatively low illumination environment, so that the target image with a relatively high signal-to-noise ratio is generated, more and high-quality feature points are extracted, and the calibration precision and the calibration effect of the image acquisition device are improved.

Description

Technical Field

[0001] The present disclosure relates to the technical field of virtual shooting, and in particular to a camera calibration method and device. Background Art

[0002] Virtual shooting refers to projecting the scene image rendered by the virtual engine onto the screen for display, and then the actors use the screen as the background for performance. The image acquisition device simultaneously captures the actors and the screen, thereby placing the real actors in the virtual scene, achieving the effect of shooting outdoor scenes or science fiction backgrounds in a studio. Before virtual shooting, it is usually necessary to calibrate the image acquisition device. However, when the lighting at the virtual shooting site is insufficient or the image captured by the image acquisition device has an uneven distribution of light and dark, the calibration effect is poor. Summary of the invention

[0003] In view of this, the present disclosure proposes a camera calibration method, device, electronic device, storage medium and computer program product.

[0004] According to one aspect of the present disclosure, a camera calibration method is provided, the method comprising:

[0005] Acquire a first image captured by an image acquisition device for a calibration image; wherein the calibration image includes at least one feature point;

[0006] Acquire a second image captured by the image acquisition device, wherein the brightness corresponding to the second image is smaller than the brightness corresponding to the first image; and the image acquisition device has the same posture when capturing the second image and the first image;

[0007] generating a target image based on the first image and the second image;

[0008] Feature points are extracted from the target image, and the image acquisition device is calibrated based on the extracted feature points.

[0009] In a possible implementation manner, generating a target image based on the first image and the second image includes:

[0010] The target image is generated by performing a difference operation on the pixel values ​​of matching pixels in the first image and the second image.

[0011] In a possible implementation manner, before acquiring the second image captured by the image acquisition device, the method further includes:

[0012] Performing feature point detection on the first image;

[0013] When the detected feature points do not meet the preset requirements, the step of acquiring the second image captured by the image acquisition device is performed.

[0014] In a possible implementation manner, the calibration image is displayed on a screen, and when the image acquisition device captures the second image, the display brightness of the screen is lower than the display brightness of the screen when the image acquisition device captures the first image;

[0015] and / or,

[0016] When the image acquisition device captures the second image, the brightness of the fill light of the image acquisition device is lower than the brightness of the fill light of the image acquisition device when the image acquisition device captures the first image.

[0017] In a possible implementation manner, the first image and the second image are color images;

[0018] The step of generating a target image based on the first image and the second image includes:

[0019] Converting the first image and the second image into grayscale images or binary images;

[0020] The target image is generated based on the converted first image and the second image.

[0021] In a possible implementation manner, the acquiring the second image captured by the image acquisition device includes:

[0022] Controlling the screen to turn off, and after detecting that the display brightness of the screen is less than a first preset threshold or after a first preset time has passed, controlling the image acquisition device to capture the second image;

[0023] or,

[0024] The image acquisition device is equipped with a fill light.

[0025] The step of acquiring a second image captured by the image acquisition device comprises:

[0026] The fill light of the image acquisition device is controlled to be turned off, and after detecting that the brightness of the fill light of the image acquisition device is less than a second preset threshold or after a second preset time has passed, the image acquisition device is controlled to capture the second image.

[0027] In a possible implementation manner, the detected feature points do not meet preset requirements, including:

[0028] The number of detected feature points is less than a third preset threshold, and / or the ratio of the number of detected feature points to the total number of feature points in the first image is less than a fourth preset threshold.

[0029] In a possible implementation, the screen is a screen used to display a rendering image during a virtual shooting process.

[0030] According to another aspect of the present disclosure, a camera calibration device is provided, the device comprising: an acquisition module, used to acquire a first image taken by an image acquisition device for a calibration image; wherein the calibration image includes at least one feature point; acquiring a second image taken by the image acquisition device, wherein the brightness corresponding to the second image is smaller than the brightness corresponding to the first image; the image acquisition device has the same posture when taking the second image and the first image; a generation module, used to generate a target image based on the first image and the second image; and a calibration module, used to extract feature points in the target image and calibrate the image acquisition device based on the extracted feature points.

[0031] According to another aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0032] According to another aspect of the present disclosure, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored, wherein the computer program instructions implement the above method when executed by a processor.

[0033] According to another aspect of the present disclosure, a computer program product is provided, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0034] Through various aspects of the present disclosure, a first image captured by an image acquisition device for a calibration image is acquired; wherein the calibration image includes at least one feature point; a second image captured by the image acquisition device is acquired, wherein the brightness corresponding to the second image is less than the brightness corresponding to the first image; the image acquisition device has the same posture when capturing the second image and the first image; a target image is generated based on the first image and the second image; feature points are extracted from the target image, and the image acquisition device is calibrated based on the extracted feature points. In this way, an image captured by the image acquisition device in a higher light environment (i.e., the first image) and an image captured in a lower light environment (i.e., the second image) are acquired, thereby generating a target image with a higher signal-to-noise ratio. Compared with the first image and the second image, more and higher-quality feature points can be extracted from the target image, thereby improving the calibration accuracy and effect of the image acquisition device. In some scenarios, it can effectively improve the calibration effect when the lighting in the calibration environment (such as a virtual shooting scene) is insufficient or the image captured by the image acquisition device has an uneven distribution of light and dark. At the same time, it avoids repeatedly changing the shooting angle of the image acquisition device to capture multiple images, saving calibration time and improving the efficiency of camera calibration. There is no need to increase lighting conditions on site, which saves costs and ensures the operability of the camera calibration method in actual scenarios.

[0035] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0037] Figure 1 A schematic diagram of a virtual shooting scene according to an embodiment of the present disclosure is shown.

[0038] Figure 2 A flowchart of a camera calibration method according to an embodiment of the present disclosure is shown.

[0039] Figure 3 A flowchart of a camera calibration method according to an embodiment of the present disclosure is shown.

[0040] Figure 4 A structural diagram of a camera calibration device according to an embodiment of the present disclosure is shown.

[0041] Figure 5 A block diagram of an electronic device 1900 according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0042] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0043] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0044] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0045] Figure 1 A schematic diagram of a virtual shooting scene according to an embodiment of the present disclosure is shown. Figure 1 As shown, the virtual shooting scene may include a display device 10 and an image acquisition device 20; wherein the display device 10 is used to display the rendered scene, and the image acquisition device 20 is used to capture the picture displayed by the display device 10. A real scene may also be arranged in front of the display device 10. An actor may stand at an appropriate position in front of the display device 10 and perform with the display device 10 as the background. The image acquisition device 20 may simultaneously capture the actual scene where the actor is located and the picture displayed by the display device 10, thereby completing the shooting of the virtual scene.

[0046] For example, the display device 10 may be a LED (Light-Emitting Diode) screen or a screen made of other materials, and may be used to display information such as text, images, and videos. The shape of the screen may be a straight screen, a curved screen, a tri-fold screen, or a special-shaped screen with a multi-faceted three-dimensional structure, etc., which is not limited in the embodiments of the present disclosure.

[0047] Exemplarily, the image acquisition device 20 may be a device with a shooting function such as a camera or a camcorder. The image acquisition device 20 may be equipped with an infrared lens, a zoom lens, a wide-angle lens or a binocular lens, etc., so as to meet different shooting requirements.

[0048] In the process of virtual shooting, the geometric model parameters of the image acquisition device 20 are crucial. Therefore, before using the image acquisition device 20 for virtual shooting, it is necessary to perform camera calibration, that is, to solve the geometric model parameters of the image acquisition device 20; the calibration of the image acquisition device 20 can usually be achieved by extracting feature points from the calibration image captured by the image acquisition device 20. For example, the calibration image can be displayed in the display device 20, or the calibration image can be displayed in other screens (i.e., screens other than the display device 20) arranged at the virtual shooting scene, or a calibration plate containing the calibration image can be arranged at the virtual shooting scene, etc., so that the image acquisition device 20 can capture the calibration image to achieve calibration; through calibration, the distortion and deformation in the image subsequently captured by the image acquisition device 20 can be eliminated, thereby improving the image quality. Exemplarily, the geometric model parameters of the image acquisition device 20 may include intrinsic parameters and / or extrinsic parameters, wherein the extrinsic parameters may include posture, that is, the position and orientation of the image acquisition device 20 in space, which can be represented by a rotation matrix R and a translation matrix T; the internal parameters may include the field of view (FOV), optical center position, focal length, distortion parameters (such as radial distortion, tangential distortion), etc. of the image acquisition device 20.

[0049] As the environment of the virtual shooting scene changes, for example, the lighting conditions, background or real scene layout of the scene change significantly, the image acquisition device 20 needs to be recalibrated to ensure the imaging effect in different environments. However, when the actual shooting scene takes place at night, indoors and other scenes, the environment of the virtual shooting scene is darker. Due to insufficient lighting, the signal-to-noise ratio of the calibration image taken by the image acquisition device 20 is reduced, and then too few feature points can be extracted from the calibration image, affecting the calibration effect; or, when the calibration image taken by the image acquisition device 20 has uneven distribution of light and dark, for example, the display device 10 is a curved screen, and the image acquisition device 20 is equipped with a wide-angle lens. Different parts of the curved screen in the field of view of the wide-angle lens of the image acquisition device 20 may be tilted to different degrees, wherein the more tilted the part of the screen is, the lower the brightness of the corresponding part of the image taken by the image acquisition device 20 is, so that uneven distribution of light and dark appears in the calibration image taken by the image acquisition device 20, and the lower the brightness of the image part is, the lower the signal-to-noise ratio is, and the fewer feature points can be extracted, affecting the calibration effect.

[0050] In the related art, the calibration images can be taken repeatedly by changing the shooting angle of the image acquisition device, and then the images with too few feature points can be deleted by filtering the data. However, this method increases the time and workload of calibration. In addition, the shooting scene can be adjusted to increase the lighting conditions on site. This method often requires more complete and expensive hardware, and has poor operability in real situations.

[0051] In order to solve the above technical problems, the embodiment of the present disclosure proposes a camera calibration method (for specific description, see below), which can be used when the lighting at a virtual shooting scene is insufficient or the image captured by the image acquisition device has an uneven distribution of light and dark. By processing the images captured in this situation and the images captured under relatively dark conditions, the signal-to-noise ratio of the images can be improved, and then more and higher-quality feature points can be extracted from the images, thereby improving the calibration effect of the image acquisition device; at the same time, it avoids repeatedly changing the shooting angle of the image acquisition device to capture multiple images, saving calibration time and improving the efficiency of camera calibration; and there is no need to increase lighting conditions on site, saving costs, and ensuring the operability of the camera calibration method in actual scenes.

[0052] It should be noted that the above-mentioned application scenarios described in the embodiments of the present disclosure are for the purpose of more clearly illustrating the technical solutions of the embodiments of the present disclosure, and do not constitute a limitation on the technical solutions provided by the embodiments of the present disclosure. Those skilled in the art may appreciate that, in response to the emergence of other similar or new scenarios, for example, in the fields of robots, autonomous driving, drones, or other shooting scenarios other than virtual shooting scenes, it is necessary to use images with insufficient illumination for calibration, or for other computer vision scenarios such as target detection, image classification, etc., the technical solutions provided in the embodiments of the present disclosure are also applicable to similar technical problems.

[0053] The following takes a virtual shooting scene as an example to describe in detail the camera calibration method provided by the embodiment of the present disclosure.

[0054] Figure 2 A flow chart of a camera calibration method according to an embodiment of the present disclosure is shown. The method can be executed by an electronic device having a data processing function such as a processor or a server. Figure 2 As shown, the method may include the following steps:

[0055] Step 201: Acquire a first image captured by an image acquisition device for a calibration image; wherein the calibration image includes at least one feature point.

[0056] Exemplarily, a calibration image may be displayed on a screen or a calibration plate containing the calibration image may be pre-arranged, and the image acquisition device may shoot the calibration image to obtain a first image; wherein the screen may be a screen used to display a rendering image during a virtual shooting process, or may be other screens pre-arranged at a virtual shooting scene except for a screen used to display a rendering image, and the like, and is not limited to this.

[0057] For example, before the image acquisition device shoots, a calibration image containing feature points can be generated and rendered on the screen, or a calibration image containing feature points can be pre-designed and a calibration plate can be prepared accordingly. The size, type, etc. of the calibration image and the number of feature points can be set according to requirements and are not limited thereto.

[0058] Among them, the feature point refers to a point with significant features in the calibration image, such as an edge, a corner point, a dot, a spot, etc. The image acquisition device can be calibrated by performing 3D-2D coordinate matching on the feature points. For example, the calibration image is a checkerboard image, wherein the corner points in the checkerboard image are feature points; for another example, the calibration image is a dot matrix image, wherein the dots in the dot matrix image are feature points; for another example, the calibration image can also include a positioning identifier with known preset position information, such as an Aruco code, and the edge of the Aruco code is a feature point; wherein the Aruco code is a synthetic square mark, consisting of a wide black border and an internal binary matrix that can determine its identification. In the disclosed embodiment, a positioning identifier can be generated based on the Aruco code, for example: the Aruco code can be used as a positioning identifier alone; other identification information can also be added on the basis of the Aruco code as a positioning identifier; the Aruco code can also be improved in shape to a certain extent, and the improved graphic can be used as a positioning identifier, and so on.

[0059] Exemplarily, a calibration image suitable for screen display can be generated according to the resolution and size of the screen; as an example, the screen is a curved screen and the image acquisition device is equipped with a wide-angle lens. Alternatively, the calibration image on the calibration plate can be designed according to the size of the calibration plate.

[0060] Step 202: Acquire a second image captured by the image acquisition device, wherein the brightness corresponding to the second image is smaller than the brightness corresponding to the first image; and the image acquisition device has the same posture when capturing the second image and the first image.

[0061] In a possible implementation, before acquiring the second image taken by the image acquisition device, the method further includes: performing feature point detection on the first image; and when the detected feature points do not meet preset requirements, executing the step of acquiring the second image taken by the image acquisition device.

[0062] When the method of extracting feature points from the target image to realize the calibration of the image acquisition device is adopted, since the number and quality of the extracted feature points have an important influence on the accuracy and effect of the calibration, after acquiring the first image, feature point detection can be performed to determine whether the detected feature points meet the preset requirements, and if they do not meet the preset requirements, the image acquisition device is obtained to take a second image under relatively dark conditions. Exemplarily, if the detected feature points meet the preset requirements, the image acquisition device can be calibrated directly by the detected feature points without performing the following steps.

[0063] As an example, the detected feature points do not meet the preset requirements, including: the number of detected feature points is less than a third preset threshold, and / or the ratio of the number of detected feature points to the total number of feature points in the first image is less than a fourth preset threshold. The specific values ​​of the third preset threshold and the fourth preset threshold can be set according to requirements, for example, the third preset threshold can be 10, and the fourth preset threshold can be 80%.

[0064] Exemplarily, the number of detected feature points can be determined by two rounds of searching. The feature points of the first round can be detected in the first image by a feature point detection algorithm, and then the feature points in the first round can be used as the center to perform a depth and breadth search to obtain the feature points of the second round. When performing the depth and breadth search, the existing search algorithm can be sampled based on the distribution of feature points in the calibration image (such as the distribution form, distribution interval, distribution direction, etc. of the feature points), so as to search for possible new feature points (i.e., the feature points of the second round), and the feature points of the first round and the feature points of the second round are summarized to determine the number of detected feature points.

[0065] Among them, feature point detection for the first image can be implemented by using a variety of different algorithms, such as scale-invariant feature transform (SIFT), oriented FAST and rotated BRIEF (ORB) and other algorithms. Exemplarily, the feature point detection algorithm used can be determined according to the type of feature points contained in the calibration image, so as to improve the detection accuracy; for example, if the calibration image is a checkerboard image, the SIFT algorithm can be used, and if the calibration image is a dot matrix image, the ORB algorithm can be used. In addition, feature point detection for the first image can also be implemented by using a trained target detection model, wherein the model type and model structure of the target detection model are not limited, and various suitable target detection models can be used to detect feature points from the first image. For example, a target detection model can be pre-trained based on a YOLO (You Only Look Once) deep learning algorithm, and then the first image can be input into the target detection model, and the target detection model identifies the feature points in the first image and can mark the feature points in the first image.

[0066] Exemplarily, the image acquisition device may first perform a first shooting to obtain a first image, then adjust the brightness of the ambient light, while the posture of the image acquisition device remains unchanged, and then perform a second shooting to obtain a second image.

[0067] The brightness corresponding to the image represents the brightness of the ambient light when the image acquisition device captures the image. Since the brightness corresponding to the second image is less than the brightness corresponding to the first image, the second image may also be referred to as a low-light image, i.e., an image captured by the image acquisition device in a relatively low-light environment, and the first image may also be referred to as a high-light image, i.e., an image captured by the image acquisition device in a relatively high-light environment. Exemplarily, the second image may be an image captured by the image acquisition device in a relatively dark environment (i.e., the brightness of the ambient light is very low).

[0068] In a possible implementation, the calibration image is displayed on a screen; when the image acquisition device captures the second image, the display brightness of the screen is lower than the display brightness of the screen when the image acquisition device captures the first image. In this way, in the case where the calibration image is displayed on the screen, by adjusting the display brightness of the screen, the image acquisition device can capture the second image in a relatively low light environment and capture the first image in a relatively high light environment.

[0069] Exemplarily, when the image acquisition device captures the first image, the display brightness of the screen can be adjusted to exceed the preset threshold, and when the image acquisition device captures the second image, the display brightness of the screen can be adjusted to be lower than the preset threshold. The specific value of the preset threshold can be set as required and is not limited to this. For example, the preset threshold can be set to 0, that is, the screen is kept in a working state when the image acquisition device captures the first image, and the screen is kept in a closed state when the image acquisition device captures the second image, so that it is more convenient to adjust, and after the calibration is completed, the screen is switched from the closed state to the working state to restore the original display brightness, reducing the impact on the on-site shooting. It can be understood that when the screen is kept in a closed state, the calibration image is no longer displayed on the screen, and accordingly, the second image captured by the image acquisition device does not contain feature points.

[0070] As an example, in this step, the acquisition of the second image captured by the image acquisition device includes: controlling the screen to turn off, and after detecting that the display brightness of the screen is less than a first preset threshold or after a first preset time, controlling the image acquisition device to capture the second image. The first preset threshold and the first preset time can be set as required. Since residual brightness of the screen display still remains in the virtual shooting scene after the screen is turned off, after the screen is switched from the off state to the working state, after a certain delay or after detecting that the display brightness of the screen is sufficiently small, the image acquisition device takes pictures again, so that the captured second image contains very few feature points or no feature points, thereby further improving the signal-to-noise ratio of the target image generated subsequently.

[0071] Exemplarily, the screen can be turned on and off by configuring a timer or a hardware control module, and the display brightness of the screen can be detected by configuring a light detection device.

[0072] In a possible implementation, when the image acquisition device captures the second image, the brightness of the fill light of the image acquisition device is lower than the brightness of the fill light of the image acquisition device when the image acquisition device captures the first image. In this way, in the case where a calibration image is displayed on the screen or a calibration plate containing a calibration image is pre-arranged, the brightness of the fill light of the image acquisition device is adjusted, so that the image acquisition device can capture the second image in a lower light environment and the first image in a higher light environment.

[0073] Exemplarily, when the image acquisition device captures the first image, the fill light of the image acquisition device can be adjusted to exceed the preset threshold, and when the image acquisition device captures the second image, the fill light of the image acquisition device can be adjusted to be lower than the preset threshold. The specific value of the preset threshold can be set as required and is not limited to this. For example, the preset threshold can be set to 0, that is, when the image acquisition device captures the second image, the fill light of the image acquisition device is kept in the off state, and when the image acquisition device captures the first image, the fill light of the image acquisition device is kept in the on state, so that it is more convenient to adjust, and after the calibration is completed, the fill light of the image acquisition device is switched from the off state to the on state, so that the original scene lighting can be restored, which effectively reduces the impact on the scene shooting.

[0074] As an example, the image acquisition device is equipped with a fill light, for example, the image acquisition device can be an infrared camera equipped with an infrared fill light, and the infrared fill light can emit infrared rays, so as to enhance the brightness of the image taken by the infrared camera when the virtual shooting scene is insufficiently illuminated. In this step, the acquisition of the second image taken by the image acquisition device includes: controlling the fill light of the image acquisition device to turn off, and controlling the image acquisition device to take the second image after detecting that the brightness of the fill light of the image acquisition device is less than a second preset threshold or after a second preset time has passed. Since the residual brightness of the fill light remains in the virtual shooting scene after the fill light of the image acquisition device is turned off, the image acquisition device takes pictures again after a certain delay or after detecting that the display brightness of the screen or the brightness of the calibration plate is sufficiently small after the fill light is switched from the off state to the working state, so that the captured second image contains as few feature points as possible, thereby further improving the signal-to-noise ratio of the target image generated subsequently.

[0075] Exemplarily, the switch of the fill light of the image acquisition device can be controlled by configuring a timer or a hardware control module, and the display brightness of the screen or the brightness of the calibration board can be detected by configuring a light detection device.

[0076] In one possible implementation, the brightness of the fill light of the image acquisition device and the display brightness of the screen can be adjusted simultaneously, so that the image acquisition device can capture the second image in a lower light environment and capture the first image in a higher light environment; the specific adjustment method can be referred to the previous description.

[0077] Step 203: Generate a target image based on the first image and the second image.

[0078] Since the first image and the second image are images taken by the image acquisition device at the same position, the pixels at the same position in the first image and the second image correspond to the same point in the three-dimensional space. For example, the pixels at the same position in the first image and the second image can be processed respectively to generate a target image; wherein the generated target image, the first image, and the second image have the same size.

[0079] In a possible implementation, the generating of the target image based on the first image and the second image includes: generating the target image by performing a difference operation on the pixel values ​​of the matching pixels in the first image and the second image. In a high-light image, the image signal is relatively strong and the noise is relatively small; while in a low-light image, the image signal is relatively weak and the noise is relatively large. Therefore, by performing a difference operation on the pixel values ​​of the matching pixels in the first image and the second image, the background noise in the first image can be separated from the image signal in the first image, thereby suppressing the background noise to a certain extent, highlighting the useful image signal (i.e., the feature point signal), and improving the ratio between the useful image signal and the background noise in the image, so as to achieve the purpose of improving the image signal-to-noise ratio. Among them, the second image can also be called a background frame, which is used to eliminate the background noise in the first image.

[0080] Exemplarily, an image processing algorithm, such as OpenCV, etc., may be used to perform a difference operation on the first image and the second image to obtain a target image with a high signal-to-noise ratio.

[0081] In a possible implementation, the first image and the second image are color images; generating a target image based on the first image and the second image includes: converting the first image and the second image into grayscale images or binary images; and generating the target image based on the converted first image and the second image.

[0082] Among them, color images contain rich color information and can represent various colors in nature. Each pixel in a color image is usually composed of multiple components. The components contained in each pixel are different depending on the color space used. For example, in the RGB color space, there are three components, R (red), G (green) and B (blue). The value of each component is between 0 and 255. The color is represented by the combination of these three components. Binary images do not contain color information and only have two colors, black and white. Each pixel in a binary image has only two possible values, usually represented by 0 and 1, 0 usually represents black, and 1 represents white. Grayscale images do not contain color information, but can represent different brightness levels. There are 256 grayscale levels from black to white. Each pixel value in a grayscale image is usually between 0 and 255, representing different grayscale levels from black (0) to white (255).

[0083] In this way, the first image and the second image are both converted into grayscale images, or are both converted into binary images. Compared with color images, the converted first image and the second image can better separate useful image signals from background noise.

[0084] Step 204: extract feature points from the target image, and calibrate the image acquisition device based on the extracted feature points.

[0085] Since the target image generated above is the image after denoising the first image, the signal-to-noise ratio of the target image is higher, and the image with a high signal-to-noise ratio can provide clearer image details and more accurate object recognition; therefore, more and high-quality feature points can be extracted from the target image, and then the image acquisition device can be calibrated based on these extracted feature points, so as to effectively improve the calibration effect. For example, for various feature points such as the cross corner points at the intersection of the longitude and latitude lines of the chessboard, or the center points of the dot dot pattern, in the case of poor image quality, it is easy to cause the corner points to be blurred and the circular part to be missing, and the feature points finally extracted have deviations. The above processing can improve the signal-to-noise ratio of the image and reduce such deviations, thereby improving the number and quality of the extracted feature points.

[0086] The method of extracting feature points may be the same as the method of detecting feature points in step 202 , which will not be described in detail herein.

[0087] Exemplarily, the image acquisition device may be calibrated using an existing feature point-based camera calibration method; for example, a calibration interface of the image acquisition device (such as an OpenCV calibration interface) may be called to obtain geometric model parameters of the image acquisition device.

[0088] In a possible implementation, calibrating the image acquisition device based on the extracted feature points may include: when a calibration image is displayed on a screen, the two-dimensional coordinates of each extracted feature point and the three-dimensional coordinates of each feature point on the screen may be determined; wherein the two-dimensional coordinates of the same feature point and the three-dimensional coordinates on the screen constitute a 2D-3D coordinate point pair; and then based on the 2D-3D coordinate point pair constituted by the two-dimensional coordinates and the three-dimensional coordinates of each feature point, the geometric model parameters of the image acquisition device may be iteratively optimized to obtain the calibration values ​​of the geometric model parameters.

[0089] Exemplarily, for a certain feature point extracted, the two-dimensional coordinates of the feature point can be expressed by the coordinate value of the feature point in the image coordinate system; wherein the image coordinate system is a two-dimensional rectangular coordinate system, and the center of the target image can be used as the coordinate origin, or a point in the lower left corner or a point in the lower right corner of the target image can be used as the coordinate origin, etc.; the X-axis and Y-axis of the image coordinate system are respectively parallel to the X-axis and Y-axis of the image acquisition device coordinate system; the image acquisition device coordinate system can take the focus center of the image acquisition device as the origin, and the optical axis of the image acquisition device as the Z axis; exemplarily, the intersection of the optical axis of the image acquisition device and the plane where the target image is located can be used as the coordinate origin of the image coordinate system. In this way, based on the established image coordinate system, after extracting each feature point from the target image as described above, the coordinate value (i.e., the two-dimensional coordinate) of each feature point in the image coordinate system can be determined. Further, the positioning identifier can be identified, so as to determine the three-dimensional coordinates of each detected feature point on the screen according to the identified positioning identifier. Exemplarily, for a certain feature point on the target image, the three-dimensional coordinates of the feature point on the screen can be represented by the coordinate value of the feature point in the screen coordinate system; wherein the screen coordinate system is a three-dimensional coordinate system set when the screen model is performed, and the definition of the screen coordinate system can be set according to the actual situation, for example: the center of the screen can be used as the coordinate origin, or a point in the lower left corner of the screen, or a point in the lower right corner of the screen can be used as the coordinate origin, etc. Exemplarily, the relative position relationship between each feature point and the positioning identifier can be obtained based on the above-extracted feature points and positioning identifiers, and then based on the relative position relationship and the position information of the positioning identifier, the position of each feature point in the picture can be determined; then, according to the position of each feature point in the picture and the two-dimensional display area of ​​the picture in the established screen model, the UV coordinates of each feature point in the screen model are determined (U represents the horizontal direction, V represents the vertical direction); finally, based on the corresponding relationship between the three-dimensional coordinates in the screen coordinate system and the UV coordinates, the three-dimensional coordinate values ​​of each feature point in the screen coordinate system, that is, the three-dimensional coordinates of each feature point on the screen, are obtained. For example, for a calibration image combining Aruco code and checkerboard, checkerboard corner point detection can be performed to obtain corner points in the target image, that is, feature points are detected, and Aruco code recognition is performed. After detecting the corner points of the checkerboard and any Aruco code, any detected Aruco code is decoded and identified to obtain identification information of any Aruco code, and based on the identification information, the position of any Aruco code on the screen is determined; then, based on the relative position relationship between each corner point and any Aruco code, the position of each corner point on the screen is obtained, and then the UV coordinates of each corner point in the screen model are determined; finally, based on the correspondence between the three-dimensional coordinates in the screen coordinate system and the UV coordinates, the three-dimensional coordinate value of each corner point in the screen coordinate system is obtained, that is, the three-dimensional coordinates of each corner point on the screen.In this way, for any feature point detected above, the two-dimensional coordinates of the feature point in the target image and the three-dimensional coordinates of the feature point on the screen are determined, thereby constructing a 2D-3D coordinate point pair.

[0090] In a possible implementation, calibrating the image acquisition device based on the extracted feature points may include: in the case of pre-arranging a calibration plate containing the calibration image, the two-dimensional coordinates of each extracted feature point and the three-dimensional coordinates of each feature point on the calibration plate may be determined; wherein the two-dimensional coordinates of the same feature point and the three-dimensional coordinates on the calibration plate constitute a 2D-3D coordinate point pair; and then based on the 2D-3D coordinate point pair composed of the two-dimensional coordinates and the three-dimensional coordinates of each feature point, the geometric model parameters of the image acquisition device may be iteratively optimized to obtain the calibration values ​​of the geometric model parameters; wherein the construction method of the 2D-3D coordinate point pair may refer to the relevant description in the previous text or be implemented through the existing technology, which will not be repeated here.

[0091] In the disclosed embodiment, a first image captured by an image acquisition device for a calibration image is acquired; wherein the calibration image includes at least one feature point; a second image captured by the image acquisition device is acquired, wherein the brightness corresponding to the second image is less than the brightness corresponding to the first image; the image acquisition device has the same posture when capturing the second image and the first image; a target image is generated based on the first image and the second image; feature points are extracted from the target image, and the image acquisition device is calibrated based on the extracted feature points. In this way, an image captured by the image acquisition device in a higher light environment (i.e., the first image) and an image captured in a lower light environment (i.e., the second image) are acquired, thereby generating a target image with a higher signal-to-noise ratio. Compared with the first image and the second image, more and higher-quality feature points can be extracted from the target image, thereby improving the calibration accuracy and effect of the image acquisition device. In some scenarios, it can effectively improve the calibration effect when the virtual shooting scene is insufficiently illuminated or the image captured by the image acquisition device has an uneven distribution of light and dark. At the same time, it avoids repeatedly changing the shooting angle of the image acquisition device to capture multiple images, saving calibration time and improving the efficiency of camera calibration. There is no need to increase lighting conditions on site, which saves costs and ensures the operability of the camera calibration method in actual scenarios.

[0092] Figure 3 FIG. 4 is a flowchart of a camera calibration method according to an embodiment of the present disclosure. Figure 3 As shown, after the image acquisition device is calibrated, the calibration image is first acquired, that is, the above Figure 2Step 201 in the above step. Then determine whether the feature points in the calibration image are abundant. If so, extract the feature points in the calibration image to calibrate the image acquisition device. If not, shoot a background frame under a lower light environment, that is, execute the above step. Figure 2 Steps 202-204 are performed to complete the calibration of the image acquisition device.

[0093] Based on the same inventive concept of the above method embodiments, an embodiment of the present disclosure further provides a camera calibration device, which can be used to execute the technical solution described in the above method embodiments.

[0094] Figure 4 FIG. 4 shows a structural diagram of a camera calibration device according to an embodiment of the present disclosure. Figure 4 As shown, the device may include: an acquisition module 401, used to acquire a first image taken by an image acquisition device for a calibration image; wherein the calibration image includes at least one feature point; acquire a second image taken by the image acquisition device, wherein the brightness corresponding to the second image is less than the brightness corresponding to the first image; the image acquisition device has the same posture when taking the second image and the first image; a generation module 402, used to generate a target image based on the first image and the second image; a calibration module 403, used to extract feature points in the target image, and calibrate the image acquisition device based on the extracted feature points.

[0095] In the disclosed embodiment, a first image captured by an image acquisition device for a calibration image is acquired; wherein the calibration image includes at least one feature point; a second image captured by the image acquisition device is acquired, wherein the brightness corresponding to the second image is less than the brightness corresponding to the first image; the image acquisition device has the same posture when capturing the second image and the first image; a target image is generated based on the first image and the second image; feature points are extracted from the target image, and the image acquisition device is calibrated based on the extracted feature points. In this way, an image captured by the image acquisition device in a higher light environment (i.e., the first image) and an image captured in a lower light environment (i.e., the second image) are acquired, thereby generating a target image with a higher signal-to-noise ratio. Compared with the first image and the second image, more and higher-quality feature points can be extracted from the target image, thereby improving the calibration accuracy and effect of the image acquisition device. In some scenarios, it can effectively improve the calibration effect when the virtual shooting scene is insufficiently illuminated or the image captured by the image acquisition device has an uneven distribution of light and dark. At the same time, it avoids repeatedly changing the shooting angle of the image acquisition device to capture multiple images, saving calibration time and improving the efficiency of camera calibration. There is no need to increase lighting conditions on site, which saves costs and ensures the operability of the camera calibration method in actual scenarios.

[0096] In a possible implementation, the generating module 402 is further configured to generate the target image by performing a difference operation on pixel values ​​of matching pixels in the first image and the second image.

[0097] In a possible implementation, the acquisition module 401 is further used to: perform feature point detection on the first image; and when the detected feature points do not meet preset requirements, execute the acquisition of the second image taken by the image acquisition device.

[0098] In a possible implementation, the calibration image is displayed on a screen; when the image acquisition device captures the second image, the display brightness of the screen is lower than the display brightness of the screen when the image acquisition device captures the first image; and / or when the image acquisition device captures the second image, the brightness of the fill light of the image acquisition device is lower than the brightness of the fill light of the image acquisition device when the image acquisition device captures the first image.

[0099] In a possible implementation, the first image and the second image are color images; the generating module 402 is further used to: convert the first image and the second image into grayscale images or binary images; and generate the target image based on the converted first image and the second image.

[0100] In a possible implementation, the acquisition module 401 is further used to: control the screen to turn off, and control the image acquisition device to capture the second image after detecting that the display brightness of the screen is less than a first preset threshold or after a first preset time has passed; or, the image acquisition device is equipped with a fill light, and the acquisition module 401 is further used to: control the fill light of the image acquisition device to turn off, and control the image acquisition device to capture the second image after detecting that the brightness of the fill light of the image acquisition device is less than a second preset threshold or after a second preset time has passed.

[0101] In a possible implementation, the detected feature points do not meet preset requirements, including: the number of detected feature points is less than a third preset threshold, and / or the proportion of the number of detected feature points to the total number of feature points in the first image is less than a fourth preset threshold.

[0102] In a possible implementation, the screen is a screen used to display a rendering image during a virtual shooting process.

[0103] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0104] The embodiment of the present disclosure also provides a computer-readable storage medium on which computer program instructions are stored, and the computer program instructions implement the above method when executed by a processor. The computer-readable storage medium can be a volatile or non-volatile computer-readable storage medium.

[0105] An embodiment of the present disclosure further proposes an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0106] The embodiments of the present disclosure also provide a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0107] Figure 5 1 shows a block diagram of an electronic device 1900 according to an embodiment of the present disclosure. For example, the electronic device 1900 may be provided as a server or a terminal device. Figure 5 , the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0108] The electronic device 1900 may also include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output interface 1958 (I / O interface). The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server 2003. TM , Mac OS X TM , Unix TM ,Linux TM , FreeBSD TM or similar.

[0109] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the electronic device 1900 to perform the above method.

[0110] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0111] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0112] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0113] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0114] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0115] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0116] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0117] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0118] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A camera calibration method, characterized in that: The method comprises: Acquire a first image captured by an image acquisition device for a calibration image; wherein the calibration image includes at least one feature point; Acquire a second image captured by the image acquisition device, wherein the brightness corresponding to the second image is smaller than the brightness corresponding to the first image; and the image acquisition device has the same posture when capturing the second image and the first image; generating a target image based on the first image and the second image; Feature points are extracted from the target image, and the image acquisition device is calibrated based on the extracted feature points.

2. The method according to claim 1, characterized in that The generating a target image based on the first image and the second image includes: The target image is generated by performing a difference operation on the pixel values ​​of matching pixels in the first image and the second image.

3. The method according to claim 1, characterized in that Before acquiring the second image captured by the image acquisition device, the method further includes: Performing feature point detection on the first image; When the detected feature points do not meet the preset requirements, the step of acquiring the second image captured by the image acquisition device is performed.

4. The method according to claim 1, characterized in that: The calibration image is displayed on a screen, and the display brightness of the screen when the image acquisition device captures the second image is lower than the display brightness of the screen when the image acquisition device captures the first image; and / or, When the image acquisition device captures the second image, the brightness of the fill light of the image acquisition device is lower than the brightness of the fill light of the image acquisition device when the image acquisition device captures the first image.

5. The method according to claim 1, characterized in that The first image and the second image are color images; The generating a target image based on the first image and the second image includes: Converting the first image and the second image into grayscale images or binary images; The target image is generated based on the converted first image and the second image.

6. The method according to claim 4, characterized in that The step of acquiring a second image captured by the image acquisition device comprises: Controlling the screen to turn off, and after detecting that the display brightness of the screen is less than a first preset threshold or after a first preset time has passed, controlling the image acquisition device to capture the second image; or, The image acquisition device is equipped with a fill light. The step of acquiring a second image captured by the image acquisition device comprises: The fill light of the image acquisition device is controlled to be turned off, and after detecting that the brightness of the fill light of the image acquisition device is less than a second preset threshold or after a second preset time has passed, the image acquisition device is controlled to capture the second image.

7. The method according to claim 3, characterized in that The detected feature points do not meet the preset requirements, including: The number of detected feature points is less than a third preset threshold, and / or the ratio of the number of detected feature points to the total number of feature points in the first image is less than a fourth preset threshold.

8. The method according to claim 4, characterized in that The screen is a screen used to display the rendering picture during the virtual shooting process.

9. A camera calibration device, characterized in that: The device includes: an acquisition module, used to acquire a first image taken by an image acquisition device for a calibration image; wherein the calibration image includes at least one feature point; acquiring a second image taken by the image acquisition device, wherein the brightness corresponding to the second image is less than the brightness corresponding to the first image; the image acquisition device has the same posture when taking the second image and the first image; a generation module, used to generate a target image based on the first image and the second image; and a calibration module, used to extract feature points in the target image and calibrate the image acquisition device based on the extracted feature points.

10. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to implement the method described in any one of claims 1 to 8 when executing the instructions stored in the memory.

11. A non-volatile computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 8 is implemented.

12. A computer program product, comprising a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code, wherein when the computer-readable code is executed in a processor of an electronic device, the processor in the electronic device executes the method according to any one of claims 1 to 8.

Images