Calibration Data Generation Method, Device, Electronic Device and Storage Medium

Abstract

The present disclosure relates to a calibration data generation method, apparatus, electronic device, and storage medium. The method includes: controlling each screen in a virtual shooting system to display its corresponding specified calibration board; controlling a camera to collect images of multiple screens in different poses to obtain multiple images; performing feature point detection on the multiple images collected by the camera to obtain the detected feature points in each image and the feature point information of the detected feature points; associating the two-dimensional coordinates of the detected feature points in each image with the three-dimensional coordinates when displayed on the screen to obtain coordinate pairs of the detected feature points in each image; wherein the calibration data includes the coordinate pairs of the detected feature points in multiple images, and the calibration data is used to calibrate the camera parameters of the camera. According to the embodiments of the present disclosure, the generation efficiency of calibration data and the acquisition accuracy can be effectively improved.

Description

Technical Field

[0001] The present disclosure relates to the technical field of camera calibration, and particularly to a calibration data generation method, apparatus, electronic device, and storage medium. Background Art

[0002] In traditional film and television production, to meet shooting requirements, a large amount of time and manpower are required to select shooting locations, produce props, and build shooting scenes. However, virtual shooting (or virtual production) technology can use virtual scenes rendered by a rendering engine to replace real sets, reducing the dependence of film and television shooting on locations and sets, and greatly reducing shooting costs. At the same time, with the ability of real-time rendering, for some special effects that originally required post-production, virtual shooting can show the finished film effect during the shooting stage, bringing post-production forward and improving production efficiency. It is precisely because of these advantages that virtual shooting has been increasingly applied to film and television production in recent years.

[0003] In virtual shooting, to achieve the integration of "virtual" and "real", it is first necessary to establish the geometric imaging model parameters of the camera through camera calibration technology, that is, to calibrate the camera parameters of the camera. Generally, the calibration board for camera calibration is printed on a rigid plane and the printed calibration board is set in the real scene. The camera calibration process usually includes: using the camera to be calibrated to capture an image containing the calibration board in the real scene, detecting feature points in the image containing the calibration board to obtain the two-dimensional image coordinates of the feature points in the image, and then corresponding the detected two-dimensional image coordinates to the preset three-dimensional coordinates of the feature points in the calibration board one by one to form calibration data for calibrating the camera, and then the camera parameters of the camera can be calibrated using the calibration data.

[0004] However, in a virtual shooting scene, the calibration board needs to be projected onto multiple large LED screens for calibration. However, the existing calibration boards cannot well adapt to camera calibration in the virtual shooting scene. For example, the checkerboard calibration board composed of alternately black and white blocks shown in Figure 1(a). The feature points in the checkerboard calibration board refer to the corner points between two adjacent black blocks in the diagonal direction. The coordinates of this corner point in the captured image are the two-dimensional image coordinates; the three-dimensional coordinates refer to the three-dimensional coordinates of each corner point in the coordinate system with the lower left corner point of the checkerboard calibration board as the coordinate origin (0, 0, 0). Since this type of checkerboard calibration board needs to be placed on a rigid plane and the entire checkerboard calibration board needs to be captured completely each time an image is collected to complete feature point detection, but for a scene with multiple LED screens, there will be an included angle between the LED screens, and it cannot be guaranteed that the checkerboard on the LED screen is fully captured, thus affecting the accuracy and success rate of feature point detection, and further affecting the accuracy and generation efficiency of the calibration data.

[0005] For another example, as shown in FIG. 1(b), a grid point calibration board composed of evenly arranged dots has its feature points at the centers of the dots. Compared with the detection of checkerboard corner points, the detection of the positions of the dot centers is more complex. And in each image acquisition, the entire calibration board needs to be photographed completely to complete the subsequent correspondence between two-dimensional coordinates and three-dimensional coordinates. However, in virtual shooting, the calibration board is displayed on a huge LED screen, and it is very difficult to ensure that the entire board is photographed. Although the existing technology can also project multiple different grid point calibration boards continuously on the LED screen and collect multiple consecutive images at a single time to locate the dot sequence of the dots in the multiple consecutive images and thus solve the problem of the correspondence between two-dimensional coordinates and three-dimensional coordinates, this method increases the image acquisition time, and the larger the LED screen, the longer the acquisition time. Moreover, when collecting multiple consecutive images at a single time, the position of the camera needs to be kept unchanged. If the position of the camera changes, even a small range change, the image data collected this time is invalid, which results in a low success rate of collecting effective images, thus significantly affecting the detection accuracy and efficiency of feature points, and further affecting the accuracy and generation efficiency of calibration data. Summary of the Invention

[0006] In view of this, the present disclosure provides a calibration data generation method, apparatus, electronic device and storage medium, which can generate a specified calibration board adapted to the virtual shooting scene, improve the success rate and stability of collecting effective images, and improve the accuracy and generation efficiency of calibration data.

[0007] According to one aspect of the present disclosure, there is provided a calibration data generation method, which is applied to a virtual shooting system. The virtual shooting system includes a camera for shooting and multiple screens for displaying a virtual scene. The method includes: controlling each screen in the virtual shooting system to display its corresponding specified calibration board, where the specified calibration board corresponding to each screen includes multiple calibration units, each calibration unit includes two identical two-dimensional identification codes, i.e., ArUco codes, which are adjacent to each other along the diagonal, and the corner point at the connection between the two identical ArUco codes in each calibration unit is the feature point of each calibration unit. The code values of the two identical ArUco codes in each calibration unit are used to represent the serial number of the feature point of each calibration unit, and the serial number of each feature point is correspondingly associated with the three-dimensional coordinates when the feature point is displayed on the screen; in the case where each of the multiple screens has displayed its corresponding specified calibration board, controlling the camera to collect images of the multiple screens in different poses, and obtaining multiple images collected by the camera; performing feature point detection on the multiple images collected by the camera to obtain the detected feature points in each image and the feature point information of the detected feature points, where the feature point information includes the serial number of the feature point and the two-dimensional coordinates of the feature point in the image; according to the serial number of the detected feature points in each image, associating the two-dimensional coordinates of the detected feature points in each image with the three-dimensional coordinates when the feature points are displayed on the screen, and obtaining the coordinate pairs of the detected feature points in each image; where the calibration data includes the coordinate pairs of the detected feature points in the multiple images, and the calibration data is used to calibrate the camera parameters of the camera.

[0008] In a possible implementation manner, the performing feature point detection on the multiple images collected by the camera to obtain the detected feature points in each image and the feature point information of the detected feature points includes: converting each image collected by the camera into a grayscale image and performing binarization processing on the grayscale image to obtain a binarized image corresponding to each image; using the ArUco detection algorithm to detect the corner points and code values at the connections between the ArUco codes in the binarized image to obtain an ArUco detection result, where the ArUco detection result includes the detected corner points and the code values of the detected ArUco codes; according to the code values of the detected ArUco codes in the ArUco detection result and the two-dimensional coordinates of the detected corner points in the binarized image, determining the feature points from the detected corner points and determining the feature point information of the feature points.

[0009] In a possible implementation manner, when there are corner points and code values of ArUco codes in the binarized image that are not detected by the ArUco detection algorithm, such that there are undetermined feature points in the binarized image, the method further includes: adding the feature point information determined by using the ArUco detection result to a preset candidate set; using a corner detection algorithm to detect the corner points in the binarized image to obtain a corner detection result, where the corner detection result includes all the corner points in the binarized image; and determining the undetermined feature points in the binarized image and the feature point information of the undetermined feature points in the binarized image according to the corner detection result and the candidate set.

[0010] In a possible implementation manner, the determining the undetermined feature points in the binarized image and the feature point information of the undetermined feature points in the binarized image according to the corner detection result and the candidate set includes: for any feature point information in the candidate set, determining, according to the first serial number in the feature point information, the second serial numbers of a plurality of nearest neighbor points corresponding to the feature point indicated by the first serial number, where the plurality of nearest neighbor points represent a plurality of feature points that are respectively the closest to the feature point indicated by the first serial number in a plurality of directions; judging whether the second serial numbers of the plurality of nearest neighbor points include a third serial number that does not exist in the candidate set according to the second serial numbers of the plurality of nearest neighbor points; in the case where the second serial numbers of the plurality of nearest neighbor points include a third serial number that does not exist in the candidate set, estimating the two-dimensional coordinates of the nearest neighbor point indicated by the third serial number according to the two-dimensional coordinates of the feature point indicated by the first serial number in the feature point information; determining, from the corner detection result, a first corner point that matches the nearest neighbor point indicated by the third serial number and determining the first corner point as a feature point according to the two-dimensional coordinates of the nearest neighbor point indicated by the third serial number, and using the third serial number as the serial number of the first corner point and using the two-dimensional coordinates of the first corner point in the binarized image as the two-dimensional coordinates of the first corner point, where the undetermined feature points in the binarized image include the first corner point.

[0011] In a possible implementation, estimating the two-dimensional coordinates of the nearest point indicated by the third serial number according to the two-dimensional coordinates of the feature points indicated by the first serial number in the feature point information includes: judging whether the fourth serial number of the reverse point corresponding to the nearest point indicated by the third serial number exists in the candidate set according to the second serial numbers of the plurality of nearest points, where the reverse point is the nearest point that is distributed in the opposite direction to the nearest point indicated by the third serial number among the plurality of nearest points; and when the fourth serial number of the reverse point exists in the candidate set, determining the two-dimensional coordinates of the nearest point indicated by the third serial number according to the vector between the reverse point and the feature point indicated by the first serial number and the two-dimensional coordinates of the feature point indicated by the first serial number.

[0012] In a possible implementation, determining, according to the two-dimensional coordinates of the nearest point indicated by the third serial number, a first corner point that matches the nearest point indicated by the third serial number from the corner detection result includes: determining, from the corner detection result, a second corner point that is the closest to the nearest point indicated by the third serial number according to the two-dimensional coordinates of the nearest point indicated by the third serial number, and judging whether the distance between the nearest point indicated by the third serial number and the second corner point is less than a preset error threshold; and when the distance between the nearest point indicated by the third serial number and the second corner point is less than the preset error threshold, determining the second corner point as the first corner point that matches the nearest point indicated by the third serial number.

[0013] In a possible implementation, the method further includes: adding the feature point information of the first corner point to the candidate set, and for the feature point information of the first corner point, re-executing determining the feature points not determined in the binary image and the feature point information of the feature points not determined in the binary image according to the corner detection result and the candidate set.

[0014] In a possible implementation, controlling each screen in the virtual shooting system to display its corresponding specified calibration board includes: in response to a calibration board configuration operation for each screen, determining the corresponding specified calibration board for each screen and controlling each screen to display its corresponding specified calibration board; where the calibration board configuration operation is used to configure at least one of the following: the size of the calibration unit in the specified calibration board, the number of rows and columns of the calibration units in the specified calibration board, and the display area of the specified calibration board on the screen.

[0015] According to another aspect of the present disclosure, a calibration data generation device is provided, which is applied to a virtual shooting system. The virtual shooting system includes a camera for shooting and multiple screens for displaying a virtual scene. The method includes: a control module for controlling each screen in the virtual shooting system to display its corresponding specified calibration board, where the specified calibration board corresponding to each screen includes multiple calibration units, each calibration unit includes two identical two-dimensional identification codes, i.e., ArUco codes, which are adjacent along the diagonal, and the corner point at the connection between the two identical ArUco codes in each calibration unit is the feature point of each calibration unit. The code values of the two identical ArUco codes in each calibration unit are used to represent the serial number of the feature point of each calibration unit, and the serial number of each feature point is correspondingly associated with the three-dimensional coordinates when the feature point is displayed on the screen; an acquisition module for, when the multiple screens have displayed their corresponding specified calibration boards, controlling the camera to perform image acquisition on the multiple screens in different poses to obtain multiple images acquired by the camera; a detection module for performing feature point detection on the multiple images acquired by the camera to obtain the detected feature points in each image and the feature point information of the detected feature points, where the feature point information includes the serial number of the feature point and the two-dimensional coordinates of the feature point in the image; an association module for, according to the serial number of the detected feature points in each image, correspondingly associating the two-dimensional coordinates of the detected feature points in each image with the three-dimensional coordinates when the feature points are displayed on the screen to obtain the coordinate pairs of the detected feature points in each image; where the calibration data includes the coordinate pairs of the detected feature points in the multiple images, and the calibration data is used to calibrate the camera parameters of the camera.

[0016] According to another aspect of the present disclosure, an electronic device is provided, including: a processor; a memory for storing instructions executable by the processor; where the processor is configured to implement the above method when executing the instructions stored in the memory.

[0017] According to another aspect of the present disclosure, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored, where the computer program instructions implement the above method when executed by a processor.

[0018] According to another aspect of the present disclosure, a computer program product is provided, including computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in the processor of an electronic device, the processor in the electronic device executes the above method.

[0019] According to an embodiment of the present disclosure, a calibration board with a new arrangement of ArUco codes is provided. Compared with the ordinary calibration board, the detection accuracy of feature points is higher, and it is easier and more efficient to detect feature points. By configuring the specified calibration board corresponding to different screens, the specified calibration board can be adapted to different screens, and it is not affected by the movement of the camera pose or the occlusion of the screen during the image acquisition process. The stability of the acquired effective image is better, and the success rate of feature point detection can also be improved, thereby improving the accuracy and generation efficiency of calibration data, which is beneficial to improving the camera calibration accuracy of the camera.

[0020] Other features and aspects of the present disclosure will become clear from the following detailed description of the exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings, which are included in and constitute a part of this specification, illustrate exemplary embodiments, features, and aspects of the present disclosure and are used to explain the principles of the present disclosure.

[0022] FIG. 1(a) shows a schematic diagram of a checkerboard calibration board in the related art.

[0023] FIG. 1(b) shows a schematic diagram of a grid point calibration board in the related art.

[0024] Figure 2 FIG. shows a schematic diagram of a virtual shooting system according to an embodiment of the present disclosure.

[0025] Figure 3 FIG. shows a flowchart of calibration data generation according to an embodiment of the present disclosure.

[0026] Figure 4 FIG. shows a schematic diagram of a specified calibration board provided by an embodiment of the present disclosure.

[0027] Figure 5 FIG. shows a schematic diagram of the display effect of the specified calibration board in an LED screen according to an embodiment of the present disclosure.

[0028] Figure 6 FIG. shows a schematic diagram of a configuration interface according to an embodiment of the present disclosure.

[0029] Figure 7 FIG. shows a schematic diagram of the display effect of the specified calibration board in another LED screen according to an embodiment of the present disclosure.

[0030] Figure 8 FIG. shows a schematic diagram of an image captured by a camera according to an embodiment of the present disclosure.

[0031] Figure 9 FIG. shows a schematic diagram of a binary image according to an embodiment of the present disclosure.

[0032] Figure 10 A schematic diagram showing an ArUco detection result according to an embodiment of the present disclosure.

[0033] Figure 11 A schematic diagram showing a corner point detection result according to an embodiment of the present disclosure.

[0034] Figure 12 A block diagram showing a calibration data generation device provided by an embodiment of the present disclosure.

[0035] Figure 13 A block diagram showing an electronic device 1900 according to an embodiment of the present disclosure. Detailed implementation manners

[0036] Various exemplary embodiments, features, and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the drawings denote elements having the same or similar functions. Although various aspects of the embodiments are shown in the drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0037] The special term "exemplary" here means "serving as an example, embodiment, or illustration". Any embodiment described as "exemplary" here need not be construed as superior to or better than other embodiments.

[0038] In addition, for a better description of the present disclosure, numerous specific details are given in the following detailed implementation manners. Those skilled in the art should understand that the present disclosure can be implemented without some specific details. In some instances, methods, means, elements, and circuits well known to those skilled in the art are not described in detail so as to highlight the gist of the present disclosure.

[0039] As described above, existing calibration plates, such as checkerboard calibration plates, grid point calibration plates, etc., cannot well adapt to virtual shooting scenarios, resulting in low accuracy and generation efficiency of calibration data. In view of this, the embodiments of the present disclosure provide a calibration data generation method, which can be applied to a virtual shooting system, can improve the success rate and stability of collecting effective images, improve the detection efficiency and detection accuracy of feature points, and thus efficiently generate calibration data with higher accuracy, which is beneficial to improving the calibration accuracy of cameras in the virtual shooting system.

[0040] Figure 2 A schematic diagram showing a virtual shooting system according to an embodiment of the present disclosure, such as Figure 2As shown, the virtual shooting system includes a camera 01 for shooting, multiple screens (021, 022, 023) for displaying virtual scenes, and a control device 03. A communication connection is established between the control device 03 and the camera 01 and the multiple screens (021, 022, 023) respectively. It should be understood that the embodiments of the present disclosure do not limit the communication connection manner between devices.

[0041] Among them, the camera 01 can be understood as a camera to be calibrated, and the embodiments of the present disclosure do not limit the type of the camera; the screens used in the virtual shooting system can be of types such as LED display screens and liquid crystal display screens, and can be of structures such as curved screens or flat screens. It should be understood that those skilled in the art can customize the type, quantity, size, resolution, etc. of the screens used in the virtual shooting system according to actual needs, and the embodiments of the present disclosure do not limit this.

[0042] Among them, the control device 03 can be an electronic device with computing, processing, and control capabilities. For example, a desktop computer, a laptop computer, etc. During virtual shooting, the control device 03 can be used to control the multiple screens (021, 022, 023) to display virtual scenes, control the shooting pose of the camera, receive and process the video data captured by the camera, perform post-production on the video data, etc.; before virtual shooting, the control device 03 can also be used to execute the calibration data generation method of the embodiments of the present disclosure to generate calibration data and calibrate the camera parameters of the camera 01. Among them, the camera parameters can include the camera internal parameters (such as the optical center, focal length) and distortion parameters (such as radial distortion, tangential distortion) of the camera.

[0043] It should be understood that the calibration data generation method of the embodiments of the present disclosure can also be executed on other electronic devices, that is, other electronic devices can be connected to and control the above-mentioned multiple screens (021, 022, 023) and the camera 01 to execute the calibration data generation method of the embodiments of the present disclosure and calibrate the camera parameters of the camera 01. The embodiments of the present disclosure do not limit the execution subject of the calibration data generation method. Among them, the electronic device can include a terminal device or a server.

[0044] The calibration data generation method according to the embodiments of the present disclosure can be deployed on various terminal devices through software or hardware transformation. The terminal devices involved in the embodiments of the present disclosure may refer to devices with wireless connection functions and / or wired connection functions. The wireless connection function means that it can be connected to other devices (such as the above-mentioned cameras, various screens, etc.) through wireless connection methods such as Wi-Fi and Bluetooth. The terminal devices involved in the embodiments of the present disclosure can also communicate with other devices through the wired connection function. The terminal devices involved in the embodiments of the present disclosure can be touch-screen, non-touch-screen, or without a screen. Touch-screen devices can be controlled by clicking, swiping, etc. on the display screen with fingers, styli, etc. Non-touch-screen devices can be connected to input devices such as mice, keyboards, and touch panels to control the terminal devices. Devices without a screen can be, for example, Bluetooth speakers without a screen. For example, the terminal devices of the present application may include, but are not limited to, user equipment (UE), mobile devices, user terminals, terminals, handheld devices, tablet computers, laptop computers, palmtop computers, computing devices, etc.

[0045] The calibration data generation method according to the embodiments of the present disclosure can also be deployed on a server. The server can be located in the cloud or locally, and can be a physical device or a virtual device, such as a virtual machine, a container, etc., and has a wireless communication function. Among them, the wireless communication function can be set in the chip (system) or other components or assemblies of the server. It can refer to a device with a wireless connection function. The wireless connection function means that it can be connected to other servers or terminal devices through wireless connection methods such as Wi-Fi and Bluetooth. The server involved in the embodiments of the present disclosure can also have the function of communicating through a wired connection. For example, the server involved in the embodiments of the present disclosure communicates with the control device 03 in the above virtual shooting system, sends the specified calibration board corresponding to each screen to the control device 03, receives multiple images captured by the camera 01 sent by the control device 03, detects the feature points and feature point information in the multiple images, determines the calibration data, and sends the calibration data to the control device 03. The control device 03 can calibrate the camera parameters of the camera 01 according to the calibration data.

[0046] The following will Figures 3 to 11 be used to introduce the calibration data generation method provided by the embodiments of the present disclosure in detail.

[0047] Figure 3 shows a flowchart of calibration data generation according to an embodiment of the present disclosure. This method can be used in the above virtual shooting system. This method can be executed by the control device in the above virtual shooting system, or can be executed by the above other electronic devices, such as Figure 3 shown, the calibration data generation method includes:

[0048] Step S31: Control each screen in the virtual shooting system to display its corresponding specified calibration board.

[0049] Among them, the specified calibration board corresponding to each screen includes multiple calibration units. Each calibration unit includes two identical two-dimensional identification codes, i.e., ArUco codes, which are adjacent to each other along the diagonal. The corner point at the connection between the two identical ArUco codes in each calibration unit is the feature point of each calibration unit. The code values of the two identical ArUco codes in each calibration unit are used to represent the serial number of the feature point of each calibration unit. The serial number of each feature point is correspondingly associated with the three-dimensional coordinates when the feature point is displayed on the screen. By using the specified calibration board provided in the embodiments of the present disclosure, the accuracy and efficiency of subsequent feature point detection can be improved.

[0050] Figure 4 The schematic diagram of a specified calibration board provided by the embodiments of the present disclosure is shown, as Figure 4 shown. The specified calibration board includes a total of 9 calibration units arranged in 3 rows × 3 columns. Each calibration unit includes two identical ArUco codes that are adjacent to each other along the diagonal. The area other than the ArUco codes in each calibration unit is a white area. Among them, the two identical ArUco codes, that is, the code values of the two ArUco codes are the same. In the embodiments of the present disclosure, the corner point at the connection between the two ArUco codes in each calibration unit is the feature point of each calibration unit. For example, Figure 4 the corner point marked in a certain calibration unit is the feature point of that calibration unit. It should be understood that Figure 4 the shown specified calibration board is an exemplary calibration board provided by the embodiments of the present disclosure. In fact, those skilled in the art can configure the required specified calibration board according to actual needs.

[0051] It can be known that the ArUco code is a square mark composed of binary codes, which consists of a wide black border and internal binary codes. The code value of the ArUco code can be obtained by identifying the binary codes. It should be understood that those skilled in the art can adopt the known ArUco code generation technology in the art to generate the ArUco codes used in the specified calibration board, and the embodiments of the present disclosure do not limit this. The code values of the ArUco codes in the calibration units of the specified calibration board are usually arranged in sequence, that is, the serial numbers of the feature points are also arranged in sequence. For example, Figure 4 in the shown specified calibration board, the code values of the ArUco codes in the calibration units can be arranged from left to right and from top to bottom as 01 to 09, that is, the serial numbers of the feature points can be arranged as 01 to 09. This can facilitate increasing the number of feature points that can be extracted from each image subsequently, and improving the accuracy and success rate of feature point detection.

[0052] As described above, multiple screens used in a virtual shooting system can have different sizes, resolutions, etc., and the cameras used can be cameras with different focal lengths. For example, they can be wide-angle cameras or telephoto cameras. The shooting ranges of cameras with different focal lengths are different, and there may also be various interference factors (such as the screen being blocked, etc.) in actual shooting that affect the stability of the calibration board. Therefore, a corresponding specified calibration board can be configured separately for each screen, so that the specified calibration board can adapt to screens of different sizes and resolutions, cameras with different focal lengths, and requirements for different feature point densities (the higher the feature point density, the more feature points in the specified calibration board), and minimize the influence of various interference factors on the stability of the specified calibration board. Among them, for example, the larger the resolution of the screen, the larger the size of the calibration unit in the specified calibration board, so that the specified calibration board can fill the entire screen. Since the longer the focal length of the camera, the narrower the field of view, the more calibration units can be in the specified calibration board when the focal length of the camera is longer, so that more feature points can be included in the images captured by the telephoto camera.

[0053] In a possible implementation, controlling each screen in the virtual shooting system to display its corresponding specified calibration board includes: in response to the calibration board configuration operation for each screen, determining the corresponding specified calibration board for each screen and controlling each screen to display its corresponding specified calibration board; where the calibration board configuration operation can be used to configure at least one of the following: the size of the calibration unit in the specified calibration board, the number of rows and columns of the calibration units in the specified calibration board, and the display area of the specified calibration board on the screen. Wherein, the size of the calibration unit can include the length and width of the calibration unit. It should be understood that after configuring and generating the specified calibration board for each screen, the three-dimensional coordinates of each feature point in the specified calibration board can be determined according to the three-dimensional coordinate system where each screen model is located or the three-dimensional coordinates of a certain vertex of the screen model, which is equivalent to determining the three-dimensional coordinates of the feature points when they are displayed on the screen. Then, the three-dimensional coordinates of each feature point in the specified calibration board can be associated with the serial number of the feature point one by one, so as to facilitate subsequent use of the serial number of the feature point to associate the two-dimensional coordinates of the feature point with the three-dimensional coordinates to obtain calibration data.

[0054] Exemplarily, assume Figure 2The three screens therein are all LED screens. Each of the three LED screens includes 9 LED cabinets arranged in 3 rows and 3 columns. Among them, the sizes of the LED cabinets in the same screen are known and the same. An LED cabinet can be understood as a constituent unit of an LED screen. In practical applications, for example, it is possible to directly configure each LED screen to use a specified calibration board with 9 calibration units arranged in 3 rows and 3 columns. The size of each calibration unit is the size of an LED cabinet. Each specified calibration board is displayed in the entire area of the LED screen, and each LED cabinet displays one specified calibration unit. Thus, the display effect of the specified calibration boards in the three LED screens as shown in Figure 5 can be obtained.

[0055] In practical applications, for example, through Figure 6 a schematic diagram of a configuration interface as shown, the specified calibration boards corresponding to the above three screens can be configured. Specifically, by configuring the size of the LED cabinet (i.e., the height and width of the LED cabinet), the number of rows or columns of the blank spaces at the top, bottom, left, and right in the LED screen, and the density of feature points displayed in the LED cabinet, it is possible to achieve the configuration of the size of the calibration unit in the calibration board, the number of rows and columns of the calibration unit in the calibration board, and the display area of the calibration board on the screen, so that the configured specified calibration board can adapt to different LED screens; Based on Figure 6 the configuration results shown in, Figure 7 the display effects of the respective specified calibration boards on the screen as shown can be obtained.

[0056] Among them, configuring the density of feature points displayed in the LED cabinet can be understood as configuring the number of feature points displayed in a single LED cabinet, or rather, configuring the number of calibration units displayed in a single LED cabinet. For example, Figure 6 the configured feature point densities for LED screens 021 and 022 are 2 respectively, which is equivalent to configuring 4 feature points, that is, 4 calibration units, to be displayed in each LED cabinet of LED screens 021 and 022. And for LED screen 023, the configured feature point density is 1, which is equivalent to configuring 1 feature point, that is, 1 calibration unit, to be displayed in each LED cabinet of LED screen 023. Then, since the size of the LED cabinet is known, after configuring the number of calibration units displayed in the LED cabinet, the size of the calibration unit in the specified calibration board can be obtained, which is equivalent to achieving the configuration of the size of the calibration unit in the calibration board;

[0057] Moreover, the number of rows or columns of blank spaces left at the top, bottom, left, and right in the LED screen can be configured. It can be understood that the number of rows and columns of calibration units in the specified calibration board that are not displayed in the LED screen is configured. At the same time, since the number of rows and columns of LED boxes in the LED screen is known, after configuring the number of calibration units displayed in the LED box and the number of rows or columns of blank spaces left at the top, bottom, left, and right in the LED screen, the number of rows and columns of calibration units in the calibration board and the display area of the calibration board on the screen can be determined. For example, Figure 6 in [reference document], for screen 021, 1 row of blank spaces is left at the top, bottom, left, and right respectively, and the feature point density is 2. That is to say, the specified calibration board for screen 021 should be a calibration board with 6 rows and 6 columns of calibration units, but there is 1 row of blank space at each of the top, bottom, left, and right, so that the calibration units in the specified calibration board displayed in LED screen 021 actually have 4 rows and 4 columns. This specified calibration board will be displayed in the middle area of the LED screen. Refer to Figure 7 the display effect of the specified calibration board in screen 021 in [reference document]; Figure 6 in [reference document], for LED screen 022, 2 rows of blank spaces are left on the upper side and the left side respectively, and no blank spaces are left on the lower side and the right side (i.e., 0 rows of blank spaces). That is to say, the specified calibration board for screen 022 should be a calibration board with 6 rows and 6 columns of calibration units, but there are 2 rows of blank spaces on the left side and the upper side respectively, so that the calibration units in the specified calibration board displayed in LED screen 022 actually have 4 rows and 4 columns. However, this specified calibration board will be displayed in the lower right area of the LED screen. Refer to Figure 7 the display effect of the specified calibration board in screen 022 in [reference document]; Figure 6 in [reference document], for LED screen 023, 0 rows of blank spaces are left at the top, bottom, left, and right, that is, no blank spaces are left, and the feature point density is 1. That is to say, the calibration board corresponding to LED screen 023 has 3 rows and 3 columns of calibration units. This specified calibration board will be displayed in the entire area of the LED screen. Refer to Figure 7 the display effect of the specified calibration board in screen 023 in [reference document]. Among them, by configuring the display area of the specified calibration, the size of the calibration unit, the number of rows and columns, etc., the specified calibration board can be made not to be displayed on the entire screen, thereby reducing the interference caused by factors such as screen occlusion, and improving the stability and effectiveness of the specified calibration board.

[0058] It should be understood that the configuration method in the above Figure 6 shown configuration interface is a possible implementation method provided by the embodiments of the present disclosure. In fact, those skilled in the art can customize the configuration method of the specified calibration board and the corresponding configuration interface according to actual needs, as long as the specified calibration board can be configured separately for each screen. The embodiments of the present disclosure do not limit this.

[0059] Step S32, when multiple screens have already displayed their respective specified calibration boards, control the camera to collect images of the multiple screens in different poses, and obtain multiple images collected by the camera.

[0060] It can be understood that, different from the traditional grid point calibration board, the designated calibration board in the embodiments of the present disclosure has a built-in feature point positioning function, that is, the serial number of the feature points can be located by recognizing the ArUco code. Therefore, when collecting images, the camera does not need to capture the entire designated calibration board (or screen), and can be photographed at any position and any angle of the screen, as long as the captured image contains the designated calibration board. Images captured in different poses (that is, different positions and different angles) can be used as valid images to complete subsequent feature point detection. If the images captured in each pose can contain multiple screens, even if the multiple screens captured in the image are not fully captured (that is, the designated calibration boards displayed on the multiple screens are not fully captured) and some ArUco codes are missing in the image, only one image needs to be captured in each pose to meet the calibration requirements, shortening the image acquisition time, improving the success rate of valid image acquisition, and having more valid images among the multiple captured images.

[0061] Step S33: Perform feature point detection on multiple images collected by the camera to obtain the detected feature points in each image and the feature point information of the detected feature points. The feature point information includes the serial number of the feature points and the two-dimensional coordinates of the feature points in the image.

[0062] In practical applications, the known ArUco detection algorithm in the art can be used to perform feature point detection on multiple images collected by the camera. The algorithm will detect the corner points at the four vertices of the ArUco code in the image and the encoded code value of this ArUco code, that is, the algorithm will detect the corner points and code values at the joints between the ArUco codes in each image. Then, based on the principle that the corner points at the joint between two identical ArUco codes are used as feature points, and the code values of two identical ArUco codes in each calibration unit are used to represent the serial number of the feature points of each calibration unit, the feature points can be determined from the detected corner points, and the code values of the two identical ArUco codes are used as the serial numbers of the feature points between the two identical ArUco codes. Moreover, the image coordinates of the feature points in the image, that is, the two-dimensional coordinates of the feature points, can be obtained, that is, the feature point information of the feature points is obtained.

[0063] In practical applications, to avoid the influence of the noise background in the image on feature point detection, the images collected by the camera can also be first converted into binary images (that is, images composed of pixels 0 and 1), and then the ArUco detection algorithm is used to perform feature point detection on the binary images. In a possible implementation manner, the performing feature point detection on multiple images collected by the camera to obtain the detected feature points in each image and the feature point information of the detected feature points includes:

[0064] Step S331: Convert each image captured by the camera into a grayscale image and perform binarization processing on the grayscale image to obtain a binarized image corresponding to each image.

[0065] Step S332: Use the ArUco detection algorithm to detect the corner points and code values at the joints between ArUco codes in the binarized image to obtain the ArUco detection result. The ArUco detection result includes the detected corner points and the code values of the detected ArUco codes.

[0066] Step S333: According to the code values of the ArUco codes detected in the ArUco detection result and the two-dimensional coordinates of the detected corner points in the binarized image, determine the feature points from the detected corner points and determine the feature point information of the feature points.

[0067] In practical applications, those skilled in the art can use image processing techniques known in the art, such as OpenCV, to implement converting each image captured by the camera into a grayscale image and performing binarization processing on the grayscale image to obtain a binarized image corresponding to each image. The embodiments of the present disclosure do not limit this. For example, Figure 8 shows an image captured by the camera. Figure 9 shows the binarized image corresponding to this image.

[0068] Exemplarily, Figure 10 shows the ArUco detection result obtained by using the ArUco detection algorithm to detect the Figure 9 binarized image. As shown in Figure 10 , the value of id marked on the ArUco code (for example, id = 20) represents the code value of the detected ArUco code, and the red dots represent the detected corner points. Furthermore, based on the principle that the corner points at the joints between two identical ArUco codes are used as feature points, and the code values of the two identical ArUco codes are used to characterize the serial numbers of the feature points, according to the Figure 10 code values of the ArUco codes detected in Figure 10 , determine the feature points and the feature information of the feature points from the corner points detected in

[0069] Considering that in actual situations, due to the low stability of the ArUco detection algorithm, in cases of image blurring, moiré patterns, or large collection angles, the corner points and code values may not be detected or detected inaccurately, reducing the detection efficiency and accuracy of feature points. That is, as shown in Figure 10As shown, in actual situations, there may also be cases where some corner points and code values of ArUco codes in the binary image are not detected by the ArUco detection algorithm, resulting in undetermined feature points in the binary image. In this case, the method may further include:

[0070] Step S334, adding the feature point information determined using the ArUco detection result to a preset candidate set;

[0071] Step S335, using a corner detection algorithm to detect the corner points in the binary image, obtaining a corner detection result, where the corner detection result includes all the corner points in the binary image;

[0072] Step S336, based on the corner detection result and the candidate set, determining the undetermined feature points in the binary image and the feature point information of the undetermined feature points in the binary image.

[0073] Among them, in step S334, the feature point information determined using the ArUco detection result, that is, the feature point information obtained from the above steps S331 to S333.

[0074] In practical applications, those skilled in the art can use known corner detection algorithms in the art, such as the Harris corner detection algorithm, the KLT corner detection algorithm, etc., to detect the corner points in the binary image and obtain all the corner points in the binary image. The embodiments of the present disclosure do not limit this. For example, Figure 11 shows the corner detection result obtained by using a corner detection algorithm to detect Figure 9 the binary image of, as Figure 11 shown, the corner detection algorithm can detect all the corner points in the image.

[0075] In a possible implementation manner, step S336, based on the corner detection result and the candidate set, determining the undetermined feature points in the binary image and the feature point information of the undetermined feature points in the binary image, includes:

[0076] Step S3361, for any feature point information in the candidate set, according to the first serial number in the feature point information, determining the second serial numbers of multiple nearest neighbor points corresponding to the feature point indicated by the first serial number, where the multiple nearest neighbor points represent multiple feature points that are respectively the closest to the feature point indicated by the first serial number in multiple directions;

[0077] Step S3362, according to the second serial numbers of the multiple nearest neighbor points, determining whether the second serial numbers of the multiple nearest neighbor points include a third serial number that does not exist in the candidate set;

[0078] Step S3363: When the third serial number that does not exist in the candidate set is included in the second serial numbers of multiple nearest points, estimate the two-dimensional coordinates of the nearest point indicated by the third serial number according to the two-dimensional coordinates of the feature point indicated by the first serial number in the feature point information.

[0079] Step S3364: Determine a first corner point that matches the nearest point indicated by the third serial number from the corner point detection result according to the two-dimensional coordinates of the nearest point indicated by the third serial number, and determine the first corner point as a feature point. Also, use the third serial number as the serial number of the first corner point, and use the two-dimensional coordinates of the first corner point in the binary image as the two-dimensional coordinates of the first corner point. Among them, the feature points not determined in the binary image include the first corner point.

[0080] As described above, the serial numbers of the feature points in the specified calibration board are arranged in sequence. In step S3361, the above steps S3361 to S3364 can be sequentially executed according to the serial numbers of the feature points. It should be noted that the prefixes such as "first", "second", and "third" in the embodiments and claims of the present disclosure are only used to distinguish different objects (such as distinguishing serial numbers), rather than to describe a specific order. For example, the first serial number is a certain serial number currently processed in the candidate set, and does not specifically refer to the first serial number. The second serial number may be before or after the first serial number. The third serial number is a certain serial number in the second serial numbers, rather than a serial number before or after.

[0081] In practical applications, since the serial numbers of the feature points in the specified calibration board are arranged in sequence, for the first serial number currently processed in the candidate set, according to the rule of sequential arrangement, the third serial numbers of multiple nearest points (that is, the nearest points) adjacent to the feature point indicated by the first serial number in the up, down, left, and right directions can be obtained. Here, the "nearest points" are feature points that should theoretically exist, but in fact, some of these feature points may not be detected in step S332 and thus do not exist in the candidate set. For example, for Figure 10 the feature point 19 (that is, the feature point indicated by the serial number "19"), the second serial numbers of the nearest points in the up, down, left, and right directions of the feature point 19 should be "13, 18, 20, 25"; then it can be judged whether each second serial number exists in the candidate set through step S3362, which is equivalent to judging whether the feature point indicated by each second serial number has been determined;

[0082] Among them, if there is a third serial number that does not exist in the candidate set among the second serial numbers, that is, there are undetermined feature points among multiple nearest points. For example, for Figure 10For the feature point indicated by the first serial number "19", if the nearest points indicated by the serial numbers "13" and "18" do not exist in the candidate set, then the serial numbers "13" and "18" are also the third serial numbers. Furthermore, through steps S3363 to S3364, the feature points indicated by the missing serial numbers "13" and "18" can be determined from the corner detection results, so as to increase the number of effective feature points that can be detected in each image, thereby increasing the number of calibration data, which is beneficial to improving the camera calibration accuracy of the camera.

[0083] In a possible implementation manner, in the above step S3362, estimating the two-dimensional coordinates of the nearest point indicated by the third serial number according to the two-dimensional coordinates of the feature point indicated by the first serial number in the feature point information includes:

[0084] According to the second serial numbers of multiple nearest points, it is judged whether the fourth serial number of the reverse point corresponding to the nearest point indicated by the third serial number exists in the candidate set, and the reverse point is the nearest point among the multiple nearest points that is distributed in the opposite direction to the nearest point indicated by the third serial number;

[0085] When the fourth serial number of the reverse point exists in the candidate set, according to the vector between the reverse point and the feature point indicated by the first serial number, and the two-dimensional coordinates of the feature point indicated by the first serial number, the two-dimensional coordinates of the nearest point indicated by the third serial number are determined.

[0086] The "opposite direction" here means the opposite direction relative to the feature point indicated by the first serial number. For example, for the first serial number 19, for Figure 10 the fourth serial number of the reverse point corresponding to the nearest point indicated by the third serial number "13" is "25", and the fourth serial number of the reverse point corresponding to the nearest point indicated by the third serial number "18" is "20", that is, the upper and lower nearest points are reverse points to each other, and the left and right nearest points are reverse points to each other. Since the fourth serial numbers "25" and "20" exist in the candidate set, therefore, the two-dimensional coordinates of the feature point indicated by the third serial number can be estimated by using the reverse point and the feature point. For example, use the two-dimensional coordinates of the reverse point 20 (that is, the nearest point indicated by the serial number 20) and the two-dimensional coordinates of the feature point 19 to calculate the vector from the feature point 20 to the feature point 19, and then add this vector to the two-dimensional coordinates of the feature point 19 to obtain the estimated two-dimensional coordinates of the nearest point indicated by the serial number "18". Due to the distortion error of the camera, there may be a difference between the two-dimensional coordinates calculated by using the vector and the actual two-dimensional coordinates of the nearest point 18 in the image. Therefore, through step S3364, according to the two-dimensional coordinates of the nearest point indicated by the third serial number, the first corner point that matches the nearest point indicated by the third serial number can be determined from the corner detection results and the first corner point is determined as the feature point.

[0087] In a possible implementation, step S3364 of determining, from the corner detection result, a first corner that matches the nearest point indicated by the third serial number according to the two-dimensional coordinates of the nearest point indicated by the third serial number includes:

[0088] Determining, from the corner detection result, a second corner that is closest in distance to the nearest point indicated by the third serial number according to the two-dimensional coordinates of the nearest point indicated by the third serial number, and determining whether the distance between the nearest point indicated by the third serial number and the second corner is less than a preset error threshold;

[0089] In the case where the distance between the nearest point indicated by the third serial number and the second corner is less than the preset error threshold, determining the second corner as the first corner that matches the nearest point indicated by the third serial number.

[0090] Since the corner detection result includes all corners in the binary image, the above process of determining the first corner can be understood as searching in the corner detection result for a second corner that is closest in distance to the nearest point indicated by the third serial number. The distance can be calculated from the two-dimensional coordinates of the two points. If the distance between the two-dimensional coordinates of this second corner and the two-dimensional coordinates of this nearest point is less than the preset error threshold, then it can be considered that this second corner is actually this nearest point, that is, the first corner that matches this nearest point. Therefore, this second corner can be determined as the feature point, and further, the third serial number can be used as the serial number of the first corner, and the two-dimensional coordinates of the first corner in the binary image can be used as the two-dimensional coordinates of the first corner.

[0091] It should be understood that in practical applications, for each determined feature point in the candidate set, the above steps S3361 to S3364 can be repeatedly executed using the corner detection result to determine the feature points and feature point information that have not been determined using the ArUco detection result until no new feature points are added in one execution process, so as to increase the number of feature points determined in each image.

[0092] In practical applications, the method further includes: adding the feature point information of the first corner to the candidate set, and for the feature point information of the first corner, repeatedly executing the determination of the feature points that have not been determined in the binary image and the feature point information of the feature points that have not been determined in the binary image according to the corner detection result and the candidate set. By this means, the number of feature points that can be detected in each image can be supplemented as much as possible, thereby increasing the number of calibration data, which is beneficial to improving the calibration accuracy of camera calibration using the calibration data.

[0093] Step 34: According to the serial numbers of the feature points detected in each image, associate the two-dimensional coordinates of the feature points detected in each image with the three-dimensional coordinates when displayed on the screen, so as to obtain the coordinate pairs of the feature points detected in each image. The calibration data includes the coordinate pairs of the feature points detected in multiple images, and the calibration data is used to calibrate the camera parameters of the camera.

[0094] It should be understood that through the above step 33, the feature points, the serial numbers and the two-dimensional coordinates of the feature points in each image captured by the camera can be detected. Since the serial numbers of the feature points are already associated with the three-dimensional coordinates when generating the specified calibration board, the serial numbers and the two-dimensional coordinates of the feature points obtained through the above step S33 are also associated. Therefore, the two-dimensional coordinates and the three-dimensional coordinates of the feature points can be associated through the serial numbers of the feature points to obtain the coordinate pairs of the feature points. It should be understood that the coordinate pair of each feature point includes the two-dimensional coordinate and the three-dimensional coordinate of this feature point.

[0095] In practical applications, after obtaining the calibration data, known camera calibration algorithms in the art can be used to implement the camera calibration of the camera. The embodiments of the present disclosure do not limit the camera calibration process.

[0096] According to the embodiments of the present disclosure, a calibration board with a new arrangement of ArUco codes is provided. Compared with the ordinary calibration board, the detection accuracy of the feature points is higher, and it is easier and more efficient to detect the feature points. Moreover, by configuring the specified calibration board corresponding to different screens, the specified calibration board can be adapted to different screens, and is not affected by the movement of the camera pose or the occlusion of the screen during the image acquisition process. The stability of collecting effective images is better, and the success rate of feature point detection can also be improved, thereby improving the accuracy and generation efficiency of the calibration data, which is beneficial to improving the camera calibration accuracy of the camera.

[0097] According to the embodiments of the present disclosure, a corner point repositioning algorithm based on breadth-first search is improved. Through the ArUco detection algorithm + corner point detection algorithm, the number of feature points that can be located in each acquisition is greatly increased, and the success rate and efficiency of data acquisition are improved, that is, the success rate of feature point detection is greatly improved, and there can also be a high recall rate in the case of blurred images, moiré patterns or large acquisition deflection angles.

[0098] According to the embodiments of the present disclosure, a calibration board generation strategy is provided, which can support synchronous acquisition of multiple LED screens and is not limited by the spatial position relationship of the LED screens, improving the efficiency of calibration data acquisition. That is, by separately configuring the corresponding specified calibration for each screen, only one corresponding calibration board needs to be displayed on each screen to complete the image acquisition, which is not affected by the movement of the camera or dynamic occlusion during the acquisition process, has better stability, and a higher success rate of collecting effective images.

[0099] Figure 12 A block diagram showing a calibration data generation device provided by an embodiment of the present disclosure, which is applied to a virtual shooting system. The virtual shooting system includes a camera for shooting and multiple screens for displaying a virtual scene. As Figure 12 shown, the device includes:

[0100] A control module 121, configured to control each screen in the virtual shooting system to display its corresponding specified calibration board. Wherein, the specified calibration board corresponding to each screen includes multiple calibration units, each calibration unit includes two identical two-dimensional identification codes, i.e., ArUco codes, which are adjacent along the diagonal. The corner point at the connection between the two identical ArUco codes in each calibration unit is the feature point of each calibration unit. The code values of the two identical ArUco codes in each calibration unit are used to represent the serial number of the feature point of each calibration unit, and the serial number of each feature point is correspondingly associated with the three-dimensional coordinates when the feature point is displayed on the screen;

[0101] An acquisition module 122, configured to control the camera to perform image acquisition on the multiple screens in different poses when the multiple screens have displayed their corresponding specified calibration boards, so as to obtain multiple images acquired by the camera;

[0102] A detection module 123, configured to perform feature point detection on the multiple images acquired by the camera, so as to obtain the detected feature points in each image and the feature point information of the detected feature points. The feature point information includes the serial number of the feature point and the two-dimensional coordinates of the feature point in the image;

[0103] An association module 124, configured to correspondingly associate the two-dimensional coordinates of the detected feature points in each image with the three-dimensional coordinates when the feature points are displayed on the screen according to the serial numbers of the detected feature points in each image, so as to obtain the coordinate pairs of the detected feature points in each image; wherein, the calibration data includes the coordinate pairs of the detected feature points in the multiple images, and the calibration data is used to calibrate the camera parameters.

[0104] In a possible implementation, feature point detection is performed on multiple images collected by the camera to obtain the detected feature points in each image and the feature point information of the detected feature points, including: converting each image collected by the camera into a grayscale image and performing binarization processing on the grayscale image to obtain a binarized image corresponding to each image; using the ArUco detection algorithm to detect the corner points and code values at the joints between ArUco codes in the binarized image to obtain an ArUco detection result, where the ArUco detection result includes the detected corner points and the code values of the detected ArUco codes; according to the code values of the detected ArUco codes in the ArUco detection result and the two-dimensional coordinates of the detected corner points in the binarized image, determining the feature points from the detected corner points and determining the feature point information of the feature points.

[0105] In a possible implementation, when there are corner points and code values of some ArUco codes in the binarized image that are not detected by the ArUco detection algorithm, resulting in undetected feature points in the binarized image, the device further includes: a corner point detection module, configured to add the feature point information determined using the ArUco detection result to a preset candidate set; using a corner point detection algorithm to detect the corner points in the binarized image to obtain a corner point detection result, where the corner point detection result includes all the corner points in the binarized image; according to the corner point detection result and the candidate set, determining the undetected feature points in the binarized image and the feature point information of the undetected feature points in the binarized image.

[0106] In a possible implementation, determining the feature points not determined in the binary image and the feature point information of the feature points not determined in the binary image according to the corner detection result and the candidate set includes: for any feature point information in the candidate set, determining, according to the first serial number in the feature point information, the second serial numbers of a plurality of nearest neighbor points corresponding to the feature point indicated by the first serial number, where the plurality of nearest neighbor points represent a plurality of feature points that are respectively the closest to the feature point indicated by the first serial number in a plurality of directions; judging whether the second serial numbers of the plurality of nearest neighbor points include a third serial number that does not exist in the candidate set according to the second serial numbers of the plurality of nearest neighbor points; in the case where the second serial numbers of the plurality of nearest neighbor points include a third serial number that does not exist in the candidate set, estimating the two-dimensional coordinates of the nearest neighbor point indicated by the third serial number according to the two-dimensional coordinates of the feature point indicated by the first serial number in the feature point information; determining, according to the two-dimensional coordinates of the nearest neighbor point indicated by the third serial number, a first corner point that matches the nearest neighbor point indicated by the third serial number from the corner detection result and determining the first corner point as a feature point, and using the third serial number as the serial number of the first corner point and using the two-dimensional coordinates of the first corner point in the binary image as the two-dimensional coordinates of the first corner point, where the feature points not determined in the binary image include the first corner point.

[0107] In a possible implementation, estimating the two-dimensional coordinates of the nearest neighbor point indicated by the third serial number according to the two-dimensional coordinates of the feature point indicated by the first serial number in the feature point information includes: judging whether a fourth serial number of a reverse point corresponding to the nearest neighbor point indicated by the third serial number exists in the candidate set according to the second serial numbers of the plurality of nearest neighbor points, where the reverse point is the nearest neighbor point that is distributed in the opposite direction to the nearest neighbor point indicated by the third serial number among the plurality of nearest neighbor points; in the case where the fourth serial number of the reverse point exists in the candidate set, determining the two-dimensional coordinates of the nearest neighbor point indicated by the third serial number according to the vector between the reverse point and the feature point indicated by the first serial number and the two-dimensional coordinates of the feature point indicated by the first serial number.

[0108] In a possible implementation, determining, from the corner detection result, a first corner that matches the nearest point indicated by the third serial number according to the two-dimensional coordinates of the nearest point indicated by the third serial number includes: determining, from the corner detection result, a second corner that is closest in distance to the nearest point indicated by the third serial number according to the two-dimensional coordinates of the nearest point indicated by the third serial number, and determining whether the distance between the nearest point indicated by the third serial number and the second corner is less than a preset error threshold; in the case where the distance between the nearest point indicated by the third serial number and the second corner is less than the preset error threshold, determining the second corner as the first corner that matches the nearest point indicated by the third serial number.

[0109] In a possible implementation, the apparatus further includes: an adding module, configured to add the feature point information of the first corner to the candidate set, so as to, for the feature point information of the first corner, perform again determining the feature points not determined in the binary image and the feature point information of the feature points not determined in the binary image according to the corner detection result and the candidate set.

[0110] In a possible implementation, controlling each screen in the virtual shooting system to display its corresponding specified calibration board includes: in response to a calibration board configuration operation for each screen, determining the corresponding specified calibration board for each screen and controlling each screen to display its corresponding specified calibration board; where the calibration board configuration operation is used to configure at least one of the following: the size of the calibration unit in the specified calibration board, the number of rows and columns of the calibration units in the specified calibration board, and the display area of the specified calibration board on the screen.

[0111] According to an embodiment of the present disclosure, a calibration board in a new arrangement of ArUco codes is provided. Compared with a common calibration board, the feature point detection accuracy is higher, it is easier and more efficient to detect feature points, and by configuring the specified calibration boards corresponding to different screens, the specified calibration boards can be adapted to different screens, and it is not affected by situations such as the movement of the camera pose during the image acquisition process and the occlusion of the screen. The stability of collecting effective images is better, and the success rate of feature point detection can also be improved, thereby improving the accuracy and generation efficiency of calibration data, which is beneficial to improving the camera calibration accuracy of the camera.

[0112] In some embodiments, the functions or modules included in the apparatus provided in the embodiments of the present disclosure can be used to execute the methods described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here.

[0113] Embodiments of the present disclosure also propose a computer-readable storage medium, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the above method is implemented. The computer-readable storage medium may be a volatile or non-volatile computer-readable storage medium.

[0114] Embodiments of the present disclosure also propose an electronic device, including: a processor; a memory for storing instructions executable by the processor; wherein, the processor is configured to implement the above method when executing the instructions stored in the memory.

[0115] Embodiments of the present disclosure also provide a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code, and when the computer-readable code runs in the processor of an electronic device, the processor in the electronic device executes the above method.

[0116] Figure 13 The block diagram of an electronic device 1900 according to an embodiment of the present disclosure is shown. For example, the electronic device 1900 may be provided as a control device, a server, or a terminal device. Referring to Figure 13 , the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by a memory 1932 for storing instructions executable by the processing component 1922, such as application programs. The application programs stored in the memory 1932 may include one or more modules each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to implement the above method.

[0117] The electronic device 1900 may further include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output interface 1958 (I / O interface). The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server TM , Mac OS X TM , Unix TM , Linux TM , FreeBSD TM or the like.

[0118] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, and the above computer program instructions can be executed by the processing component 1922 of the electronic device 1900 to complete the above method.

[0119] The present disclosure may be a system, a method, and / or a computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions thereon for causing a processor to implement aspects of the present disclosure.

[0120] A computer-readable storage medium may be a tangible device that can retain and store instructions for use by an instruction execution device. A computer-readable storage medium may be, for example—but not limited to—an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium include: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), a memory stick, a floppy disk, a mechanically encoded device such as a punch card or raised structures in a groove having instructions stored thereon, and any suitable combination of the foregoing. The computer-readable storage medium as used herein is not to be construed as a transitory signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through an optical fiber cable), or an electrical signal transmitted through a wire.

[0121] The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to respective computing / processing devices, or may be downloaded to an external computer or an external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include copper transmission cables, optical fiber transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in each computing / processing device.

[0122] The computer program instructions for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine - related instructions, microcode, firmware instructions, state - setting data, or source code or object code written in any combination of one or more programming languages, including object - oriented programming languages such as Smalltalk, C++, etc., and conventional procedural programming languages such as the "C" language or similar programming languages. The computer - readable program instructions may be executed entirely on the user's computer, partially on the user's computer, executed as a stand - alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider). In some embodiments, by using the state information of the computer - readable program instructions to customize an electronic circuit, such as a programmable logic circuit, a field - programmable gate array (FPGA), or a programmable logic array (PLA), the electronic circuit can execute the computer - readable program instructions to implement various aspects of the present disclosure.

[0123] Aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer - readable program instructions.

[0124] These computer - readable program instructions can be provided to a processor of a general - purpose computer, a special - purpose computer, or other programmable data - processing apparatus to produce a machine such that the instructions, when executed by the processor of the computer or other programmable data - processing apparatus, create a means for implementing the functions / acts specified in one or more blocks of the flowchart and / or block diagram. These computer - readable program instructions can also be stored in a computer - readable storage medium, which causes a computer, a programmable data - processing apparatus, and / or other devices to operate in a particular manner, so that the computer - readable medium storing the instructions includes a manufacture, which includes instructions for implementing various aspects of the functions / acts specified in one or more blocks of the flowchart and / or block diagram.

[0125] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device, causing a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process such that the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in one or more boxes of the flowchart and / or block diagram.

[0126] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, a segment of code, or a portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two consecutive blocks may in fact be executed substantially in parallel, or they may sometimes be executed in the reverse order, depending upon the functionality involved. It should also be noted that each block of the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented by a dedicated hardware-based system that performs the specified functions or acts, or by a combination of dedicated hardware and computer instructions.

[0127] The embodiments of the present disclosure have been described above. The above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The choice of terms used herein is intended to best explain the principles of the embodiments, the practical application, or improvements made to the technology in the marketplace, or to enable other ordinary skilled in the art to understand the embodiments disclosed herein.

Claims

1. A calibration data generation method, applied to a virtual shooting system, the virtual shooting system including a camera for shooting and multiple screens for displaying a virtual scene, characterized in that The method includes: Controlling each screen in the virtual shooting system to display its corresponding specified calibration board. Each specified calibration board corresponding to a screen includes a plurality of calibration units. Each calibration unit includes two identical two-dimensional identification codes, i.e., ArUco codes, which are distributed adjacent to each other along the diagonal. The corner point at the connection between the two identical ArUco codes in each calibration unit is the feature point of each calibration unit. The code values of the two identical ArUco codes in each calibration unit are used to represent the serial number of the feature point of each calibration unit. The serial number of each feature point is correspondingly associated with the three-dimensional coordinates when the feature point is displayed on the screen; When the multiple screens have displayed their corresponding specified calibration boards, controlling the camera to collect images of the multiple screens in different poses to obtain multiple images collected by the camera; Performing feature point detection on the multiple images collected by the camera to obtain the detected feature points in each image and the feature point information of the detected feature points. The feature point information includes the serial number of the feature point and the two-dimensional coordinates of the feature point in the image; According to the serial number of the detected feature points in each image, correspondingly associating the two-dimensional coordinates of the detected feature points in each image with the three-dimensional coordinates when they are displayed on the screen to obtain the coordinate pairs of the detected feature points in each image; Among them, the calibration data includes the coordinate pairs of the detected feature points in the multiple images, and the calibration data is used to calibrate the camera parameters.

2. The method according to claim 1, characterized in that, The performing feature point detection on the multiple images collected by the camera to obtain the detected feature points in each image and the feature point information of the detected feature points includes: Converting each image collected by the camera into a grayscale image and performing binarization processing on the grayscale image to obtain the binarized image corresponding to each image; Using the ArUco detection algorithm to detect the corner points and code values at the connections between ArUco codes in the binarized image to obtain the ArUco detection result. The ArUco detection result includes the detected corner points and the code values of the detected ArUco codes; According to the code values of the detected ArUco codes in the ArUco detection result and the two-dimensional coordinates of the detected corner points in the binarized image, determining the feature points from the detected corner points and determining the feature point information of the feature points.

3. The method according to claim 2, wherein When there are corner points and code values of some ArUco codes in the binarized image that are not detected by the ArUco detection algorithm, resulting in undetected feature points in the binarized image, the method further includes: Adding the feature point information determined by using the ArUco detection result to a preset candidate set; Using a corner point detection algorithm to detect the corner points in the binarized image to obtain the corner point detection result. The corner point detection result includes all the corner points in the binarized image; According to the corner point detection result and the candidate set, determining the undetected feature points in the binarized image and the feature point information of the undetected feature points in the binarized image.

4. The method according to claim 3, characterized in that Determining the feature points not determined in the binary image and the feature point information of the feature points not determined in the binary image according to the corner detection result and the candidate set includes: For any feature point information in the candidate set, according to the first serial number in the feature point information, determine the second serial numbers of a plurality of nearest neighbor points corresponding to the feature point indicated by the first serial number, where the plurality of nearest neighbor points represent a plurality of feature points that are respectively the closest to the feature point indicated by the first serial number in a plurality of directions; According to the second serial numbers of the plurality of nearest neighbor points, determine whether a third serial number that does not exist in the candidate set is included in the second serial numbers of the plurality of nearest neighbor points; When a third serial number that does not exist in the candidate set is included in the second serial numbers of the plurality of nearest neighbor points, estimate the two-dimensional coordinates of the nearest neighbor point indicated by the third serial number according to the two-dimensional coordinates of the feature point indicated by the first serial number in the feature point information; According to the two-dimensional coordinates of the nearest neighbor point indicated by the third serial number, determine a first corner point that matches the nearest neighbor point indicated by the third serial number from the corner detection result and determine the first corner point as a feature point, and use the third serial number as the serial number of the first corner point and use the two-dimensional coordinates of the first corner point in the binary image as the two-dimensional coordinates of the first corner point, where the feature points not determined in the binary image include the first corner point.

5. The method according to claim 4, characterized in that, The estimating the two-dimensional coordinates of the nearest neighbor point indicated by the third serial number according to the two-dimensional coordinates of the feature point indicated by the first serial number in the feature point information includes: According to the second serial numbers of the plurality of nearest neighbor points, determine whether a fourth serial number of a reverse point corresponding to the nearest neighbor point indicated by the third serial number exists in the candidate set, where the reverse point is the nearest neighbor point that is distributed in the opposite direction to the nearest neighbor point indicated by the third serial number among the plurality of nearest neighbor points; When the fourth serial number of the reverse point exists in the candidate set, determine the two-dimensional coordinates of the nearest neighbor point indicated by the third serial number according to the vector between the reverse point and the feature point indicated by the first serial number and the two-dimensional coordinates of the feature point indicated by the first serial number.

6. The method according to claim 4, wherein The determining a first corner point that matches the nearest neighbor point indicated by the third serial number from the corner detection result according to the two-dimensional coordinates of the nearest neighbor point indicated by the third serial number includes: According to the two-dimensional coordinates of the nearest neighbor point indicated by the third serial number, determine a second corner point that is the closest to the nearest neighbor point indicated by the third serial number from the corner detection result, and determine whether the distance between the nearest neighbor point indicated by the third serial number and the second corner point is less than a preset error threshold; When the distance between the nearest neighbor point indicated by the third serial number and the second corner point is less than the preset error threshold, determine the second corner point as the first corner point that matches the nearest neighbor point indicated by the third serial number.

7. The method according to any one of claims 4 to 5, characterized in that The method further includes: Add the feature point information of the first corner point to the candidate set, and for the feature point information of the first corner point, re - execute the determination of the feature points not determined in the binary image and the feature point information of the feature points not determined in the binary image according to the corner point detection result and the candidate set.

8. The method according to claim 1, characterized in that, The controlling each screen in the virtual shooting system to display its corresponding specified calibration board includes: Responding to the calibration board configuration operation for each screen, determining the corresponding specified calibration board for each screen and controlling each screen to display its corresponding specified calibration board; wherein, the calibration board configuration operation is used to configure at least one of the following: the size of the calibration unit in the specified calibration board, the number of rows and columns of the calibration units in the specified calibration board, and the display area of the specified calibration board on the screen.

9. A calibration data generation device is applied to a virtual shooting system, and the virtual shooting system includes a camera for shooting and multiple screens for displaying a virtual scene, characterized in that, The device includes: A control module, configured to control each screen in the virtual shooting system to display its corresponding specified calibration board, where the specified calibration board corresponding to each screen includes a plurality of calibration units, each calibration unit includes two identical two - dimensional identification codes, ArUco codes, distributed adjacent to each other along the diagonal, the corner point at the connection between the two identical ArUco codes in each calibration unit is the feature point of each calibration unit, the code values of the two identical ArUco codes in each calibration unit are used to represent the serial number of the feature point of each calibration unit, and the serial number of each feature point is corresponding and associated with the three - dimensional coordinates when the feature point is displayed on the screen; An acquisition module, configured to control the camera to perform image acquisition on the multiple screens in different poses to obtain multiple images acquired by the camera when the multiple screens have displayed their corresponding specified calibration boards; A detection module, configured to perform feature point detection on the multiple images acquired by the camera to obtain the detected feature points in each image and the feature point information of the detected feature points, where the feature point information includes the serial number of the feature point and the two - dimensional coordinates of the feature point in the image; An association module, configured to associate the two - dimensional coordinates of the detected feature points in each image with the three - dimensional coordinates when displayed on the screen according to the serial numbers of the detected feature points in each image to obtain the coordinate pairs of the detected feature points in each image; wherein, the calibration data includes the coordinate pairs of the detected feature points in the multiple images, and the calibration data is used to calibrate the camera parameters.

10. An electronic device, characterized in that, Includes: A processor; A memory for storing instructions executable by the processor; Wherein, the processor is configured to implement the method according to any one of claims 1 to 8 when executing the instructions stored in the memory.

11. A non-volatile computer-readable storage medium having computer program instructions stored thereon, characterized in that, The computer program instructions, when executed by the processor, implement the method according to any one of claims 1 to 8.

Images