Calibration data generation method, device, electronic device and storage medium

Abstract

The present disclosure relates to a calibration data generation method, device, electronic device and storage medium, the method comprising: controlling multiple screens to sequentially display multiple calibration images in a calibration image sequence; obtaining a target image sequence; performing feature point detection on each target image in the target image sequence to obtain a feature point set corresponding to each target image; determining the sequence number of the target image in which each feature point appears according to the two-dimensional coordinates of the feature points in the feature point set; determining the number of each feature point according to the sequence number of the target image in which each feature point appears; according to the number of each feature point, correspondingly associating the two-dimensional coordinates of the feature point with the three-dimensional coordinates of the feature point to obtain the coordinate pair corresponding to the feature point; the calibration data includes the coordinate pairs corresponding to the feature points detected in the multiple target images. According to the embodiments of the present disclosure, the requirements for the integrity and clarity of the captured images in camera calibration can be reduced, and the efficiency of collecting calibration data can be improved.

Description

Technical Field

[0001] The present disclosure relates to the technical field of camera calibration, and in particular to a calibration data generation method, device, electronic device and storage medium. Background Art

[0002] In order to meet the shooting needs, traditional film and television production needs to spend a lot of time and manpower to select shooting locations, make props, and build shooting scenes. However, virtual shooting (or virtual production) technology can use virtual scenes rendered by rendering engines to replace real scenes, reducing the dependence of film and television shooting on locations and scenery, and greatly reducing shooting costs. At the same time, with the ability of real-time rendering, for some special effects that originally required post-production, virtual shooting can see the effect of the film during the shooting stage, and the post-production is advanced, which improves production efficiency. It is precisely because of these advantages that virtual shooting has been increasingly used in film and television production in recent years.

[0003] In virtual shooting, if you want to achieve the fusion of "virtual" and "real", you first need to use camera calibration technology to establish the geometric imaging model parameters of the camera, that is, to calibrate the camera parameters of the camera. Generally, camera calibration is to print the calibration plate on a rigid plane and set the printed calibration plate in the real scene. The camera calibration process usually includes: using the camera to be calibrated to shoot the calibration plate in the real scene to obtain an image containing the calibration plate, performing feature point detection on the image containing the calibration plate to obtain the two-dimensional image coordinates of the feature points in the image, and then one-to-one correspondence between the detected two-dimensional image coordinates and the preset three-dimensional coordinates of the feature points in the calibration plate to form calibration data for calibrating the camera, and then the calibration data can be used to calibrate the camera parameters of the camera.

[0004] However, in a virtual shooting scene, the calibration plate needs to be projected onto multiple large LED screens for camera calibration, and the existing calibration plates cannot adapt well to the camera calibration in the virtual shooting scene. For example, the checkerboard calibration plate composed of alternating black and white blocks shown in FIG1(a) and the grid point calibration plate composed of evenly arranged dots shown in FIG1(b), wherein the feature point in the checkerboard calibration plate is the corner point between two adjacent black blocks in the diagonal direction, and the feature point of the grid point calibration plate is the center of the dot. These two types of calibration plates need to be completely photographed each time an image is acquired in order to complete feature point detection and the subsequent correspondence between two-dimensional coordinates and three-dimensional coordinates. However, in a virtual shooting scene, the calibration plate will be displayed on multiple huge LED screens, and it cannot be guaranteed that the camera will take a complete and clear picture of the calibration plate on the LED screen, which affects the success rate of feature point detection and further affects the efficiency of generating calibration data. Summary of the invention

[0005] In view of this, the present disclosure proposes a calibration data generation method, device, electronic device and storage medium, which can be applicable to virtual shooting scenes, reduce the requirements for the integrity and clarity of captured images in camera calibration, and improve the efficiency of calibration data generation.

[0006] According to one aspect of the present disclosure, a calibration data generation method is provided, which is applied to a virtual shooting system, wherein the virtual shooting system includes a camera for shooting and a plurality of screens for displaying virtual scenes, wherein the method includes: controlling the plurality of screens to sequentially display a plurality of calibration images in a calibration image sequence, wherein each calibration image includes feature points that should be displayed by the plurality of screens, wherein the sequence number of the calibration image in which the feature point appears is determined by the number of the feature point, and the number of the feature point is correspondingly associated with the three-dimensional coordinates of the feature point when displayed on the screen; acquiring a target image sequence, wherein the target image sequence includes a plurality of target images arranged in sequence, and the plurality of target images are images captured by the camera for each calibration image sequentially displayed by the plurality of screens; performing feature point detection on each target image in the target image sequence, and obtaining the feature points corresponding to each target image; A feature point set, wherein the feature point set corresponding to each target image includes feature points detected in each target image and the two-dimensional coordinates of the feature points in the target image; according to the two-dimensional coordinates of the feature points in the feature point set corresponding to each target image, the sequence number of the target image in which each feature point in the feature point set appears is determined; according to the sequence number of the target image in which each feature point in the feature point set appears, the serial number of each feature point in the feature point set is determined; according to the serial number of each feature point in the feature point set, the two-dimensional coordinates of the feature points detected in the multiple target images are correspondingly associated with the three-dimensional coordinates of the feature points when displayed on the screen, so as to obtain the coordinate pairs corresponding to the feature points detected in the multiple target images; wherein the calibration data includes the coordinate pairs corresponding to the feature points detected in the multiple target images, and the calibration data is used to calibrate the camera parameters of the camera.

[0007] In a possible implementation, before controlling the multiple screens to sequentially display multiple calibration images in the calibration image sequence, the method further includes: in response to the calibration plate configuration operation for each of the screens, determining the grid point calibration plates corresponding to each of the screens, wherein the grid point calibration plates corresponding to each screen include multiple feature points; based on the total number of feature points in the multiple grid point calibration plates corresponding to the multiple screens, determining the unique numbers of all the feature points in the multiple grid point calibration plates, and correspondingly associating the numbers of the respective feature points in the multiple grid point calibration plates with the three-dimensional coordinates of the center of each feature point when it is to be displayed on the screen; determining, according to the numbers of the respective feature points in the multiple grid point calibration plates, the serial numbers of the calibration images in which the respective feature points in the multiple grid point calibration plates should appear; and generating the calibration image sequence according to the serial numbers of the calibration images in which the respective feature points in the multiple grid point calibration plates should appear and the position information of the respective feature points in the multiple grid point calibration plates.

[0008] In a possible implementation, determining the serial number of the calibration image in which each feature point in the multiple grid point calibration plates should appear according to the respective serial number of each feature point in the multiple grid point calibration plates includes: determining the number of bits of the binary code based on the total number, and converting the serial number of each feature point in the multiple grid point calibration plates into a binary code based on the number of bits of the binary code, wherein the number of bits of the binary code is also used to indicate the number of calibration images to be included in the calibration image sequence; determining the serial number of the calibration image in which each feature point should appear according to the binary code corresponding to the serial number of each feature point in the multiple grid point calibration plates, wherein feature points with the same number should appear in at least one calibration image, and feature points with different numbers should appear in different serial numbers of calibration images.

[0009] In a possible implementation, determining the serial number of the calibration image in which each feature point should appear according to the binary code corresponding to the number of each feature point in the multiple grid point calibration plates includes: determining the serial number of the calibration image in which each feature point should appear according to the position of code 1 in the binary code corresponding to the number of each feature point.

[0010] In a possible implementation, the controlling the multiple screens to sequentially display each calibration image in the calibration image sequence includes: controlling the multiple screens to first display a white image according to a preconfigured image dwell frame number, and then sequentially display each calibration image in the calibration image sequence; wherein the image dwell frame number is used to control the number of frames at which the white image and each calibration image remain when displayed on the multiple screens; wherein obtaining the target image sequence includes: when the camera starts to capture images on the multiple screens, performing white frame detection on the image currently captured by the camera, the white frame detection is used to detect whether the currently captured image is a white frame image, and the white frame image is the image captured by the camera when the multiple screens display white images; when it is detected that the image currently captured by the camera is a white frame image, controlling the camera to capture images on the multiple screens with the image dwell frame number as the capture period to obtain the target image sequence captured by the camera.

[0011] In a possible implementation, the method further includes: acquiring a white frame image captured by the camera, and obtaining a screen area where the multiple screens in the white frame image are located by detecting the outline of the white area in the white frame image, wherein the screen area in the white frame image is used to indicate the screen area in each target image captured by the camera; wherein, performing feature point detection on each target image in the target image sequence to obtain a set of feature points corresponding to each target image comprises: performing feature point detection on each target image to obtain the feature points detected in each target image and the two-dimensional coordinates of the center of the feature points in the target image; filtering out noise feature points outside the screen area in each target image according to the screen area in the white frame image to obtain a set of feature points corresponding to each target image.

[0012] In a possible implementation, the target image sequence includes I target images, the I target images correspond to I feature point sets, I is a positive integer, wherein the determining, according to the two-dimensional coordinates of the feature points in the feature point sets corresponding to the respective target images, the sequence number of the target image in which each feature point in the feature point set appears comprises: for the f-th feature point in the i-th feature point set corresponding to the i-th target image, determining, according to the two-dimensional coordinates of the f-th feature point, the feature points in the I feature point sets that are closest to the f-th feature point, and placing the feature points in the I feature point sets that are closest to the f-th feature point The nearest feature point is used as the I nearest neighboring points corresponding to the f-th feature point, where 1≤i≤I, 1≤f≤F, and F is the total number of feature points in the i-th feature point set; it is determined whether the distance between the j-th nearest neighboring point among the I nearest neighboring points and the f-th feature point is less than an error threshold, 1≤j≤I; when the distance between the j-th nearest neighboring point and the f-th feature point is less than the error threshold, it is determined that the j-th nearest neighboring point and the f-th feature point are feature points with the same number, and it is determined that the serial number of the target image in which the f-th feature point appears includes the serial number of the target image to which the j-th nearest neighboring point belongs.

[0013] In a possible implementation, the numbering of each feature point in the feature point set is determined according to the serial number of the target image in which each feature point in the feature point set appears, including: determining the binary code corresponding to the f-th feature point in the ith feature point set according to the serial number of the target image in which the f-th feature point appears, wherein the serial number of the target image in which the f-th feature point appears is used to indicate the position of code 1 in the binary code corresponding to the f-th feature point, and the total number of bits of the binary code is I; converting the binary code corresponding to the f-th feature point into a decimal code to obtain the number of the f-th feature point.

[0014] In one possible implementation, the calibration plate configuration operation is used to configure at least one of the following: the number of rows and columns of feature points in the grid point calibration plate, the display area of ​​the grid point calibration plate on the screen, and the number of image retention frames when the calibration image is displayed on multiple screens.

[0015] According to another aspect of the present disclosure, a calibration data generating device is provided, which is applied to a virtual shooting system, wherein the virtual shooting system includes a camera for shooting and a plurality of screens for displaying virtual scenes, and the method includes: a control module, which is used to control the plurality of screens to sequentially display each calibration image in a calibration image sequence, wherein each calibration image includes a feature point that should be displayed by each of the plurality of screens, and the sequence number of the calibration image in which the feature point appears is determined by the number of the feature point, and the number of the feature point is correspondingly associated with the three-dimensional coordinates of the feature point when displayed on the screen; an acquisition module, which is used to acquire a target image sequence, wherein the target image sequence includes a plurality of target images arranged in sequence, and the plurality of target images are images captured by the camera for each calibration image sequentially displayed by the plurality of screens; and a detection module, which is used to perform feature point detection on each target image in the target image sequence, and obtain the feature points corresponding to each target image. A set of feature points corresponding to each target image includes feature points detected in each target image and two-dimensional coordinates of the feature points in the target image; a serial number determination module is used to determine the serial number of each feature point in the feature point set according to the two-dimensional coordinates of the feature points in the feature point set corresponding to each target image; a serial number determination module is used to determine the serial number of each feature point in the feature point set according to the serial number of the target image in which each feature point in the feature point set appears; an association module is used to associate the two-dimensional coordinates of the feature points detected in the multiple target images with the three-dimensional coordinates of the feature points when displayed on the screen according to the serial number of each feature point in the feature point set, so as to obtain the coordinate pairs corresponding to the feature points detected in the multiple target images; wherein the calibration data includes the coordinate pairs corresponding to the feature points detected in the multiple target images, and the calibration data is used to calibrate the camera parameters of the camera.

[0016] According to another aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0017] According to another aspect of the present disclosure, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored, wherein the computer program instructions implement the above method when executed by a processor.

[0018] According to another aspect of the present disclosure, a computer program product is provided, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0019] According to the embodiments of the present disclosure, by supporting the camera to capture multiple screens synchronously, the number and duration of target image sequences required for camera calibration can be reduced. Since the serial number of the calibration image in which the feature points appear is determined by the serial number of the feature points, even if the target image captured by the camera does not contain all the feature points in the calibration image, the serial number of the feature points can be determined by the feature points detected in the target image. Moreover, even if the camera focus is not on the screen, making the target image blurry, as long as the feature points in the target image can be detected, the two-dimensional coordinates of the feature points can be used to determine the serial number of the target image in which the feature points appear, and then determine the serial number of the feature points. The system has good blur resistance and can overall reduce the requirements for the integrity and clarity of the captured images in camera calibration, thereby facilitating the improvement of the efficiency of generating calibration data, and further facilitating the improvement of the calibration efficiency of the camera in the virtual shooting system.

[0020] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0022] FIG. 1( a ) shows a schematic diagram of a chessboard calibration plate in the related art.

[0023] FIG. 1( b ) shows a schematic diagram of a grid point calibration plate of the related art.

[0024] Figure 2 A schematic diagram of a virtual shooting system according to an embodiment of the present disclosure is shown.

[0025] Figure 3 A flowchart of generating calibration data according to an embodiment of the present disclosure is shown.

[0026] Figure 4 A flowchart of a calibration image sequence generation process according to an embodiment of the present disclosure is shown.

[0027] Figure 5 A schematic diagram of a configuration interface according to an embodiment of the present disclosure is shown.

[0028] Figure 6 A schematic diagram showing the display effect of a grid point calibration plate on a screen according to an embodiment of the present disclosure.

[0029] Figure 7 A schematic diagram of a grid point calibration plate according to an embodiment of the present disclosure is shown.

[0030] Figure 8(a) to Figure 8(e)A schematic diagram showing multiple calibration images according to an embodiment of the present disclosure is shown.

[0031] Fig. 9 A schematic diagram showing another calibration image according to an embodiment of the present disclosure is shown.

[0032] Fig.10 A schematic diagram showing a white frame image according to an embodiment of the present disclosure.

[0033] Fig.11 A block diagram of a calibration data generating device provided by an embodiment of the present disclosure is shown.

[0034] Fig.12 A block diagram of an electronic device 1900 according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0035] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0036] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0037] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0038] As mentioned above, existing calibration plates, such as checkerboard calibration plates, grid point calibration plates, etc., cannot adapt well to virtual shooting scenes, resulting in low efficiency in generating calibration data. In view of this, the disclosed embodiment provides a calibration data generation method that can be applied to a virtual shooting system, can support the camera to synchronously capture multiple screens, even if the camera does not capture the complete calibration images displayed on the multiple screens, it can also perform subsequent feature point detection and other processing, and has good blur resistance, can well adapt to the camera calibration requirements of different focal lengths and different focus distances, even if the camera focus is not on the screen so that the captured target image is relatively blurry, the blurred target image can be used for subsequent feature point detection and other processing, and can also reduce the time and number required for image capture, which is conducive to improving the generation efficiency of calibration data, and thus is conducive to improving the calibration efficiency of cameras in virtual shooting systems.

[0039] Figure 2A schematic diagram of a virtual shooting system according to an embodiment of the present disclosure is shown. Figure 2 As shown, the virtual shooting system includes a camera 01 for shooting, multiple screens (021, 022, 023) for displaying virtual scenes, and a control device 03. The control device 03 is connected to the camera 01 and the multiple screens (021, 022, 023) respectively. It should be understood that the embodiment of the present disclosure does not limit the communication connection method between devices.

[0040] Among them, camera 01 can be understood as the camera to be calibrated, and the embodiment of the present disclosure does not limit the type of camera; the screen used in the virtual shooting system can be an LED display screen, an LCD display screen, etc., and can be a curved screen or a flat screen. It should be understood that technical personnel in this field can customize the type, quantity, size, resolution, etc. of the screen used in the virtual shooting system according to actual needs, and the embodiment of the present disclosure does not limit this.

[0041] Among them, the control device 03 can be an electronic device with computing and processing control capabilities, such as a desktop computer, a laptop computer, etc. During the virtual shooting process, the control device 03 can be used to control multiple screens (021, 022, 023) to display virtual scenes, control the shooting posture of the camera, receive and process the video data shot by the camera, and perform post-production on the video data; before performing virtual shooting, the control device 03 can also be used to execute the calibration data generation method of the embodiment of the present disclosure to generate calibration data and calibrate the camera parameters of the camera 01, wherein the camera parameters may include the camera intrinsic parameters (such as optical center, focal length) and distortion parameters (such as radial distortion, tangential distortion) of the camera.

[0042] It should be understood that the calibration data generation method of the embodiment of the present disclosure can also be executed on other electronic devices, that is, other electronic devices can be connected and controlled by the above-mentioned multiple screens (021, 022, 023) and the camera 01 to execute the calibration data generation method of the embodiment of the present disclosure and calibrate the camera parameters of the camera 01. The embodiment of the present disclosure does not limit the execution subject of the calibration data generation method. Among them, the electronic device can include a terminal device or a server.

[0043] The calibration data generation method of the embodiment of the present disclosure can be deployed on various terminal devices through software or hardware modification. The terminal device involved in the embodiment of the present disclosure may refer to a device with a wireless connection function and / or a wired connection function. The wireless connection function refers to the ability to connect to other devices (such as the above-mentioned camera, each screen, etc.) through wireless connection methods such as wifi and Bluetooth. The terminal device involved in the embodiment of the present disclosure can also communicate with other devices through a wired connection function. The terminal device involved in the embodiment of the present disclosure can be a touch screen, a non-touch screen, or a screen-free terminal device. The touch screen can be controlled by clicking and sliding on the display screen with a finger or a stylus. The non-touch screen device can be connected to an input device such as a mouse, a keyboard, a touch panel, and the terminal device is controlled through the input device. For example, the device without a screen can be a Bluetooth speaker without a screen. For example, the terminal device of the present application may include but is not limited to a user equipment (UE), a mobile device, a user terminal, a terminal, a handheld device, a tablet computer, a laptop, a PDA, a computing device, etc.

[0044] The calibration data generation method of the embodiment of the present disclosure can also be deployed on a server, which can be located in the cloud or locally, and can be a physical device or a virtual device, such as a virtual machine, a container, etc., and has a wireless communication function, wherein the wireless communication function can be set in the chip (system) or other parts or components of the server. It can refer to a device with a wireless connection function, and the wireless connection function means that it can be connected to other servers or terminal devices through wireless connection methods such as Wi-Fi and Bluetooth. The server involved in the embodiment of the present disclosure may also have the function of communicating through a wired connection. For example, the server involved in the embodiment of the present disclosure communicates with the control device 03 in the above-mentioned virtual shooting system, sends a calibration image sequence required to be displayed on multiple screens to the control device 03, receives a target image sequence shot by the camera 01 sent by the control device 03, executes the calibration data generation method of the embodiment of the present disclosure to determine the calibration data and send the calibration data to the control device 03, and the control device 03 can calibrate the camera parameters of the camera 01 according to the calibration data.

[0045] The following Figures 3 to 10 , the calibration data generation method provided in the embodiment of the present disclosure is introduced in detail.

[0046] Figure 3 The flowchart of calibration data generation according to an embodiment of the present disclosure is shown. The method can be used in the above virtual shooting system, and the method can be executed by the control device in the above virtual shooting system, and can also be executed by the above other electronic devices, such as Figure 3 As shown, the calibration data generation method includes:

[0047] Step S31, controlling multiple screens to sequentially display multiple calibration images in the calibration image sequence, each calibration image includes feature points that should be displayed on the multiple screens respectively, the sequence number of the calibration image in which the feature point appears is determined by the number of the feature point, and the number of the feature point is correspondingly associated with the three-dimensional coordinates of the feature point when displayed on the screen.

[0048] Step S32, obtaining a target image sequence, wherein the target image sequence includes a plurality of target images arranged in sequence, and the plurality of target images are images captured by the camera for each calibration image sequentially displayed on a plurality of screens;

[0049] Step S33, performing feature point detection on each target image in the target image sequence to obtain a feature point set corresponding to each target image, wherein the feature point set corresponding to each target image includes the feature points detected in each target image and the two-dimensional coordinates of the feature points in the target image;

[0050] Step S34, determining the sequence number of each feature point in the feature point set in the target image according to the two-dimensional coordinates of the feature points in the feature point set corresponding to each target image;

[0051] Step S35, determining the serial number of each feature point in the feature point set according to the serial number of each feature point in the target image in which the feature point appears;

[0052] Step S36, according to the number of each feature point in the feature point set, the two-dimensional coordinates of the feature points detected in the multiple target images are correspondingly associated with the three-dimensional coordinates of the feature points when displayed on the screen, so as to obtain the coordinate pairs corresponding to the feature points detected in the multiple target images; wherein the calibration data includes the coordinate pairs corresponding to the feature points detected in the multiple target images, and the calibration data is used to calibrate the camera parameters of the camera.

[0053] In practical applications, in step S31, the number of feature points that can be displayed in multiple screens can be pre-configured and each feature point can be uniquely numbered, and then the serial number of the calibration image in which it should appear is assigned to each feature point according to the number of each feature point, and multiple calibration images are generated based on the serial numbers of the calibration images in which each feature point should appear, wherein the serial numbers of the calibration images in which different numbers should appear are different, and feature points with the same number can appear in at least one calibration image, and the positions of feature points with the same number in at least one calibration image in which they appear should be the same, so as to facilitate the subsequent determination of feature points with the same number in different target images based on the positions of the feature points, that is, to achieve point sequence positioning of the feature points; wherein the multiple calibration images in the calibration image sequence are arranged in sequence, and the serial number of each calibration image is the sequential serial number of each calibration image in the calibration image sequence.

[0054] In a possible implementation, the present disclosure provides a method as follows: Figure 4 The calibration image sequence generation process shown in Figure 4 As shown, before controlling the multiple screens to sequentially display the multiple calibration images in the calibration image sequence in step S31, the method further includes:

[0055] Step S41, in response to the calibration plate configuration operation for each screen, determining the grid point calibration plate corresponding to each screen, wherein the grid point calibration plate corresponding to each screen includes a plurality of feature points;

[0056] Step S42, based on the total number of feature points in the multiple grid point calibration plates corresponding to the multiple screens, determining the unique numbers of all feature points in the multiple grid point calibration plates, and correspondingly associating the numbers of the respective feature points in the multiple grid point calibration plates with the three-dimensional coordinates of the center of each feature point when it is to be displayed on the screen;

[0057] Step S43, determining the serial number of the calibration image in which each feature point in the plurality of grid point calibration plates should appear, according to the serial number of each feature point in the plurality of grid point calibration plates;

[0058] Step S44 , generating a calibration image sequence according to the sequence number of the calibration image in which each feature point in the multiple grid point calibration plates should appear and the position information of each feature point in the multiple grid point calibration plates.

[0059] Wherein, in step S41, the grid point calibration plate may be in the form of feature points arranged in an array as shown in FIG1(b). The calibration plate configuration operation may be used to configure at least one of the following: the number of rows and columns of feature points in the grid point calibration plate, the display area of ​​the grid point calibration plate on the screen, and the number of image retention frames when the calibration image is displayed on multiple screens. Wherein, the feature points in the grid point calibration plate may be in any shape such as a circle or a square. In practical applications, circular feature points may be specifically used. In this way, the detection of circular feature points is more accurate than other detection methods such as corner point detection, and the detection success rate of circular feature points in a target image with low definition is also higher, which can be applied to the calibration of camera parameters of cameras at various focal lengths and various focus distances.

[0060] Among them, the size of the feature points in the grid point calibration plate (such as the radius of the circular feature point) can adopt the default size, and of course, the configuration options of the feature point size can also be provided to the user, so that the user can configure the required feature point size. It should be understood that after configuring and generating the grid point calibration plate for each screen, the three-dimensional coordinates of each feature point in the grid point calibration plate can be determined according to the three-dimensional coordinate system corresponding to the screen model of each screen or the three-dimensional coordinates of a vertex of the screen model. Specifically, the three-dimensional coordinates of the center of each feature point (such as the center of a circle) can be determined, which is equivalent to determining the three-dimensional coordinates of the feature point when it is displayed on the screen. After that, the three-dimensional coordinates of each feature point can be associated with the number of the feature point one by one, so that the two-dimensional coordinates of the feature point can be associated with the three-dimensional coordinates by using the number of the feature point in the future to obtain calibration data.

[0061] Among them, the number of image retention frames is used to indicate the number of frames that each calibration image stays on when displayed on the screen. It should be understood that the screen refresh rate is certain (that is, the number of frames displayed per second on the screen is certain). By configuring the number of image retention frames for each calibration image, it is equivalent to configuring the length of time each calibration image is displayed on the screen. In practical applications, the number of image retention frames can be configured according to the refresh rate of the screen in the virtual shooting system and the maximum acquisition frequency of the camera. For example, the screen refresh rate is 50 Hz, and the maximum acquisition frequency of the camera is 25 Hz, which means that when the camera performs image acquisition at the highest acquisition frequency, the camera only captures 1 frame of image for every 2 frames of image displayed on the screen. Therefore, the number of image retention frames can be set to 3 frames, 5 frames, 7 frames, etc., that is, the length of time each calibration image is displayed on the screen for 3 frames, 5 frames or 7 frames can be set, so as to ensure that the camera captures each calibration image displayed on the screen.

[0062] Among them, since the multiple screens used in the virtual shooting system can have different sizes, resolutions, etc., and the cameras used can be cameras with different focal lengths, such as wide-angle cameras and telephoto cameras. The ranges captured by cameras with different focal lengths are different, and there may be various interference factors (such as screen being blocked, etc.) in actual shooting that affect the stability of the calibration plate. Therefore, a corresponding grid point calibration plate can be configured for each screen separately, so that the grid point calibration plate can adapt to screens of different sizes and resolutions, cameras of different focal lengths, and different feature point densities (the higher the feature point density, the more feature points in the grid point calibration plate), and minimize the influence of various interference factors on the stability of the grid point calibration plate; for example, the larger the screen resolution or the larger the screen size, the more feature points can be in the grid point calibration plate and the larger the size of the feature points can be; since the longer the focal length of the camera, the narrower the field of view, the longer the focal length of the camera, the more feature points can be in the grid point calibration plate, so that the image captured by the telephoto camera can contain more feature points.

[0063] For example, assuming Figure 2 The three screens in the figure are all LED screens, and each of the three LED screens includes 9 LED boxes in 3 rows and 3 columns. The sizes of the LED boxes in the same screen are known and the same. The LED boxes can be understood as the components of the LED screen. Based on the three LED screens, for example, Figure 5 A schematic diagram of a configuration interface is shown, in which the grid point calibration plates corresponding to the three screens are configured, and the number of image retention frames (eg Figure 5 The configured image retention frame number is 7). Specifically, the number of rows and columns of feature points in the grid point calibration plate and the display area of ​​the grid point calibration plate in the screen can be configured by the number of blank rows or columns in the upper, lower, left and right of the LED screen and the density of feature points displayed in a single LED box, so that the configured grid point calibration plate can be adapted to different LED screens; wherein, knowing the display area of ​​the grid point calibration plate in the screen can also determine the three-dimensional coordinates of each feature point in the grid point calibration plate when it is displayed on the screen based on the screen model; based on Figure 5 The configuration results in can be obtained Figure 6 The display effect of the calibration plate of each grid point on the screen is shown.

[0064] in, Figure 5 The density of feature points configured in is equivalent to the number of feature points displayed in a single LED cabinet. For example Figure 5The feature point density configured for LED screens 021 and 022 is 2, which is equivalent to configuring each LED box in LED screens 021 and 022 to display 4 feature points, and the feature point density configured for LED screen 023 is 1, which is equivalent to configuring each LED box in LED screen 023 to display 1 feature point. Since the number of rows and columns of LED boxes in each LED screen is known, after configuring the feature point density, the number of rows and columns of feature points required when the grid point calibration plate fills the entire LED screen can be obtained; and, the number of blank rows or columns in the upper, lower, left, and right sides of the LED screen is configured, which is equivalent to configuring the grid point calibration plate to have several rows and columns of feature points that are not displayed in the LED screen. At the same time, since the number of rows and columns of LED boxes in the LED screen is known, after configuring the number of feature points displayed in a single LED box and the number of blank rows or columns in the upper, lower, left, and right sides of the LED screen, the number of rows and columns of feature points that the grid point calibration plate should actually contain and the display area of ​​the grid point calibration plate in the screen can be determined. For example, Figure 5 In the setting of screen 021, 1 row is left blank at the top, bottom, left and right, and the density of feature points is 2, which means that the grid point calibration plate of screen 021 should have 6 rows and 6 columns of feature points, but there is a row left blank at the top, bottom, left and right, so that the grid point calibration plate actually displayed in LED screen 021 has 4 rows and 4 columns of feature points. The grid point calibration plate will be displayed in the middle area of ​​the LED screen. Figure 6 The display effect of the middle grid point calibration plate on screen 021; Figure 5 The upper and left sides of the LED screen 022 are set to leave 1 blank row respectively, and the lower and right sides are not left blank (that is, 0 blank rows), which means that the grid point calibration plate of the screen 022 should have 6 rows and 6 columns of feature points, but 1 row is left blank on the left and upper sides, so that the feature points in the grid point calibration plate displayed in the LED screen 022 actually have 5 rows and 5 columns, but the grid point calibration plate will be displayed in the lower right area of ​​the LED screen, refer to Figure 6 The display effect of the middle grid point calibration plate on screen 022; Figure 5 For the LED screen 023, 0 rows are left blank on the top, bottom, left, and right sides, that is, no blank is left, and the feature point density is 1, which means that the feature points in the grid point calibration plate corresponding to the LED screen 023 have 3 rows and 3 columns. The grid point calibration plate will be displayed in the entire area of ​​the LED screen. Figure 6 The display effect of the grid point calibration plate on the screen 023. Among them, by configuring the number of rows and columns of feature points in the grid point calibration plate, the display area of ​​the grid point calibration plate on the screen, etc., the grid point calibration plate does not need to be displayed on the entire screen, and can adapt to the needs of different screens and different cameras, reduce the interference caused by factors such as screen occlusion, and improve the stability and effectiveness of the grid point calibration plate.

[0065] It should be understood that the above Figure 5 The configuration method in the configuration interface shown is a possible implementation method provided by the embodiment of the present disclosure. In fact, those skilled in the art can customize the configuration method of the grid point calibration plate and the content required to be configured in the configuration interface according to actual needs, and the embodiment of the present disclosure is not limited to this. For example, the size of the feature points can also be configured, or the calibration image display frame can be configured to trigger image acquisition, etc. (for example, if the image stays for 7 frames, the calibration image can be configured to trigger image acquisition when the 4th frame is displayed). Of course, the above content can directly use the default value, such as the default feature point size and the default camera triggering image acquisition when the calibration image displays the middle frame.

[0066] After configuring the grid point calibration plates corresponding to each screen in step S41, the total number of feature points in the multiple grid point calibration plates corresponding to the multiple screens can be counted. Figure 6 The total number of feature points in the grid point calibration plate shown in the figure is N=16+25+9=50; then in step S42, each feature point in the multiple grid point calibration plates can be numbered from 1 to N based on the total number of feature points counted, for example, 50 feature points are numbered from 1 to 50 respectively, and the unique number of all feature points in the multiple grid point calibration plates is obtained, and then the number of each feature point can be associated with the three-dimensional coordinates of the center of each feature point when it is displayed on the screen.

[0067] In step S43, the serial number of the feature point can be converted into a binary code, and then the serial number of the calibration image in which the feature point should appear can be determined based on the binary code. Specifically, in a possible implementation, according to the serial number of each feature point in the multiple grid point calibration plates, the serial number of the calibration image in which each feature point in the multiple grid point calibration plates should appear can be determined, including:

[0068] Based on the total number, the number of bits of the binary code is determined, and based on the number of bits of the binary code, the number of each feature point in the multiple grid point calibration plates is converted into a binary code, and the number of bits of the binary code is also used to indicate the number of calibration images that should be included in the calibration image sequence; according to the binary code corresponding to the number of each feature point in the multiple grid point calibration plates, the serial number of the calibration image in which each feature point should appear is determined, wherein the feature point with the same number should appear in at least one calibration image, and the serial numbers of the calibration images in which the feature points with different numbers should appear are different. In this way, the serial number of the calibration image in which the feature point should appear can be quickly and regularly determined using binary coding.

[0069] Among them, based on the total number of feature points, determining the number of bits of binary coding is equivalent to determining how many bits of binary coding are used to encode the number of feature points. For example, the formula To achieve the total number of feature points based on the determination of the number of bits S of the binary code, where Represents rounding up. For example, when N=50, S is 6, which means that 6 bits of binary code are needed to encode the numbers of 50 feature points. Then, the numbers of each feature point can be converted into binary codes according to the number of bits of the binary code. For example, the 6-bit binary code of the number "1" is "000001" and the 6-bit binary code of the number "50" is "110010".

[0070] The number of bits of the binary code is also used to indicate the number of calibration images contained in the calibration image sequence (that is, the length of the calibration image sequence), that is, the number of calibration images in the calibration image sequence is consistent with the number of bits of the binary code. For example, a calibration image sequence corresponding to a 6-bit binary code contains 6 calibration images. Based on this, in a possible implementation, the serial number of the calibration image in which each feature point should appear is determined according to the binary code corresponding to the number of each feature point in the calibration plate of multiple grid points, including: determining the serial number of the calibration image in which each feature point should appear according to the position of the code 1 in the binary code corresponding to the number of each feature point.

[0071] For example, the 6-bit binary code of number "1" is "000001", that is, the first bit in the 6-bit binary code of number "1" is 1, then the feature point numbered "1" should appear in the first calibration image of the calibration image sequence, that is, the serial number of the calibration image in which the feature point numbered "1" should appear is 1; the binary code of numbered "3" is "000011", then the feature point numbered "3" should appear in the first and second calibration images of the calibration image sequence, that is, the serial numbers of the calibration images in which the feature point numbered "3" should appear include 1 and 2; the 6-bit binary code of numbered "50" is "110010", then the feature point numbered "50" should appear in the second, fifth and sixth calibration images of the calibration image sequence, that is, the serial numbers of the calibration images in which the feature point numbered "50" should appear include 2, 5 and 6; the feature points of other numbers are determined by analogy to determine the serial numbers of the calibration images in which they should appear.

[0072] It should be understood that the above-mentioned determination of the sequence number of the calibration image where the feature point should appear based on the position of the code 1 is a possible implementation method provided by the embodiment of the present disclosure. In fact, the sequence number of the calibration image where the feature point should appear can also be determined based on the position of the code 0. Among them, the method of determining the sequence number based on the position of the code 1 can reduce the number of feature points that should appear in the calibration image, which is conducive to reducing the time consumption of subsequent feature point detection and improving the efficiency of feature point detection.

[0073] After determining the serial numbers of the calibration images in which each feature point in the multiple grid point calibration plates should appear through the above step S43, a calibration image sequence can be generated in step S44 according to the serial numbers of the calibration images in which each feature point should appear and the position information of each feature point in the multiple grid point calibration plates. The position information of each feature point in the grid point calibration plate is also the position of the feature point in the grid point calibration plate, and the position information can indicate the position of the same coded feature point in the calibration image in which it should appear, wherein the positions of the feature points with the same number in the calibration image in which it should appear are the same, then the serial numbers of the calibration images in which the feature points should appear and the positions of the feature points in the calibration images in which they should appear are known, and each calibration image in the calibration image sequence can be generated.

[0074] For example, Figure 7 The three grid point calibration plates corresponding to the three screens shown in the figure and the numbers of the feature points in the grid point calibration plates, each grid point calibration plate has 3 rows and 3 columns with a total of 27 circular feature points, and the 27 feature points are numbered from 1 to 27. Through the above steps S43 to S44, the following can be obtained: Figure 8(a) to Figure 8(e) Schematic diagram of 5 calibration images when displayed on 3 screens, where: Figure 8(a) to Figure 8(e) The order of arrangement represents the order of each calibration image in the calibration image sequence, such as Figure 8(a) to Figure 8(e) As shown in FIG. 8 , feature points with the same number appear in the same position in the calibration images, that is, the positions of feature points with the same number when displayed on the screen are the same. For example, feature point number 1 only appears in the first calibration image shown in FIG. 8 (a), feature point number 2 only appears in the second calibration image shown in FIG. 8 (b), feature point number 3 appears in the first and second calibration images shown in FIG. 8 (a) and FIG. 8 (b), and the same applies to other feature points.

[0075] It should be understood that Figure 8(a) to Figure 8(e) The display mode of black dots on a white background is an exemplary implementation mode provided by the embodiment of the present disclosure. In fact, the display mode of white dots on a black background can also be used to display the feature points in the calibration image. For example, Fig. 9 In the calibration image shown, white dots on a black background are used to represent feature points.

[0076] According to the implementation method of the above-mentioned steps S41 to S44 in the embodiment of the present disclosure, the calibration image in which the feature point should appear is determined by determining the binary code corresponding to the number of the feature point, which facilitates the subsequent point sequence positioning of the feature point and greatly reduces the number of calibration images required for locating the feature point, which is beneficial to improving the acquisition efficiency of the subsequent acquisition of the target image sequence and improving the detection efficiency of the subsequent feature point detection, thereby improving the efficiency of generating calibration data.

[0077] After generating the target image sequence, in order to ensure that the camera can capture each calibration image displayed on the screen, the camera can be started to capture images of multiple screens, and then the multiple screens can be controlled to display each calibration image in the calibration image sequence in turn. Specifically, the multiple screens can be controlled to display each calibration image in turn according to the image capture frame number configured above, wherein the position of the camera during the image capture process should remain unchanged as much as possible and the camera is controlled to capture images of the screen with the above image capture frame number as the capture period, so that the target image sequence captured by the camera is a valid image captured for each calibration image displayed on the multiple screens in turn.

[0078] In practical applications, in order to enable the camera to know when to start collecting the target image sequence during the image collection phase, before controlling the screen to display the calibration image sequence, each screen can be controlled to display a white image as a start signal, and then the calibration images in the calibration image sequence are sequentially displayed on the screen, and the number of frames that the white image and each calibration image stay on the screen is determined by the configured image retention frame number. In a possible implementation, in the above step S31, controlling multiple screens to sequentially display the calibration images in the calibration image sequence includes:

[0079] Control multiple screens to first display a white image according to a pre-configured number of image dwell frames, and then display each calibration image in the calibration image sequence in sequence; wherein the number of image dwell frames is used to control the number of frames that the white image and each calibration image stay when displayed on the multiple screens; based on this, in the above step S32, obtaining the target image sequence includes: when the camera starts to capture images on the multiple screens, performing white frame detection on the image currently captured by the camera, the white frame detection is used to detect whether the currently captured image is a white frame image, and the white frame image is the image captured by the camera when the multiple screens display white images; when it is detected that the image currently captured by the camera is a white frame image, control the camera to capture images on the multiple screens with the number of image dwell frames as the acquisition cycle, and obtain the target image sequence captured by the camera. In this way, the camera can capture the effective target image when the screen displays each calibration image, and the number and duration required for image capture are also less, which is conducive to improving the efficiency of generating calibration data.

[0080] It should be understood that those skilled in the art can use image detection technology known in the art to implement white frame detection of the target image, and the embodiments of the present disclosure are not limited to this. After the white frame image is detected, it means that the screen starts to display the calibration image sequence, so after the white frame image is detected, it is possible to wait for the length of the image retention frame number, and start to capture images of multiple screens with the image retention frame number as the acquisition cycle. For example, if the image retention frame number is 7 frames, the camera can capture a frame of the target image every 7 frames.

[0081] Among them, since the camera uses the image retention frame number as the acquisition cycle to capture images of multiple screens, and each calibration image in the calibration image sequence is displayed on the screen in sequence according to the image retention frame number, the sequence numbers of multiple target images in the target image sequence captured by the camera correspond to the sequence numbers of multiple calibration images in the calibration image sequence, that is, the camera captures the target image when the calibration image is displayed on the screen. Since the number of calibration images in the calibration image sequence is known, the image acquisition can be stopped when the number of target images captured by the camera reaches the number of calibration images in the calibration image sequence, which can significantly reduce the number and duration of image acquisition, that is, reduce the number of target images required to generate calibration data and the acquisition time.

[0082] In step S33, those skilled in the art may use feature point detection techniques known in the art to perform feature point detection on each target image in the target image sequence, and obtain feature points in each target image and the two-dimensional coordinates of the feature points. The two-dimensional coordinates of the feature points may be the two-dimensional coordinates of the center of the feature points in the target image. For example, for circular feature points, a circular point detection algorithm may be used to obtain the circular feature points in the target image and the two-dimensional coordinates of the center of the circular feature points in the target image.

[0083] Considering that in an actual scene, the target image captured by the camera may contain not only the screen area displaying the calibration image, but also a useless noise background area, and the background area may contain noise feature points. Based on this, in a possible implementation, the method may also include: obtaining a white frame image captured by the camera, and obtaining the screen area where multiple screens are located in the white frame image by detecting the contour of the white area in the white frame image, wherein the screen area in the white frame image is used to indicate the screen area in each target image captured by the camera, for example Fig.10 A white frame image that can be collected is shown. Fig.10 The white area in the image may represent the screen area.

[0084] In practical applications, for example, the contour detection algorithm can be used to detect the contour of the white area in the white frame image to obtain the screen area in the white frame image. It should be understood that since the position of the camera during the image acquisition process is unchanged, the screen area in the white frame image taken at the same position should be the same as the screen area in the target image, and the accuracy of the screen area obtained by detecting the white frame image is also high, so the screen area in the target image can be indicated by detecting the screen area in the white frame image, and then the screen area can be used to filter out the noise feature points outside the screen area in the target image.

[0085] In a possible implementation, in the above step S33, feature point detection is performed on each target image in the target image sequence to obtain a feature point set corresponding to each target image, including: feature point detection is performed on each target image to obtain the feature points detected in each target image and the two-dimensional coordinates of the center of the feature point in the target image; according to the screen area in the white frame image, the noise feature points outside the screen area in each target image are filtered out to obtain a feature point set corresponding to each target image. In this way, the feature points in the feature point set can be all valid feature points, which is conducive to improving the accuracy of determining the feature point number using the feature point set in the future.

[0086] After obtaining the feature point sets corresponding to each target image through step S33, since the positions of the feature points with the same number displayed on the screen are the same, and the two-dimensional coordinates of the feature points with the same number in the target image in which they appear in the target image sequence captured by the camera with unchanged posture should also be the same, in step S34, the feature points with the same number in each feature point set can be determined according to the two-dimensional coordinates of the feature points in the feature point sets corresponding to each target image, and the serial numbers of the target images in which the feature points with the same number appear can be obtained. Since the serial number of the target image corresponds to the serial number of the calibration image, this is equivalent to determining the serial number of the calibration image in which the feature points with the same number appear. Since the serial number of the calibration image in which the feature points appear is determined by the serial number of the feature points, the number of the feature points can be determined in step S35 based on the serial number of the target image in which the feature points appear.

[0087] In a possible implementation, if the calibration image sequence includes I calibration images, then the target image sequence captured by the camera also includes I target images, and the I target images correspond to I feature point sets, where I is a positive integer. In step S34, according to the two-dimensional coordinates of the feature points in the feature point sets corresponding to the respective target images, the sequence number of the target image in which each feature point in the feature point set appears is determined, including:

[0088] For the f-th feature point in the i-th feature point set corresponding to the i-th target image, determine the feature points in the I feature point sets that are closest to the f-th feature point according to the two-dimensional coordinates of the f-th feature point, and use the feature points in the I feature point sets that are closest to the f-th feature point as the I nearest neighboring points corresponding to the f-th feature point, where 1≤i≤I, 1≤f≤F, and F is the total number of feature points in the i-th feature point set;

[0089] Determine whether the distance between the jth nearest neighbor point among the I nearest neighbor points and the fth feature point is less than the error threshold, 1≤j≤I;

[0090] When the distance between the jth nearest neighbor point and the fth feature point is less than the error threshold, it is determined that the jth nearest neighbor point and the fth feature point are feature points with the same number, and it is determined that the serial number of the target image in which the fth feature point appears includes the serial number of the target image to which the jth nearest neighbor point belongs.

[0091] For example, suppose the target image sequence is that multiple screens display the above Figure 8(a) to Figure 8(e) The image sequence collected when the calibration image sequence is shown, then I=5, the first feature point set corresponding to the first target image includes 5 feature points, and the 5 feature points in the first feature point set can be traversed to determine the nearest points corresponding to the 5 feature points in turn. For example, for the fth feature point P0 in the first feature point set, a feature point with the closest distance to the feature point P0 can be calculated from the first feature point set to the fifth feature point set as the nearest neighbor point, then a total of 5 nearest neighbors are obtained. Assuming that the 5 nearest neighbors are represented as P1, P2, P3, P4, and P5, respectively, it should be understood that the nearest neighbor point P1 with the closest distance to the feature point P0 determined in the first feature point set is actually the feature point P0 itself; then it is judged It is determined whether the distances between the feature point P0 and P1, P2, P3, P4, and P5 are respectively less than an error threshold. Optionally, the error threshold may be the minimum value of the distances between the feature points contained in each of the above-mentioned I feature point sets. Specifically, the distances between the feature points contained in the first feature point set to the I feature point set may be calculated respectively, and then the minimum value of the distances is determined as the error threshold. For example, assuming that there are two feature point sets, 5 feature points in the first feature point set can calculate 15 distances, and 6 feature points in the second feature point set can calculate 21 distances. The error threshold may be the minimum value of the 36 distances; or the error threshold may be a value set based on historical experience, which is not limited to this embodiment of the present disclosure. If it is determined that the distance between feature points P1 and P2 and feature point P0 is less than the error threshold, then P1 and P2 are considered to be feature points with the same number as P0. In other words, P1 and P2 may be in the same position as P0, or should be in the same position, but there is a slight deviation in position due to error. Then it is confirmed that the target image in which feature point P0 appears is numbered 1 and 2. If it is determined that the distance between feature points P1 and P4 and feature point P0 is less than the error threshold, then P1 and P4 are considered to be feature points with the same number as P0. Then it is determined that the target image in which feature point P0 appears is numbered 1 and 4. By analogy, the sequence numbers of the target images in which each feature point in the I feature point set appears can be determined.

[0092] It should be understood that the positions of feature points with the same number in the target image in which they appear should be the same, but since the camera may be unstable and the position may change during the actual shooting process, this may cause the actual position of the feature points with the same number in the target image in which they appear to have errors, and the position error (that is, the distance) between feature points with the same number should be less than the above error threshold, that is, the distance between feature points with the same number should be less than the minimum value of the distance between adjacent feature points in the set of I feature points; if the distance between two feature points is greater than or equal to the error threshold, it is considered that the two feature points are not feature points with the same number. Among them, the minimum value of the distance between adjacent feature points in the set of I feature points is the minimum distance between two feature points in multiple groups of adjacent positions in the set of I feature points. In the above manner, the serial number of each feature point in the target image can be accurately determined, so as to facilitate the subsequent accurate determination of the number of the feature points.

[0093] As described above, the disclosed embodiments also support configuring the number of rows and columns of feature points in the grid point calibration plate, which is equivalent to supporting adjustment of the minimum spacing between adjacent feature points. Therefore, when configuring the number of rows and columns of feature points in the grid point calibration plate, the stability of the camera in the actual scene can also be considered. For example, for the case where the stability of the camera in the actual scene is poor, that is, the camera whose posture is prone to change, the number of rows and columns of feature points in the grid point calibration plate can be reduced (that is, the distance between adjacent feature points is increased). This is equivalent to increasing the error threshold, thereby increasing the tolerance for changes in camera posture, which is beneficial to improving the success rate of the camera in acquiring a valid target image sequence, and is also beneficial to accurately determining the number of feature points.

[0094] As described above, when determining the serial number of the calibration image in which the feature point in the grid point calibration plate should appear, the serial number of the feature point can be converted into a binary code, and then the serial number of the calibration image in which the feature point should appear can be determined based on the binary code. Based on this, after determining the serial number of the target image in which each feature point of the I feature point set appears through the above step S34, in step S35, according to the serial number of the target image in which each feature point in the feature point set appears, the serial number of each feature point in the feature point set is determined, which may include:

[0095] According to the sequence number of the target image where the fth feature point in the i-th feature point set appears, the binary code corresponding to the fth feature point is determined, wherein the sequence number of the target image where the fth feature point appears is used to indicate the position of code 1 in the binary code corresponding to the fth feature point, and the total number of bits of the binary code is 1; the binary code corresponding to the fth feature point is converted into a decimal code to obtain the number of the fth feature point. In this way, the number of the feature point can be determined efficiently and accurately, that is, the point sequence positioning of the feature point in the target image sequence can be accurately achieved.

[0096] For example, if it is determined in the above example that the sequence numbers of the target images in which the feature point P0 appears are 1 and 2, then it can be determined that the 1st and 2nd bits in the binary code corresponding to the feature point P0 are 1. Since I is 5, that is, there are 5 calibration images in the target image sequence, and the number of bits of the binary code is 5, then the 3rd to 5th bits in the binary code corresponding to the feature point P0 should be 0, and the binary code corresponding to the feature point P0 is 00011, and the decimal code of 00011 is 3, so the feature point P0 is numbered 3; if the sequence numbers of the target images in which the feature point P0 appears are 1 and 4, it means that the binary code corresponding to the feature point P0 is 01001, and the decimal code of 01001 is 9, then the feature point P0 is numbered 9, and so on and so forth, the numbers corresponding to each feature point in the I feature point set can be determined.

[0097] It should be understood that the numbers of each feature point in the target image sequence captured by the camera and the two-dimensional coordinates of the feature point can be determined through the above steps S33 to S35. Since the numbers of the feature points have been correspondingly associated with the three-dimensional coordinates of the feature points when displayed on the screen when the grid point calibration plate is generated, the numbers of the feature points obtained through the above step S35 are also correspondingly associated with the two-dimensional coordinates. Therefore, in step S36, the two-dimensional coordinates of the feature points can be correspondingly associated with the three-dimensional coordinates based on the numbers of the feature points to obtain the coordinate pairs of the feature points; it should be understood that the coordinate pair of each feature point includes the two-dimensional coordinates and the three-dimensional coordinates of the feature point. Coordinate pairs can be obtained for all feature points appearing in the target image sequence.

[0098] In practical applications, after the calibration data is obtained through step S36, a camera calibration algorithm known in the art can be used to implement camera calibration of the camera. The embodiment of the present disclosure does not limit the camera calibration process.

[0099] According to the calibration data generation method of the embodiment of the present disclosure, by supporting the camera to synchronously capture multiple screens, the number and duration of target image sequences required for camera calibration can be reduced. Since the serial number of the calibration image in which the feature points appear is determined by the serial number of the feature points, even if the target image captured by the camera does not contain all the feature points in the calibration image, the serial number of the feature points detected in the target image can be used to determine the serial number of the feature points. Moreover, even if the camera focus is not on the screen, making the target image blurry, as long as the feature points in the target image can be detected, the two-dimensional coordinates of the feature points can be used to determine the serial number of the target image in which the feature points appear, and then determine the serial number of the feature points. The method has good blur resistance and can overall reduce the requirements for the integrity and clarity of the captured images in camera calibration, thereby facilitating improving the efficiency of generating calibration data, and further facilitating improving the calibration efficiency of cameras in virtual shooting systems.

[0100] According to the calibration data generation method of the disclosed embodiment, a calibration data generation method based on binary coding and a point sequence positioning method of feature points are proposed, which greatly reduces the number of calibration images required for feature point positioning and improves the efficiency of generating dynamic calibration data; the use of circular feature points can also improve the accuracy of feature point detection and the success rate of detecting blurred images, and can be applicable to the calibration of lens groups at various focal lengths and various focusing distances.

[0101] Fig.11 A block diagram of a calibration data generating device provided by an embodiment of the present disclosure is shown, which is applied to a virtual shooting system, wherein the virtual shooting system includes a camera for shooting and multiple screens for displaying virtual scenes, such as Fig.11 As shown, the device comprises:

[0102] The control module 111 is used to control the multiple screens to sequentially display the calibration images in the calibration image sequence, each calibration image includes the feature points that should be displayed by the multiple screens, the sequence number of the calibration image in which the feature point appears is determined by the number of the feature point, and the number of the feature point is correspondingly associated with the three-dimensional coordinates of the feature point when it is displayed on the screen;

[0103] An acquisition module 112, configured to acquire a target image sequence, wherein the target image sequence includes a plurality of target images arranged in sequence, and the plurality of target images are images acquired by the camera for each calibration image sequentially displayed on the plurality of screens;

[0104] The detection module 113 is used to perform feature point detection on each target image in the target image sequence to obtain a feature point set corresponding to each target image, wherein the feature point set corresponding to each target image includes the feature points detected in each target image and the two-dimensional coordinates of the feature points in the target image;

[0105] A sequence number determination module 114, configured to determine the sequence number of each feature point in the feature point set in the target image according to the two-dimensional coordinates of the feature points in the feature point set corresponding to each target image;

[0106] A serial number determination module 115, configured to determine the serial number of each feature point in the feature point set according to the serial number of each feature point in the target image in which the feature point appears;

[0107] The association module 116 is used to associate the two-dimensional coordinates of the feature points detected in the multiple target images with the three-dimensional coordinates of the feature points when displayed on the screen according to the numbers of the feature points in the feature point set, so as to obtain the coordinate pairs corresponding to the feature points detected in the multiple target images; wherein the calibration data includes the coordinate pairs corresponding to the feature points detected in the multiple target images, and the calibration data is used to calibrate the camera parameters of the camera.

[0108] In a possible implementation, before controlling the multiple screens to sequentially display multiple calibration images in the calibration image sequence, the device further includes: a calibration image sequence generation module, which is used to: determine the grid point calibration plates corresponding to each of the screens in response to the calibration plate configuration operation for each of the screens, wherein the grid point calibration plates corresponding to each screen include multiple feature points; determine the unique numbers of all the feature points in the multiple grid point calibration plates based on the total number of feature points in the multiple grid point calibration plates corresponding to the multiple screens, and associate the numbers of the respective feature points in the multiple grid point calibration plates with the three-dimensional coordinates of the center of each feature point when it is to be displayed on the screen; determine the serial numbers of the calibration images in which the respective feature points in the multiple grid point calibration plates should appear according to the serial numbers of the respective feature points in the multiple grid point calibration plates; and generate the calibration image sequence according to the serial numbers of the calibration images in which the respective feature points in the multiple grid point calibration plates should appear and the position information of the respective feature points in the multiple grid point calibration plates.

[0109] In a possible implementation, determining the serial number of the calibration image in which each feature point in the multiple grid point calibration plates should appear according to the respective serial number of each feature point in the multiple grid point calibration plates includes: determining the number of bits of the binary code based on the total number, and converting the serial number of each feature point in the multiple grid point calibration plates into a binary code based on the number of bits of the binary code, wherein the number of bits of the binary code is also used to indicate the number of calibration images to be included in the calibration image sequence; determining the serial number of the calibration image in which each feature point should appear according to the binary code corresponding to the serial number of each feature point in the multiple grid point calibration plates, wherein feature points with the same number should appear in at least one calibration image, and feature points with different numbers should appear in different serial numbers of calibration images.

[0110] In a possible implementation, determining the serial number of the calibration image in which each feature point should appear according to the binary code corresponding to the number of each feature point in the multiple grid point calibration plates includes: determining the serial number of the calibration image in which each feature point should appear according to the position of code 1 in the binary code corresponding to the number of each feature point.

[0111] In a possible implementation, the controlling the multiple screens to sequentially display each calibration image in the calibration image sequence includes: controlling the multiple screens to first display a white image according to a preconfigured image dwell frame number, and then sequentially display each calibration image in the calibration image sequence; wherein the image dwell frame number is used to control the number of frames at which the white image and each calibration image remain when displayed on the multiple screens; wherein obtaining the target image sequence includes: when the camera starts to capture images on the multiple screens, performing white frame detection on the image currently captured by the camera, the white frame detection is used to detect whether the currently captured image is a white frame image, and the white frame image is the image captured by the camera when the multiple screens display white images; when it is detected that the image currently captured by the camera is a white frame image, controlling the camera to capture images on the multiple screens with the image dwell frame number as the capture period to obtain the target image sequence captured by the camera.

[0112] In a possible implementation, the device also includes: a white frame detection module, which is used to obtain the white frame image captured by the camera, and obtain the screen area where the multiple screens in the white frame image are located by detecting the outline of the white area in the white frame image, wherein the screen area in the white frame image is used to indicate the screen area in each target image captured by the camera; wherein, the feature point detection is performed on each target image in the target image sequence to obtain a set of feature points corresponding to each target image, including: feature point detection is performed on each target image to obtain the feature points detected in each target image and the two-dimensional coordinates of the center of the feature points in the target image; according to the screen area in the white frame image, the noise feature points in each target image that are outside the screen area are filtered out to obtain a set of feature points corresponding to each target image.

[0113] In a possible implementation, the target image sequence includes I target images, the I target images correspond to I feature point sets, I is a positive integer, wherein the determining, according to the two-dimensional coordinates of the feature points in the feature point sets corresponding to the respective target images, the sequence number of the target image in which each feature point in the feature point set appears comprises: for the f-th feature point in the i-th feature point set corresponding to the i-th target image, determining, according to the two-dimensional coordinates of the f-th feature point, the feature points in the I feature point sets that are closest to the f-th feature point, and placing the feature points in the I feature point sets that are closest to the f-th feature point The nearest feature point is used as the I nearest neighboring points corresponding to the f-th feature point, where 1≤i≤I, 1≤f≤F, and F is the total number of feature points in the i-th feature point set; it is determined whether the distance between the j-th nearest neighboring point among the I nearest neighboring points and the f-th feature point is less than an error threshold, 1≤j≤I; when the distance between the j-th nearest neighboring point and the f-th feature point is less than the error threshold, it is determined that the j-th nearest neighboring point and the f-th feature point are feature points with the same number, and it is determined that the serial number of the target image in which the f-th feature point appears includes the serial number of the target image to which the j-th nearest neighboring point belongs.

[0114] In a possible implementation, the numbering of each feature point in the feature point set is determined according to the serial number of the target image in which each feature point in the feature point set appears, including: determining the binary code corresponding to the f-th feature point in the ith feature point set according to the serial number of the target image in which the f-th feature point appears, wherein the serial number of the target image in which the f-th feature point appears is used to indicate the position of code 1 in the binary code corresponding to the f-th feature point, and the total number of bits of the binary code is I; converting the binary code corresponding to the f-th feature point into a decimal code to obtain the number of the f-th feature point.

[0115] In one possible implementation, the calibration plate configuration operation is used to configure at least one of the following: the number of rows and columns of feature points in the grid point calibration plate, the display area of ​​the grid point calibration plate on the screen, and the number of image dwell frames when the calibration image is displayed on multiple screens.

[0116] According to the embodiments of the present disclosure, the camera can be supported to simultaneously capture multiple screens. Even if the camera does not capture the complete calibration plates displayed on the multiple screens, subsequent feature point detection and other processing can be performed. It has good blur resistance and can adapt well to camera calibration requirements of different focal lengths and different focusing distances. Even if the camera focus is not on the screen, making the captured target image blurry, the blurred target image can be used for subsequent feature point detection and other processing. It can also reduce the time and number required for image capture, which is beneficial to improving the efficiency of generating calibration data, and further beneficial to improving the calibration efficiency of cameras in the virtual shooting system.

[0117] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0118] The embodiment of the present disclosure also provides a computer-readable storage medium on which computer program instructions are stored, and the computer program instructions implement the above method when executed by a processor. The computer-readable storage medium can be a volatile or non-volatile computer-readable storage medium.

[0119] An embodiment of the present disclosure further proposes an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0120] The embodiments of the present disclosure also provide a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0121] Fig.12 1 is a block diagram of an electronic device 1900 according to an embodiment of the present disclosure. For example, the electronic device 1900 may be provided as a server or a terminal device. Fig.12, the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0122] The electronic device 1900 may also include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output interface 1958 (I / O interface). The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server 2003. TM , Mac OS X TM , Unix TM ,Linux TM , FreeBSD TM or similar.

[0123] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the electronic device 1900 to perform the above method.

[0124] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0125] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0126] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0127] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0128] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0129] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0130] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0131] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0132] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A calibration data generation method, applied to a virtual shooting system, wherein the virtual shooting system comprises a camera for shooting and a plurality of screens for displaying virtual scenes, characterized in that: The method comprises: Controlling the plurality of screens to sequentially display a plurality of calibration images in a calibration image sequence, each calibration image including a feature point that should be displayed by each of the plurality of screens, the sequence number of the calibration image in which the feature point appears is determined by the number of the feature point, and the number of the feature point is correspondingly associated with the three-dimensional coordinate of the feature point when displayed on the screen; Acquire a target image sequence, wherein the target image sequence includes a plurality of target images arranged in sequence, and the plurality of target images are images acquired by the camera for each calibration image sequentially displayed on the plurality of screens; Performing feature point detection on each target image in the target image sequence to obtain a feature point set corresponding to each target image, wherein the feature point set corresponding to each target image includes the feature points detected in each target image and the two-dimensional coordinates of the feature points in the target image; Determine, according to the two-dimensional coordinates of the feature points in the feature point sets corresponding to the respective target images, the sequence numbers of the target images in which the feature points in the feature point sets appear; wherein the sequence numbers of the target images in which the feature points appear include: the sequence numbers of the target images in which the feature points with the same number appear, determined according to the distances between the two-dimensional coordinates of the feature points in the feature point sets corresponding to the respective target images, wherein the distances between the two-dimensional coordinates of the feature points with the same number appear are less than an error threshold; Determine the serial number of each feature point in the feature point set according to the serial number of each feature point in the target image where the feature point appears; According to the serial number of each feature point in the feature point set, the two-dimensional coordinates of the feature points detected in the multiple target images are correspondingly associated with the three-dimensional coordinates of the feature points when displayed on the screen to obtain coordinate pairs corresponding to the feature points detected in the multiple target images; The calibration data includes coordinate pairs corresponding to feature points detected in the multiple target images, and the calibration data is used to calibrate camera parameters of the camera.

2. The method according to claim 1, characterized in that Before controlling the plurality of screens to sequentially display the plurality of calibration images in the calibration image sequence, the method further includes: In response to the calibration plate configuration operation for each of the screens, determining a grid point calibration plate corresponding to each of the screens, wherein the grid point calibration plate corresponding to each screen includes a plurality of feature points; Based on the total number of feature points in the multiple grid point calibration plates corresponding to the multiple screens, determine the unique numbers of all feature points in the multiple grid point calibration plates, and associate the numbers of the feature points in the multiple grid point calibration plates with the three-dimensional coordinates of the centers of the feature points when they are to be displayed on the screens; Determining, according to the respective serial numbers of the respective feature points in the plurality of grid point calibration plates, the serial numbers of the calibration images in which the respective feature points in the plurality of grid point calibration plates should appear; The calibration image sequence is generated according to the sequence number of the calibration image in which each feature point in the plurality of grid point calibration plates should appear and the position information of each feature point in the plurality of grid point calibration plates.

3. The method according to claim 2, characterized in that The determining, according to the respective serial numbers of the respective feature points in the plurality of grid point calibration plates, the serial numbers of the calibration images in which the respective feature points in the plurality of grid point calibration plates should appear, comprises: Based on the total number, determining the number of bits of the binary code, and based on the number of bits of the binary code, converting the number of each feature point in the plurality of grid point calibration plates into a binary code, wherein the number of bits of the binary code is also used to indicate the number of calibration images that should be included in the calibration image sequence; According to the binary codes corresponding to the numbers of the feature points in the multiple grid point calibration plates, the serial numbers of the calibration images in which the feature points should appear are determined, wherein the feature points with the same number should appear in at least one calibration image, and the feature points with different numbers should appear in different serial numbers in the calibration images.

4. The method according to claim 3, characterized in that The step of determining the sequence number of the calibration image in which each feature point should appear according to the binary code corresponding to the number of each feature point in the plurality of grid point calibration plates comprises: According to the position of code 1 in the binary code corresponding to the number of each feature point, the sequence number of the calibration image where each feature point should appear is determined.

5. The method according to claim 1, characterized in that The controlling the plurality of screens to sequentially display the calibration images in the calibration image sequence comprises: Controlling the multiple screens to first display a white image according to a pre-configured number of image retention frames, and then sequentially displaying each calibration image in the calibration image sequence; wherein the number of image retention frames is used to control the number of frames that the white image and each calibration image remain when displayed on the multiple screens; Wherein, acquiring the target image sequence comprises: When the camera starts to capture images of the multiple screens, a white frame detection is performed on the image currently captured by the camera, wherein the white frame detection is used to detect whether the currently captured image is a white frame image, and the white frame image is an image captured by the camera when the multiple screens display white images; When it is detected that the image currently captured by the camera is a white frame image, the camera is controlled to capture images of the multiple screens with the number of frames at which the image stays as a capture period to obtain a target image sequence captured by the camera.

6. The method according to claim 5, characterized in that The method further comprises: Acquire a white frame image captured by the camera, and obtain the screen area where the multiple screens are located in the white frame image by detecting the outline of the white area in the white frame image, wherein the screen area in the white frame image is used to indicate the screen area in each target image captured by the camera; The step of performing feature point detection on each target image in the target image sequence to obtain a set of feature points corresponding to each target image includes: Performing feature point detection on each target image to obtain the feature points detected in each target image and the two-dimensional coordinates of the centers of the feature points in the target image; According to the screen area in the white frame image, noise feature points outside the screen area in each target image are filtered out to obtain a feature point set corresponding to each target image.

7. The method according to any one of claims 1 to 6, characterized in that: The target image sequence includes I target images, and the I target images correspond to I feature point sets, where I is a positive integer, wherein determining the sequence number of the target image in which each feature point in the feature point set appears according to the two-dimensional coordinates of the feature points in the feature point set corresponding to each target image includes: For the f-th feature point in the i-th feature point set corresponding to the i-th target image, determine the feature points in the I feature point sets that are closest to the f-th feature point according to the two-dimensional coordinates of the f-th feature point, and use the feature points in the I feature point sets that are closest to the f-th feature point as the I nearest neighboring points corresponding to the f-th feature point, where 1≤i≤I, 1≤f≤F, and F is the total number of feature points in the i-th feature point set; Determine whether the distance between the jth nearest neighbor point among the I nearest neighbor points and the fth feature point is less than an error threshold, 1≤j≤I; When the distance between the jth nearest neighbor point and the fth feature point is less than an error threshold, it is determined that the jth nearest neighbor point and the fth feature point are feature points with the same number, and it is determined that the serial number of the target image in which the fth feature point appears includes the serial number of the target image to which the jth nearest neighbor point belongs.

8. The method according to claim 7, characterized in that The step of determining the serial number of each feature point in the feature point set according to the serial number of each feature point in the target image in which the feature point in the feature point set appears comprises: According to the sequence number of the target image in which the f-th feature point in the i-th feature point set appears, determine the binary code corresponding to the f-th feature point, wherein the sequence number of the target image in which the f-th feature point appears is used to indicate the position of code 1 in the binary code corresponding to the f-th feature point, and the total number of bits of the binary code is 1; The binary code corresponding to the f-th feature point is converted into a decimal code to obtain the number of the f-th feature point.

9. The method according to claim 2, characterized in that: The calibration plate configuration operation is used to configure at least one of the following: the number of rows and columns of feature points in the grid point calibration plate, the display area of ​​the grid point calibration plate on the screen, and the number of image retention frames when the calibration image is displayed on multiple screens.

10. A calibration data generating device, applied to a virtual shooting system, wherein the virtual shooting system comprises a camera for shooting and a plurality of screens for displaying virtual scenes, characterized in that: The device comprises: A control module, used for controlling the plurality of screens to sequentially display each calibration image in the calibration image sequence, each calibration image including a feature point to be displayed by each of the plurality of screens, the sequence number of the calibration image in which the feature point appears is determined by the number of the feature point, and the number of the feature point is correspondingly associated with the three-dimensional coordinate of the feature point when displayed on the screen; An acquisition module, used for acquiring a target image sequence, wherein the target image sequence includes a plurality of target images arranged in sequence, and the plurality of target images are images acquired by the camera for each calibration image sequentially displayed on the plurality of screens; A detection module is used to perform feature point detection on each target image in the target image sequence to obtain a feature point set corresponding to each target image, wherein the feature point set corresponding to each target image includes the feature points detected in each target image and the two-dimensional coordinates of the feature points in the target image; A sequence number determination module is used to determine the sequence number of the target image in which each feature point in the feature point set appears according to the two-dimensional coordinates of the feature points in the feature point set corresponding to each target image; wherein the sequence number of the target image in which each feature point appears includes: the sequence number of the target image in which the feature points with the same number appear determined according to the distance between the two-dimensional coordinates of the feature points in the feature point set corresponding to each target image, wherein the distance between the two-dimensional coordinates of the feature points with the same number is less than the error threshold; A number determination module, used to determine the number of each feature point in the feature point set according to the sequence number of each feature point in the target image where the feature point appears; An association module is used to associate the two-dimensional coordinates of the feature points detected in the multiple target images with the three-dimensional coordinates of the feature points when displayed on the screen according to the numbers of the respective feature points in the feature point set, so as to obtain coordinate pairs corresponding to the feature points detected in the multiple target images; wherein the calibration data includes the coordinate pairs corresponding to the feature points detected in the multiple target images, and the calibration data is used to calibrate the camera parameters of the camera.

11. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to implement the method described in any one of claims 1 to 9 when executing the instructions stored in the memory.

12. A non-volatile computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images