Method, device, electronic device and storage medium for determining positional relationship between screens

Abstract

The present disclosure relates to a method, device, electronic device and storage medium for determining the positional relationship between screens. The method includes: when S screens have displayed their respective corresponding calibration plate images, obtaining multiple screen images captured by a camera on the S screens at different positions; determining the calibration data sets corresponding to the S screens according to the multiple screen images; determining the camera intrinsic parameters of the camera and the reference screen in the S screens according to the calibration data sets corresponding to the S screens; determining the camera pose set corresponding to the S screens according to the camera intrinsic parameters of the camera and the calibration data sets corresponding to the S screens; determining the positional relationship between other screens in the S screens and the reference screen according to the camera pose set corresponding to the S screens. Thus, the precise positional relationship between the screens in the virtual shooting system can be quickly determined, reducing labor costs and hardware costs, which is conducive to improving the construction efficiency and restoration accuracy of the screen model.

Description

Technical Field

[0001] The present disclosure relates to the field of virtual shooting technology, and in particular to a method, device, electronic device and storage medium for determining a position relationship between screens. Background Art

[0002] In order to meet the shooting needs, traditional film and television production needs to spend a lot of time and manpower to select shooting locations, make props, and build shooting scenes. However, virtual shooting (or virtual production) technology can use virtual scenes rendered by rendering engines to replace real scenes, reducing the dependence of film and television shooting on locations and scenery, and greatly reducing shooting costs. At the same time, with the ability of real-time rendering, for some special effects that originally required post-production, virtual shooting technology can be used to see the effects of the film during the shooting stage, improving the efficiency of film production.

[0003] In virtual shooting technology, it is usually necessary to place a virtual camera in the virtual scene to simulate the displacement and rotation of the physical camera. It is also necessary to restore the screen model of the LED screen used to display the virtual scene in the real world 1:1, and project the virtual picture in the virtual camera's field of view onto the screen model after three-dimensional projection transformation, and then map it to the LED screen in the real world. In this way, the physical camera can not only shoot the real scenery and actors, but also the virtual scene displayed on the LED screen, creating an augmented reality experience with overlapping virtual and real. To achieve this virtual shooting process that combines virtual and real, it is necessary to restore the screen model of the LED screen in the real world 1:1 in the virtual engine, which includes both modeling of each LED screen and accurate measurement of the positional relationship between each LED screen. The combination of the two can restore the screen models of multiple LED screens in the real world in the virtual engine.

[0004] Currently, there are two main ways to determine the positional relationship, one is manual measurement, and the other is 3D scanning. Manual measurement is to manually measure the positional relationship between screens, and the accuracy of manual measurement is unstable, especially when the screens in the real world are far apart, the accuracy of positional relationship measurement will be significantly reduced, resulting in large errors in manual measurement and unable to meet the accuracy requirements of virtual shooting. The 3D scanning method is to obtain information such as the positional relationship between screens by performing 3D scanning on the on-site LED screen. Although the 3D scanning method has high accuracy, it relies on expensive hardware scanning equipment, has high hardware costs, and the scanned results often require manual adjustment. The above two methods require manual labor to spend a long time and have low efficiency. Summary of the invention

[0005] In view of this, the present disclosure proposes a method, device, electronic device and storage medium for determining the positional relationship between screens, which can quickly determine the precise positional relationship between each screen in a virtual shooting system, reduce labor costs and hardware costs, and help improve the construction efficiency and restoration accuracy of the screen model.

[0006] According to one aspect of the present disclosure, a method for determining a positional relationship between screens is provided, which is applied to a virtual shooting system, wherein the virtual shooting system includes a camera for shooting and S screens for displaying a virtual scene, where S≥2, and the method includes: when the S screens have displayed their respective corresponding calibration plate images, obtaining a plurality of screen images captured by the camera of the S screens at different postures, wherein the calibration plate image displayed on each screen includes a plurality of feature points; determining calibration data sets corresponding to the S screens according to the plurality of screen images, wherein the calibration data set corresponding to the s-th screen includes: a set of feature points consisting of feature points belonging to the s-th screen detected from at least one first screen image, wherein the first screen image is a plurality of screen images in which the feature points appearing in the plurality of screen images; a screen image of the s-th screen, s∈[1,S]; determining the camera intrinsic parameters of the camera and a reference screen in the S blocks of screens according to the calibration data sets corresponding to each of the S blocks of screens; determining the camera pose sets corresponding to each of the S blocks of screens according to the camera intrinsic parameters of the camera and the calibration data sets corresponding to each of the S blocks of screens, wherein the camera pose set corresponding to the s-th screen includes: at least one group of camera poses determined based on a set of feature points respectively belonging to the s-th screen in the at least one first screen image, one group of camera poses corresponding to one first screen image; determining the positional relationship between other screens in the S blocks of screens and the reference screen according to the camera pose sets corresponding to each of the S blocks of screens, wherein the other screens include screens in the S blocks of screens except the reference screen.

[0007] In a possible implementation, a plurality of feature points in the calibration plate image displayed on each screen are respectively provided with unique numbers, and the feature points in the calibration plate images corresponding to different screens have different numbers, and the number of the feature point is associated with the three-dimensional coordinates of the feature point when displayed on the screen, wherein determining the calibration data set corresponding to each of the S screens according to the multiple screen images includes: determining the feature points detected in each screen image and the feature point information of the detected feature points by performing feature point detection on the multiple screen images respectively, wherein the feature point information includes the number of the feature point and the two-dimensional coordinates of the feature point in the screen image; for the sth screen, determining from the multiple screen images the location where the sth screen appears according to the number of the feature points detected in each screen image N first screen images and feature point information belonging to the s-th screen in each first screen image, N≥1; according to the feature point information belonging to the s-th screen in each first screen image, determine a feature point set constituted by the feature points belonging to the s-th screen in each first screen image, wherein the feature point set constituted by the feature points belonging to the s-th screen in the n-th first screen image includes: the two-dimensional coordinates of each feature point belonging to the s-th screen in the n-th first screen image and the three-dimensional coordinates of each feature point belonging to the s-th screen in the n-th first screen image when displayed in the s-th screen, wherein the two-dimensional coordinates of the feature points in the same first screen image are associated with the three-dimensional coordinates, n∈[1,N].

[0008] In a possible implementation, determining the camera intrinsic parameters of the camera and a reference screen in the S blocks of screens according to the calibration data sets corresponding to each of the S blocks of screens includes: calibrating the camera intrinsic parameters respectively using the calibration data sets corresponding to each of the S blocks of screens to obtain S groups of calibrated camera intrinsic parameters, and determining a calibration error value corresponding to each group of camera intrinsic parameters in the S groups of camera intrinsic parameters, wherein one screen corresponds to a group of camera intrinsic parameters, and the calibration error value is used to characterize the error between the calibrated camera intrinsic parameters and the actual camera intrinsic parameters of the camera; determining the camera intrinsic parameters corresponding to the minimum calibration error value as the camera intrinsic parameters of the camera, and determining the screen corresponding to the camera intrinsic parameters corresponding to the minimum calibration error value as the reference screen.

[0009] In a possible implementation manner, determining the calibration error value corresponding to each group of camera intrinsic parameters in the S groups of camera intrinsic parameters includes: for the sth group of camera intrinsic parameters, determining a projection transformation relationship corresponding to the sth group of camera intrinsic parameters according to the sth group of camera intrinsic parameters, wherein the projection transformation relationship represents a coordinate transformation relationship between three-dimensional coordinates and two-dimensional coordinates of feature points; according to the projection transformation relationship corresponding to the sth group of camera intrinsic parameters, converting the three-dimensional coordinates of feature points in a calibration data set corresponding to the sth group of camera intrinsic parameters into two-dimensional projection coordinates; and determining the calibration error value corresponding to the sth group of camera intrinsic parameters according to a deviation between the two-dimensional projection coordinates of the feature points in the calibration data set corresponding to the sth group of camera intrinsic parameters and the actual two-dimensional coordinates of the feature points in the screen image.

[0010] In a possible implementation, the other screens in the S blocks of screens include: at least one first other screen that appears in the same screen image simultaneously with the reference screen; wherein, determining the positional relationship between the other screens in the S blocks of screens and the reference screen based on the camera pose sets corresponding to each of the S blocks of screens includes: for any first other screen, determining from the multiple screen images a second screen image in which the first other screen and the reference screen appear simultaneously; determining the positional relationship between the first other screen and the reference screen based on a camera pose determined by a feature point set belonging to the first other screen in the second screen image and a camera pose determined by a feature point set belonging to the reference screen.

[0011] In a possible implementation, the first other screen and the reference screen appear simultaneously in multiple second screen images, wherein the positional relationship between the first other screen and the reference screen is determined based on the camera pose determined by the feature point set belonging to the first other screen in the second screen image and the camera pose determined by the feature point set belonging to the reference screen, including: determining the reprojection error of the feature points displayed in the reference screen projected into each second screen image based on the camera pose determined by the feature point set belonging to the reference screen in each second screen image and the camera intrinsic parameters of the camera; determining the positional relationship between the first other screen and the reference screen based on the camera pose determined by the feature point set belonging to the first other screen in the second screen image corresponding to the minimum reprojection error and the camera pose determined by the feature point set belonging to the reference screen.

[0012] In a possible implementation, the other screens in the S blocks of screens also include: at least one second other screen that does not appear in the same screen image at the same time as the reference screen; wherein, determining the positional relationship between the other screens in the S blocks of screens and the reference screen based on the camera pose sets corresponding to each of the S blocks of screens includes: for any second other screen, determining the first other screen that appears in the same screen image at the same time as the second other screen, and determining from the multiple screen images a third screen image in which the second other screen and the first other screen appear at the same time; determining the positional relationship between the second other screen and the first other screen based on the camera pose determined by the feature point set belonging to the second other screen in the third screen image and the camera pose determined by the feature point set belonging to the first other screen; determining the positional relationship between the second other screen and the reference screen based on the positional relationship between the second other screen and the first other screen and the positional relationship between the first other screen and the reference screen.

[0013] In one possible implementation, the screen includes an LED screen, each LED screen includes multiple LED boxes, and the method further includes: obtaining screen information of each screen in the S screens, the screen information including the number of rows, columns, resolution and box size of the LED boxes in the screen; generating a calibration plate image corresponding to each screen according to the screen information of each screen, wherein multiple feature points in the calibration plate image corresponding to each screen cover each LED box in each screen.

[0014] According to another aspect of the present disclosure, a device for determining a positional relationship between screens is provided, which is applied to a virtual shooting system, wherein the virtual shooting system includes a camera for shooting and S screens for displaying a virtual scene, S≥2, and the device includes: an image acquisition module, which is used to acquire a plurality of screen images captured by the camera of the S screens at different postures when the S screens have displayed their respective corresponding calibration plate images, wherein the calibration plate image displayed by each screen includes a plurality of feature points; a calibration data set determination module, which is used to determine the calibration data sets corresponding to each of the S screens based on the plurality of screen images, wherein the calibration data set corresponding to the s-th screen includes: a feature point set consisting of feature points belonging to the s-th screen detected from at least one first screen image, wherein the first screen image is a plurality of screen images in which the s-th screen appears. A screen image, s∈[1,S]; a reference screen determination module, used to determine the camera intrinsic parameters of the camera and the reference screen in the S blocks of screens according to the calibration data sets corresponding to the S blocks of screens; a camera pose set determination module, used to determine the camera pose sets corresponding to the S blocks of screens according to the camera intrinsic parameters of the camera and the calibration data sets corresponding to the S blocks of screens, wherein the camera pose set corresponding to the s-th screen includes: at least one group of camera poses determined based on the feature point sets respectively belonging to the s-th screen in the at least one first screen image, and one group of camera poses corresponding to one first screen image; a position relationship determination module, used to determine the position relationship between other screens in the S blocks of screens and the reference screen according to the camera pose sets corresponding to the S blocks of screens, wherein the other screens include screens in the S blocks of screens except the reference screen.

[0015] According to another aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0016] According to another aspect of the present disclosure, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored, wherein the computer program instructions implement the above method when executed by a processor.

[0017] According to another aspect of the present disclosure, a computer program product is provided, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0018] According to the embodiments of the present disclosure, the calibration plate image displayed on the screen in the virtual shooting system and multiple screen images shot by a common camera can be used, and then the camera intrinsic parameters and the reference screen used by the camera can be determined according to the calibration data sets corresponding to each screen determined by the multiple screen images, and then the camera pose set of each screen can be determined by using the camera intrinsic parameters and the calibration data set, and finally the positional relationship between other screens in the system and the reference screen can be determined by using the camera pose set corresponding to each screen. Compared with manual measurement or 3D scanning, there is no need to introduce additional expensive hardware equipment. Only the existing screens and cameras in the virtual shooting system need to be used to quickly determine the positional relationship between the screens in the system with high precision. In addition, the time required to determine the positional relationship between the screens is short, the tolerance for manual operation errors is high, the operation is simple, the calculation speed is faster, the labor cost and hardware cost are lower, the accuracy is higher, it is not limited by the screen position in the real world, and the application is wider.

[0019] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0021] Figure 1 A schematic diagram of a virtual shooting system according to an embodiment of the present disclosure is shown.

[0022] FIG. 2( a ) shows a schematic diagram of three LED screens in the real world according to an embodiment of the present disclosure.

[0023] FIG. 2( b ) shows a schematic diagram of a screen model according to an embodiment of the present disclosure.

[0024] Figure 3 A flow chart of a method for determining a position relationship between screens according to an embodiment of the present disclosure is shown.

[0025] FIG. 4( a ) shows a schematic diagram of a checkerboard calibration plate according to an embodiment of the present disclosure.

[0026] FIG4( b ) shows a schematic diagram of a grid point calibration plate according to an embodiment of the present disclosure.

[0027] Figure 5 A schematic diagram showing a calibration plate image displayed on a screen according to an embodiment of the present disclosure.

[0028] 6( a ) and 6 ( b ) are schematic diagrams showing screen images according to an embodiment of the present disclosure.

[0029] Figure 7 A schematic diagram showing a feature point numbering result according to an embodiment of the present disclosure.

[0030] Figure 8 A block diagram of a device for determining a positional relationship between screens according to an embodiment of the present disclosure is shown.

[0031] Fig. 9 A block diagram of an electronic device 1900 according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0032] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0033] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0034] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0035] As mentioned above, currently, 3D scanning and manual measurement are generally used to obtain the positional relationship between screens. The accuracy of the manual measurement method is unstable and may not meet the accuracy requirements of virtual shooting. Although the 3D scanning method has high accuracy, the hardware cost is also high, and both methods require a long time cost for manual labor. In view of this, the embodiment of the present disclosure provides a method for determining the positional relationship between screens, which can be applied to a virtual shooting system. It can use a common camera in a virtual shooting system to capture multiple screen images of the screen, and then use the multiple screen images captured by the camera to quickly and automatically calculate the precise positional relationship between multiple screens in the virtual shooting system, which can reduce labor costs and hardware costs, and is conducive to improving the construction efficiency and restoration accuracy of the screen model.

[0036] Figure 1 A schematic diagram of a virtual shooting system according to an embodiment of the present disclosure is shown. Figure 1As shown, the virtual shooting system includes a camera 01 for shooting, three screens (021, 022, 023) for displaying virtual scenes, and a control device 03. The control device 03 is respectively connected to the camera 01 and the three screens (021, 022, 023). It should be understood that the embodiment of the present disclosure does not limit the communication connection method between devices.

[0037] Among them, camera 01 is a camera used to capture screen images, and the embodiment of the present disclosure does not limit the type of camera; the screen used in the virtual shooting system can be an LED display screen, a liquid crystal display screen, etc., and can be a curved screen or a flat screen. It should be understood that technical personnel in this field can customize the type, quantity, size, resolution, etc. of the screen used in the virtual shooting system according to actual needs, and the embodiment of the present disclosure does not limit this.

[0038] Among them, the control device 03 can be an electronic device with computing and processing control capabilities, such as a desktop computer, a laptop computer, etc. During the virtual shooting process, a virtual engine can be deployed in the control device 03, so that a virtual scene, a virtual camera and a screen model can be constructed in the virtual engine of the control device 03, and multiple screens (021, 022, 023) can be controlled to display virtual scenes, and the shooting posture of the camera 01 can be controlled, the video data shot by the camera 01 can be received and processed, and the video data can be post-produced, etc.; before constructing the screen model, the control device 03 can also be used to execute the method for determining the position relationship between screens of the embodiment of the present disclosure to obtain the position relationship between each screen in the virtual shooting system, so as to construct a virtual screen model using the position relationship between each screen and the screen information (such as shape, size, etc.) of each screen. Exemplarily, three LED screens in the real world shown in FIG2 (a) can construct the screen model shown in FIG2 (b).

[0039] It should be understood that the method for determining the positional relationship between screens provided in the embodiment of the present disclosure can also be executed on other electronic devices, that is, other electronic devices can be connected and controlled by the above-mentioned multiple screens (021, 022, 023) and the camera 01 to execute the method for determining the positional relationship between screens in the embodiment of the present disclosure. The embodiment of the present disclosure does not limit the execution subject of the method for determining the positional relationship between screens. Among them, the electronic device may include a terminal device or a server.

[0040] The method for determining the position relationship between screens in the embodiment of the present disclosure can be deployed on various terminal devices through software or hardware modification. The terminal device involved in the embodiment of the present disclosure may refer to a device with a wireless connection function and / or a wired connection function. The wireless connection function refers to the ability to connect to other devices (such as the above-mentioned camera, each screen, etc.) through wireless connection methods such as wifi and Bluetooth. The terminal device involved in the embodiment of the present disclosure can also communicate with other devices through a wired connection function. The terminal device involved in the embodiment of the present disclosure can be a touch screen, a non-touch screen, or a screen-free terminal device. The touch screen can be controlled by clicking and sliding on the display screen with a finger or a stylus. The non-touch screen device can be connected to an input device such as a mouse, a keyboard, a touch panel, and the terminal device is controlled through the input device. For example, the device without a screen can be a Bluetooth speaker without a screen. For example, the terminal device of the present application may include but is not limited to user equipment (UE), mobile devices, user terminals, terminals, handheld devices, tablet computers, laptops, PDAs, computing devices, etc.

[0041] The method for determining the positional relationship between screens in the embodiment of the present disclosure can also be deployed on a server, which can be located in the cloud or locally, and can be a physical device or a virtual device, such as a virtual machine, a container, etc., and has a wireless communication function, wherein the wireless communication function can be set in the chip (system) or other parts or components of the server. It can refer to a device with a wireless connection function, and the wireless connection function means that it can be connected to other servers or terminal devices through wireless connection methods such as Wi-Fi and Bluetooth. The server involved in the embodiment of the present disclosure can also have a function of communicating through a wired connection. For example, the server involved in the embodiment of the present disclosure communicates with the control device 03 in the above-mentioned virtual shooting system, receives multiple screen images taken by the camera 01 sent by the control device 03, and executes the method for determining the positional relationship between screens in the embodiment of the present disclosure based on the multiple screen images, determines the positional relationship between the three screens (021, 022, 023) and feeds it back to the control device 03, and the control device 03 can construct the screen model corresponding to the three screens (021, 022, 023) according to the positional relationship between the three screens (021, 022, 023) fed back by the server.

[0042] The following Figures 3 to 7 , a method for determining the position relationship between screens provided in an embodiment of the present disclosure is introduced in detail.

[0043] Figure 3 A flowchart of a method for determining a positional relationship between screens according to an embodiment of the present disclosure is shown. The method can be applied to various virtual shooting systems (such as the above Figure 1The virtual shooting system shown in FIG. 1 includes a camera for shooting (eg, Figure 1 The camera 01 in FIG. 1 and the S screens for displaying the virtual scene (such as Figure 1 The method may be specifically executed by the control device 03 in the virtual shooting system, or by other electronic devices. Figure 3 As shown, the method for determining the position relationship between screens includes:

[0044] Step S31, when S screens have displayed their corresponding calibration plate images, multiple screen images captured by the camera at different positions of the S screens are obtained, and the calibration plate image displayed on each screen includes multiple feature points.

[0045] Among them, the calibration plate image can be an image formed by any calibration plate that meets the camera calibration requirements. For example, a checkerboard calibration plate composed of alternating black and white blocks as shown in Figure 4(a) can be used. The feature point in the checkerboard calibration plate refers to the corner point between two adjacent black blocks in the diagonal direction. A grid point calibration plate composed of evenly arranged dots as shown in Figure 4(b) can also be used. The feature point in the grid point calibration plate can be the center point of the dots. The embodiments of the present disclosure do not limit the type of calibration plate used in the calibration plate image.

[0046] Taking into account that the sizes and resolutions of various screens in the real world are different, a corresponding calibration plate image can be generated for each screen so that the calibration plate image can adapt to different screens, or in other words, the feature points in the calibration plate image can cover the entire screen; illustratively, a configuration interface can be provided to the user to obtain screen information such as the size (length and width) and resolution of each screen entered by the user in the configuration interface, so that the calibration plate image corresponding to each screen can be automatically generated according to the screen information entered by the user.

[0047] As described above, the screen in the virtual shooting system can be an LED screen. It can be known that the LED screen is usually composed of multiple LED boxes, that is, each LED screen includes multiple LED boxes, and the LED box is equivalent to the display unit of the LED screen. Based on this, the method also includes: obtaining screen information of each screen in the S blocks of screens, the screen information includes the number of rows, columns, resolution and box size of the LED boxes in the screen; according to the screen information of each screen, generating a calibration plate image corresponding to each screen, wherein multiple feature points in the calibration plate image corresponding to each screen cover each LED box in each screen.

[0048] It should be understood that knowing the number of rows, columns and box size of the LED boxes in each screen is equivalent to knowing the size of the screen, thereby obtaining the size of the calibration plate image; the number of feature points in the calibration plate image can match the number of rows, columns and box size of the LED boxes in each screen. For example, if the box size is small, one feature point can be displayed in each LED box. For example, for an LED screen with 3 rows and 3 columns of LED boxes, its corresponding calibration plate image can contain 9 feature points in 3 rows and 3 columns. If the box size is large, Each LED box can display multiple feature points. For example, for an LED screen with 3 rows and 3 columns of LED boxes, its corresponding calibration plate image can also contain 6 rows and 6 columns with a total of 36 feature points, that is, each LED box can display 2 rows and 2 columns with a total of 4 feature points. In this way, the multiple feature points in the calibration plate image corresponding to each screen can cover each LED box in each screen; wherein, the resolution of the screen in the screen information can be used to indicate the resolution that the calibration plate image should have, for example, the resolution of the calibration plate image can be the same as the resolution of the screen. It can be known that the screen is usually rectangular, so the calibration plate image can be generated by assuming that the screen is rectangular. If the screen is of other shapes, the screen information can also include the actual shape of the screen to generate the calibration plate image in combination with the actual shape.

[0049] As described above, the screen can be a flat screen or a curved screen. For a flat screen, the screen information of each screen input by the user can be obtained through the configuration interface, and then the corresponding calibration plate image can be generated according to the screen information input by the user; if the screen is a curved screen, the screen information can be obtained by importing the screen model of the curved screen. At this time, the screen information can also include information such as the curvature of the curved screen to generate a calibration plate image adapted to the curved screen.

[0050] After generating the calibration plate images corresponding to the respective screens according to the screen information of the respective screens, the respective screens can be controlled to display the calibration plate images corresponding to the respective screens. For example, Figure 5 What is shown may be a schematic diagram of three screens displaying the calibration plate images respectively, such as Figure 5 As shown, the three screens display the grid point calibration plate respectively, and the grid point calibration plate displayed on each screen contains 3 rows and 3 columns of feature points (i.e., the black dots in the figure). Then the camera can be placed or moved to different positions, and the images of each screen that has displayed the calibration plate image can be captured at different positions. It should be understood that at least one screen image can be captured at each position, and multiple screen images can be captured at multiple positions. Each screen image can include at least two screens out of the S screens. For example, based on Figure 56(a) shows a captured screen image including screen 021 and screen 022, and FIG6(b) shows a captured screen image including screen 022 and 023. It should be understood that the number of captured screen images may depend on the number of screens in the system and the number of postures during shooting, and the embodiment of the present disclosure does not limit the number of captured screen images.

[0051] In practical applications, after generating the calibration plate images corresponding to each screen, each feature point in the calibration plate image of each screen can also be numbered, that is, a unique number is set for each feature point in the calibration plate image corresponding to each screen, and the feature points in the calibration plate images corresponding to different screens have different numbers. For example, by Figure 5 By numbering each feature point in the calibration plate image displayed on the three screens shown, we can get Figure 7 A feature point numbering result is shown as Figure 7 As shown, the feature points numbered 1-9 belong to the left screen 021, the feature points numbered 10-18 belong to the right screen 022, and the feature points numbered 19-27 belong to the bottom screen 023. After completing the numbering of each feature point in the calibration plate image corresponding to each screen, the number of the feature point can also be associated with the three-dimensional coordinates of the feature point when it is displayed on the screen, so as to facilitate the subsequent determination of the feature point set using the number of the feature point and the associated three-dimensional coordinates, wherein the three-dimensional coordinates of the feature point displayed on each screen can be the coordinate value in the space coordinate system established with the center point (or vertex) of each screen as the origin.

[0052] Step S32, determining the calibration data sets corresponding to each of the S screens based on the multiple screen images, wherein the calibration data set corresponding to the s-th screen includes: a set of feature points consisting of feature points belonging to the s-th screen detected from at least one first screen image, the first screen image being a screen image in which the s-th screen appears among the multiple screen images, s∈[1,S].

[0053] It should be understood that the screen image captured by the camera may include part or all of the S screens. For the s-th screen, the calibration data set corresponding to the s-th screen can be determined based on at least one screen image in which the s-th screen appears. For example, feature point detection can be performed on each screen image to obtain the feature points belonging to the s-th screen in each first screen image, and then the two-dimensional coordinates and three-dimensional coordinates of the feature points belonging to the s-th screen in each first screen image constitute a feature point set belonging to the s-th screen in each first screen image.

[0054] As described above, the multiple feature points in the calibration plate image displayed on each screen are respectively provided with unique numbers, and the feature points in the calibration plate images corresponding to different screens have different numbers, and the numbers of the feature points are associated with the three-dimensional coordinates of the feature points when displayed on the screen. Therefore, in a possible implementation, the above-mentioned determination of the calibration data sets corresponding to each of the S screens based on the multiple screen images includes:

[0055] Step S321, by performing feature point detection on a plurality of screen images respectively, determining the feature points detected in each screen image and the feature point information of the detected feature points, wherein the feature point information includes the number of the feature point and the two-dimensional coordinates of the feature point in the screen image;

[0056] Step S322, for the s-th screen, according to the number of the feature points detected in each screen image, determine N first screen images where the s-th screen appears and feature point information belonging to the s-th screen in each first screen image from the multiple screen images, N≥1;

[0057] Step S323, based on the feature point information belonging to the s-th screen in each first screen image, determine a feature point set constituted by the feature points belonging to the s-th screen in each first screen image, wherein the feature point set constituted by the feature points belonging to the s-th screen in the n-th first screen image includes: the two-dimensional coordinates of each feature point belonging to the s-th screen in the n-th first screen image and the three-dimensional coordinates of each feature point belonging to the s-th screen in the n-th first screen image when displayed in the s-th screen, wherein the two-dimensional coordinates of the feature points in the same first screen image are associated with the three-dimensional coordinates, n∈[1,N].

[0058] In step S321, according to the type of calibration plate used in the calibration plate image, a feature point detection technology known in the art can be selected to perform feature point detection on multiple screen images respectively, and obtain the two-dimensional coordinates of the feature points in each screen image and the number of the feature points (that is, the point sequence positioning). For example, for the checkerboard calibration plate, a corner point detection algorithm known in the art can be selected, and for the grid point calibration plate, a circle point detection algorithm known in the art can be selected. It should be understood that the embodiment of the present disclosure does not limit the feature point detection method, as long as the feature points in the screen image and the feature point information of the feature points can be detected.

[0059] In step S322, since the number of feature points in the calibration plate image corresponding to different screens is different and the number of feature points displayed in the s-th screen is known, it is possible to determine at least one first screen image (that is, N first screen images) where the s-th screen appears based on the number of feature points detected in each screen image, and at the same time, the feature point information belonging to the s-th screen in each first screen image where the s-th screen appears is obtained.

[0060] In step S323, since the number of the feature point is associated with the three-dimensional coordinates of the feature point when it is displayed on the screen, and the two-dimensional coordinates of the feature point in the screen image actually correspond to the number of the feature point, the two-dimensional coordinates and three-dimensional coordinates of each feature point belonging to the s-th screen in each first screen image can be determined based on the feature point information, and then a feature point set consisting of the feature points belonging to the s-th screen in each first screen image can be obtained.

[0061] For example, assuming that the feature points belonging to the sth screen in the nth first screen image are feature points numbered f1, f2, ..., fi, the coordinate pairs corresponding to the i feature points (i.e., two-dimensional coordinates and three-dimensional coordinates) can be expressed as (f1_l2, f1_l3), (f2_l2, f2_l3), ..., (fi_l2, fi_l3), respectively, where l2 represents the two-dimensional coordinates, l3 represents the three-dimensional coordinates, fi_l2 represents the two-dimensional coordinates of the feature point fi, fi_l3 represents the three-dimensional coordinates of the feature point fi, and the other coordinate pairs are similar. Then the feature point set Fn composed of the feature points belonging to the sth screen in the nth first screen image can be expressed as Fn = {(f1_l2, f1_l3), (f2_l2, f2_l3), ..., (fi_l2, fi_l3)}, and Fn can be understood as the feature point set composed of the feature points on the sth screen detected from the nth first screen image. The calibration data set corresponding to each screen can be determined quickly and efficiently through the above method.

[0062] It should be understood that a feature point set can be obtained for each first screen image, and thus, the calibration data set D corresponding to the s-th screen s It can be expressed as D s ={F1,F2,…,F N}, s∈[1,S]. That is to say, after detecting the feature point information in each screen image, all the feature points detected in multiple screen images can be split and grouped according to the screens to which the feature points belong. Each screen will get a calibration data set composed of the feature point sets belonging to each screen in different screen images. Then the calibration data sets corresponding to each of the S screens can be expressed as {D1,D2,…,D S}.

[0063] Step S33, determining the camera intrinsic parameters of the camera and the reference screen in the S blocks of screens according to the calibration data sets corresponding to the S blocks of screens.

[0064] After obtaining the calibration data set corresponding to each screen through the above step S32, the calibration data set corresponding to each screen can be used to calculate the camera intrinsic parameters respectively, wherein the camera calibration algorithm used in any known camera calibration technology in the art, such as the Zhang Zhengyou calibration method, can be used to calibrate the camera intrinsic parameters, that is, calculate the camera intrinsic parameters of the camera. The embodiment of the present disclosure does not limit the calibration process of the camera intrinsic parameters, wherein the camera intrinsic parameters may include parameters such as optical center, focal length, radial distortion, tangential distortion, etc. Among them, the calibration data set D corresponding to the s-th screen s The camera internal parameters C of the sth group can be calibrated s , then the S calibration data sets corresponding to the S screen blocks can be calibrated to obtain the S groups of camera internal parameters {C1, C2, …, C S Each set of camera intrinsic parameters corresponds to a calibration error value, which is used to characterize the error between the calibrated camera intrinsic parameters and the actual camera intrinsic parameters of the camera. The smaller the calibration error value, the higher the precision and accuracy of the camera intrinsic parameters. Therefore, a set of camera intrinsic parameters with the smallest calibration error value can be selected as the camera intrinsic parameters used by the camera, and the screen corresponding to the camera intrinsic parameters with the smallest calibration error value can be used as the reference screen.

[0065] Based on this, in a possible implementation, the camera intrinsic parameters of the camera and the reference screen in the S blocks of screens are determined according to the calibration data sets corresponding to the S blocks of screens, including: using the calibration data sets corresponding to the S blocks of screens to calibrate the camera intrinsic parameters respectively, obtain the calibrated S groups of camera intrinsic parameters, and determine the calibration error values ​​corresponding to each group of camera intrinsic parameters in the S groups of camera intrinsic parameters, wherein one screen corresponds to a group of camera intrinsic parameters, and the calibration error value is used to characterize the error between the calibrated camera intrinsic parameters and the actual camera intrinsic parameters of the camera; the camera intrinsic parameters corresponding to the minimum calibration error value are determined as the camera intrinsic parameters of the camera, and the screen corresponding to the camera intrinsic parameters corresponding to the minimum calibration error value is determined as the reference screen. In this way, more accurate camera intrinsic parameters and more precise reference screens can be effectively obtained, which is conducive to the subsequent use of the high-precision camera intrinsic parameters and reference screens to obtain a more accurate positional relationship between screens.

[0066] As described above, the Zhang Zhengyou calibration method can be used to calculate the camera intrinsic parameters. The camera parameters include the camera intrinsic parameters and the camera extrinsic parameters. The camera extrinsic parameters are also the camera pose. For example, based on the camera imaging principle in the Zhang Zhengyou calibration method, the projection transformation relationship between the pixel coordinate system of the camera and the world coordinate system can be expressed as formula (1).

[0067]

[0068] Among them, (X, Y, Z) represents the three-dimensional coordinates in the world coordinate system. represents the camera intrinsic parameter matrix, f x and f y represents the focal length, c x and c y Represents the heart of light, Represents the camera extrinsic matrix, R represents the rotation matrix (that is, the posture of the camera pose), t represents the translation matrix (that is, the position in the camera pose), (u, v) represents the two-dimensional coordinates in the pixel coordinate system, and α represents the scaling factor.

[0069] It can be known that, based on the Zhang Zhengyou calibration method, the camera parameters can be solved by using an optimization method (such as the least square method, the maximum likelihood estimation algorithm, etc.) using the calibration data set corresponding to the s-th screen combined with the above formula (1) to solve the optimal solution of the camera intrinsic parameters and camera extrinsic parameters. In other words, the above formula (1) can be understood as a loss function in the optimization method. The two-dimensional coordinates and three-dimensional coordinates of the feature points in the calibration data set can be used to iteratively find the optimal camera intrinsic parameters and camera extrinsic parameters, so that the error of the calculated loss function is minimized. Among them, since the calibration data set corresponding to each screen includes a feature point set consisting of feature points belonging to each screen in at least one screen image appearing on each screen, and the camera pose when the camera captures each screen image can be different, a set of calibration data sets corresponding to a screen is calculated to obtain a set of camera intrinsic parameters and at least one set of camera extrinsic parameters.

[0070] In a possible implementation, the determination of the calibration error value corresponding to each camera intrinsic parameter in the S groups of camera intrinsic parameters may include: for the sth group of camera intrinsic parameters, according to the sth group of camera intrinsic parameters, determining the projection transformation relationship corresponding to the sth group of camera intrinsic parameters, the projection transformation relationship representing the coordinate transformation relationship between the three-dimensional coordinates and the two-dimensional coordinates of the feature points; according to the projection transformation relationship corresponding to the sth group of camera intrinsic parameters, converting the three-dimensional coordinates of the feature points in the calibration data set corresponding to the sth group of camera intrinsic parameters into two-dimensional projection coordinates; according to the deviation between the two-dimensional projection coordinates of the feature points in the calibration data set corresponding to the sth group of camera intrinsic parameters and the actual two-dimensional coordinates of the feature points in the screen image, determining the calibration error value corresponding to the sth group of camera intrinsic parameters. It should be understood that a calibration error value can be calculated for each camera intrinsic parameter in the S groups of camera intrinsic parameters. This method is equivalent to reprojecting the three-dimensional coordinates into two-dimensional projection coordinates using the calibrated camera intrinsic parameters, so that the deviation between the reprojected two-dimensional projection coordinates and the actual two-dimensional coordinates can be used to accurately reflect the calibration error of the camera intrinsic parameters.

[0071] As described above, the camera extrinsic parameters are obtained simultaneously during the process of calibrating the camera intrinsic parameters. Then, based on the projection transformation relationship shown in the above formula (1), the known camera intrinsic parameters and the camera extrinsic parameters are equivalent to the coordinate transformation relationship between the known three-dimensional coordinates and the two-dimensional coordinates, that is, the projection transformation relationship corresponding to the s-th group of camera intrinsic parameters can be expressed in the form of formula (1). Since a group of calibration data sets may determine at least one group of camera extrinsic parameters, the s-th group of camera intrinsic parameters may correspond to at least one projection transformation relationship. Then, based on the at least one group of projection transformation relationships, the three-dimensional coordinates of each feature point in the calibration data set corresponding to the s-th group of camera intrinsic parameters can be converted into two-dimensional projection coordinates respectively. For example, the three-dimensional coordinates of the feature points in any screen image can be substituted into the projection transformation relationship determined by a group of camera extrinsic parameters corresponding to the screen image to obtain the two-dimensional projection coordinates of the feature points in the screen image. It can be seen that, in theory, the two-dimensional projection coordinates of the feature point should be the same as the actual two-dimensional coordinates of the feature point in the screen image. However, due to errors in the camera parameter calibration process, there is also a deviation between the two-dimensional projection coordinates of the feature point determined by the calibrated camera parameters and the actual two-dimensional coordinates of the feature point in the screen image. Therefore, the deviation between the two-dimensional projection coordinates of the feature point and the actual two-dimensional coordinates can be used to determine the calibration error of the camera intrinsic parameters, that is, the error between the calibrated camera intrinsic parameters and the actual camera intrinsic parameters of the camera.

[0072] Optionally, the deviation between the two-dimensional projection coordinates of the feature point and the actual two-dimensional coordinates can be expressed as the distance between the two-dimensional projection coordinates and the actual two-dimensional coordinates. It is understandable that, since the calibration data set corresponding to the s-th group of camera intrinsic parameters contains a feature point set consisting of feature points belonging to the s-th screen in at least one screen image, and the feature point set of each screen image contains multiple feature points belonging to the s-th screen, the calibration error value corresponding to the s-th group of camera intrinsic parameters can be obtained by calculating the average value, variance value, etc. of the distance between the two-dimensional projection coordinates of all or part of the feature points in the calibration data set corresponding to the s-th screen and the actual two-dimensional coordinates, and the embodiments of the present disclosure are not limited to this.

[0073] Among them, the camera intrinsic parameter corresponding to the minimum calibration error value is determined as the camera intrinsic parameter of the camera, and the screen corresponding to the camera intrinsic parameter corresponding to the minimum calibration error value is determined as the reference screen. It can be understood that the camera intrinsic parameter with the smallest calibration error value among the S calibration error values ​​corresponding to the S groups of camera intrinsic parameters is used as the camera intrinsic parameter used by the camera, and the screen corresponding to the camera intrinsic parameter with the smallest calibration error value is determined as the reference screen.

[0074] Step S34, determining the camera pose sets corresponding to the S screens according to the camera intrinsic parameters and the calibration data sets corresponding to the S screens, wherein the camera pose set corresponding to the s-th screen includes: at least one group of camera poses determined based on a set of feature points respectively belonging to the s-th screen in at least one first screen image, and one group of camera poses corresponding to one first screen image.

[0075] It should be understood that the camera intrinsic parameters of the camera used in step S34 are the camera intrinsic parameters corresponding to the minimum calibration error value determined in step S33, wherein the camera pose (i.e., camera extrinsic parameters) can be calculated based on the projection transformation relationship used in the camera calibration shown in the above formula (1) using the camera intrinsic parameters corresponding to the minimum calibration error value and the calibration data sets corresponding to each of the S screens. That is, at this time, the camera extrinsic parameters can be taken as unknown quantities, and the camera intrinsic parameters and the two-dimensional coordinates and three-dimensional coordinates in the calibration data set can be taken as known quantities in combination with the above formula (1) to solve the camera pose.

[0076] As described above, since the calibration data set corresponding to each screen includes a feature point set consisting of feature points belonging to each screen in at least one screen image appearing on each screen, and the camera pose when the camera captures each screen image may be different, a set of calibration data sets can calculate at least one set of camera extrinsics. That is, for the N first screen images appearing on the s-th screen, a set of camera poses corresponding to the n-th first screen image can be determined based on the feature point set belonging to the s-th screen in the n-th first screen image in the calibration data set corresponding to the s-th screen. n , the n It can represent the camera pose calculated by using the feature point set belonging to the s-th screen in the n-th first screen image. It should be understood that a set of camera poses can be calculated for each of the N first screen images appearing on the s-th screen. Then the camera pose set E corresponding to the s-th screen is s It can be expressed as E s ={e1,e2,…,e N}.

[0077] Step S35, determining the positional relationship between other screens in the S blocks of screens and the reference screen according to the camera pose sets corresponding to the S blocks of screens, wherein the other screens include screens in the S blocks of screens except the reference screen.

[0078] As described above, the three-dimensional coordinates of the feature points displayed on each screen can be the coordinate values ​​in the spatial coordinate system established with the center point (or vertex) of each screen as the origin, that is, the three-dimensional coordinates of the feature points displayed on different screens are determined based on different spatial coordinate systems. Then, for at least two screens appearing in the same screen image, the camera poses determined respectively using the feature point sets corresponding to the at least two screens appearing in the same screen image are actually camera poses determined based on different spatial coordinate systems. That is to say, the at least two screens appearing in the same screen image each correspond to a camera pose in its own spatial coordinate system, and theoretically, the camera pose of the same screen image captured by the same camera is the same. Then, the camera poses corresponding to the at least two screens appearing in the same screen image can be used to infer the positional relationship between the at least two screens appearing in the same screen image. In this way, the positional relationship between the screens in the system can be quickly and accurately determined.

[0079] As described above, a screen image captured by a camera includes at least two screens, and thus, the other screens in the S screens include: at least one first other screen that appears in the same screen image as the reference screen. Then, the positional relationship between the first other screen and the reference screen can be determined by using the first other screen image that appears in the same screen image as the reference screen and the camera poses corresponding to the reference screens. Specifically, the above-mentioned determination of the positional relationship between the other screens in the S screens and the reference screen based on the camera pose sets corresponding to the S screens includes:

[0080] For any first other screen, determine the second screen image in which the first other screen and the reference screen appear simultaneously from multiple screen images; determine the positional relationship between the first other screen and the reference screen based on the camera posture determined by the feature point set belonging to the first other screen in the second screen image and the camera posture determined by the feature point set belonging to the reference screen.

[0081] For example, if Figure 5Screen 022 among the three screens is the reference screen, and both of FIG6(a) and FIG6(b) show the second screen images, then screen 021 in the second screen image shown in FIG6(a) is the first other screen, and the positional relationship between screen 021 and screen 022 can be calculated by using the camera poses corresponding to screen 022 and screen 021 in the second screen image shown in FIG6(a), and similarly, the positional relationship between screen 023 and screen 022 can be calculated by using the camera poses corresponding to screen 022 and screen 023 in the second screen image shown in FIG6(b), thereby obtaining the positional relationship between the other screens 021 and 023 among the three screens and the reference screen 022. It should be understood that at least one first other screen may appear in the same screen image, and the positional relationship between each first other screen and the reference screen can be calculated separately.

[0082] As described above, the feature points displayed in different screens are numbered differently. The second screen image in which the reference screen appears can be determined from multiple screen images based on the number of feature points displayed in the reference screen. The other screens appearing in the second screen image are also the first other screens. Furthermore, for any first other screen, the positional relationship between the first other screen and the reference screen can be determined based on the camera pose determined by the feature point set in the second screen image belonging to the first other screen and the camera pose determined by the feature point set belonging to the reference screen.

[0083] For example, assuming that the reference screen and a first other screen appear simultaneously in the second screen image, the number is recorded as m, and the camera pose determined by the feature point set belonging to the reference screen in the second screen image is recorded as e. m0 The camera pose determined by the feature point set belonging to the first other screen in the second screen image is recorded as e m1 , then the camera pose pair from the first other screen to the reference screen can be expressed as {e m0 ,e m1}, according to this {e m0 ,e m1}The coordinate system transformation relationship from the spatial coordinate system of the first other screen to the spatial coordinate system of the reference screen can be obtained, which is equivalent to obtaining the positional relationship between the first other screen and the reference screen.

[0084] Specifically, let e m1 = {q m1 ,t m1},e m0 = {q m0 ,t m0}, where q m1 and q m0 is a quaternion representing the rotation in the camera pose, with q m1 ={ωm1 ,x m1 ,y m1 ,z m1)}, q m0 ={ω m0 ,x m0 ,y m0 ,z m0)},ω m0 and ω m0 Represent the camera pose e m1 and e m0 The attitude angle in t m1 and t m0 is a three-dimensional vector representing the position in the camera pose, with t m1 ={x m1 ,y m1 ,z m1}、t m0 ={x m0 ,y m0 ,z m0}, x m1 ,y m1 ,z m1 Represents the camera pose e m1 The position coordinates in x m0 ,y m0 ,z m0 Represents the camera pose e m0 The position coordinates in; first calculate q m1 Relative to q m0 The reverse rotation Inv_q m1 =Inv(q m1 )*q m0 ,in Calculate t again m1 Relative to t m0 Inverse displacement Inv_t m1 =Inv_q m1 *(t m0 -t m1 ), then the coordinate system transformation relationship from the spatial coordinate system of the first other screen to the spatial coordinate system of the reference screen can be expressed as e m1-m0 ={Inv_q m1 ,Inv_t m1}, thus, for each spatial position of the first other screen, it can be obtained by e m1-m0 Transformed to the spatial coordinate system corresponding to the reference screen, for example, assuming that the spatial position of the screen vertex (for example, any vertex among the four vertices of the rectangular screen) in the first other screen is v1 = {x, y, z}, then the position of the spatial position v1 transformed to the spatial coordinate system of the reference screen is v trans1_0 =Inv_q m1*v1+Inv_t m1 , that is, the positional relationship between the first other screen and the reference screen can be expressed as v trans1_0 =Inv_q m1 *v1+Inv_t m1 .

[0085] It should be understood that, for any first other screen, if the first other screen and the reference screen appear simultaneously in only one second screen image, the positional relationship between the first other screen and the reference screen is determined directly according to the camera pose determined by the feature point set belonging to the first other screen in the one second screen image and the camera pose determined by the feature point set belonging to the reference screen; if the first other screen and the reference screen appear simultaneously in multiple second screen images, a screen image can be selected from the multiple second screen images to determine the positional relationship between the first other screen and the reference screen based on the selected screen image. Specifically, the determining the positional relationship between the first other screen and the reference screen according to the camera pose determined by the feature point set belonging to the first other screen in the second screen image and the camera pose determined by the feature point set belonging to the reference screen includes:

[0086] Determine a reprojection error of the feature points displayed in the reference screen projected into each second screen image according to the camera pose determined by the feature point set belonging to the reference screen in each second screen image and the camera intrinsic parameters of the camera;

[0087] The positional relationship between the first other screen and the reference screen is determined according to the camera pose determined by the feature point set belonging to the first other screen in the second screen image corresponding to the minimum reprojection error and the camera pose determined by the feature point set belonging to the reference screen.

[0088] Among them, the above-mentioned method for determining the calibration error value can be referred to to realize the camera posture determined by the feature point set belonging to the reference screen in each second screen image and the camera intrinsic parameters of the camera, and determine the reprojection error of the feature points displayed in the reference screen projected into each second screen image. Specifically, when the camera posture corresponding to each second screen is known, that is, when the camera extrinsic parameters are known, the projection transformation relationship between the pixel coordinate system and the world coordinate system as shown in formula (1) can be constructed based on the camera intrinsic parameters and the camera extrinsic parameters. Then, the three-dimensional coordinates of the feature points belonging to the reference screen in each second screen image can be substituted into the above-mentioned projection transformation relationship to obtain the two-dimensional projection coordinates of the three-dimensional coordinates of the feature points projected into the second screen image. Then, the reprojection error can be determined according to the deviation between the two-dimensional projection coordinates of the feature points in the reference screen and the actual two-dimensional coordinates of the feature points in the second screen image. It should be understood that one reprojection error can be determined corresponding to one second screen image.

[0089] After obtaining the reprojection errors corresponding to the multiple second screens, a second screen image with the smallest reprojection error can be selected to determine the positional relationship between the first other screen and the reference screen, that is, the positional relationship between the first other screen and the reference screen is determined based on the camera pose determined by the feature point set belonging to the first other screen in the second screen image corresponding to the smallest reprojection error and the camera pose determined by the feature point set belonging to the reference screen. In this way, when there are multiple second screen images, a more accurate positional relationship between the first other screen and the reference screen is determined using the second screen image with the smallest reprojection error.

[0090] It is understandable that, in addition to the first other screen that appears in the same screen image as the reference screen, there may also be other screens that do not appear in the same screen image at the same time as the reference screen in the S blocks of screens in the system, that is, the other screens in the S blocks of screens may also include: at least one second other screen that does not appear in the same screen image at the same time as the reference screen. For this type of second other screen, the positional relationship between the second other screen and the first other screen can be determined first, and then the positional relationship between the second other screen and the reference screen can be determined based on the positional relationship between the first other screen and the reference screen. Therefore, in one possible implementation, the positional relationship between the other screens in the S blocks of screens and the reference screen is determined based on the camera pose sets corresponding to each of the S blocks of screens, including:

[0091] For any second other screen, determine a first other screen that appears simultaneously with the second other screen in the same screen image, and determine a third screen image in which the second other screen and the first other screen appear simultaneously from the plurality of screen images;

[0092] Determine a positional relationship between the second other screen and the first other screen according to a camera pose determined by a feature point set belonging to the second other screen and a camera pose determined by a feature point set belonging to the first other screen in the third screen image;

[0093] The positional relationship between the second other screen and the first other screen is determined based on the positional relationship between the second other screen and the first other screen, and the positional relationship between the first other screen and the reference screen.

[0094] For example, if Figure 5If screen 021 among the three screens is the reference screen, then the screen image shown in FIG6(a) is still the second screen image, and FIG6(b) is the third screen image. Screen 022 in FIG6(a) is the first other screen. Then, the positional relationship between screen 022 and screen 021 can be calculated by using the camera poses corresponding to each other in the second screen image shown in FIG6(a). Then, the positional relationship between screen 023 and screen 022 can be calculated according to the camera poses corresponding to each other in the third screen image shown in FIG6(b). Then, the positional relationship between screen 023 and screen 022 can be determined in combination with the positional relationship between screen 022 and screen 021 calculated in FIG6(a). Thus, the positional relationship between other screens 022 and 023 in the three screens and the reference screen 021 can be obtained. It should be understood that at least one second other screen may appear in the same screen image, and the positional relationship between each second other screen and the first other screen may be calculated first, and then the positional relationship between each second other screen and the reference screen may be calculated.

[0095] As described above, the feature points displayed in different screens have different numbers. The first other screen that appears simultaneously with the second other screen in the same screen image and the third screen image in which the second other screen and the first other screen appear simultaneously can be determined based on the number of the feature points displayed in the first other screen. Then, the positional relationship between the second other screen and the first other screen can be determined by referring to the method for determining the positional relationship between the first other screen and the reference screen. The positional relationship between the second other screen and the first other screen can be determined based on the camera pose determined by the feature point set belonging to the second other screen in the third screen image and the camera pose determined by the feature point set belonging to the first other screen. That is, if the image number of the third screen image in which the first other image and the second other image appear simultaneously is represented as m at this time, then the above e m0 At this time, it can represent the camera posture determined by the feature point set belonging to the first other screen in the third screen image, e m1 At this time, it can represent the camera pose determined by the feature point set belonging to the second other screen in the third screen image, then {e m0 ,e m1} can represent the camera position of the second other screen to the first other screen, and then at this time, according to {e m0 ,e m1}Get the coordinate system transformation relationship between the second other screen and the first other screen, that is, get the position relationship between the second other screen and the first other screen.

[0096] It should be understood that the first other screen may be one or more. For any second other screen, if the second other screen only appears in a third screen image simultaneously with one first other screen, the positional relationship between the second other screen and the first other screen can be directly determined based on the camera pose determined by the feature point set belonging to the second other screen in the third screen image and the camera pose determined by the feature point set belonging to the first other screen; if the second other screen and one or more first other screens appear in multiple third screen images simultaneously, that is, a second other screen may appear in multiple third screen images simultaneously with a first other screen, or may appear in multiple third screen images simultaneously with multiple first other screen images, then the implementation method of selecting a second screen image with the smallest reprojection error from multiple second screen images to determine the positional relationship between the first other screen and the reference screen can be referred to, and a third screen image with the smallest reprojection error can be selected from multiple third screen images to determine the positional relationship between the second other screen and a first other screen based on the selected third screen image.

[0097] Specifically, the reprojection error of the feature points displayed in each block of the first other screen projected into each third screen image can be determined according to the camera pose determined by the feature point set belonging to each block of the first other screen in each third screen image and the camera internal parameters of the camera; the positional relationship between the second other screen and the first other screen can be determined according to the camera pose determined by the feature point set belonging to the second other screen in the third screen image corresponding to the minimum reprojection error and the camera pose determined by the feature point set belonging to the first other screen. Among them, the reprojection error of the feature points displayed in each block of the first other screen projected into each third screen image can be determined by referring to the above-mentioned implementation method of determining the reprojection error of the feature points displayed in the reference screen projected into each second screen image, which will not be described in detail here.

[0098] It should be understood that after determining the positional relationship between the second other screen and the first other screen, it is equivalent to determining the coordinate system transformation relationship from the spatial coordinate system of the second other screen to the spatial coordinate system of the first other screen. Then, based on the positional relationship between the second other screen and the first other screen, and the positional relationship between the first other screen and the reference screen, the positional relationship between the second other screen and the reference screen is determined, which is equivalent to first converting each spatial position of the second other screen to the spatial coordinate system of the first other screen, and then converting it to the spatial coordinate system of the reference screen.

[0099] Exemplarily, as described above, the coordinate system transformation relationship of the spatial coordinate system of the first other screen to the spatial coordinate system of the reference screen is expressed as e m1-m0 ={Inv_qm1 ,Inv_t m1}, please refer to the above e m1-m0 The determination method is to calculate the coordinate system transformation relationship e from the spatial coordinate system of the second other screen to the spatial coordinate system of the first other screen. m2-m1 ={Inv_q m2 ,Inv_t m2}, thus, for each spatial position of the second other screen, it can be obtained by e m1-m0 and e m2-m1 Transformed to the spatial coordinate system corresponding to the reference screen, for example, assuming that the spatial position of the screen vertex in the second other screen is represented by v2 = {x, y, z}, then the position of the spatial position v2 transformed to the spatial coordinate system of the first other screen is Then the spatial position v2 is transformed to the position in the spatial coordinate system of the reference screen as That is, the positional relationship between the second other screen and the reference screen can be expressed as

[0100] It should be understood that there may be a third other screen in the S block of screens that does not appear in the same screen image as the reference screen or the first other screen. In this case, the positional relationship between the third other screen and the second other screen may be determined first, and then the positional relationship between the third other screen and the reference screen may be determined in combination with the positional relationship between the second other screen and the reference screen, and so on, until the positional relationship between all other screens in the S block of screens and the reference screen is determined. In order to facilitate the determination of the positional relationship between all other screens in the S block of screens and the reference screen, an embodiment of the present disclosure provides a process for determining the positional relationship between screens, including:

[0101] Step a, applying the camera intrinsic parameters determined in step S33 above, and referring to step S34 above, calculating the camera pose for each feature point set in the calibration data set corresponding to each screen, that is, for each screen, a camera pose set E is obtained. s ={e1,e2,…,e N};

[0102] Step b, establish screen set K, and set the base screen K base Add to the screen set K, then K={K base};

[0103] Step c: Establish a transformation sequence set T, and transform K base The corresponding camera pose set E base Add to the set T, then T={E base};

[0104] Step d, traverse all other screens Kq that are not in the set K. For each other screen Kq, query whether there is a screen Kj in the set K that appears in the same screen image as the other screen Kq. If so, record the image number of the same screen image as m. Then the camera pose pair from screen Kq to screen Kj is {e mj ,e mq},e mj represents the camera pose determined by the set of feature points belonging to the screen Kj, e mq represents the camera pose determined by the feature point set belonging to the screen Kq; then Kq can be added to the above screen set K, and then K={K base , Kq}, and the camera pose pair {e mj ,e mq} is added to the above transformation sequence set T, and now T={E base ,{e mj ,e mq}};

[0105] Step e, repeating the above step d until all screens in the system are added to the screen set K, and obtaining a transformation sequence set T containing the coordinate system transformation relationship between each screen;

[0106] Step f: according to the camera pose pairs between the screens in the transformation sequence set T, with reference to the implementation of the above step S35, calculate the positional relationship between all other screens in the system and the reference screen.

[0107] Through the above process, it is equivalent to first determining the first other screen that appears in the same screen image as the reference screen and the camera pose pair between the two, then determining the second other screen that appears in the same screen image as the first other screen and the camera pose pair between the two, then determining the third other screen that appears in the same screen image as the second other screen and the camera pose pair between the two, and so on, until the camera pose pairs between all the screens are determined, and then determining the positional relationship between all other screens and the reference screen.

[0108] According to the embodiments of the present disclosure, the calibration plate image displayed on the screen in the virtual shooting system and multiple screen images shot by a common camera can be used, and then the camera intrinsic parameters and the reference screen used by the camera can be determined according to the calibration data sets corresponding to each screen determined by the multiple screen images, and then the camera pose set of each screen can be determined by using the camera intrinsic parameters and the calibration data set. Finally, the positional relationship between other screens in the system and the reference screen can be determined by using the camera pose set corresponding to each screen. Compared with manual measurement or 3D scanning, there is no need to introduce additional expensive hardware equipment. Only the existing screens and cameras in the virtual shooting system need to be used to quickly determine the positional relationship between the screens in the system with high precision. In addition, the time required to determine the positional relationship between the screens is short, the tolerance for manual operation errors is high, the operation is simple, the calculation speed is faster, the labor cost and hardware cost are lower, the accuracy is higher, it is not limited by the screen position in the real world, and the application is wider.

[0109] According to the embodiments of the present disclosure, by utilizing the camera equipment necessary for the shooting scene to capture several screen images, the precise positional relationship between the screens and the screen model can be automatically output through an algorithm, which can greatly improve the efficiency and effect of the virtual shooting process.

[0110] According to the method for determining the positional relationship between screens in the embodiment of the present disclosure, the positional relationship between each screen in the system can be accurately calculated within minutes, with high speed and low cost. It is beneficial to accurately calculate the positional relationship between each other screen and the reference screen after completing the modeling of the reference screen, thereby improving the overall restoration accuracy of the screen model and significantly reducing labor costs.

[0111] The method for determining the positional relationship between screens according to the embodiment of the present disclosure is a method in the art for automatically measuring the positional relationship between screens in a virtual shooting system through an algorithm. Compared with conventional manual measurement or 3D scanning, it has advantages in cost and accuracy. In the virtual shooting process, any calibration data that meets the camera calibration requirements can be used, that is, the calibration data can be shared with the camera calibration task, and there is no need to add additional manual operations to obtain the calibration data, thereby minimizing labor costs and improving processing efficiency. In terms of algorithm design, the method of inversely calculating the positional relationship between screens through camera posture is adopted, which has higher accuracy and faster calculation speed.

[0112] Figure 8 A block diagram of a device for determining a positional relationship between screens according to an embodiment of the present disclosure is shown. The device can be applied to a virtual shooting system, which includes a camera for shooting and S screens for displaying a virtual scene, where S≥2. Figure 8 As shown, the device comprises:

[0113] The image acquisition module 801 is used to acquire a plurality of screen images captured by the camera at different positions of the S screens when the S screens have displayed their respective corresponding calibration plate images, wherein the calibration plate image displayed on each screen includes a plurality of feature points;

[0114] The calibration data set determination module 802 is used to determine the calibration data sets corresponding to the S screen blocks according to the multiple screen images, wherein the calibration data set corresponding to the s-th screen block includes: a feature point set consisting of feature points belonging to the s-th screen block detected from at least one first screen image, the first screen image being a screen image in which the s-th screen block appears among the multiple screen images, s∈[1,S];

[0115] A reference screen determination module 803 is used to determine the camera intrinsic parameters of the camera and the reference screen in the S blocks of screens according to the calibration data sets corresponding to the S blocks of screens;

[0116] The camera pose set determination module 804 is used to determine the camera pose sets corresponding to the S screens according to the camera intrinsic parameters of the camera and the calibration data sets corresponding to the S screens, wherein the camera pose set corresponding to the s-th screen includes: at least one group of camera poses determined based on the feature point sets respectively belonging to the s-th screen in the at least one first screen image, and one group of camera poses corresponding to one first screen image;

[0117] The position relationship determination module 805 is used to determine the position relationship between other screens in the S blocks of screens and the reference screen according to the camera pose sets corresponding to the S blocks of screens, wherein the other screens include screens in the S blocks of screens except the reference screen.

[0118] In a possible implementation, a plurality of feature points in the calibration plate image displayed on each screen are respectively provided with unique numbers, and the feature points in the calibration plate images corresponding to different screens have different numbers, and the number of the feature point is associated with the three-dimensional coordinates of the feature point when displayed on the screen, wherein determining the calibration data set corresponding to each of the S screens according to the multiple screen images includes: determining the feature points detected in each screen image and the feature point information of the detected feature points by performing feature point detection on the multiple screen images respectively, wherein the feature point information includes the number of the feature point and the two-dimensional coordinates of the feature point in the screen image; for the sth screen, determining from the multiple screen images the location where the sth screen appears according to the number of the feature points detected in each screen image N first screen images and feature point information belonging to the s-th screen in each first screen image, N≥1; according to the feature point information belonging to the s-th screen in each first screen image, determine a feature point set constituted by the feature points belonging to the s-th screen in each first screen image, wherein the feature point set constituted by the feature points belonging to the s-th screen in the n-th first screen image includes: the two-dimensional coordinates of each feature point belonging to the s-th screen in the n-th first screen image and the three-dimensional coordinates of each feature point belonging to the s-th screen in the n-th first screen image when displayed in the s-th screen, wherein the two-dimensional coordinates of the feature points in the same first screen image are associated with the three-dimensional coordinates, n∈[1,N].

[0119] In a possible implementation, determining the camera intrinsic parameters of the camera and a reference screen in the S blocks of screens according to the calibration data sets corresponding to each of the S blocks of screens includes: calibrating the camera intrinsic parameters respectively using the calibration data sets corresponding to each of the S blocks of screens to obtain S groups of calibrated camera intrinsic parameters, and determining a calibration error value corresponding to each group of camera intrinsic parameters in the S groups of camera intrinsic parameters, wherein one screen corresponds to a group of camera intrinsic parameters, and the calibration error value is used to characterize the error between the calibrated camera intrinsic parameters and the actual camera intrinsic parameters of the camera; determining the camera intrinsic parameters corresponding to the minimum calibration error value as the camera intrinsic parameters of the camera, and determining the screen corresponding to the camera intrinsic parameters corresponding to the minimum calibration error value as the reference screen.

[0120] In a possible implementation manner, determining the calibration error value corresponding to each group of camera intrinsic parameters in the S groups of camera intrinsic parameters includes: for the sth group of camera intrinsic parameters, determining a projection transformation relationship corresponding to the sth group of camera intrinsic parameters according to the sth group of camera intrinsic parameters, wherein the projection transformation relationship represents a coordinate transformation relationship between three-dimensional coordinates and two-dimensional coordinates of feature points; according to the projection transformation relationship corresponding to the sth group of camera intrinsic parameters, converting the three-dimensional coordinates of feature points in a calibration data set corresponding to the sth group of camera intrinsic parameters into two-dimensional projection coordinates; and determining the calibration error value corresponding to the sth group of camera intrinsic parameters according to a deviation between the two-dimensional projection coordinates of the feature points in the calibration data set corresponding to the sth group of camera intrinsic parameters and the actual two-dimensional coordinates of the feature points in the screen image.

[0121] In a possible implementation, the other screens in the S blocks of screens include: at least one first other screen that appears in the same screen image simultaneously with the reference screen; wherein, determining the positional relationship between the other screens in the S blocks of screens and the reference screen based on the camera pose sets corresponding to each of the S blocks of screens includes: for any first other screen, determining from the multiple screen images a second screen image in which the first other screen and the reference screen appear simultaneously; determining the positional relationship between the first other screen and the reference screen based on a camera pose determined by a feature point set belonging to the first other screen in the second screen image and a camera pose determined by a feature point set belonging to the reference screen.

[0122] In a possible implementation, the first other screen and the reference screen appear simultaneously in multiple second screen images, wherein the positional relationship between the first other screen and the reference screen is determined based on the camera pose determined by the feature point set belonging to the first other screen in the second screen image and the camera pose determined by the feature point set belonging to the reference screen, including: determining the reprojection error of the feature points displayed in the reference screen projected into each second screen image based on the camera pose determined by the feature point set belonging to the reference screen in each second screen image and the camera intrinsic parameters of the camera; determining the positional relationship between the first other screen and the reference screen based on the camera pose determined by the feature point set belonging to the first other screen in the second screen image corresponding to the minimum reprojection error and the camera pose determined by the feature point set belonging to the reference screen.

[0123] In a possible implementation, the other screens in the S blocks of screens also include: at least one second other screen that does not appear in the same screen image at the same time as the reference screen; wherein, determining the positional relationship between the other screens in the S blocks of screens and the reference screen based on the camera pose sets corresponding to each of the S blocks of screens includes: for any second other screen, determining the first other screen that appears in the same screen image at the same time as the second other screen, and determining from the multiple screen images a third screen image in which the second other screen and the first other screen appear at the same time; determining the positional relationship between the second other screen and the first other screen based on the camera pose determined by the feature point set belonging to the second other screen in the third screen image and the camera pose determined by the feature point set belonging to the first other screen; determining the positional relationship between the second other screen and the reference screen based on the positional relationship between the second other screen and the first other screen and the positional relationship between the first other screen and the reference screen.

[0124] In one possible implementation, the screen includes an LED screen, each LED screen includes multiple LED boxes, and the device also includes: a screen information acquisition module, used to obtain screen information of each screen in the S screens, the screen information includes the number of rows, columns, resolution and box size of the LED boxes in the screen; an image generation module, used to generate a calibration plate image corresponding to each screen according to the screen information of each screen, wherein multiple feature points in the calibration plate image corresponding to each screen cover each LED box in each screen.

[0125] According to the embodiments of the present disclosure, the calibration plate image displayed on the screen in the virtual shooting system and multiple screen images shot by a common camera can be used, and then the camera intrinsic parameters and the reference screen used by the camera can be determined according to the calibration data sets corresponding to each screen determined by the multiple screen images, and then the camera pose set of each screen can be determined by using the camera intrinsic parameters and the calibration data set. Finally, the positional relationship between other screens in the system and the reference screen can be determined by using the camera pose set corresponding to each screen. Compared with manual measurement or 3D scanning, there is no need to introduce additional expensive hardware equipment. Only the existing screens and cameras in the virtual shooting system need to be used to quickly determine the positional relationship between the screens in the system with high precision. In addition, the time required to determine the positional relationship between the screens is short, the tolerance for manual operation errors is high, the operation is simple, the calculation speed is faster, the labor cost and hardware cost are lower, the accuracy is higher, it is not limited by the screen position in the real world, and the application is wider.

[0126] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0127] The embodiment of the present disclosure also provides a computer-readable storage medium on which computer program instructions are stored, and the computer program instructions implement the above method when executed by a processor. The computer-readable storage medium can be a volatile or non-volatile computer-readable storage medium.

[0128] An embodiment of the present disclosure further proposes an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0129] The embodiments of the present disclosure also provide a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0130] Fig. 9 1 is a block diagram of an electronic device 1900 according to an embodiment of the present disclosure. For example, the electronic device 1900 may be provided as a server or a terminal device. Fig. 9 , the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0131] The electronic device 1900 may also include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output interface 1958 (I / O interface). The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server 2003. TM , Mac OS X TM , Unix TM ,Linux TM , FreeBSD TM or similar.

[0132] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the electronic device 1900 to perform the above method.

[0133] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0134] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0135] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0136] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0137] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0138] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0139] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0140] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0141] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for determining the position relationship between screens, applied to a virtual shooting system, wherein the virtual shooting system comprises a camera for shooting and S screens for displaying a virtual scene, S≥2, characterized in that: The method comprises: When the S screens have displayed their respective corresponding calibration plate images, a plurality of screen images captured by the camera at different positions of the S screens are obtained, wherein the calibration plate image displayed on each screen includes a plurality of feature points; Determine, according to the multiple screen images, calibration data sets corresponding to the S screens, wherein the calibration data set corresponding to the s-th screen comprises: a feature point set consisting of feature points belonging to the s-th screen that are respectively detected from at least one first screen image, the first screen image being a screen image in which the s-th screen appears among the multiple screen images, s∈[1,S]; Determine the camera intrinsic parameters of the camera and a reference screen among the S blocks of screens according to the calibration data sets corresponding to the S blocks of screens; Determine, according to the camera intrinsic parameters of the camera and the calibration data sets corresponding to the S screens, the camera pose sets corresponding to the S screens, wherein the camera pose set corresponding to the s screen includes: at least one group of camera poses determined based on the feature point sets respectively belonging to the s screen in the at least one first screen image, and one group of camera poses corresponding to one first screen image; According to the camera pose sets corresponding to the S blocks of screens, the positional relationship between other screens in the S blocks of screens and the reference screen is determined, wherein the other screens include screens in the S blocks of screens except the reference screen.

2. The method according to claim 1, characterized in that: A plurality of feature points in the calibration plate image displayed on each screen are respectively provided with unique numbers, and the feature points in the calibration plate images corresponding to different screens have different numbers, and the numbers of the feature points are associated with the three-dimensional coordinates of the feature points when displayed on the screen, wherein determining the calibration data sets corresponding to each of the S screens according to the plurality of screen images includes: By performing feature point detection on the multiple screen images respectively, determining the feature points detected in each screen image and feature point information of the detected feature points, the feature point information including the number of the feature point and the two-dimensional coordinates of the feature point in the screen image; For the s-th screen, according to the number of feature points detected in each screen image, determine N first screen images in which the s-th screen appears and feature point information belonging to the s-th screen in each first screen image from the multiple screen images, where N≥1; According to the feature point information belonging to the sth screen in each first screen image, a feature point set constituted by the feature points belonging to the sth screen in each first screen image is determined, wherein the feature point set constituted by the feature points belonging to the sth screen in the nth first screen image includes: the two-dimensional coordinates of each feature point belonging to the sth screen in the nth first screen image in the nth first screen image and the three-dimensional coordinates of each feature point belonging to the sth screen in the nth first screen image when displayed in the sth screen, wherein the two-dimensional coordinates of the feature points in the same first screen image are associated with the three-dimensional coordinates, n∈[1,N].

3. The method according to claim 1, characterized in that The step of determining the camera intrinsic parameters of the camera and the reference screen in the S blocks of screens according to the calibration data sets corresponding to the S blocks of screens respectively includes: Calibrate camera intrinsic parameters respectively using calibration data sets corresponding to the S screens to obtain S groups of calibrated camera intrinsic parameters, and determine calibration error values ​​corresponding to each group of camera intrinsic parameters in the S groups of camera intrinsic parameters, wherein one screen corresponds to one group of camera intrinsic parameters, and the calibration error value is used to characterize the error between the calibrated camera intrinsic parameters and the actual camera intrinsic parameters of the camera; The camera intrinsic parameter corresponding to the minimum calibration error value is determined as the camera intrinsic parameter of the camera, and the screen corresponding to the camera intrinsic parameter corresponding to the minimum calibration error value is determined as the reference screen.

4. The method according to claim 3, characterized in that The determining of the calibration error value corresponding to each group of camera intrinsic parameters in the S groups of camera intrinsic parameters includes: For the sth group of camera intrinsic parameters, determine a projection transformation relationship corresponding to the sth group of camera intrinsic parameters according to the sth group of camera intrinsic parameters, where the projection transformation relationship represents a coordinate transformation relationship between a three-dimensional coordinate and a two-dimensional coordinate of a feature point; According to the projection transformation relationship corresponding to the sth group of camera intrinsic parameters, the three-dimensional coordinates of the feature points in the calibration data set corresponding to the sth group of camera intrinsic parameters are converted into two-dimensional projection coordinates; The calibration error value corresponding to the sth group of camera intrinsic parameters is determined according to the deviation between the two-dimensional projection coordinates of the feature points in the calibration data set corresponding to the sth group of camera intrinsic parameters and the actual two-dimensional coordinates of the feature points in the screen image.

5. The method according to any one of claims 1 to 4, characterized in that: The other screens in the S blocks of screens include: at least one first other screen that appears in the same screen image simultaneously with the reference screen; The step of determining the positional relationship between other screens in the S blocks of screens and the reference screen according to the camera pose sets corresponding to the S blocks of screens includes: For any first other screen, determining, from the plurality of screen images, a second screen image in which the first other screen and the reference screen appear simultaneously; The positional relationship between the first other screen and the reference screen is determined according to the camera pose determined by the feature point set belonging to the first other screen in the second screen image and the camera pose determined by the feature point set belonging to the reference screen.

6. The method according to claim 5, characterized in that The first other screen and the reference screen appear simultaneously in a plurality of second screen images, wherein determining the positional relationship between the first other screen and the reference screen according to the camera pose determined by the feature point set belonging to the first other screen in the second screen image and the camera pose determined by the feature point set belonging to the reference screen comprises: Determine, based on the camera pose determined by the feature point set belonging to the reference screen in each second screen image and the camera intrinsic parameters of the camera, a reprojection error of the feature points displayed in the reference screen projected into each second screen image; The positional relationship between the first other screen and the reference screen is determined according to the camera pose determined by the feature point set belonging to the first other screen in the second screen image corresponding to the minimum reprojection error and the camera pose determined by the feature point set belonging to the reference screen.

7. The method according to claim 5, characterized in that The other screens in the S blocks of screens also include: at least one second other screen that does not appear in the same screen image at the same time as the reference screen; The step of determining the positional relationship between other screens in the S blocks of screens and the reference screen according to the camera pose sets corresponding to the S blocks of screens includes: For any second other screen, determining a first other screen that appears simultaneously with the second other screen in the same screen image, and determining a third screen image in which the second other screen and the first other screen appear simultaneously from the multiple screen images; Determine a positional relationship between the second other screen and the first other screen according to a camera pose determined by a feature point set belonging to the second other screen and a camera pose determined by a feature point set belonging to the first other screen in the third screen image; The positional relationship between the second other screen and the first other screen is determined based on the positional relationship between the second other screen and the first other screen, and the positional relationship between the first other screen and the reference screen.

8. The method according to claim 1, characterized in that The screen includes an LED screen, each LED screen includes a plurality of LED boxes, and the method further includes: Obtaining screen information of each of the S screens, wherein the screen information includes the number of rows, the number of columns, the resolution, and the size of the LED boxes in the screen; According to the screen information of each screen, a calibration plate image corresponding to each screen is generated, wherein a plurality of feature points in the calibration plate image corresponding to each screen covers each LED box in each screen.

9. A device for determining the position relationship between screens, applied to a virtual shooting system, wherein the virtual shooting system comprises a camera for shooting and S screens for displaying a virtual scene, S≥2, characterized in that: The device comprises: An image acquisition module, used for acquiring a plurality of screen images captured by the camera on the S screens at different positions when the S screens have displayed their respective corresponding calibration plate images, wherein the calibration plate image displayed on each screen includes a plurality of feature points; a calibration data set determination module, configured to determine calibration data sets corresponding to each of the S screens according to the plurality of screen images, wherein the calibration data set corresponding to the s-th screen comprises: a feature point set consisting of feature points belonging to the s-th screen detected from at least one first screen image, the first screen image being a screen image in which the s-th screen appears among the plurality of screen images, s∈[1,S]; A reference screen determination module, configured to determine the camera intrinsic parameters of the camera and the reference screen in the S blocks of screens according to the calibration data sets corresponding to the S blocks of screens; A camera pose set determination module, configured to determine a camera pose set corresponding to each of the S screens according to the camera intrinsic parameters of the camera and the calibration data sets corresponding to each of the S screens, wherein the camera pose set corresponding to the s-th screen includes: at least one group of camera poses determined based on a set of feature points respectively belonging to the s-th screen in the at least one first screen image, and one group of camera poses corresponding to one first screen image; The position relationship determination module is used to determine the position relationship between other screens in the S blocks of screens and the reference screen according to the camera posture sets corresponding to the S blocks of screens, wherein the other screens include screens in the S blocks of screens except the reference screen.

10. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to implement the method described in any one of claims 1 to 8 when executing the instructions stored in the memory.

11. A non-volatile computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images