A multi-station virtual shooting method and device, electronic equipment and storage medium

Abstract

The present disclosure relates to a multi-camera virtual shooting method and device, electronic equipment and storage medium. The method comprises: determining an overlapping area on a display screen corresponding to the inner viewing cone of at least two roller shutter cameras in a plurality of roller shutter cameras in a first display period; determining, for the first overlapping area, the pixels in the image sensor of each roller shutter camera corresponding to each display unit on the first overlapping area; controlling each display unit on the first overlapping area to display the inner viewing cone picture of each roller shutter camera in turn in the first display period; and when any display unit displays the inner viewing cone picture of any roller shutter camera, controlling the corresponding pixels in the image sensor of the roller shutter camera to be exposed synchronously. Through the present disclosure, the display of the inner viewing cone picture of the roller shutter camera on each display unit in the overlapping area is adapted to the exposure of the corresponding pixels in the image sensor of the roller shutter camera, thereby realizing excellent multi-camera virtual shooting.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of virtual shooting, and in particular to a multi-camera virtual shooting method and device, electronic equipment and a storage medium. BACKGROUND

[0002] Virtual production is a film and television production technology that combines real-time 3D rendering, game engine technology and traditional studio shooting, uses computer graphics and film production technology to create visual content, and uses large light emitting diode (LED) display technology and live shooting technology to shoot high-definition, realistic film and television content in real time. In order to improve the production quality of film and television works, while reducing the time cost and economic cost of film and television production, virtual production technology has become a popular solution in the industry.

[0003] Rolling shutter cameras are widely used in virtual shooting scenes of high-quality film and television works; however, current virtual production technology is mostly based on a single rolling shutter camera, and the virtual production effect for multiple rolling shutter cameras is not good, which restricts the application of multi-camera virtual production. SUMMARY

[0004] Therefore, the present disclosure provides a multi-camera virtual shooting method, device, electronic equipment, storage medium and computer program product.

[0005] According to an aspect of the present disclosure, a multi-camera virtual shooting method is provided, which is applied to virtual shooting of multiple rolling shutter cameras, and the method comprises:

[0006] determining an overlap area corresponding to inner view cones of at least two rolling shutter cameras in the multiple rolling shutter cameras on a display screen in a first display period; the display screen comprises a plurality of regularly arranged display units;

[0007] for a first overlap area, determining pixels corresponding to each display unit on the first overlap area in each rolling shutter camera image sensor related to the first overlap area; wherein the first overlap area is any one of the overlap areas corresponding to the inner view cones of at least two rolling shutter cameras in the multiple rolling shutter cameras on the display screen;

[0008] in the first display period, controlling each display unit on the first overlap area to display the inner view cone pictures of the rolling shutter cameras in turn; and when any display unit displays the inner view cone picture of any rolling shutter camera, controlling the pixels corresponding to the display unit in the rolling shutter camera image sensor to be exposed synchronously.

[0009] In a possible implementation, the controlling, in the first display period, the display units in the first overlapping area to display the inner view cone pictures of the respective rolling shutter cameras in sequence; and the controlling, when any display unit displays the inner view cone picture of any rolling shutter camera, the pixel in the image sensor of the rolling shutter camera corresponding to the display unit to perform synchronous exposure, comprises:

[0010] determining a first exposure time of a first pixel in the first display period; wherein the first pixel is a pixel in a first rolling shutter camera image sensor corresponding to a first display unit, the first display unit is any display unit in the first overlapping area, and the first rolling shutter camera is any rolling shutter camera in the respective rolling shutter cameras;

[0011] determining, based on the first exposure time of the first pixel in the first display period, a first display time of the first display unit in the first display period for displaying the inner view cone picture of the first rolling shutter camera, wherein the first display time covers the first exposure time;

[0012] controlling, in the first display period, the first display unit to display the inner view cone picture of the first rolling shutter camera in the first display time, and controlling the first pixel to perform exposure in the first exposure time.

[0013] In a possible implementation, the determining the first exposure time of the first pixel in the first display period comprises:

[0014] determining, based on the shooting frame rates of the respective rolling shutter cameras, exposure start times of the respective rolling shutter camera image sensors for the first overlapping area in the first display period;

[0015] determining the first exposure time based on the exposure start time of the first rolling shutter camera image sensor for the first overlapping area, the position of the first pixel in the first rolling shutter camera image sensor, the shutter time length of the first rolling shutter camera, and the rolling shutter time length of the first rolling shutter camera.

[0016] In a possible implementation, the determining, based on the first exposure time of the first pixel in the first display period, the first display time of the first display unit in the first display period for displaying the inner view cone picture of the first rolling shutter camera comprises:

[0017] determining the first exposure time as a first estimated time of the first display unit in the first display period for displaying the inner view cone picture of the first rolling shutter camera;

[0018] determine a display correction duration of the first display unit based on a length of the first display period, and exposure times of pixels of the respective roll shutter cameras corresponding to the first display unit in the first display period;

[0019] correct the first estimated time based on the display correction duration to obtain the first display time.

[0020] In a possible implementation, the controlling, in the first display period, the display units in the first overlapping area to display the inner cone of view pictures of the respective roll shutter cameras in sequence includes:

[0021] obtaining heights of the inner cones of view of the respective roll shutter cameras in the first display period;

[0022] determining display orders of the inner cone of view pictures of the respective roll shutter cameras in the first display period based on the heights of the inner cones of view of the respective roll shutter cameras, wherein the display order of any inner cone of view picture of any roll shutter camera is negatively correlated with the height of the inner cone of view of the roll shutter camera;

[0023] controlling, in the first display period, the display units to display the inner cone of view pictures of the respective roll shutter cameras in sequence according to the display orders of the inner cone of view pictures of the respective roll shutter cameras.

[0024] In a possible implementation, the method further includes:

[0025] controlling, in the first display period, the display units in the first overlapping area to display at least one second picture in a time gap between displaying the inner cone of view pictures of any two roll shutter cameras.

[0026] In a possible implementation, the controlling, in the first display period, the display units in the first overlapping area to display at least one second picture in a time gap between displaying the inner cone of view pictures of any two roll shutter cameras includes:

[0027] determining display frame rates of the display units based on the photographing frame rates of the respective roll shutter cameras, wherein the display frame rates are integer multiples of a sum of the photographing frame rates of the respective roll shutter cameras;

[0028] determining exposure start times of the image sensors of the respective roll shutter cameras with respect to the first overlapping area in the first display period based on the display frame rates;

[0029] determining display times of the display units for displaying the second picture based on the exposure start times of the image sensors of the respective roll shutter cameras with respect to the first overlapping area in the first display period;

[0030] In the first display period, the display units display corresponding second pictures in display time for displaying second pictures.

[0031] In a possible implementation, the determining, for the first overlapping area, the pixels in the image sensor of each of the rolling shutter cameras corresponding to each of the display units in the first overlapping area comprises:

[0032] determining a range of position coordinates of the first overlapping area in the inner viewing cone of the first rolling shutter camera;

[0033] determining, based on a first correspondence between the position coordinates in the inner viewing cone of the first rolling shutter camera and the pixels in the image sensor of the first rolling shutter camera and the range of position coordinates, a second correspondence between the position coordinates in the first overlapping area and the pixels in the image sensor of the first rolling shutter camera;

[0034] determining the position coordinates of each of the display units in the first overlapping area;

[0035] determining, based on the position coordinates of each of the display units in the first overlapping area and the second correspondence, the pixels in the image sensor of the first rolling shutter camera corresponding to each of the display units.

[0036] According to another aspect of the present disclosure, a multi-position virtual shooting device is provided for virtual shooting of multiple rolling shutter cameras, the device comprising:

[0037] an overlapping area module configured to determine, in a first display period, overlapping areas on a display screen corresponding to inner viewing cones of at least two of the multiple rolling shutter cameras; the display screen comprising a plurality of regularly arranged display units;

[0038] a correspondence module configured to determine, for a first overlapping area, pixels in the image sensor of each of the rolling shutter cameras corresponding to each of the display units in the first overlapping area; wherein the first overlapping area is any of the overlapping areas on the display screen corresponding to the inner viewing cones of the at least two of the multiple rolling shutter cameras;

[0039] a synchronous exposure module configured to control, in the first display period, the display units in the first overlapping area to display the inner viewing cone pictures of the rolling shutter cameras in turn; and when any of the display units displays the inner viewing cone picture of any of the rolling shutter cameras, control the pixels in the image sensor of the rolling shutter camera corresponding to the display unit to be synchronously exposed.

[0040] According to another aspect of the present disclosure, there is provided an electronic device comprising a memory, a processor, and a computer program stored on the memory, the processor executing the computer program to implement the steps of the above method.

[0041] According to another aspect of the present disclosure, there is provided a non-transitory computer readable storage medium having stored thereon a computer program, the computer program, when executed by a processor, implementing the steps of the above method.

[0042] According to another aspect of the present disclosure, there is provided a computer program product comprising a computer program, or a non-transitory computer readable storage medium having a computer program stored thereon, the computer program, when executed by a processor, implementing the steps of the above method.

[0043] According to the aspects of the present disclosure, it is determined that, in a first display period, the inner view cones of at least two of the plurality of rolling shutter cameras correspond to overlapping areas on a display screen; the display screen comprises a plurality of regularly arranged display units; for each of the first overlapping areas, the pixels in the image sensor of each rolling shutter camera corresponding to each display unit on the first overlapping area are determined respectively; wherein the first overlapping area is any one of the overlapping areas on the display screen corresponding to the inner view cones of at least two of the plurality of rolling shutter cameras; in the first display period, the inner view cone images of the rolling shutter cameras are displayed on each display unit on the first overlapping area in turn; and when any display unit displays the inner view cone image of any rolling shutter camera, the pixels in the image sensor of the rolling shutter camera corresponding to the display unit are controlled to be exposed synchronously. In this way, in the multi-camera virtual shooting process, for the overlapping areas on the display screen corresponding to the inner view cones of the rolling shutter cameras, the display of the inner view cone images of the rolling shutter cameras on each display unit on the overlapping area and the exposure of the corresponding pixels in the image sensor of the rolling shutter camera are adapted, so that the pixels in the image sensor of the rolling shutter camera can be exposed accurately and completely to the inner view cone images of the rolling shutter camera displayed on each display unit, thereby realizing excellent multi-camera virtual shooting, and greatly improving the production efficiency of virtual shooting and the application range of multi-camera virtual shooting.

[0044] Other features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the drawings. BRIEF DESCRIPTION OF DRAWINGS

[0045] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the present disclosure and serve to explain the principles of the present disclosure.

[0046] Figure 1 FIG. 1 shows a schematic diagram of a scene for multi-camera virtual shooting using a plurality of rolling shutter cameras according to an embodiment of the present disclosure;

[0047] Figure 2 A schematic diagram showing a region corresponding to the inner view cones of multiple rolling shutter cameras on a display screen according to an embodiment of the present disclosure;

[0048] Figure 3 A schematic diagram showing display of inner view cone pictures of three rolling shutter cameras in a time-division manner according to an embodiment of the present disclosure;

[0049] Figure 4 A schematic diagram showing a correspondence between positions in an inner view cone picture and exposure start times of rows of pixels of an image sensor according to an embodiment of the present disclosure;

[0050] Figure 5 A structural diagram of a multi-camera virtual shooting system according to an embodiment of the present disclosure;

[0051] Figure 6 A flowchart of a multi-camera virtual shooting method according to an embodiment of the present disclosure;

[0052] Figure 7 A schematic diagram showing an overlapping region corresponding to the inner view cones of at least two rolling shutter cameras in multiple rolling shutter cameras on a display screen according to an embodiment of the present disclosure;

[0053] Figure 8 A flowchart of a method for determining corresponding pixels of a display unit according to an embodiment of the present disclosure;

[0054] Figure 9 A flowchart of a multi-camera virtual shooting method according to an embodiment of the present disclosure;

[0055] Figure 10 A schematic diagram showing display times of inner view cone pictures of three rolling shutter cameras according to an embodiment of the present disclosure;

[0056] Figure 11 A schematic diagram showing display times of inner view cone pictures of three rolling shutter cameras according to an embodiment of the present disclosure;

[0057] Figure 12 A schematic diagram showing expansion of display times of inner view cone pictures of three rolling shutter cameras according to an embodiment of the present disclosure;

[0058] Figure 13 A flowchart of a method for determining display times of a display unit according to an embodiment of the present disclosure;

[0059] Figure 14 A schematic diagram showing display times of inner view cone pictures of three rolling shutter cameras according to an embodiment of the present disclosure;

[0060] Figure 15A flow chart of a multi-position virtual shooting method according to an embodiment of the present disclosure is shown;

[0061] Figure 16 A schematic diagram of displaying an inner view frustum picture and a second picture according to an embodiment of the present disclosure is shown;

[0062] Figure 17 A schematic diagram of inner view frustum picture display time of two rolling shutter cameras according to an embodiment of the present disclosure is shown;

[0063] Figure 18 A structural diagram of a multi-position virtual shooting device according to an embodiment of the present disclosure is shown;

[0064] Figure 19 A block diagram of an electronic device 1900 according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0065] Various exemplary embodiments, features and aspects of the present disclosure will be explained in detail below with reference to the accompanying drawings. The same reference numerals in the drawings denote the same or similar elements / functionality. Although various aspects of the embodiments are illustrated in the drawings, the drawings are not necessarily drawn to scale unless specifically indicated.

[0066] As used herein, the terms "comprise", "comprising", "have", "having", "include", "including", "contain", "containing", or variants thereof, are open-ended and include one or more stated features, integers, elements, steps, components or functions but do not preclude the presence or addition of one or more other features, integers, elements, steps, components, functions or groups thereof.

[0067] When an element is referred to as being "connected", "coupled", "responsive", or "related" to another element, it can be directly connected, coupled, responsive, or related to the other element, or intervening elements can be present.

[0068] Although the terms first, second, third, etc. can be used herein to describe various elements / operations, these elements / operations should not be limited by these terms. These terms are only used to distinguish one element / operation from another element / operation. Thus, a first element / operation in some embodiments can be termed a second element / operation in other embodiments without departing from the teachings of the present inventive concept.

[0069] The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any implementation described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other implementations.

[0070] In addition, for a better illustration of the present disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand, however, that the present disclosure can be practiced without certain specific details, which are set forth in the following detailed description. In some instances, well-known methods, schemes, elements, and circuits have not been described in detail in order to avoid obscuring the present disclosure.

[0071] Virtual shooting is an innovative shooting method combining real-time computer graphics technology and traditional film and television production methods. It fuses virtual scenes, special effects and live-action content in real time, so that directors, photographers and other creators can view the final picture effect in real time during the shooting process without waiting for the post-production to be completed. This shooting method greatly improves the efficiency of film and television production and provides greater creative flexibility for creators. Based on the number of cameras used, virtual shooting can be divided into single-camera virtual shooting and multi-camera virtual shooting. Multi-camera virtual shooting can effectively improve the efficiency of virtual shooting and meet the needs of different shooting angles.

[0072] Figure 1 A scene schematic diagram of multi-camera virtual shooting using multiple rolling shutter cameras according to an embodiment of the present disclosure is shown. As shown in Figure 1 The live scene of virtual shooting can be deployed with a display screen and a rolling shutter camera.

[0073] The number of display screens can be one or more, and the display screen can be a flat screen, a curved screen, a three-fold screen, or a special-shaped screen in a multi-faceted three-dimensional structure, etc. Illustratively, the display screen can be an LED screen, also known as an LED curtain wall (LED Wall), which is formed by a large number of LED beads as basic light-emitting units, and the LED beads can present different colors. The number of rolling shutter cameras is multiple, such as rolling shutter camera 1, rolling shutter camera 2, …, and rolling shutter camera N in the figure, where N is a positive integer, and different rolling shutter cameras capture images at different angles. The rolling shutter camera is an imaging device that uses line-by-line or column-by-column scanning exposure. Each time the camera is shooting, the pixels in the image sensor are not exposed at the same time, but are exposed in a top-to-bottom order or a left-to-right order. The rolling shutter time and the shutter speed of the rolling shutter camera determine the time required for the rolling shutter camera to complete one shooting. The rolling shutter time refers to the time difference between the exposure start time of the earliest scanned row of pixels and the exposure start time of the latest scanned row of pixels (or the exposure start time of the earliest scanned column of pixels and the exposure start time of the latest scanned column of pixels) of the image sensor using the rolling shutter. It is usually expressed in milliseconds. The shutter speed refers to the time interval from the opening to the closing of the camera shutter, which is used to control the length of time that light is exposed to the image sensor. It can also be referred to as the exposure duration of each pixel.

[0074] Compared with the single-camera virtual shooting scene using a single rolling shutter camera, the above-mentioned Figure 1 multi-camera virtual shooting scene using multiple rolling shutter cameras is more complex. During multi-camera virtual shooting, the virtual scene image is displayed on the display screen, and the actor can perform in front of the display screen. At the same time, each rolling shutter camera captures images at different angles, and the images captured by each rolling shutter camera have the actor as the foreground and the virtual scene image on the display screen as the background. Different rolling shutter cameras can capture images at different angles to meet the multi-angle shooting requirement. Since the shooting angles of the rolling shutter cameras are different during shooting, in order to ensure that each rolling shutter camera can capture accurate images, the virtual scene image displayed on the display screen needs to match the angles of each rolling shutter camera when each rolling shutter camera is shooting, especially the inner cone image displayed in the area on the display screen corresponding to the inner cone of each rolling shutter camera (i.e., the area on the display screen that each rolling shutter camera can capture) needs to match the angle of each rolling shutter camera.

[0075] However, the areas on the display screen corresponding to the inner cones of different rolling shutter cameras can partially or completely overlap. For example, Figure 2A schematic diagram showing the area on a display screen corresponding to the inner viewing cones of a plurality of roller shutter cameras according to an embodiment of the present disclosure, such as... Figure 2 As shown, the inner viewing cones of roller shutter cameras 1, 2, and N correspond to regions 1, 2, and N on the display screen, respectively, where regions 1 and 2 partially overlap. For the overlapping region, the display screen needs to display the inner viewing cone image of any one of the roller shutter cameras (1 or 2) in that overlapping region to match its viewing angle.

[0076] In related technologies, the inner viewing cones of different roller shutter cameras correspond to overlapping areas on the display screen, and the inner viewing cone images of each roller shutter camera are displayed in time slots. For example, Figure 3 This diagram illustrates a time-segmented display of the inner cone images of three rolling shutter cameras according to an embodiment of the present disclosure. The inner cone images of rolling shutter cameras A, B, and C correspond to the same and overlapping areas on an LED screen. The overlapping area includes multiple rows of LED beads. The rolling shutter duration of each of the three cameras is 10ms, the shutter speed is 5ms, and the frame rate is 25fps. The display cycle duration is set to 40ms. The display cycle refers to the minimum time interval for each rolling shutter camera's inner cone image to complete one round of display. In each display cycle, each rolling shutter camera takes pictures sequentially according to a preset shooting order, and each rolling shutter camera takes pictures at least once within the same display cycle. Figure 3 As shown, the vertical axis represents the row number of the LED beads, and the horizontal axis represents time. "A" represents the inner cone image of rolling shutter camera A, "B" represents the inner cone image of rolling shutter camera B, and "C" represents the inner cone image of rolling shutter camera C. Within the display period of [1~40ms], the LED beads in each row synchronously display the inner cone image. Specifically, in the period of [1~14ms], the LED beads in rows 1~10 all display the inner cone image of rolling shutter camera A, in the period of [15~27ms], the LED beads in rows 1~10 all display the inner cone image of rolling shutter camera B, and in the period of [28~40ms], the LED beads in rows 1~10 all display the inner cone image of rolling shutter camera C. This multi-camera virtual shooting method with time-segmented display has poor shooting effect and hinders the application of multi-camera virtual shooting.

[0077] Analysis revealed that when any rolling shutter camera captures an overlapping area, the duration for which the inner cone image of that rolling shutter camera needs to be displayed in the overlapping area is limited by both the rolling shutter's rolling time and shutter speed. Due to this characteristic of rolling shutter cameras, the duration of the inner cone image displayed in each time period in the aforementioned multi-camera virtual shooting method cannot meet the exposure time requirements of all pixels in the rolling shutter camera's image sensor, resulting in suboptimal shooting quality.

[0078] Taking a rolling shutter camera image sensor using a rolling shutter exposure mode as an example, Figure 4 A schematic diagram showing the correspondence between each position in the inner view frustum picture and the exposure start time of each row of pixels of the image sensor according to an embodiment of the present disclosure is shown as follows: Figure 4 As shown in the figure, each position in the inner view frustum picture of the rolling shutter camera displayed on the display screen is exposed by the pixel in the rolling shutter camera image sensor in turn in the vertical direction, wherein the start time T of the exposure of each position by the pixel in the rolling shutter camera image sensor is proportional to the row coordinate H of the position in the inner view frustum picture, that is, the lower the position in the vertical direction of the inner view frustum picture displayed on the display screen, the later the start time (T_shutter_start) of the exposure of the corresponding pixel in the image sensor, wherein the exposure start time T_shutter_start(first_row) corresponding to the first row of positions is the earliest, and the exposure start time T_shutter_start(last_row) corresponding to the last row of positions is the latest, and the two have the following relationship:

[0079] T_shutter_start(last_row) = T_shutter_start(first_row) + T_rollingtime

[0080] That is, the interval between the exposure start times corresponding to the first row of positions and the last row of positions in the inner view frustum picture is the rolling shutter time (T_rollingtime) of the rolling shutter camera. Therefore, in order to complete the exposure of the inner view frustum picture by all the pixels in the rolling shutter camera image sensor, the time (T_slice_led) during which the inner view frustum picture is displayed on the display screen needs to satisfy the following formula:

[0081] T_slice_led >= T_rollingtime + T_shutter

[0082] Wherein, T_rollingtime represents the rolling shutter time of the rolling shutter camera, and T_shutter represents the shutter time of the rolling shutter camera.

[0083] As can be seen, in the multi-camera virtual shooting scene, when any rolling shutter camera shoots the overlapping area, the time during which the overlapping area displays the inner view frustum picture of the rolling shutter camera needs to be no less than the sum of the rolling shutter time and the shutter time of the rolling shutter camera. However, the multi-camera virtual shooting mode of the above-mentioned related art displays in time periods, and the time length of the display period limits the time length of the inner view frustum picture of the rolling shutter camera displayed in each time period, which cannot meet the requirements; still taking the above-mentioned Figure 3For example, since the rolling shutter camera has a rolling shutter duration of 10ms, a shutter speed of 5ms, and a shooting frame rate of 25fps, the duration T_slice_led for displaying the inner cone image of each rolling shutter camera in the overlapping area needs to be at least 15ms. Correspondingly, to complete the display of 25 frames, the total display time should theoretically be 3*25*15ms = 1125ms. However, the shooting time for 25 frames is 1 second, which is clearly contradictory. Therefore, in order to meet the shooting frame rate of the rolling shutter camera, the display time is compressed in each time period, resulting in insufficient time for the inner cone image of some LED-lit rolling shutter cameras to be fully exposed by the pixels on the rolling shutter camera's image sensor. Figure 3 In the diagram, the gray area represents the time required for the LED beads in rows 1-10 to expose the inner cone image of the rolling shutter camera A to the pixels of the rolling shutter camera A's image sensor; the medium yellow area represents the time required for the LED beads in rows 1-10 to expose the inner cone image of the rolling shutter camera B to the pixels of the rolling shutter camera B's image sensor; and the light yellow area represents the time required for the LED beads in rows 1-10 to expose the inner cone image of the rolling shutter camera C to the pixels of the rolling shutter camera C's image sensor. The gray area specifically refers to the 10th row of LED beads displaying the inner cone image of the rolling shutter camera... The time required for the inner cone image of the roller shutter camera A to be exposed by the pixels of the image sensor of the roller shutter camera A is 11ms-15ms. However, at 15ms, the inner cone image of the roller shutter camera B has already been displayed by the 10th row of LED beads. As a result, the pixels of the image sensor of the roller shutter camera A do not fully expose the inner cone image of the roller shutter camera A displayed by the 10th row of LED beads, but instead expose the wrong image (i.e., the red mark B in the figure). Similarly, this type of incorrect exposure occurs in the medium yellow area and the light yellow area (i.e., the red marks C and A in the figure).

[0084] To address the aforementioned technical problems, this disclosure proposes a multi-camera virtual shooting method suitable for roller shutter cameras (detailed description below). This method adapts the exposure of the inner cone image of the roller shutter camera displayed on each display unit in the overlapping area to the corresponding pixel exposure in the roller shutter camera's image sensor. This ensures that the pixels in the roller shutter camera's image sensor can accurately and completely expose the inner cone image of the roller shutter camera displayed on each display unit, thereby achieving excellent multi-camera virtual shooting results. This significantly improves the production efficiency of virtual shooting and expands the application scope of multi-camera virtual shooting.

[0085] Figure 5 This diagram illustrates a structural diagram of a multi-camera virtual shooting system according to an embodiment of the present disclosure; as shown below. Figure 5As shown, the multi-position virtual shooting system can include: a control device 10, a rendering device 20, a display screen 30, a rolling shutter camera 40; wherein the number of various devices can be flexibly configured according to actual needs, and no limitation is made in this regard; in addition, other devices or fewer devices can be configured in the system according to needs in actual application, and one or more devices in the system can also be integrated into one device according to needs, for example, the rendering device 20 and the control device 10 can be integrated into one device.

[0086] Exemplarily, the control device 10 can be various types of terminals such as a notebook computer, a tablet computer, a desktop computer, a smart phone, a smart watch, a smart television, etc.; the control device 10 can control the display of the display screen 30, for example, can control the display of the display screen 30 at a specific display frame rate, and can control the time of displaying each frame of picture; in addition, the control device 10 can also regulate and control the display parameters of the display screen 30, such as brightness adjustment, color correction, gray scale control, etc. The control device 10 can also control the shooting of the rolling shutter camera 40, for example, can control the rolling shutter camera 40 to shoot at a specific shooting frame rate, and can control the starting time of each shooting.

[0087] The rendering device 20, also known as a rendering engine, can construct a virtual space corresponding to the real space, and place a virtual camera and a virtual screen in the virtual space; wherein the virtual camera corresponds one-to-one to the rolling shutter camera 40 in the real space, the motion and the view angle of the virtual camera in the virtual space are kept synchronous with the motion and the view angle of the corresponding rolling shutter camera 40 in the real space; the virtual screen corresponds one-to-one to the display screen 30 in the real space, the position, the size, the distribution of the lamp beads of the virtual screen in the virtual space are kept consistent with the position, the size, the distribution of the lamp beads of the corresponding display screen 30 in the real space. In the virtual shooting process, based on the model geometry data, the lighting data, the environment data, etc. in the virtual space, the rendering device 20 can construct a virtual scene that the user plans to shoot in the virtual space; then for each rolling shutter camera 40, the rendering device 20 can determine the pose of the virtual camera corresponding to the rolling shutter camera 40 in the virtual space based on the pose of the rolling shutter camera 40, and then project the picture of the virtual scene in the view range of the virtual camera in the virtual space onto the virtual screen after three-dimensional projection transformation, as the internal view cone picture of the rolling shutter camera 40, the rendering device 20 renders the internal view cone picture and displays it on the corresponding position of the display screen 30 corresponding to the virtual screen in the real space, to adapt to the internal view cone of the rolling shutter camera 40.

[0088] The display screen 30 can be the above-mentioned Figure 1The display screen 30 can be a LED screen. Exemplarily, the parameters of the display screen 30 can be calibrated before virtual shooting. The parameters of the display screen 30 include spatial position, distribution of lamp beads, size, etc.

[0089] The rolling shutter cameras 40 can be the rolling shutter cameras 1, the rolling shutter cameras 2, …, and the rolling shutter cameras N. Figure 1 The rolling shutter cameras 40 can be calibrated before virtual shooting. The parameters of the rolling shutter cameras 40 include intrinsic parameters (such as focal length, optical center offset, distortion parameters, etc.), extrinsic parameters (such as pose), rolling shutter time length, shutter time length, etc. As an example, the rolling shutter cameras 40 can be installed with a motion capture device to capture the real-time pose of the rolling shutter cameras 40 during virtual shooting.

[0090] A multi-camera virtual shooting method provided by the present disclosure is described in detail below.

[0091] Figure 6 A flowchart of a multi-camera virtual shooting method according to an embodiment of the present disclosure is shown. Exemplarily, the method can be performed by some or all components of the virtual shooting system shown in the above Figure 5 The method can include the following steps: Figure 6

[0092] Step 601: Determine the overlapping area on the display screen corresponding to the inner viewing cone of at least two rolling shutter cameras in the plurality of rolling shutter cameras in a first display period.

[0093] The first display period represents any display period in the process of multi-camera virtual shooting using the plurality of rolling shutter cameras. Exemplarily, the time length of the first display period can be determined based on the shooting frame rate of the plurality of rolling shutter cameras. For example, the shooting frame rate of the plurality of rolling shutter cameras can be converted into the shooting time length of each frame (i.e., the reciprocal of the shooting frame rate), and the least common multiple of these shooting time lengths is calculated as the first display period, to ensure that each rolling shutter camera can shoot at least once in each first display period. As an example, the shooting frame rate of any two rolling shutter cameras in the plurality of rolling shutter cameras is the same or in a multiple relationship. For example, for a three-camera virtual shooting scene, if the shooting frame rates of the three rolling shutter cameras are 25 frames / second, 50 frames / second, and 25 frames / second, respectively, the time length of the first display period is 40 ms (i.e., the least common multiple of 40 ms, 20 ms, and 40 ms); if the shooting frame rates of the three rolling shutter cameras are 25 frames / second, 25 frames / second, and 25 frames / second, respectively, the time length of the first display period is 40 ms; and if the shooting frame rates of the three rolling shutter cameras are 5 frames / second, 25 frames / second, and 50 frames / second, respectively, the time length of the first display period is 200 ms.

[0094] ​The display screen comprises a plurality of regularly arranged display units; for example, the display screen can be an LED screen, and the display unit can comprise one or more adjacent LED lamp beads in the LED screen; for example, a plurality of adjacent LED lamp beads in the same row or the same column, or a plurality of adjacent LED lamp beads in a plurality of continuous rows or columns. The number and arrangement of LED lamp beads in each display unit can be set according to requirements; for example, an LED lamp box can be taken as a display unit, and a plurality of regularly arranged LED lamp beads (for example, 256*256 arranged LED lamp beads) can be contained in each LED box; for another example, LED lamp beads controlled by the same switch controller (usually 16*16 arranged LED lamp beads) can be taken as a display unit; for another example, each LED lamp bead can be taken as a display unit.

[0095] The overlapping area on the display screen corresponding to the inner view cones of the at least two rolling shutter cameras represents the overlapping part of the area in the plane where the display screen is located, which is projected from the three-dimensional space to the inner view cones of the rolling shutter cameras (i.e., the area on the display screen corresponding to the inner view cones of the rolling shutter cameras). For example, Figure 7 A schematic diagram showing the overlapping area on the display screen corresponding to the inner view cones of the at least two rolling shutter cameras in the plurality of rolling shutter cameras according to an embodiment of the present disclosure is shown; as Figure 7 As shown, for the rolling shutter camera A, the rolling shutter camera B and the rolling shutter camera C, the area on the display screen corresponding to the inner view cone of the rolling shutter camera A comprises the A area, the AB area, the AC area and the ABC area, the area on the display screen corresponding to the inner view cone of the rolling shutter camera B comprises the B area, the AB area, the BC area and the ABC area, and the area on the display screen corresponding to the inner view cone of the rolling shutter camera C comprises the C area, the AC area, the BC area and the ABC area; wherein the ABC area represents the overlapping area on the display screen corresponding to the inner view cones of the rolling shutter camera A, the rolling shutter camera B and the rolling shutter camera C; the AB area represents the overlapping area on the display screen corresponding to the inner view cones of the rolling shutter camera A and the rolling shutter camera B; the AC area represents the overlapping area on the display screen corresponding to the inner view cones of the rolling shutter camera A and the rolling shutter camera C; the BC area represents the overlapping area on the display screen corresponding to the inner view cones of the rolling shutter camera B and the rolling shutter camera C; the A area represents the area on the display screen corresponding to the inner view cone of the rolling shutter camera A except the overlapping area; the B area represents the area on the display screen corresponding to the inner view cone of the rolling shutter camera B except the overlapping area; and the C area represents the area on the display screen corresponding to the inner view cone of the rolling shutter camera C except the overlapping area. The ABC area, the AB area, the AC area and the BC area are the overlapping area on the display screen corresponding to the inner view cones of the at least two rolling shutter cameras among the rolling shutter camera A, the rolling shutter camera B and the rolling shutter camera C.

[0096] For example, the roller shutter camera can be the above Figure 5 The central roller shutter camera 40, the display screen can be as described above Figure 5 The central display screen is 30. Figure 5 The rendering device 20 can determine the overlapping area on the display screen 30 corresponding to the inner view cones of at least two of the multiple rolling cameras 40 during the first display cycle.

[0097] In one possible implementation, the rendering device 20 can acquire the poses of multiple rolling cameras 40 during the first display cycle; for any rolling camera 40, based on the poses of each rolling camera 40, the poses of the corresponding virtual cameras in the virtual space are synchronously adjusted, and then the area on the virtual screen corresponding to the inner view cone of each virtual camera can be determined; furthermore, the overlapping portion of the areas on the virtual screen corresponding to the inner view cones of at least two virtual cameras is determined as the virtual overlapping area on the virtual screen, and the area on the display screen 30 corresponding to the virtual overlapping area is the overlapping area.

[0098] In some scenarios, during multi-camera virtual shooting, the rendering device 20 can acquire the poses of multiple rolling cameras in real time within each display cycle. For example, in the first display cycle, the real-time poses of each rolling camera 40 can be captured using motion capture devices mounted on each rolling camera 40. In other scenarios, before multi-camera virtual shooting, camera pre-playing can be performed based on the shooting plan to acquire the pre-playing poses of multiple rolling cameras 40 within each display cycle. Here, camera pre-playing refers to ensuring shooting efficiency and image quality by pre-planning the camera's motion trajectory, shooting angle, and the position of actors / objects. Then, during multi-camera virtual shooting, the pre-playing poses of each rolling camera 40 within the first display cycle are used as the poses of each rolling camera 40 within the first display cycle.

[0099] Step 602: For the first overlapping region, determine the pixels in each roller shutter camera image sensor related to the first overlapping region that correspond to each display unit on the first overlapping region; wherein, the first overlapping region is any overlapping region on the display screen corresponding to the inner viewing cones of at least two of the plurality of roller shutter cameras.

[0100] The number of pixels corresponding to each display unit in the first overlapping region can be one or more. As an example, the view cone of the rolling shutter camera and the display screen are both kept horizontal. That is, the pixels in the same row of the rolling shutter camera image sensor are arranged horizontally, and the pixels in the same column are arranged vertically. In the first overlapping region, the display units are regularly arranged in the horizontal and vertical directions. For example, each rolling shutter camera is in a line-by-line scanning mode. For any display unit, one or more rows of pixels in the rolling shutter camera image sensor corresponding to the display unit can be determined. For another example, each rolling shutter camera is in a column-by-column scanning mode. For any display unit, one or more columns of pixels in the rolling shutter camera image sensor corresponding to the display unit can be determined.

[0101] Figure 8 A flow chart of a method for determining the pixels corresponding to a display unit is shown according to an embodiment of the present disclosure, as shown in Figure 8 may include the following steps:

[0102] Step 60201, determining the position coordinate range of the first overlapping region in the inner view cone of the first rolling shutter camera.

[0103] The position coordinate in the inner view cone of the first rolling shutter camera can be a position coordinate in a plane coordinate system in which the inner view cone of the first rolling shutter camera is located.

[0104] For example, the region corresponding to the inner view cone of the first rolling shutter camera on the display screen can be determined, as well as the position coordinate range of the region in the inner view cone of the first rolling shutter camera. In the case where the display screen covers the inner view cone of the first rolling shutter camera, the position coordinate range of the region in the inner view cone of the first rolling shutter camera is equal to the position coordinate range of the entire inner view cone. In the case where the display screen does not cover the inner view cone of the first rolling shutter camera (i.e., the environment outside the display screen can be seen in the field of view of the first rolling shutter camera), the position coordinate range of the region in the inner view cone of the first rolling shutter camera is smaller than the position coordinate range of the entire inner view cone. Then, the position coordinate range of the first overlapping region in the inner view cone of the first rolling shutter camera is determined in combination with the position of the first overlapping region in the region and the position coordinate range of the region in the inner view cone of the first rolling shutter camera. For example, in the above Figure 7 , the display screen covers the inner view cone of the rolling shutter camera A, and the region corresponding to the inner view cone of the rolling shutter camera A on the display screen includes four regions A, AB, AC, and ABC. The position coordinate range of the four regions in the inner view cone of the rolling shutter camera A is equal to the position coordinate range of the entire inner view cone. If the first overlapping region is the ABC region, the position coordinate range of the ABC region in the inner view cone of the rolling shutter camera A can be determined based on the position of the ABC region in the four regions.

[0105] Step 60202, based on the first corresponding relationship between the position coordinates in the inner view cone of the first roller shutter camera and the pixels in the image sensor of the first roller shutter camera, and the range of the position coordinates, determine the second corresponding relationship between the position coordinates in the first overlap region and the pixels in the image sensor of the first roller shutter camera.

[0106] Exemplarily, the distribution of the pixels in the image sensor of each roller shutter camera can be determined through pre-calibration. If the plane where the inner view cone of the first roller shutter camera is located is parallel to the plane where the image sensor of the first roller shutter camera is located, then the first corresponding relationship between the position coordinates in the inner view cone of the first roller shutter camera and the pixels in the image sensor of the first roller shutter camera can be determined based on the distribution of the pixels in the image sensor of the first roller shutter camera. Furthermore, for any coordinate position within the range of the position coordinates in the inner view cone of the first roller shutter camera in the first overlap region, the pixel in the image sensor of the first roller shutter camera corresponding to the position coordinate can be determined based on the first corresponding relationship. One position coordinate can correspond to one or more pixels, or one or more position coordinates can correspond to one pixel, and the pixels corresponding to different coordinate positions can be partially or entirely the same. All the position coordinates are traversed, and thus the second corresponding relationship between the position coordinates in the first overlap region and the pixels in the image sensor of the first roller shutter camera can be obtained.

[0107] Step 60203, determine the position coordinates of the display units in the first overlap region.

[0108] Exemplarily, the positions of the display units on the display screen, the distribution of the display units in the display screen, and the like can be determined through pre-calibration. Thus, for the first overlap region, the position coordinates of any display unit on the first overlap region can be determined.

[0109] Step 60204, based on the position coordinates of the display units in the first overlap region and the second corresponding relationship, determine the pixels in the image sensor of the first roller shutter camera corresponding to the display units.

[0110] Exemplarily, for any display unit, the pixel corresponding to the position coordinate of the display unit in the first overlap region can be found based on the second corresponding relationship between the position coordinates in the first overlap region and the pixels in the image sensor of the first roller shutter camera. All the display units are traversed, and thus the pixels in the image sensor of the first roller shutter camera corresponding to the display units can be determined.

[0111] Thus, through the above steps 60201-60204, the pixels in the image sensor of each roller shutter camera corresponding to the display units on any overlap region are determined respectively for the overlap region.

[0112] In the first display period, the display units in the first overlapping region are controlled to display the inner view frustums of the respective rolling shutter cameras in sequence; and when any display unit displays the inner view frustum of any rolling shutter camera, the pixels in the image sensor of the rolling shutter camera corresponding to the display unit are controlled to be exposed synchronously.

[0113] In a possible implementation, in the first display period, the inner view frustums of the respective rolling shutter cameras to be displayed by the display units in the first overlapping region can be determined. The number of frames of the inner view frustums of the respective rolling shutter cameras to be displayed by each display unit is the same and is equal to the number of frames of the inner view frustums of the respective rolling shutter cameras to be displayed in the first overlapping region. It can be understood that, for any frame of the inner view frustums to be displayed in the first overlapping region, each display unit displays a part of the frame of the inner view frustum (i.e., the inner view frustum to be displayed by each display unit), thereby completing the complete display of the frame of the inner view frustum in the first overlapping region.

[0114] For example, for each rolling shutter camera, the number of frames of the inner view frustum of the rolling shutter camera to be displayed in the first display period is positively correlated with the shooting frame rate of the rolling shutter camera, that is, the higher the shooting frame rate of the rolling shutter camera, the more frames of the inner view frustum of the rolling shutter camera to be displayed in the first display period; conversely, the lower the shooting frame rate of the rolling shutter camera, the fewer frames of the inner view frustum of the rolling shutter camera to be displayed in the first display period. As an example, the shooting frame rates of the respective rolling shutter cameras in the plurality of rolling shutter cameras are the same, and in the first display period, the number of frames of the inner view frustum of each rolling shutter camera to be displayed in the first overlapping region is one frame. For example, in the above example, the frame rates of the rolling shutter camera A, the rolling shutter camera B, and the rolling shutter camera C are the same, and in the first display period, the ABC region needs to display one frame of the inner view frustum of the rolling shutter camera A, one frame of the inner view frustum of the rolling shutter camera B, and one frame of the inner view frustum of the rolling shutter camera C. Figure 7

[0115] In a possible implementation, in the first display period, the display order corresponding to the inner view frustums of the respective rolling shutter cameras can be determined.

[0116] In some scenarios, if the image sensors of the respective rolling shutter cameras are exposed row by row in the order from top to bottom, considering that the heights of the inner view frustums of different rolling shutter cameras in the respective rolling shutter cameras related to the first overlapping region can be different, the height difference can affect the exposure of the pixels of the image sensors of the respective rolling shutter cameras. For example, in the above example, the heights of the inner view frustums of the rolling shutter camera A, the rolling shutter camera B, and the rolling shutter camera C are different, and the height of the inner view frustum of the rolling shutter camera A is the highest, the height of the inner view frustum of the rolling shutter camera B is the second highest, and the height of the inner view frustum of the rolling shutter camera C is the lowest. In the first display period, the ABC region needs to display one frame of the inner view frustum of the rolling shutter camera A, one frame of the inner view frustum of the rolling shutter camera B, and one frame of the inner view frustum of the rolling shutter camera C. If the image sensors of the respective rolling shutter cameras are exposed row by row in the order from top to bottom, the image sensor of the rolling shutter camera A needs to be exposed first, the image sensor of the rolling shutter camera B needs to be exposed second, and the image sensor of the rolling shutter camera C needs to be exposed last. However, the inner view frustum of the rolling shutter camera A has the highest height, and the inner view frustum of the rolling shutter camera C has the lowest height. Therefore, the exposure of the image sensor of the rolling shutter camera A is affected, and the exposure of the image sensor of the rolling shutter camera C is affected. Figure 7 ​In the first display period, since the image sensor of the rolling shutter camera A exposes the ABC region latest in one shot, and the image sensor of the rolling shutter camera C exposes the ABC region earliest, the two exposure times can overlap. If the ABC region is in the first display period, the inner frustum picture of the rolling shutter camera A is displayed first, and then the inner frustum picture of the rolling shutter camera C is displayed, a conflict can occur. For example, the exposure of the image sensor of the rolling shutter camera A to the ABC region has not been completed, and the ABC region needs to display the inner frustum picture of the rolling shutter camera C to adapt to the exposure of the image sensor of the rolling shutter camera C to the ABC region. Therefore, in the case that the frame numbers of the inner frustum pictures of the rolling shutter cameras to be displayed in the first display period are the same, the display order corresponding to the inner frustum pictures of the rolling shutter cameras can be determined according to the heights of the inner frustums of the rolling shutter cameras.

[0117] In a possible implementation, the controlling, in the first display period, the display units on the first overlapping region to display the inner frustum pictures of the rolling shutter cameras in sequence includes: obtaining the heights of the inner frustums of the rolling shutter cameras in the first display period; determining the display order corresponding to the inner frustum pictures of the rolling shutter cameras in the first display period based on the heights of the inner frustums of the rolling shutter cameras, wherein the display order corresponding to any inner frustum picture of a rolling shutter camera is negatively related to the height of the inner frustum of the rolling shutter camera; and controlling the display units to display the inner frustum pictures of the rolling shutter cameras in sequence according to the display order corresponding to the inner frustum pictures of the rolling shutter cameras in the first display period.

[0118] For example, considering that the sizes of the regions corresponding to the inner frustums of different rolling shutter cameras on the display screen can be different, the average height or the maximum height of the region corresponding to the inner frustum of each rolling shutter camera on the display screen can be taken as the height of the inner frustum of the rolling shutter camera. Furthermore, the display order corresponding to the inner frustum pictures of the rolling shutter cameras can be determined in the order from low to high according to the heights of the inner frustums of the rolling shutter cameras. For example, in the above example, the heights of the inner frustums of the rolling shutter cameras A, B, and C are different, the height of the inner frustum of the rolling shutter camera A is the highest, and the height of the inner frustum of the rolling shutter camera C is the lowest. Therefore, the display order corresponding to the inner frustum pictures of the three rolling shutter cameras in the first display period can be determined as: the inner frustum picture of the rolling shutter camera C, the inner frustum picture of the rolling shutter camera B, and the inner frustum picture of the rolling shutter camera A. In this way, the lower the height of the inner frustum of a rolling shutter camera, the earlier the inner frustum picture of the rolling shutter camera is displayed in the first display period, so that the inner frustum picture of any rolling shutter camera can be completely exposed in the first overlapping region. Figure 7 In the first display period, since the image sensor of the rolling shutter camera A exposes the ABC region latest in one shot, and the image sensor of the rolling shutter camera C exposes the ABC region earliest, the two exposure times can overlap. If the ABC region is in the first display period, the inner frustum picture of the rolling shutter camera A is displayed first, and then the inner frustum picture of the rolling shutter camera C is displayed, a conflict can occur. For example, the exposure of the image sensor of the rolling shutter camera A to the ABC region has not been completed, and the ABC region needs to display the inner frustum picture of the rolling shutter camera C to adapt to the exposure of the image sensor of the rolling shutter camera C to the ABC region. Therefore, in the case that the frame numbers of the inner frustum pictures of the rolling shutter cameras to be displayed in the first display period are the same, the display order corresponding to the inner frustum pictures of the rolling shutter cameras can be determined according to the heights of the inner frustums of the rolling shutter cameras.

[0119] In some scenarios, the display order of the inner view cone pictures of the respective rolling shutter cameras in the first display period can be determined according to the order in which the respective rolling shutter cameras start shooting, or a preset order, or a randomly generated order. For example, if the heights of the inner view cones of different rolling shutter cameras are different, causing some pictures to be shot abnormally, the abnormal picture parts shot by the respective rolling shutter cameras can be removed through post-processing such as cropping.

[0120] In a possible implementation, a first display unit can be determined, which is used to display the display time of each frame of the inner view cone pictures of the first rolling shutter camera in the first display period and the exposure time of the corresponding pixels, wherein the display time can be represented by one or more of a display start time, a display end time, and a display duration, and the interval between the display start time and the display end time is the display duration. The exposure time can be represented by one or more of an exposure start time, an exposure end time, and an exposure duration, and the interval between the exposure start time and the exposure end time is the exposure duration. Due to the characteristics of the rolling shutter camera, the exposure duration of each pixel on the image sensor of the same rolling shutter camera is the same, which is the shutter duration of the rolling shutter camera. It should be noted that, in the embodiments of the present disclosure, for a frame of the inner view cone pictures displayed in the first display period, the display time of the frame of the inner view cone pictures displayed by any display unit indicates that at least one of the display start time, the display end time, and the display duration corresponding to the frame of the inner view cone pictures is located in the first display period; and the exposure time of the corresponding pixels of any display unit indicates that at least one of the exposure start time, the exposure end time, and the exposure duration corresponding to the frame of the inner view cone pictures is located in the first display period.

[0121] For example, when determining the display time of each display unit for displaying the frusta image in each frame and the exposure time of the corresponding pixel in the first display period, for any frusta image in each frame, the following constraints need to be met: 1. For any display unit, the display start time of the display unit for displaying the frusta image in the frame is not later than the exposure start time of the corresponding pixel, and the display end time of the display unit for displaying the frusta image in the frame is not earlier than the exposure end time of the corresponding pixel. 2. The interval of the exposure start time of the different rolling shutter camera image sensors for the first overlapping area is not less than the shutter time of the rolling shutter camera (when the shutter times of the rolling shutter cameras are the same, it is not less than the same shutter time, and when the shutter times of the rolling shutter cameras are different, it is not less than the maximum shutter time). 3. For any display unit, the interval between the display start time of the display unit for displaying the frusta image in the frame and the display start time of the display unit for displaying the adjacent frusta image in the frame is not less than the rolling shutter time of the rolling shutter camera (when the shutter times of the rolling shutter cameras are the same, it is not less than the same shutter time, and when the shutter times of the rolling shutter cameras are different, it is not less than the maximum shutter time). 4. For different display units, the exposure start time of the display unit corresponding pixel is positively correlated with the display start time of the display unit for displaying the frusta image in the frame. In this way, based on the above constraints, by reasonably configuring the display time of each display unit for displaying the frusta image of each rolling shutter camera and the exposure time of the corresponding pixel on the first overlapping area, it is ensured that the display time of each display unit for displaying the frusta image of each rolling shutter camera on the overlapping area is adapted to the exposure time of the corresponding pixel of the rolling shutter camera image sensor; and when the rolling shutter camera image sensor and each pixel corresponding to the display unit on the overlapping area are exposed, the display unit corresponding to each pixel in the overlapping area can display the frusta image to be collected during the exposure of each pixel, so that each rolling shutter camera can capture a clear and complete frusta image each time.

[0122] In some scenarios, if there are multiple overlapping areas in the first display period, the display time of each display unit in the overlapping area corresponding to a larger number of rolling shutter cameras and the exposure time of the corresponding pixel can be determined first, and then based on the display time of each display unit in the overlapping area and the exposure time of the corresponding pixel, the display time of each display unit corresponding to the same rolling shutter camera in other overlapping areas and the exposure time of the corresponding pixel are determined, so that the display time of the frusta image of the same rolling shutter camera on the display screen is adapted to the exposure time of the rolling shutter camera during one shooting; for example, the above Figure 7In some scenarios, the display time of each display unit on the ABC region and the exposure time of the corresponding pixels can be determined first, and then the display time of each display unit on the AB region, the AC region and the BC region and the exposure time of the corresponding pixels can be determined, so that the display time of the same rolling shutter camera in different regions of a frame of the frustum image is matched with the exposure time of one-time shooting of the rolling shutter camera. In other scenarios, if there is an overlapping region in the first display period, i.e., the inner view cones of multiple rolling shutter cameras coincide, then only the display time of each display unit on the overlapping region and the exposure time of the corresponding pixels can be determined.

[0123] Exemplarily, the display time of each display unit on the first overlapping region and the exposure time of the corresponding pixels can be determined by the above-mentioned Figure 5 The control device 10 can control each display unit on the first overlapping region to display the inner view cone image of each rolling shutter camera 40 in turn, and control the pixels corresponding to each display unit in the image sensor of each rolling shutter camera 40 to be exposed synchronously. As an example, in the virtual shooting process, the control device 10 can obtain the real-time pose of each rolling shutter camera 40, and combine the parameters of each rolling shutter camera 40 and the parameters of the display screen 30 to calculate the exposure time of the pixels in the image sensor of each rolling shutter camera 40 and the display time of each display unit to display each frame of the inner view cone image in the first display period in real time. The rendering device 20 can obtain the real-time pose of each rolling shutter camera 40, and then render each frame of the inner view cone image displayed by each display unit in the first display period according to the shooting plan. The control device 10 can control each display unit to display the corresponding inner view cone image at the display time of each frame of the inner view cone image, and control the rolling shutter camera 40 to be exposed at the exposure time of the pixels in the image sensor of the rolling shutter camera 40. In this way, the display time of each display unit to display the inner view cone image of the first rolling shutter camera is matched with the exposure time of the pixels corresponding to the image sensor of the first rolling shutter camera, so that each rolling shutter camera can accurately capture a complete inner view cone image. It should be noted that, since the display start time of different display units to display the same frame of the inner view cone image is different in the first display period, for the last displayed inner view cone image in the first display period, the display unit with a later display start time may have completed the display of the inner view cone image and may have started the display of the next image; therefore, at the end of the first display period, there may be a part of the display units that have started to display a frame of the inner view cone image but have not completed the display process, and correspondingly, the display unit will continue to display the frame of the inner view cone image in the next display period.

[0124] In the embodiments of the present disclosure, the overlapping area corresponding to the inner view cones of at least two of the plurality of rolling shutter cameras in the first display period is determined on the display screen; the display screen comprises a plurality of regularly arranged display units; for the first overlapping area, the pixels in the image sensor of each rolling shutter camera corresponding to each display unit on the first overlapping area are determined respectively; the first overlapping area is any one of the overlapping areas corresponding to the inner view cones of at least two of the plurality of rolling shutter cameras on the display screen; in the first display period, each display unit on the first overlapping area is controlled to display the inner view cone picture of each rolling shutter camera in turn; and when any display unit displays the inner view cone picture of any rolling shutter camera, the pixel in the image sensor of the rolling shutter camera corresponding to the display unit is controlled to be exposed synchronously. In this way, in the multi-position virtual shooting process, for the overlapping area corresponding to the inner view cones of the rolling shutter cameras on the display screen, the display of the inner view cone picture of the rolling shutter camera on each display unit in the overlapping area is adapted to the exposure of the corresponding pixel in the image sensor of the rolling shutter camera, so that the pixel in the image sensor of the rolling shutter camera can be accurately and completely exposed to the inner view cone picture of the rolling shutter camera displayed on each display unit, thereby realizing excellent multi-position virtual shooting, and greatly improving the production efficiency of virtual shooting and the application range of multi-position virtual shooting.

[0125] The specific process of the synchronous exposure of the display of the inner view cone picture of the rolling shutter camera on each display unit and the pixel in the image sensor of the rolling shutter camera in the virtual shooting process described above will be described exemplarily.

[0126] Figure 9 A flowchart of a multi-position virtual shooting method according to an embodiment of the present disclosure is shown as follows, Figure 9 The method comprises the following steps:

[0127] In step 901, the first exposure time of the first pixel in the first display period is determined; the first pixel is the pixel in the first rolling shutter camera image sensor corresponding to the first display unit, the first display unit is any display unit on the first overlapping area, and the first rolling shutter camera is any rolling shutter camera in the plurality of rolling shutter cameras.

[0128] The first exposure time represents the time for the first display unit to display the inner view cone picture of the first rolling shutter camera for exposure, the number of the first exposure time is the same as the number of the inner view cone pictures to be displayed by the first rolling shutter camera in the first display period, and there is a one-to-one correspondence.

[0129] In a possible implementation, the determining the first exposure time of the first pixel in the first display period comprises: determining exposure start times of the image sensors of the respective rolling shutter cameras for the first overlapping area in the first display period based on shooting frame rates of the respective rolling shutter cameras; and determining the first exposure time based on the exposure start time of the first image sensor for the first overlapping area, the position of the first pixel in the first image sensor, a shutter time length of the first rolling shutter camera, and a rolling time length of the first rolling shutter camera.

[0130] Exemplarily, the number of frames of the inner view frustum picture to be displayed in the first display period can be determined based on the shooting frame rates of the respective rolling shutter cameras, and then the exposure start times of the image sensors of the respective rolling shutter cameras for the first overlapping area can be determined based on the number of frames so as to divide the first display period as much as possible into the target. For example, if the first overlapping area is related to three rolling shutter cameras and the shooting frame rates of the three rolling shutter cameras are the same, the first display period can be divided into three equal parts.

[0131] Exemplarily, the first exposure time can be calculated by the following formula:

[0132] StartTime = T0 + pH * T’(rolling time), EndTime = StartTime + T(shutter),

[0133] wherein, StartTime represents the exposure start time of the first pixel; EndTime represents the exposure end time of the first pixel, T0 represents the exposure start time of the first rolling shutter camera image sensor for the first overlapping area, for example, one of the three exposure start times obtained after the above-mentioned trisection. T(shutter) represents the shutter time length of the first rolling shutter camera, and the first rolling shutter camera image sensor exposes row by row from top to bottom; PH represents the normalized height value corresponding to the position of the first pixel in the first rolling shutter camera image sensor, and the value of PH is the ratio of the distance from the position of the first pixel in the first rolling shutter camera image sensor to the upper edge of the region of the first rolling shutter camera image sensor corresponding to the first overlapping area to the height of the region of the first rolling shutter camera image sensor corresponding to the first overlapping area (i.e. the distance between the upper and lower edges); 0 represents the top of the region, and 1 represents the bottom of the region. T'(rollingtime) represents the rolling shutter time length corresponding to the region of the first rolling shutter camera image sensor corresponding to the first overlapping area, and the value thereof is the product of the proportion of the height of the region of the first rolling shutter camera image sensor corresponding to the first overlapping area in the entire height of the first rolling shutter camera image sensor and the rolling shutter time length T(rollingtime) of the first rolling shutter camera; as an example, if the first overlapping area and the inner viewing cone of the first rolling shutter camera are the same in height, i.e. the upper edges of the first overlapping area and the inner viewing cone of the first rolling shutter camera are at the same height, and the lower edges are also at the same height, at this time, the rolling shutter time length T'(rollingtime) corresponding to the region of the first rolling shutter camera image sensor corresponding to the first overlapping area is the same as the rolling shutter time length T(rollingtime) of the first rolling shutter camera.

[0134] In this way, the normalized height value corresponding to the position of the first pixel in the first rolling shutter camera image sensor can be determined based on the distribution of each pixel in the first rolling shutter camera image sensor and the region of the first rolling shutter camera image sensor corresponding to the first overlapping area; and the proportion of the height of the region of the first rolling shutter camera image sensor corresponding to the first overlapping area in the entire height of the first rolling shutter camera image sensor can be determined, and the product of the proportion and the rolling shutter time length of the first rolling shutter camera can be calculated to obtain the rolling shutter time length corresponding to the region of the first rolling shutter camera image sensor corresponding to the first overlapping area; further, the product of the normalized height value and the rolling shutter time length corresponding to the region of the first rolling shutter camera image sensor corresponding to the first overlapping area is calculated based on the above formula, and the sum of the product and the exposure start time of the first rolling shutter camera image sensor for the first overlapping area is calculated, so as to obtain the exposure start time of the first pixel.

[0135] At step 902, a first display time of the first display unit in the first display period for displaying the first lenticular camera's inner view frustum picture is determined based on a first exposure time of the first pixel in the first display period, wherein the first display time covers the first exposure time.

[0136] Exemplarily, the first display time can be represented by a first display start time and a first display duration, and the first exposure time can be represented by a first exposure start time and a first exposure duration. The first display time covering the first exposure time is represented as: the first display start time is not later than the first exposure start time, and the first display duration is not shorter than the first exposure duration.

[0137] In some scenarios, the first exposure time can be determined as the first display time, i.e., for any display unit, the display time of the display unit for displaying the inner view frustum picture of a certain lenticular camera is the same as the exposure time of the pixel in the image sensor of the lenticular camera corresponding to the display unit. In this way, for any lenticular camera, the image sensor of the lenticular camera can be exposed while the display units display the inner view frustum picture of the lenticular camera, so that the lenticular camera can capture the corresponding inner view frustum picture.

[0138] Figure 10 A schematic diagram showing the display time of the inner view frustum pictures of three lenticular cameras according to an embodiment of the present disclosure is shown in FIG. 15. Figure 10 As shown, for the same moment, there are different display units displaying the inner view frustum pictures of different lenticular cameras. As shown in the 15th s in the figure, the LED lamp beads in the first row display the inner view frustum picture of lenticular camera B, while the LED lamp beads in the tenth row display the inner view frustum picture of lenticular camera A.

[0139] Taking the multi-camera virtual shooting of three lenticular cameras cameraA, cameraB and cameraC as an example, the shooting frame rate, the rolling time T(rolling), and the shutter time T(shutter) of the three lenticular cameras are the same, and the duration T(period) of the display period (i.e., the first display period) is the same. In any display period, for the overlapping area of the three lenticular cameras on the display screen, the exposure start time of cameraA is t0(cameraA), the exposure start time of cameraB is t0(cameraB), and the exposure start time of cameraC is t0(cameraC).

[0140] For a display unit corresponding to the overlapping area on the display screen of the inner view cones of the three rolling cameras cameraA, cameraB, cameraC, the display unit is determined to correspond to pixels on the image sensors of the three rolling cameras respectively, and the normalized height values pH(A), pH(B), pH(C) corresponding to the positions of the pixels in the image sensors are calculated.

[0141] Further, the display start times StartTime(A), StartTime(B), StartTime(C) of the display unit displaying the inner view cone pictures of the three rolling cameras can be calculated by the following formula, wherein:

[0142] StartTime(A) = t0(cameraA) + pH(A) * TA(rollingtime),

[0143] StartTime(B) = t0(cameraB) + pH(B) * TB(rollingtime),

[0144] StartTime(C) = t0(cameraC) + pH(C) * TC(rollingtime);

[0145] Wherein, TA(rollingtime) represents the rolling time corresponding to the area of the cameraA image sensor corresponding to the overlapping area, TB(rollingtime) represents the rolling time corresponding to the area of the cameraB image sensor corresponding to the overlapping area, and TC(rollingtime) represents the rolling time corresponding to the area of the cameraC image sensor corresponding to the overlapping area.

[0146] Correspondingly, the display end times EndTime(A), EndTime(B), EndTime(C) of the display unit displaying the inner view cone pictures of the three rolling cameras can be obtained, wherein:

[0147] EndTime(A) = StartTime(A) + T(shutter),

[0148] EndTime(B) = StartTime(B) + T(shutter),

[0149] EndTime(C) = StartTime(C) + T(shutter).

[0150] Wherein, EndTime(A) is not greater than StartTime(B), EndTime(B) is not greater than StartTime(C), and EndTime(C) is not greater than (StartTime(A) + T(period))

[0151] For example, assuming that the frame rates of the above three rolling shutter cameras cameraA, cameraB and cameraC are all 25 fps, the rolling time T(rolling time) of each rolling shutter camera is 10 ms, and the shutter time T(shutter) of each rolling shutter camera is 5 ms, for the purpose of easy understanding, it is assumed that the inner view cones of the three rolling shutter cameras correspond to the overlapping area on the display screen and the inner view cones of the three rolling shutter cameras are the same in height, i.e., the upper edges of the inner view cones of the three rolling shutter cameras are at the same height, and the lower edges are also at the same height. At this time, the rolling time TA(rolling time), TB(rolling time) and TC(rolling time) corresponding to the areas of the image sensors of the three rolling shutter cameras and the areas corresponding to the overlapping area are all the same as the rolling time T(rolling time). The display period time T(period) can be set to 40 ms. Figure 11 A schematic diagram of the inner view cone picture display time of three rolling shutter cameras according to an embodiment of the present disclosure is shown as follows. Figure 11 As shown in the [0-40 ms] display period, the exposure start time t0(cameraA) of cameraA is set to 0 ms, the exposure start time t0(cameraB) of cameraB is set to 13.3 ms, and the exposure start time t0(cameraC) of cameraC is set to 26.6 ms. Taking the display unit displayed earliest in the overlapping area of the inner view cones of the three rolling shutter cameras (i.e., pH(A) = 0, pH(B) = 0, and pH(C) = 0) as an example, the display start time expStartTime(A) = 0, expStartTime(B) = 13.3 ms, and expStartTime(C) = 26.6 ms corresponding to the display unit can be calculated by the above formula, and the display end time expEndTime(A) = 5 ms, expEndTime(B) = 18.3 ms, and expEndTime(C) = 31.6 ms. Thus, the display time T(A) = [0 ms-5 ms] of the inner view cone picture of cameraA, the display time T(B) = [13.3 ms-18.3 ms] of the inner view cone picture of cameraB, and the display time T(C) = [26.6 ms-31.6 ms] of the inner view cone picture of cameraC can be obtained.

[0152] In some scenarios, the display time of each display unit can be extended, i.e., the display start time can be moved forward by a certain time length, and / or the display end time can be moved backward by a certain time length; and the extended display time can be used as the final display time. Exemplarily, the range of the display time extension of each display unit can be the same or different for different display units. In this way, for any rolling shutter camera, the display time length of each display unit for displaying the inner view cone picture of the rolling shutter camera exceeds the exposure time length required for the image sensor pixels of the rolling shutter camera to be exposed, and thus, when the image sensor of the rolling shutter camera is exposed to the first display unit, the first display unit displays the inner view cone picture of the rolling shutter camera, so that the rolling shutter camera has enough time to capture a complete inner view cone picture, thereby ensuring that any rolling shutter camera can capture a correct and complete inner view cone picture.

[0153] For example, Figure 12 A schematic diagram showing the display time extension of the inner view cone pictures of three rolling shutter cameras according to an embodiment of the present disclosure is shown. As shown in Figure 12 Based on the above Figure 10 , for each display unit, the display time of the display unit for displaying any frame of the inner view cone picture can be extended by 3ms, wherein the display start time is moved forward by 1ms and the display end time is moved backward by 2ms.

[0154] Figure 13 A flowchart showing the determination of the display time of a display unit according to an embodiment of the present disclosure is shown. As shown in Figure 13 , the method comprises the following steps:

[0155] Step 90201, determining the first exposure time as a first estimated time of the first display unit for displaying the inner view cone picture of the first rolling shutter camera in the first display period.

[0156] Exemplarily, the first estimated time can be represented by a first estimated start time and a first estimated end time.

[0157] Taking the above three rolling shutter cameras cameraA, cameraB, and cameraC as an example, the first estimated start time expStartTime(A), expStartTime(B), and expStartTime(C) of the display unit for displaying the inner view cone pictures of the three rolling shutter cameras can be calculated by the following formula, wherein:

[0158] expStartTime(A) = t0(cameraA) + pH(A) * TA(rollingtime),

[0159] expStartTime(B) = t0(cameraB) + pH(B) * TB(rolling time),

[0160] expStartTime(C) = t0(cameraC) + pH(C) * TC(rolling time);

[0161] wherein, pH(A), pH(B), pH(C) represent the normalized height values corresponding to the positions of the pixels of the display unit in the image sensors of the three rolling shutter cameras, t0(cameraA), t0(cameraB), t0(cameraC) represent the exposure start times of the three rolling shutter cameras for the overlapping area respectively, TA(rolling time) represents the rolling time corresponding to the area of the image sensor of cameraA corresponding to the overlapping area, TB(rolling time) represents the rolling time corresponding to the area of the image sensor of cameraB corresponding to the overlapping area, TC(rolling time) represents the rolling time corresponding to the area of the image sensor of cameraC corresponding to the overlapping area.

[0162] The first estimated end times expEndTime(A), expEndTime(B), expEndTime(C) of the display unit for displaying the inner cone pictures of the three rolling shutter cameras can be calculated by the following formula, wherein:

[0163] expEndTime(A) = expStartTime(A) + T(shutter),

[0164] expEndTime(B) = expStartTime(B) + T(shutter),

[0165] expEndTime(C) = expStartTime(C) + T(shutter);

[0166] wherein, T(shutter) represents the shutter time, and T(period) represents the display period time.

[0167] It is required that expEndTime(A) is not greater than expStartTime(B), expEndTime(B) is not greater than expStartTime(C), and expEndTime(C) is not greater than (expStartTime(A) + T(period)).

[0168] Step 90202, determining a display correction duration of the first display unit based on the length of the first display period, and the exposure time of the pixels of the respective rolling shutter cameras corresponding to the first display unit within the first display period.

[0169] The exposure time of the pixels of the respective rolling shutter cameras corresponding to the first display unit within the first display period is the estimated start time and the estimated end time of the first display unit for displaying the inner view frustum pictures of the respective rolling shutter cameras.

[0170] Taking the above three rolling shutter cameras cameraA, cameraB, and cameraC as an example, the display correction duration phase1, phase2, and phase3 of the first display unit for the inner view frustum pictures of the three rolling shutter cameras can be calculated by the following formula:

[0171] phase1=(expStartTime(A)+T(period)-expEndTime(C)) / 2;

[0172] phase2=(expStartTime(B)-expEndTime(A)) / 2;

[0173] phase3=(expStartTime(C)-expEndTime(B)) / 2;

[0174] Step 90203, correcting the first estimated time based on the display correction duration to obtain the first display time.

[0175] Taking the above three rolling shutter cameras cameraA, cameraB, and cameraC as an example, the first estimated start time of the display unit for displaying the inner view frustum pictures of the three rolling shutter cameras can be corrected to obtain the first display start time StartTime(A), StartTime(B), and StartTime(C) by the following formula:

[0176] StartTime(A)=expStartTime(A)-phase1,

[0177] StartTime(B)=expStartTime(B)-phase2,

[0178] StartTime(C)=expStartTime(C)-phase3,

[0179] The first estimated end time of the inner view cone picture of the three rolling shutter cameras displayed by the display unit is corrected by the following formula to obtain the first display end time EndTime(A), EndTime(B), and EndTime(C):

[0180] EndTime(A) = expEndTime(A) + phase2,

[0181] EndTime(B) = expEndTime(B) + phase3,

[0182] EndTime(C) = expEndTime(C) + phase1.

[0183] Further, the corrected first display time of the display unit for displaying the inner view cone pictures of the three rolling shutter cameras in the display period can be T(A), T(B), and T(C), wherein:

[0184] T(A) = [StartTime(A) ~ EndTime(A)]

[0185] T(B) = [StartTime(B) ~ EndTime(B)]

[0186] T(C) = [StartTime(C) ~ EndTime(C)]

[0187] Through the above steps 90201-90203, the display time of each display unit for displaying each frame of the inner view cone picture can be extended, so that complete inner view cone pictures can be captured in the case of delay or advance of the rolling shutter camera shooting time, or delay error caused by pose data acquisition.

[0188] For example, assuming that the shooting frame rate of the above cameraA, cameraB, and cameraC is 25fps, the rolling time T(rollingtime) is 10ms, and the shutter time T(shutter) is 5ms, and the inner view cones of the three rolling shutter cameras correspond to the overlapping area of the display screen and the inner view cones of the three rolling shutter cameras are the same in height (i.e., TA(rollingtime), TB(rollingtime), and TC(rollingtime) are the same as T(rollingtime)), the display period time T(period) can be set to 40ms. Figure 14 A schematic diagram of the inner view cone picture display time of the three rolling shutter cameras according to an embodiment of the present disclosure is shown as follows: Figure 14As shown, in the display period of [0~40ms], the overlapping area on the display screen corresponding to the inner view cones of the three rolling shutter cameras, the exposure start time t0(cameraA) of cameraA is set to 0ms, the exposure start time t0(cameraB) of cameraB is set to 13.3ms, and the exposure start time t0(cameraC) of cameraC is set to 26.6ms. Taking the earliest displayed display unit in the overlapping area of the inner view cones of the three rolling shutter cameras as an example, i.e., pH(A)=0, pH(B)=0, and pH(C)=0, the three display start times of the display unit can be calculated based on the above formula, expStartTime(A)=0, expStartTime(B)=13.3ms, and expStartTime(C)=26.6ms, and the three display end times expEndTime(A)=5ms, expEndTime(B)=18.3ms, and expEndTime(C)=31.6ms; further, the three display correction durations phase1=4.2ms, phase2=4.15ms, and phase3=4.15ms are calculated; thus, the corrected three display times T(A)=[-4.2ms~9.15ms], T(B)=[9.15ms~22.45ms], and T(C)=[22.45ms~35.8ms] are obtained. The negative value represents the time in the previous display period. Further, the display unit can be controlled to display the inner view cone picture of cameraA in [-4.2ms~9.15ms], the inner view cone picture of cameraB in [9.15ms~22.45ms], and the inner view cone picture of cameraC in [22.45ms~35.8ms].

[0189] Step 903, in the first display period, controlling the first display unit to display the inner view cone picture of the first rolling shutter camera at the first display time, and controlling the first pixel to expose at the first exposure time.

[0190] The steps 901-903 can be a possible implementation manner of step 603 in the above Figure 6 .

[0191] Thus, by steps 901-903, the first display time of the first display unit in the first display period for displaying the inner view frustum picture of the first rolling shutter camera is determined based on the first exposure time of the first pixel in the first display period, and then the first display unit is controlled to display the inner view frustum picture of the first rolling shutter camera at the first display time, and the first pixel is controlled to be exposed at the first exposure time; since the first display time of the first display unit corresponding to the first pixel for the inner view frustum picture covers the first exposure time of the first pixel for the inner view frustum picture, the first display unit displays the inner view frustum picture of the first rolling shutter camera when the first pixel is exposed at the first exposure time in the first display period, the synchronization of display and shooting is realized, and thus it is ensured that the first pixel can collect a correct and complete inner view frustum picture.

[0192] In the embodiments of the present disclosure, the first exposure time of the first pixel in the first display period is determined; the first display time of the first display unit in the first display period for displaying the inner view frustum picture of the first rolling shutter camera is determined based on the first exposure time of the first pixel in the first display period, wherein the first display time covers the first exposure time; in the first display period, the first display unit is controlled to display the inner view frustum picture of the first rolling shutter camera at the first display time, and the first pixel is controlled to be exposed at the first exposure time. In this way, considering the influence of the rolling shutter time and the shutter time of the rolling shutter camera on the display time of the inner view frustum picture of the rolling shutter camera, in the multi-camera virtual shooting process, the time (i.e., the display time) of each display unit for displaying the inner view frustum picture of the rolling shutter camera is dynamically adjusted according to the real-time change of the exposure time of the corresponding pixel, so that the time of each display unit for displaying the inner view frustum picture of the rolling shutter camera is adapted to the exposure time of the corresponding pixel in the image sensor of the rolling shutter camera, and it is ensured that the inner view frustum picture displayed by each display unit can be normally and completely shot by the rolling shutter camera.

[0193] Further, considering that in the first display period, in addition to the time for displaying each inner view frustum picture, each display unit in the first overlapping area can also include other remaining time; since in these remaining time, each rolling shutter camera will not scan the first overlapping area, therefore, the content displayed in the remaining time will not affect the normal shooting of each rolling shutter camera. For example, in the remaining time, no content can be displayed, or the display content of the remaining time can be flexibly configured as needed, for example, a preset image, a random image, or the inner view frustum picture can be continuously displayed, and the like.

[0194] In some scenarios, the second picture can be displayed in the remaining time in the first display period. The second picture can be a preset image, a random image, or a FOV picture, etc. For example, in a double-camera virtual shooting scenario, due to the characteristics of the rolling shutter camera, there can be a situation of visual flicker in the double-camera virtual shooting. For example, the shooting frame rate of two rolling shutter cameras A and B is 25 fps, the rolling shutter time is 10 ms, the shutter time is 5 ms, and the display period is 40 ms. Since the display time of a FOV picture in a frame can be 15 ms, if a time-sharing manner is used, for example, the FOV picture of the rolling shutter camera A is displayed in each display unit from 0 to 20 ms, and the FOV picture of the rolling shutter camera B is displayed in each display unit from 20 to 40 ms. However, there is a problem: the display screen flickers at a frequency of 25 hz, and the human eye has a strong flicker, which can interfere with the performance of the performers in front of the display screen. According to the subjective characteristics of human visual perception, if the flicker frequency is high enough, the subjective perception of a person will be very small. Experiments have found that if the flicker frequency can be increased to 50 hz or above, the flicker can be basically ignored. Therefore, in the embodiments of the present disclosure, by displaying a second picture in the remaining time and reasonably configuring the display time of the second picture and the FOV picture, the flicker frequency can be increased, that is, the display frame rate in the first display period is increased, so as to eliminate the subjective flicker and improve the comfort of the performers in the double-camera virtual shooting scenario.

[0195] Figure 15 A flowchart of a multi-camera virtual shooting method according to an embodiment of the present disclosure is shown as follows. Figure 15 The method includes the following steps:

[0196] Step 1401: Determine the overlapping area on the display screen corresponding to the FOV of at least two rolling shutter cameras in the first display period.

[0197] Step 1402: For the first overlapping area, determine the pixels in the image sensor of each rolling shutter camera corresponding to each display unit in the first overlapping area; wherein the first overlapping area is any one of the overlapping areas on the display screen corresponding to the FOV of at least two rolling shutter cameras.

[0198] Step 1403: In the first display period, control each display unit in the first overlapping area to display the FOV picture of each rolling shutter camera in turn; and when any display unit displays the FOV picture of any rolling shutter camera, control the pixels in the image sensor of the rolling shutter camera corresponding to the display unit to be exposed synchronously.

[0199] The above steps 1401-1403 are the same as the above steps 1101-1103. Figure 6The steps 601-603 are the same, and will not be described here.

[0200] Step 1404, in the first display period, controlling each display unit on the first overlapping area to display at least one second picture in the time interval of displaying the inner viewing cone pictures of any two rolling shutter cameras.

[0201] The step 1404 can be performed simultaneously with the step 1403 described above. In this way, in the first display period, the display of the inner viewing cone pictures of each rolling shutter camera and the second picture is realized, thereby improving the display frame rate of the first display period and reducing flicker.

[0202] Exemplarily, the number and display time of the second picture in the first display period can be configured according to requirements; wherein, for any display unit, the total time of displaying each second picture and the total time of displaying the inner viewing cone pictures of each rolling shutter camera in the first display period does not exceed the time length of the first display period.

[0203] Exemplarily, the time length of displaying each second picture and the time length of each inner viewing cone picture in the first display period are as close as possible, thereby further reducing flicker.

[0204] Figure 16 A schematic diagram of displaying the inner viewing cone picture and the second picture according to an embodiment of the present disclosure is shown, as shown in Figure 16 "D" represents the second picture, and the display unit displays the second picture in the interval between the inner viewing cone picture of the rolling shutter camera A and the inner viewing cone picture of the rolling shutter camera B.

[0205] In a possible implementation, in this step, the step of controlling each display unit on the first overlapping area to display at least one second picture in the time interval of displaying the inner viewing cone pictures of any two rolling shutter cameras in the first display period can include: determining the display frame rate of each display unit based on the shooting frame rate of each rolling shutter camera; wherein the display frame rate is an integer multiple of the sum of the shooting frame rates of the rolling shutter cameras; determining the exposure start time of the image sensor of each rolling shutter camera for the first overlapping area in the first display period based on the display frame rate; determining the display time of each display unit for displaying the second picture based on the exposure start time of the image sensor of each rolling shutter camera for the first overlapping area in the first display period; and controlling each display unit to display the corresponding second picture in the display time for displaying the second picture in the first display period.

[0206] Exemplarily, the shooting frame rates of each rolling shutter camera can be the same; and twice the sum of the shooting frame rates of each rolling shutter camera can be taken as the display frame rate of each display unit.

[0207] Exemplarily, determining the exposure start time of each rolling shutter camera image sensor for the first overlapping area in the first display period based on the display frame rate can include: determining the exposure start time corresponding to each picture (including the inner cone picture and the second picture) as a target of equally dividing the first display period as much as possible, wherein the number of exposure start times (i.e., the number of display pictures) is determined based on the display frame rate of each display unit.

[0208] Exemplarily, determining the display time of the display unit for displaying the second picture based on the exposure start time of each rolling shutter camera image sensor for the first overlapping area in the first display period can include: determining the display time of the first display unit for displaying the inner cone picture of each rolling shutter camera in the first display period based on the exposure start time of each rolling shutter camera image sensor for the first overlapping area in the first display period; and further determining the display time of the first display unit for displaying each second picture in the first display period based on the length of the first display period and the display time of the first display unit for displaying the inner cone picture of each rolling shutter camera in the first display period. As an example, the display time corresponding to each second picture is the same as the display time of the inner cone picture of each rolling shutter camera.

[0209] Taking cameraA and cameraB as an example, the shooting frame rate, the rolling time T (rollingtime), the shutter time T (shutter) of cameraA and cameraB are the same, and the length of the display period T (period) is the same. Two frames of inner cone pictures of rolling shutter camera A and two frames of inner cone pictures of rolling shutter camera B are displayed in the same display period, i.e., the second picture includes one frame of inner cone picture of rolling shutter camera A and one frame of inner cone picture of rolling shutter camera B, wherein the display time of the first frame of inner cone picture of rolling shutter camera A, the first frame of inner cone picture of rolling shutter camera B, the second frame of inner cone picture of rolling shutter camera A, and the second frame of inner cone picture of rolling shutter camera B is equally divided in the same display period as much as possible.

[0210] The first estimated start time expStartTime(A), expStartTime(B) and the first estimated end time expEndTime(A), expEndTime(B) of the display unit for displaying one frame of inner cone picture of two rolling shutter cameras can be calculated by the following formula, wherein:

[0211] expStartTime(A) = t0(cameraA) + pH(A) * TA(rollingtime),

[0212] expStartTime(B) = t0(cameraB) + pH(B) * TB(rolling time),

[0213] expEndTime(A) = expStartTime(A) + T(shutter),

[0214] expEndTime(B) = expStartTime(B) + T(shutter),

[0215] wherein, pH(A), pH(B) represent the normalized height value corresponding to the position of the display unit corresponding pixel in each rolling shutter camera image sensor, t0(cameraA), t0(cameraB) represent the exposure start time of the two rolling shutter cameras for the overlapping region respectively, T(shutter) represents the shutter time length, TA(rolling time) represents the rolling time length corresponding to the region of the cameraA image sensor corresponding to the overlapping region, TB(rolling time) represents the rolling time length corresponding to the region of the cameraB image sensor corresponding to the overlapping region.

[0216] It is required to satisfy expEndTime(A) is not greater than expStartTime(B), expEndTime(B) is not greater than (expStartTime(A) + T(period)), wherein, T(period) represents the display period time length;

[0217] Further, the display time of the display unit displaying the inner cone picture of the two rolling shutter cameras can be calculated by the following formula:

[0218] phase1 = expStartTime(A) - (T(period) - expEndTime(B)) / 2;

[0219] phase2 = max(expEndTime(A), phase1 + TA(rolling time));

[0220] phase4 = min(expStartTime(B), phase1 + T(period) - TB(rolling time));

[0221] phase3 = (phase2 + phase4) / 2;

[0222] The first frame of the inner view frustum of the rolling shutter camera A can be displayed in [phase1-phase2], the first frame of the inner view frustum of the rolling shutter camera B can be displayed in [phase2-phase3], the second frame of the inner view frustum of the rolling shutter camera A can be displayed in [phase3-phase4], and the second frame of the inner view frustum of the rolling shutter camera B can be displayed in [phase4-T(period)+phase1], so that the display frame rate of each display unit is improved, and the screen flicker of the dual-camera virtual shooting is effectively reduced.

[0223] For example, taking two rolling shutter cameras cameraA and cameraB as an example, the shooting frame rate of cameraA and cameraB is 25 fps, the rolling time T(rolling time) of cameraA and cameraB is 10 ms, and the shutter time T(shutter) of cameraA and cameraB is 5 ms. Assuming that the inner view frustum of the two rolling shutter cameras corresponds to the overlapping area of the display screen and the inner view frustum of the three rolling shutter cameras is the same in height (i.e., TA(rolling time) and TB(rolling time) are the same as T(rolling time)), the display period T(period) can be set to 40 ms, the display frame rate is set to (25+25)*2=100 fps, and the two frames of the inner view frustum of cameraA and the two frames of the inner view frustum of cameraB can be displayed in the display period of [0-40 ms]. Figure 17 A schematic diagram of the display time of the inner view frustum of two rolling shutter cameras according to an embodiment of the present disclosure is shown as follows. Figure 17 As shown in the figure, taking the earliest display unit in the overlapping area of the inner view frustum of the two rolling shutter cameras (i.e., pH(A)=0 and pH(B)=0) as an example, t0(cameraA)=0 ms and t0(cameraB)=30 ms. According to the above formula, the display time T(A1) of the first frame of the inner view frustum of cameraA is [-2.5 ms-7.5 ms], the display time T(B1) of the first frame of the inner view frustum of cameraB is [7.5 ms-17.5 ms], the display time T(A2) of the second frame of the inner view frustum of cameraA is [17.5 ms-27.5 ms], and the display time T(B2) of the second frame of the inner view frustum of cameraB is [27.5 ms-37.5 ms].

[0224] In the first display period, the display units in the first overlapping area are controlled to display the inner view cone pictures of the respective rolling shutter cameras in sequence; when any display unit displays the inner view cone picture of any rolling shutter camera, the pixels in the image sensor of the rolling shutter camera corresponding to the display unit are controlled to be synchronously exposed; and the display units in the first overlapping area are controlled to display at least one second picture in the time interval of displaying the inner view cone pictures of any two rolling shutter cameras. In this way, the display frame rate of the display screen is greatly improved, thereby effectively reducing the influence of flicker and improving the comfort of the performers in the multi-camera virtual shooting process.

[0225] Based on the same inventive concept of the above method embodiments, the embodiments of the present disclosure also provide a multi-camera virtual shooting device, which can be used to execute the technical solutions described in the above method embodiments.

[0226] Figure 18 A structural diagram of a multi-camera virtual shooting device according to an embodiment of the present disclosure is shown, which is applied to virtual shooting of multiple rolling shutter cameras. As shown in the figure, the device comprises: Figure 18

[0227] An overlapping area module 1601 is configured to determine, in a first display period, overlapping areas on a display screen corresponding to inner view cones of at least two rolling shutter cameras in the multiple rolling shutter cameras; the display screen comprises multiple regularly arranged display units;

[0228] A corresponding module 1602 is configured to determine, for each first overlapping area, pixels in image sensors of respective rolling shutter cameras corresponding to display units in the first overlapping area; the first overlapping area is any one of the overlapping areas on the display screen corresponding to the inner view cones of the at least two rolling shutter cameras in the multiple rolling shutter cameras;

[0229] A synchronous exposure module 1603 is configured to, in the first display period, control the display units in the first overlapping area to display the inner view cone pictures of the respective rolling shutter cameras in sequence; and when any display unit displays the inner view cone picture of any rolling shutter camera, control the pixels in the image sensor of the rolling shutter camera corresponding to the display unit to be synchronously exposed.

[0230] ​In the embodiments of the present disclosure, the overlapping area corresponding to the inner view cones of at least two of the plurality of rolling shutter cameras in the first display period is determined on the display screen; the display screen comprises a plurality of regularly arranged display units; for the first overlapping area, the pixels in the image sensor of each rolling shutter camera corresponding to each display unit on the first overlapping area are determined respectively; the first overlapping area is any one of the overlapping areas corresponding to the inner view cones of at least two of the plurality of rolling shutter cameras on the display screen; in the first display period, each display unit on the first overlapping area is controlled to display the inner view cone picture of each rolling shutter camera in turn; and when any display unit displays the inner view cone picture of any rolling shutter camera, the pixel in the image sensor of the rolling shutter camera corresponding to the display unit is controlled to be exposed synchronously. In this way, in the multi-position virtual shooting process, for the overlapping area corresponding to the inner view cones of the rolling shutter cameras on the display screen, the display of the inner view cone picture of the rolling shutter camera on each display unit in the overlapping area and the exposure of the corresponding pixel in the image sensor of the rolling shutter camera are adapted, so that the pixels in the image sensor of the rolling shutter camera can be exposed accurately and completely to the inner view cone picture of the rolling shutter camera displayed on each display unit, thereby realizing excellent multi-position virtual shooting, and greatly improving the production efficiency of virtual shooting and the application range of multi-position virtual shooting.

[0231] In a possible implementation, the synchronization exposure module 1603 is further configured to: determine a first exposure time of a first pixel in the first display period; the first pixel is a pixel in a first rolling shutter camera image sensor corresponding to a first display unit, the first display unit is any display unit on the first overlapping area, and the first rolling shutter camera is any rolling shutter camera in the rolling shutter cameras; based on the first exposure time of the first pixel in the first display period, determine a first display time of the first display unit in the first display period for displaying the inner view cone picture of the first rolling shutter camera, wherein the first display time covers the first exposure time; in the first display period, control the first display unit to display the inner view cone picture of the first rolling shutter camera in the first display time, and control the first pixel to be exposed in the first exposure time.

[0232] In a possible implementation, the synchronization exposure module 1603 is further configured to: based on the shooting frame rate of the rolling shutter cameras, determine the exposure start time of the image sensor of each rolling shutter camera for the first overlapping area in the first display period; based on the exposure start time of the first rolling shutter camera image sensor for the first overlapping area, the position of the first pixel in the first rolling shutter camera image sensor, the shutter time length of the first rolling shutter camera, and the rolling shutter time length of the first rolling shutter camera, determine the first exposure time.

[0233] In a possible implementation, the synchronization exposure module 1603 is further configured to: determine the first exposure time as a first estimated time for the first display unit to display the inner view cone picture of the first rolling shutter camera in the first display period; determine a display correction time length of the first display unit based on a time length of the first display period and exposure times of pixels of the respective rolling shutter cameras corresponding to the first display unit in the first display period; and correct the first estimated time based on the display correction time length to obtain the first display time.

[0234] In a possible implementation, the synchronization exposure module 1503 is further configured to: obtain heights of inner view cones of the respective rolling shutter cameras in the first display period; determine display orders of the inner view cone pictures of the respective rolling shutter cameras in the first display period based on the heights of the inner view cones of the respective rolling shutter cameras; wherein the display order of any inner view cone picture of any rolling shutter camera is negatively correlated with the height of the inner view cone of the rolling shutter camera; and control the respective display units to display the inner view cone pictures of the respective rolling shutter cameras in the first display period in the display orders of the inner view cone pictures of the respective rolling shutter cameras.

[0235] In a possible implementation, the synchronization exposure module 1603 is further configured to: control the display units in the first overlapping area to display at least one second picture in a time gap for displaying the inner view cone pictures of any two rolling shutter cameras in the first display period.

[0236] In a possible implementation, the synchronization exposure module 1603 is further configured to: determine display frame rates of the respective display units based on the photographing frame rates of the respective rolling shutter cameras; wherein the display frame rates are integer multiples of a sum of the photographing frame rates of the respective rolling shutter cameras; determine exposure start times of the image sensors of the respective rolling shutter cameras for the first overlapping area in the first display period based on the display frame rates; determine display times of the respective display units for displaying second pictures based on the exposure start times of the image sensors of the respective rolling shutter cameras for the first overlapping area in the first display period; and control the respective display units to display corresponding second pictures in the display times for displaying second pictures in the first display period.

[0237] In a possible implementation, the correspondence module 1602 is further configured to: determine a range of position coordinates of the first overlap region in an inner view cone of the first roller shutter camera; determine a second correspondence between position coordinates in the first overlap region and pixels in the image sensor of the first roller shutter camera based on the first correspondence between the position coordinates in the inner view cone of the first roller shutter camera and the pixels in the image sensor of the first roller shutter camera, and the range of position coordinates; and determine the pixels in the image sensor of the first roller shutter camera corresponding to the display units based on the position coordinates of the display units in the first overlap region and the second correspondence.

[0238] In some embodiments, the apparatus provided by the embodiments of the present disclosure has functions or includes modules that can be used to perform the methods described in the above method embodiments, and the specific implementation can refer to the description of the above method embodiments. For brevity, details are not repeated here.

[0239] The embodiments of the present disclosure further provide an electronic device, including a memory, a processor, and a computer program stored in the memory, and the processor executes the computer program to implement the steps of the above method.

[0240] The embodiments of the present disclosure further provide a non-volatile computer readable storage medium having a computer program stored thereon, and the computer program is executed by a processor to implement the steps of the above method.

[0241] The embodiments of the present disclosure further provide a computer program product, including a computer program or a non-volatile computer readable storage medium carrying the computer program, and the computer program is executed by a processor to implement the steps of the above method.

[0242] Figure 19 A block diagram of an electronic device 1900 according to an embodiment of the present disclosure is shown. For example, the electronic device 1900 can be provided as a server or a terminal device. Referring to Figure 19 , the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932, for storing instructions executable by the processing component 1922, such as an application program. The application program stored in the memory 1932 can include one or more than one module each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute the instructions to perform the above method.

[0243] The electronic device 1900 can further include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output interface 1958 (I / O interface). The electronic device 1900 can operate based on an operating system stored in the memory 1932, such as Windows Server TM , Mac OS X TM , Unix TM , Linux TM , FreeBSD TM , or the like.

[0244] In an exemplary embodiment, there is also provided a non-transitory computer readable storage medium, such as the memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the electronic device 1900 to perform the above method.

[0245] The computer readable storage medium can be a tangible device that can retain and store instructions for execution by a processor. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or punched tape, a

[0246] The computer program (or computer readable program instructions) described herein can be downloaded from a computer readable storage medium to various computing / processing devices by way of a network, e.g., the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0247] Computer readable program instructions for carrying out operations of the present disclosure can be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions can execute entirely on the user's computing / processing device, partly on the user's computing / processing device, as a stand-alone software package, partly on the user's computing / processing device and partly on a remote computing / processing device or entirely on the remote computing / processing device or server. In the latter scenario, the remote computing / processing device can be connected to the user's computing / processing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computing / processing device, for example, through the Internet using an Internet Service Provider. In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present disclosure.

[0248] The computer readable program instructions can also be loaded onto a computing / processing device, other programmable data processing apparatus, or other device to cause a series of operations to be performed on the computing / processing device, other programmable apparatus or other device to produce a computer implemented process, such that the instructions which execute on the computing / processing device, other programmable apparatus, or other device implement the operations specified in the flow diagrams and / or block diagrams.

[0249] These computer readable program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can include a non-transitory computer readable storage medium that can be a computer- readable storage medium having no data storage cycles that change state. The instructions can be executed by one or more processors of a computer, to cause a series of operational steps to be performed on the computer to produce a computer-implemented process. The instructions can also cause one or more processors of a computer or other programmable data processing apparatus to

[0250] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer implemented process, such that the instructions which execute on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0251] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer implemented process, such that the instructions which execute on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0252] Embodiments of the present disclosure have been described above, and the description is intended to be illustrative of the embodiments and not restrictive of the disclosure. Many modifications and variations of the described embodiments are possible in light of this disclosure without departing from the scope and spirit of the described embodiments. The choice of words in this document is intended to best explain the principles of the embodiments, the practical application, or technical improvement over prior art, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A multi-station virtual shooting method, characterized in that, The method is applied to virtual shooting of multiple rolling shutter cameras, and the method comprises the following steps: determining overlapping areas corresponding to inner view cones of at least two rolling shutter cameras in the multiple rolling shutter cameras on a display screen in a first display period; the display screen comprises a plurality of regularly arranged display units; for a first overlapping area, determining pixels corresponding to each display unit on the first overlapping area in each rolling shutter camera image sensor related to the first overlapping area; wherein the first overlapping area is any one of overlapping areas corresponding to inner view cones of at least two rolling shutter cameras in the multiple rolling shutter cameras on the display screen; in the first display period, controlling each display unit on the first overlapping area to display inner view cone pictures of the rolling shutter cameras in turn; and when any display unit displays an inner view cone picture of any rolling shutter camera, controlling pixels corresponding to the display unit in the rolling shutter camera image sensor to be exposed synchronously.

2. The method of claim 1, wherein, the step of controlling each display unit on the first overlapping area to display inner view cone pictures of the rolling shutter cameras in turn in the first display period; and when any display unit displays an inner view cone picture of any rolling shutter camera, controlling pixels corresponding to the display unit in the rolling shutter camera image sensor to be exposed synchronously, comprises: determining a first exposure time of a first pixel in the first display period; wherein the first pixel is a pixel corresponding to a first display unit in a first rolling shutter camera image sensor, the first display unit is any display unit on the first overlapping area, and the first rolling shutter camera is any rolling shutter camera in the rolling shutter cameras; based on the first exposure time of the first pixel in the first display period, determining a first display time of the first display unit in the first display period for displaying an inner view cone picture of the first rolling shutter camera, wherein the first display time covers the first exposure time; in the first display period, controlling the first display unit to display the inner view cone picture of the first rolling shutter camera in the first display time, and controlling the first pixel to be exposed in the first exposure time.

3. The method of claim 2, wherein, the step of determining the first exposure time of the first pixel in the first display period, comprises: based on a shooting frame rate of the rolling shutter cameras, determining exposure start times of the rolling shutter camera image sensors respectively for the first overlapping area in the first display period; based on the exposure start time of the first rolling shutter camera image sensor for the first overlapping area, the position of the first pixel in the first rolling shutter camera image sensor, the shutter time length of the first rolling shutter camera, and the rolling shutter time length of the first rolling shutter camera, determining the first exposure time.

4. The method of claim 2, wherein, the step of determining the first display time of the first display unit in the first display period for displaying the inner view cone picture of the first rolling shutter camera based on the first exposure time of the first pixel in the first display period, comprises: determining the first exposure time as a first estimated time of the first display unit in the first display period for displaying the inner view cone picture of the first rolling shutter camera; determine a display correction duration of the first display unit based on a length of the first display period, and exposure times of pixels of the respective roller shutter cameras corresponding to the first display unit in the first display period; correct the first estimated time based on the display correction duration to obtain the first display time.

5. The method of claim 1, wherein, The controlling, in the first display period, the display units in the first overlapping area to display the inner viewing cone pictures of the respective roller shutter cameras in sequence includes: obtaining heights of the inner viewing cones of the respective roller shutter cameras in the first display period; determining display orders of the inner viewing cone pictures of the respective roller shutter cameras in the first display period based on the heights of the inner viewing cones of the respective roller shutter cameras; wherein the display order of any inner viewing cone picture of any roller shutter camera is negatively correlated with the height of the inner viewing cone of the roller shutter camera; controlling, in the first display period, the display units to display the inner viewing cone pictures of the respective roller shutter cameras in sequence according to the display orders of the inner viewing cone pictures of the respective roller shutter cameras.

6. The method of claim 1, wherein, The method further includes: controlling, in the first display period, the display units in the first overlapping area to display at least one second picture in a time gap between displaying the inner viewing cone pictures of any two roller shutter cameras.

7. The method of claim 6, wherein, The controlling, in the first display period, the display units in the first overlapping area to display at least one second picture in a time gap between displaying the inner viewing cone pictures of any two roller shutter cameras includes: determining display frame rates of the display units based on shooting frame rates of the respective roller shutter cameras; wherein the display frame rates are integer multiples of the sum of the shooting frame rates of the respective roller shutter cameras; determining exposure start times of image sensors of the respective roller shutter cameras with respect to the first overlapping area in the first display period based on the display frame rates; determining display times of the display units for displaying second pictures based on the exposure start times of the image sensors of the respective roller shutter cameras with respect to the first overlapping area in the first display period; controlling, in the first display period, the display units to display corresponding second pictures in the display times for displaying second pictures.

8. The method of claim 1, wherein, The determining, for the first overlapping area, pixels of image sensors of the respective roller shutter cameras related to the first overlapping area corresponding to the display units in the first overlapping area includes: determining a range of position coordinates of the first overlapping area in an inner viewing cone of a first roller shutter camera; determining a second correspondence between position coordinates in the first overlapping area and pixels of the image sensor of the first roller shutter camera based on a first correspondence between the position coordinates in the inner viewing cone of the first roller shutter camera and the pixels of the image sensor of the first roller shutter camera, and the range of position coordinates; determining position coordinates of the display units in the first overlapping area; determining the pixels of the image sensor of the first roller shutter camera corresponding to the display units based on the position coordinates of the display units in the first overlapping area and the second correspondence.

9. A multi-camera virtual shooting device, characterized in that, The device is applied to virtual shooting of a plurality of roller shutter cameras, and the device includes: An overlapping area module is configured to determine overlapping areas on a display screen corresponding to inner view cones of at least two of the plurality of rolling shutter cameras in a first display period; the display screen comprises a plurality of regularly arranged display units; A corresponding module is configured to determine, for a first overlapping area, pixels in each rolling shutter camera image sensor corresponding to each display unit in the first overlapping area; the first overlapping area is any one of the overlapping areas on the display screen corresponding to inner view cones of at least two of the plurality of rolling shutter cameras; A synchronous exposure module is configured to control the display units in the first overlapping area to display the inner view cones of the rolling shutter cameras in sequence in the first display period; and control the pixels in the rolling shutter camera image sensor corresponding to the display unit to be synchronously exposed when the display unit displays the inner view cone of the rolling shutter camera.

10. An electronic device comprising a memory, a processor, and a computer program stored on the memory, wherein the computer program, when executed by the processor, is arranged to perform the method of any one of claims 1 to 9. The processor executes the computer program to implement the steps of the method of any one of claims 1 to 8.

11. A non-transitory computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 8.

12. A computer program product comprising a computer program or a non-transitory computer-readable storage medium bearing a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 8.

Images