Display control method and device of LED screen, electronic equipment, storage medium and program product

Abstract

The invention relates to a display control method and device for an LED screen, electronic equipment, a storage medium and a program product, and the method comprises the steps: obtaining the pose information of a roller shutter camera in a virtual shooting process; according to the pose information and a screen model of the LED screen, determining an inner view cone picture and pixel position information of a plurality of target LED lamp beads; determining the scanned time sequence of the target LED lamp bead according to the shooting starting time of the roller shutter camera, the pixel position information and the roller shutter parameters of the roller shutter camera; according to the scanned time sequence of the target LED lamp beads, the shutter duration and the continuous display duration of the LED lamp beads, the display time sequence of the multiple target LED lamp beads is determined; and controlling the plurality of target LED lamp beads in the LED screen to perform color display according to the display time sequence of the plurality of target LED lamp beads. Therefore, the roller shutter camera can be applied to a virtual shooting scene with a higher frame rate picture refreshing requirement, and the display acquisition ratio of an LED screen is greatly improved.

Description

Technical Field

[0001] The present disclosure relates to the field of virtual shooting, and particularly to a method and device for controlling the display of an LED screen, an electronic device, a storage medium, and a program product. Background Art

[0002] Virtual Production is an innovative shooting method that combines real-time computer graphics technology with traditional film and television production means. By fusing virtual scenes, special effects, and real-shot content in real time, directors, cinematographers, and other creative personnel can directly see the final effect during the shooting process without waiting for post-production to be completed. This technology greatly improves production efficiency and provides greater flexibility for creators. In recent years, in order to improve the production quality of film and television works while reducing the time cost and economic cost of film and television production, virtual production technology has increasingly become a popular solution in the industry.

[0003] A rolling shutter camera is an imaging device that uses progressive scanning exposure. Its sensor does not capture the entire image at once, but reads the image data line by line from top to bottom; due to the rolling shutter design of its sensor, it does not require the hardware cost advantage of configuring an independent storage unit for each pixel (which global shutter requires); therefore, applying a rolling shutter camera to virtual shooting has become a development trend.

[0004] However, due to the rolling shutter feature of progressive scanning exposure of the rolling shutter camera, compared with a global camera, the time taken to capture one frame of the image is longer, resulting in a lower display acquisition ratio of the LED screen, and thus may not be able to meet the virtual shooting scenarios with higher frame rate image refresh requirements. Summary of the Invention

[0005] In view of this, the present disclosure proposes a method and device for controlling the display of an LED screen, an electronic device, a storage medium, and a program product, which can apply a rolling shutter camera to virtual shooting scenarios with higher frame rate image refresh requirements, without being troubled by the rolling shutter duration, and greatly improve the display acquisition ratio of the LED screen.

[0006] According to one aspect of the present disclosure, a display control method for an LED screen is provided, which is applied to a virtual shooting scenario where a rolling shutter camera is used to shoot the LED screen. The method includes: during virtual shooting, obtaining the pose information of the rolling shutter camera; according to the pose information of the rolling shutter camera and the screen model of the LED screen, determining the inner frustum view within the inner frustum range of the rolling shutter camera to be displayed on the LED screen and the pixel position information of a plurality of target LED beads within the inner frustum range of the rolling shutter camera in the LED screen, where the pixel position information includes the pixel positions where the target LED beads are projected onto the imaging plane of the rolling shutter camera; according to the starting shooting moment of the rolling shutter camera, the pixel position information of the plurality of target LED beads, and the rolling shutter parameter of the rolling shutter camera, determining the scanning sequence of each target LED bead; where the rolling shutter parameter represents the rolling shutter duration of the rolling shutter camera, and the scanning sequence includes the starting moment when each target LED bead starts to be scanned by the rolling shutter camera; according to the scanning sequence of each target LED bead, the shutter duration of the rolling shutter camera, and the continuous display duration of a single LED bead, determining the display sequence of the plurality of target LED beads for displaying the inner frustum view, where the display sequence represents the starting display moment and the ending display moment of each target LED bead; according to the display sequence of the plurality of target LED beads for displaying the inner frustum view, controlling the plurality of target LED beads in the LED screen to perform color display.

[0007] In a possible implementation manner, the determining the scanning sequence of each target LED bead according to the starting shooting moment of the rolling shutter camera, the pixel position information of the plurality of target LED beads, and the rolling shutter parameter of the rolling shutter camera includes: for the i-th row of pixels on the imaging plane of the rolling shutter camera, according to the pixel position information of the plurality of target LED beads, determining the i-th group of beads corresponding to the i-th row of pixels, where the i-th group of beads includes one row or a plurality of target LED beads projected onto the i-th row of pixels, 1 ≤ i ≤ N, and N is the total pixel height of the imaging plane; when i is 1, according to the starting shooting moment of the rolling shutter camera, the rolling shutter duration and the shutter duration of the rolling shutter camera, determining the starting moment when the rolling shutter camera waits to start scanning the first group of beads corresponding to the first row of pixels; when i is greater than 1, according to the pixel height of the i-th row of pixels, the rolling shutter parameter, the total pixel height of the imaging plane, and the starting moment when the rolling shutter camera waits to start scanning the first group of beads corresponding to the first row of pixels, determining the starting moment when the rolling shutter camera waits to start scanning the i-th group of beads.

[0008] In a possible implementation manner, determining the display timing of the multiple target LED beads for displaying the endoscope cone view according to the scanned timing of each target LED bead, the shutter duration of the rolling shutter camera, and the continuous display duration of a single LED bead includes: taking the difference between the shutter duration of the rolling shutter camera and the continuous display duration of a single LED bead as the redundant duration; for the i-th group of beads corresponding to the i-th row of pixels, determining the start display moment of the i-th group of beads according to the start moment when the rolling shutter camera is to start scanning the i-th group of beads and the redundant duration, where the start display moment of the i-th group of beads is earlier than the start moment when it is to be started to be scanned by the rolling shutter camera; taking the sum of the start display moment of the i-th group of beads and the continuous display duration of a single LED bead as the end display moment of the i-th group of beads.

[0009] In a possible implementation manner, the rolling duration represents the time difference between the start exposure time of the uppermost row of pixels and the start exposure time of the lowermost row of pixels of the rolling shutter camera. The calibration process of the rolling parameters of the rolling shutter camera includes: during the process of controlling the LED screen to alternately display a first image frame and a second image frame and continuously adjusting the display phase of the LED beads in the LED screen, using the rolling shutter camera to capture the LED screen at a preset acquisition frame rate to obtain an acquisition video; where the picture contents of the first image frame and the second image frame are different; by analyzing the video picture of the acquisition video, obtaining a first moment when the first mixed picture exactly becomes the picture content of the second image frame, and a second moment when the picture content of the second image frame exactly does not show the second mixed picture; where the first mixed picture includes the picture content of the first image frame in the top area of the video picture and the picture content of the second image frame in other areas, and the second mixed picture includes the picture content of the first image frame in the bottom area of the video picture and the picture content of the second image frame in other areas; determining the rolling parameters of the rolling shutter camera according to the first moment, the second moment, the single-frame display duration of the second image frame, and the shutter duration of the rolling shutter camera.

[0010] In a possible implementation, determining the in-view cone image to be displayed on the LED screen and the pixel position information of a plurality of target LED beads within the in-view cone range of the rolling shutter camera in the LED screen according to the pose information of the rolling shutter camera and the screen model of the LED screen includes: determining the relative pose between the virtual camera in the virtual shooting environment and the screen model and the in-view cone image to be displayed on the LED screen according to the pose information of the rolling shutter camera, where the pose and internal parameters of the virtual camera are the same as those of the rolling shutter camera; determining the in-view cone area within the in-view cone range of the virtual camera in the screen model based on the relative pose, and taking the plurality of LED beads within the in-view cone area as target LED beads; projecting each target LED bead in the screen model onto the imaging plane of the virtual camera based on the spatial positions of the respective target LED beads in the screen model to obtain the pixel position information of the plurality of target LED beads within the in-view cone range of the rolling shutter camera.

[0011] In a possible implementation, the in-view cone image to be displayed is any one of at least two images to be alternately displayed. Among them, controlling the color display of the plurality of target LED beads in the LED screen according to the display timing of the in-view cone image displayed by the plurality of target LED beads includes: determining the target display timing for the plurality of target LED beads to alternately display at least two images according to the display timing of any one image displayed by the plurality of target LED beads and the continuous display duration of a single LED bead; controlling the color display of each target LED bead in the LED screen according to the target display timing for the plurality of target LED beads to alternately display at least two images.

[0012] According to another aspect of the present disclosure, there is provided a display control device for an LED screen, which is applied to a virtual shooting scenario in which a rolling shutter camera is used to shoot the LED screen. The device includes: an acquisition module, configured to acquire pose information of the rolling shutter camera during virtual shooting; a first determination module, configured to determine, according to the pose information of the rolling shutter camera and the screen model of the LED screen, an in-view cone picture within the in-view cone range of the rolling shutter camera to be displayed on the LED screen and pixel position information of a plurality of target LED beads within the in-view cone range of the rolling shutter camera in the LED screen, where the pixel position information includes the pixel positions where the target LED beads are projected onto the imaging plane of the rolling shutter camera; a second determination module, configured to determine the scanning timings of the respective target LED beads according to the starting shooting time of the rolling shutter camera, the pixel position information of the plurality of target LED beads, and the rolling shutter parameters of the rolling shutter camera; where the rolling shutter parameters represent the rolling shutter duration of the rolling shutter camera, and the scanning timings include the starting times when the respective target LED beads are started to be scanned by the rolling shutter camera; a third determination module, configured to determine the display timings of the plurality of target LED beads for displaying the in-view cone picture according to the scanning timings of the respective target LED beads, the shutter duration of the rolling shutter camera, and the continuous display duration of a single LED bead, where the display timings represent the starting display times and the ending display times of the respective target LED beads; a display control module, configured to control the plurality of target LED beads in the LED screen to perform color display according to the display timings of the plurality of target LED beads for displaying the in-view cone picture.

[0013] According to another aspect of the present disclosure, there is provided an electronic device, including a memory, a processor, and a computer program stored on the memory, where the processor executes the computer program to implement the steps of the above method.

[0014] According to another aspect of the present disclosure, there is provided a non-volatile computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0015] According to another aspect of the present disclosure, there is provided a computer program product, including a computer program, or a non-volatile computer-readable storage medium carrying the computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0016] According to various aspects of the present disclosure, by obtaining the pose information of the rolling shutter camera, the in-cone view within the in-cone range and the pixel positions of multiple target LED beads can be determined. Then, based on the pixel positions of the respective target LED beads and the rolling shutter parameters of the rolling shutter camera, the scanned timing of each target LED bead scanned by the rolling shutter camera can be obtained. This scanned timing is related to the positions of the respective target LED beads. The later the start time of the rolling shutter camera scanning the target LED bead closer to the bottom. Then, by combining the continuous display duration of a single LED bead, the display timing of each target LED bead adapted to the line-by-line scanning characteristics of the rolling shutter camera can be obtained. Then, controlling each target LED bead to perform color display according to this display timing is equivalent to matching the display of the in-cone view on the screen with the working timing of the line-by-line scanning exposure of the rolling shutter camera. Thus, the rolling shutter camera can be applied to virtual shooting scenarios with higher frame rate requirements for screen refreshing (i.e., shorter continuous display duration of LED beads), without being troubled by the rolling shutter duration (i.e., the rolling time), and greatly improving the display acquisition ratio of the LED beads in the LED screen, that is, improving the utilization rate of the LED screen.

[0017] Other features and aspects of the present disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings, which are included in and constitute a part of this specification, illustrate exemplary embodiments, features, and aspects of the present disclosure together with the specification and are used to explain the principles of the present disclosure.

[0019] Figure 1 FIG. shows a structural diagram of a virtual shooting system according to an embodiment of the present disclosure.

[0020] Figure 2 FIG. shows a schematic diagram of the relationship between the starting exposure time and pixel position of pixels within the in-cone of a rolling shutter camera.

[0021] Figure 3a FIG. shows a schematic diagram of the scanning result when the dynamic phase scheme of the embodiment of the present disclosure is not adopted.

[0022] Figure 3b FIG. shows a schematic diagram of the scanning result when the dynamic phase scheme of the embodiment of the present disclosure is adopted.

[0023] Figure 4 FIG. shows a flowchart of a method for controlling the display of an LED screen according to an embodiment of the present disclosure.

[0024] Figure 5 FIG. shows a schematic diagram of a method for determining a target LED bead according to an embodiment of the present disclosure.

[0025] Figure 6aSchematic diagram showing a rolling curtain parameter calibration system according to an embodiment of the present disclosure.

[0026] Figure 6b Schematic diagram showing a rolling curtain duration analysis process according to an embodiment of the present disclosure.

[0027] Figure 7 Schematic diagram showing the camera scanning situation in the case of simultaneous display of lamp beads according to the related art.

[0028] Figure 8 Schematic diagram showing the camera scanning situation in the case of controlling the lamp beads to be displayed sequentially based on the display timing according to an embodiment of the present disclosure.

[0029] Figure 9 Block diagram showing another virtual shooting system proposed by an embodiment of the present disclosure.

[0030] Figure 10 Block diagram showing a display control device for an LED screen according to an embodiment of the present disclosure.

[0031] Figure 11 Block diagram showing an electronic device 1900 according to an embodiment of the present disclosure. Detailed implementation manners

[0032] Hereinafter, various exemplary embodiments, features, and aspects of the present disclosure will be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements having the same or similar functions. Although various aspects of the embodiments are shown in the drawings, the drawings do not have to be drawn to scale unless otherwise specified.

[0033] As used herein, the terms "comprising", "including", "having", or variations thereof are open-ended and include one or more stated features, wholes, elements, steps, components, or functions, but do not exclude the presence or addition of one or more other features, wholes, elements, steps, components, functions, or groups thereof.

[0034] When an element is referred to as being "connected", "coupled", "responsive" or variations thereof to another element, it can be directly connected, coupled, or responsive to the other element, or there can be intermediate elements.

[0035] Although the terms first, second, third, etc. may be used herein to describe various elements / operations, these elements / operations should not be limited by these terms. These terms are only used to distinguish one element / operation from another element / operation. Thus, without departing from the teachings of the inventive concept, a first element / operation in some embodiments can be referred to as a second element / operation in other embodiments.

[0036] As used herein, the term "exemplary" means "serving as an example, embodiment, or illustration". Any embodiment described herein as "exemplary" should not necessarily be construed as superior to or better than other embodiments.

[0037] In addition, for a better illustration of the present disclosure, numerous specific details are given in the following detailed description. Those skilled in the art should understand that the present disclosure can also be implemented without some of these specific details. In some instances, methods, means, elements, and circuits well known to those skilled in the art are not described in detail so as to highlight the gist of the present disclosure.

[0038] Figure 1 FIG. shows a structural diagram of a virtual shooting system according to an embodiment of the present disclosure; as Figure 1 shown, the virtual shooting system may include: a main control device 101, an Unreal rendering device 102, a broadcast control device 103, an LED screen 104, a shooting device 105, and a motion capture device 106; wherein, the number of various devices can be flexibly configured according to actual needs, and no limitation is imposed thereon; in addition, in practical applications, other devices may also be configured in the virtual shooting system according to needs, such as a mobile terminal or a network device, etc.; one or more devices in the virtual shooting system can also be integrated into one device according to requirements, for example, the main control device 101, the Unreal rendering device 102, and the broadcast control device 103 can be integrated into one device.

[0039] Among them, the main control device 101 can be connected to the Unreal rendering device 102, and the motion capture device 106 can be connected to the main control device 101. The specific connection method can be selected according to actual needs and device compatibility. For example, wired or wireless connections can be made through a local area network or the Internet, and network transmission protocols can be used for communication. The motion capture device 106 can capture the pose information of the shooting device 105 and send it to the main control device 101. The main control device 101 can send control instructions to the Unreal rendering device 102 connected thereto based on the pose information of the shooting device 105 to control the Unreal rendering device 102 to render the picture to be displayed on the LED screen 104; alternatively, the motion capture device 106 can also be connected to the Unreal rendering device 102 to directly send the captured pose information to the Unreal rendering device, so that the Unreal rendering device 102 performs picture rendering based on the pose information of the shooting device 105.

[0040] Among them, the virtual rendering device 102 can be deployed with a rendering engine, such as the UE (Unreal Engine) engine. In the rendering engine, a virtual shooting environment consistent with the real shooting environment (including a virtual camera, a screen model, etc.) can be constructed. Among them, the pose of the internal parameters of the virtual camera is consistent with the internal parameters and pose of the shooting device 105, and the positional relationship between the virtual camera and the screen model is consistent with the positional relationship between the shooting device 105 and the LED screen 104; the screen model can be understood as a screen model that restores the real LED screen 104 1:1, and the shape, size, number of boxes and layout of the screen model, the number and layout of the light beads on each box, etc. are all consistent with the real LED screen 104.

[0041] In practical applications, those skilled in the art can calibrate the internal parameters of the shooting device 105 through known camera calibration techniques in the art, and can use known model construction techniques in the art to construct the screen model of the real LED screen 104. The embodiments of the present disclosure do not limit this. And known spatial calibration techniques in the art can be used to calibrate the positional relationship between the virtual camera and the screen model to be consistent with the positional relationship between the real shooting device 105 and the real LED screen 104.

[0042] Among them, the virtual rendering device 102 can be connected to the broadcast control device 103, and the broadcast control device 103 is connected to the LED screen 104; the specific connection method can be selected according to actual needs and device compatibility. Among them, the broadcast control device 103 can be used for data transmission and decoding. For example, the broadcast control device 103 can receive the rendered image from the virtual rendering device 102 and decode it into a format suitable for display on the LED screen 104, and can also send control instructions to the LED screen 104 to control and manage the LED screen 104 connected thereto, such as overall control and scheduling of the LED screen 104, including brightness adjustment, color correction, gray level control, etc.; it can also be used for zoning management of the LED screen 104. The LED screen 104 can be divided into multiple independent areas, and different contents can be displayed in each area. It can also control the display color, display duration, etc. of any LED light bead in the LED screen.

[0043] Among them, the LED screen 104 can be in the structure of a curved screen or a flat screen, etc. The type, number, size, resolution, etc. of the LED screen 104 in the virtual shooting system can be custom-set according to actual needs, and this is not limited. It can be known that the LED screen is usually composed of multiple LED boxes, and each LED box contains evenly arranged LED light beads at equal intervals. Each LED light bead can be used for color display. By controlling the color and duration of each LED light bead, it is possible to control the LED screen to display the corresponding image and the display duration of each frame of the image.

[0044] Among them, the master control device 101 can also be connected to the shooting device 105; the specific connection method can be selected according to actual needs and device compatibility. The shooting device 105 can transmit the captured video to the master control device 101 to display the shooting result to the user in the master control device 101. The embodiments of the present disclosure do not limit the type of the shooting device 105; as an example, according to different lenses used, the shooting device 105 can be a telephoto camera, a wide-angle camera, etc.; as another example, according to different shutters used, the shooting device 105 can be a rolling shutter camera, a global shutter camera, etc.; among them, the rolling shutter camera is configured with a rolling shutter, and the rolling shutter scans and exposes image data row by row in sequence; the global shutter camera is configured with a global shutter, and the global shutter can expose all pixels on the image simultaneously.

[0045] During the virtual shooting process, the master control device 101 in the virtual shooting system controls the virtual rendering device 102 to perform screen rendering, that is, it can drive the virtual camera in the rendering engine to render and generate a virtual image and project the virtual image onto the screen model after three-dimensional projection transformation, and then map it onto the LED screen 104, that is, project the rendered image onto the LED screen 104 through the playback control device 103 for display (that is, realize the image on the screen). At the same time, the actor can perform in front of the LED screen 104, and the master control device 101 can control the shooting device 105 (such as a rolling shutter camera) to shoot. In the captured video, the actor is the foreground and the image on the LED screen is the background, thus completing the virtual shooting.

[0046] In practical applications, a synchronization signal generator can be configured in the virtual shooting system to achieve signal synchronization between the shooting device 105 and the playback control device 103, that is, to achieve synchronization between the screen display of the LED screen and the video shooting of the shooting device 105, so that when the playback control device 103 controls the LED screen 104 to perform screen display, the shooting device 105 can capture the complete image.

[0047] Based on the above virtual shooting system, when the shooting device 105 uses a rolling shutter camera, such as Figure 2As shown, the rolling shutter camera captures the LED screen. Due to the rolling shutter characteristic of the camera, on the inner view cone image of the rolling shutter camera, the lower the vertical position (i.e., the lower the row coordinate of the camera image), the later the starting exposure time of each row of pixels by the camera. The exposure of the last row of pixels (i.e., the pixels corresponding to the maximum number of rows H of the image) is the latest. Among them, the starting exposure time T_shutter_start(last_row) of the last row of pixels is equal to the sum of the starting exposure time T_shutter_start(first_row) of the topmost row of pixels and a rolling shutter duration T_roll. That is, T_shutter_start(last_row) = T_shutter_start(first_row) + T_roll. Then, the difference between the starting exposure times of the topmost row and the last row of pixels is one rolling shutter time (T_roll, i.e., the rolling shutter duration). In this case, for the rolling shutter camera to perform well in shooting, generally, the continuous display duration of the LED lamp beads, that is, the duration of one frame of image display (T_slice_led), needs to satisfy: T_slice_led ≥ T_roll + t_exp, where t_exp is the shutter duration. The shutter duration refers to the time interval from when the camera shutter is opened to when it is closed, which is used to control the duration of light irradiating the image sensor and affects the image brightness and dynamic performance. If we assume that T_roll is 10 ms and t_exp is 5 ms, then T_slice_led needs to be guaranteed to be above 15 ms. This is not suitable for virtual shooting scenarios with higher frame rate content refresh requirements (i.e., high screen refresh rate). At this time, other solutions may be considered, such as using a global shutter camera.

[0048] Taking a single-camera green screen insertion shooting as an example (a typical scenario in virtual shooting. To balance the flexibility in the later stage, both the background-containing image and the green background image need to be captured. In this way, the content displayed on the LED screen alternates between the content and the green screen. After shooting, the entire video is divided into a content background and a green screen background through odd-even frame splitting), assuming that the acquisition frame rate of the rolling shutter camera is 100 fps (equivalent to requiring a screen refresh rate of 100 fps), then the display frame rates of the green screen and the content are both 50 fps. Let the shutter duration be 5 ms and the rolling shutter parameter be 10 ms. If the display control method proposed in this embodiment of the present disclosure is not adopted, each frame of image requires the LED lamp beads to display for 15 ms, but the acquisition frame rate of 100 fps requires that each frame of content lasts at most 10 ms.

[0049] To solve the above problems and enable the rolling shutter camera to be free from the rolling shutter time constraint and meet the shooting requirements of more high-refresh-rate LED screen scenarios, the embodiments of the present disclosure propose a display control method for a rolling shutter camera to shoot an LED screen, which can overcome the influence of the rolling shutter duration of the rolling shutter camera, greatly improve the display acquisition ratio of the LED screen, and thus meet the advanced shooting requirements of various virtual shooting business scenarios. The core of the method lies in adjusting the display time of each LED bead on the LED screen according to the exposure time of the rolling shutter camera to it, which can be a dynamic phase adjustment scheme for LED beads, simply referred to as the dynamic phase scheme. After adopting the dynamic phase scheme proposed in the embodiments of the present disclosure, in the above single-camera green insertion shooting example, each frame of the picture only requires a single LED bead to display for 5 ms. Considering some redundant time, it can be assumed that a single LED bead displays for 8 ms. Then the display frame rate can reach 125 fps, and at this time, the rolling shutter camera can be used for normal shooting. That is, by using the display control method of the embodiments of the present disclosure in virtual shooting, only the display timing of each row of target LED beads needs to be calculated in real time, and according to this display timing, each target LED bead is controlled to perform color display, and the rolling shutter camera can be used to complete virtual shooting at a high screen refresh rate.

[0050] Exemplarily, Figure 3a A schematic diagram showing the scanning result when the dynamic phase scheme of the embodiments of the present disclosure is not adopted. Figure 3b A schematic diagram showing the scanning result when the dynamic phase scheme of the embodiments of the present disclosure is adopted, where the gray area represents the display time when the bead is lit, the small horizontal line represents the time when the bead is scanned by the rolling shutter camera, the duration occupied by a row of small horizontal lines represents the shutter duration (i.e., 5 ms), and the red vertical line represents the bead that cannot be normally photographed by the rolling shutter camera. As Figure 3a shown, when the dynamic phase scheme is not adopted, each LED bead within the inner cone range of the rolling shutter camera is lit simultaneously and continuously displays for 10 ms. The closer the LED bead is to the bottom of the cone, the later it is scanned by the rolling shutter camera. And because the delay of the rolling shutter camera's progressive scanning (i.e., it takes until the 14th ms to complete the scanning) is greater than the LED bead's continuous display duration of 10 ms, the rolling shutter camera has not completed the scanning yet, and the LED beads near the bottom of the cone have already gone out, indicating that the current shooting of the rolling shutter camera cannot meet the 10 ms continuous display duration of the LED beads. And as Figure 3b shown, when the dynamic phase scheme is adopted, the scanning time of each LED bead is within the display time of the LED bead. In this way, through the arrangement of different display times for different LED beads, it is possible to use the rolling shutter camera to achieve shooting with a 5 ms shutter under the condition that the LED beads continuously display for 8 ms.

[0051] The following is a detailed introduction to the display control method (i.e., the dynamic phase scheme) of the LED screen proposed in the embodiments of the present disclosure through Figures 4 to 9 this.

[0052] Figure 4 The flowchart shows a display control method for an LED screen according to an embodiment of the present disclosure. This method is applied to a virtual shooting scenario where a rolling shutter camera is used to shoot an LED screen. In this virtual shooting scenario, for example, the main control device 101 in the above virtual shooting system can execute this method. Of course, it can also be executed by other terminal devices or servers, and the embodiments of the present disclosure do not limit this. As Figure 4 shown, the method includes: step S11 to step S15.

[0053] In step S11, during the virtual shooting process, obtain the pose information of the rolling shutter camera.

[0054] As described above, the virtual shooting system may include a motion capture device. The motion capture device may adopt a motion capture system such as OptiTrack, and the motion capture device can be used to collect the pose information of the rolling shutter camera (i.e., the position and orientation of the rolling shutter camera). Of course, other known pose capture technologies in the art can also be used, such as wheel odometers and inertial measurement units, etc., and the embodiments of the present disclosure do not limit this.

[0055] In step S12, according to the pose information of the rolling shutter camera and the screen model of the LED screen, determine the frustum plane that is within the frustum range of the rolling shutter camera and to be displayed on the LED screen, and the pixel position information of multiple target LED beads within the frustum range of the rolling shutter camera in the LED screen. Among them, the pixel position information includes the pixel positions where the target LED beads are projected onto the imaging plane of the rolling shutter camera, and multiple target LED beads are to display the frustum plane.

[0056] As described above, a virtual shooting environment consistent with the real shooting environment can be constructed in an Unreal rendering engine. The virtual shooting environment includes a virtual camera and a screen model. The camera internal parameters of the virtual camera (such as lens, focal length, etc.) are consistent with those of the rolling shutter camera, and the pose of the virtual camera is the same as that of the rolling shutter camera. The positional relationship between the virtual camera and the screen model is consistent with the positional relationship between the rolling shutter camera and the LED screen. Therefore, the above-mentioned determining the frustum plane that is within the frustum range of the rolling shutter camera and to be displayed on the LED screen, and the pixel position information of multiple target LED beads within the frustum range of the rolling shutter camera in the LED screen according to the pose information of the rolling shutter camera and the screen model of the LED screen includes:

[0057] According to the pose information of the rolling shutter camera, determine the relative pose between the virtual camera and the screen model in the virtual shooting environment and the frustum plane to be displayed on the LED screen;

[0058] Determine the inner cone region within the inner cone range of the virtual camera in the screen model based on the relative pose, and use multiple LED beads within the inner cone region as target LED beads;

[0059] Based on the spatial positions of the respective target LED beads in the screen model, project the respective target LED beads in the screen model onto the imaging plane of the virtual camera to obtain the pixel position information of multiple target LED beads within the inner cone range of the rolling shutter camera.

[0060] It should be understood that given the pose information of the rolling shutter camera, that is, knowing the pose of the virtual camera in the virtual shooting environment, and the position of the screen model in the virtual shooting environment remains unchanged. Thus, the relative pose between the virtual camera and the screen model is known. At the same time, based on the inner cone range of the virtual camera in this relative pose, the image can be rendered, that is, the inner cone image to be displayed on the LED screen can be obtained. At the same time, the coordinates of the inner cone region within the inner cone range of the virtual camera in the screen model (that is, the regional position of the inner cone of the rolling shutter camera on the LED screen) can also be obtained. Then, multiple LED beads within the inner cone region of the screen model can be used as target LED beads, that is, multiple target LED beads for displaying the inner cone image within the inner cone range of the rolling shutter camera on the LED screen can be obtained.

[0061] Among them, projecting the respective target LED beads in the screen model onto the imaging plane of the virtual camera is equivalent to using the rolling shutter camera to project the target LED beads in the three-dimensional space onto the imaging plane of the rolling shutter camera to obtain the pixel positions of the respective target LED beads on the imaging plane. For example, based on the pose and internal parameters of the virtual camera, the imaging mapping relationship from the three-dimensional space to the two-dimensional plane can be determined, and then based on this mapping relationship, the spatial positions of the respective target LED beads can be converted into pixel positions. In the case where the resolution of the rolling shutter camera is lower than the resolution of the screen, there may be a situation where multiple LED beads are projected onto the same pixel position.

[0062] Exemplarily, Figure 5 The schematic diagram showing a method for determining target LED beads is as Figure 5As shown, the LED screen can be spatially calibrated in advance to obtain the position information of the LED screen (i.e., obtain the position of the screen model in the virtual shooting environment). During the shooting process, a spatial tracking device (i.e., a motion capture device) can be used to track the pose information (i.e., tracking data) of the rolling shutter camera and send it to a computing device (equivalent to the main control device and the rendering device). The computing device can determine the relative pose between the virtual camera and the screen model in the virtual shooting environment based on the pose information of the rolling shutter camera and the position information of the LED screen. Furthermore, the coordinates of the inner frustum region within the inner frustum range of the rolling shutter camera in the LED screen (i.e., the frustum screen area coordinates) can be determined. Then, the box coordinates within the inner frustum region in the LED screen (i.e., box positioning) can be located, and thus the spatial coordinates of each target LED bead on each box within the inner frustum region are obtained. Furthermore, based on the mapping relationship between the LED screen projected onto the imaging plane of the rolling shutter camera, the pixel coordinates of each target LED bead can be obtained through coordinate mapping. Among them, at the same time, the computing device can also perform frustum rendering (i.e., render the inner frustum image) based on the pose information of the rolling shutter camera.

[0063] In step S13, according to the starting shooting time of the rolling shutter camera, the pixel position information of multiple target LED beads, and the rolling shutter parameter of the rolling shutter camera, the scanning time sequence of each target LED bead is determined; wherein, the rolling shutter parameter represents the rolling shutter duration of the rolling shutter camera, and the scanning time sequence includes the starting time when each target LED bead starts to be scanned by the rolling shutter camera.

[0064] Among them, the starting shooting time of the rolling shutter camera can be understood as the time when the rolling shutter camera starts shooting. It should be understood that this starting shooting time is known; wherein, the rolling shutter duration represents the time difference between the starting exposure time of the topmost row of pixels and the starting exposure time of the bottommost row of pixels of the rolling shutter camera, or, represents the time difference between the start of exposure of the topmost row of pixels and the start of exposure of the bottommost row of pixels of the rolling shutter camera. This rolling shutter parameter can be calibrated in advance. For example, the high-speed moving object method (i.e., using the deformation of an object with a known speed (such as a rotating disk, a fast-moving LED lamp) in the image to calculate the rolling shutter duration through geometric relationships), the synchronous flashing light source method (i.e., by controlling the flashing frequency of the light source to be asynchronous with the row scanning of the rolling shutter, observing the width of the light band in the image to calculate the rolling shutter duration), the direct measurement method of a high-speed camera (i.e., using a high-speed camera with a higher frame rate to synchronously shoot the working process of the rolling shutter, directly recording the start and end times of each row of exposure, and then calculating the rolling shutter duration), etc.

[0065] In a possible implementation manner, the calibration process of the rolling shutter parameter of the rolling shutter camera includes:

[0066] During the process of controlling the LED screen to alternately display the first image frame and the second image frame and continuously adjusting the display phase of the LED beads in the LED screen, a rolling shutter camera is used to capture the LED screen at a preset capture frame rate to obtain a captured video; wherein, the picture contents of the first image frame and the second image frame are different;

[0067] By analyzing the video picture of the captured video, the first moment when the first mixed picture exactly becomes the picture content of the second image frame and the second moment when the picture content of the second image frame exactly does not show the second mixed picture are obtained; wherein, the first mixed picture includes that the top area of the video picture is the picture content of the first image frame and the other areas are the picture content of the second image frame, and the second mixed picture includes that the bottom area of the video picture is the picture content of the first image frame and the other areas are the picture content of the second image frame;

[0068] According to the first moment, the second moment, the single-frame display duration of the second image frame, and the shutter duration of the rolling shutter camera, the rolling shutter parameters of the rolling shutter camera are determined.

[0069] Exemplarily, such as Figure 6aA rolling shutter parameter calibration system is shown. The LED screen alternately displays the content of frame A (i.e., the first image frame) and frame B (i.e., the second image frame). There should be an obvious difference between the picture contents of frame A and frame B. For example, frame A is a picture filled with diagonal lines, and frame B is a picture filled with a single color. The synchronization signal generator is used for time synchronization between the rolling shutter camera and the screen controller (i.e., the playback control device). After the rolling shutter camera starts shooting, the screen controller can continuously adjust the display phase of the LED beads in the LED screen (equivalent to continuously adjusting the start display time of the first image frame and the second image frame). For example, assuming that the preset acquisition frame rate of the rolling shutter camera is fixed at 50 fps, and it is set that frame A and frame B are alternately displayed for 20 ms each. After the rolling shutter camera starts shooting and keeping the acquisition frame rate of the rolling shutter camera unchanged, the screen controller can make the start display times of frame A and frame B shift continuously relative to the shutter time of the rolling shutter camera (it can be shifted earlier or later) by adjusting the display phase of the LED beads in the LED screen (that is, adjusting the offset of the LED bead lighting time). During this process, the boundary between frame A and frame B in the video picture captured by the rolling shutter camera moves continuously, and there will be situations where the bottom of the frame B picture just appears the AB mixed area or the bottom AB mixed area just disappears, and the top of the frame B picture just appears the AB mixed area or the top AB mixed area just disappears, etc. Therefore, by analyzing the video picture of the captured video, the first moment when the first mixed picture just becomes the picture content of the second image frame can be obtained, that is, the moment when the top AB mixture just disappears, and the second moment when the picture content of the second image frame just does not show the second mixed picture, that is, the moment when the bottom AB mixed area just does not appear (i.e., the moment before the bottom AB mixed area just appears); or rather, assuming that during the continuous adjustment of the display phase of the LED beads, the content of the frame B picture in the video picture is moving downward continuously, then the moment when the lower boundary of the frame B picture just touches the bottom can be recorded as t1. As the display phase of the LED beads is continuously adjusted, the content of the frame B picture can move downward at a constant speed, then the moment when the upper boundary of the frame B picture just touches the top can be recorded as t2.

[0070] More vividly, as Figure 6b shown, keeping the acquisition frame rate of the rolling shutter camera unchanged and adjusting the display phase of the LED beads is also equivalent to keeping the trapezoidal shadow in the figure fixed and moving the position of the rectangle in the figure left and right. Thus, the first moment t1 when the top AB mixture just disappears and the second moment t2 when the bottom AB mixture just does not appear can be obtained. Combining the single-frame display duration of the second image frame (i.e., the frame duration t_pic) and the shutter duration of the rolling shutter camera (i.e., t_exp), the rolling duration t_roll of the rolling shutter camera can be calculated as: t_roll = t_pic - t_exp - |t2 - t1|.

[0071] Based on the rolling shutter parameters obtained from the above calibration, in a possible implementation, step S13, determining the scanned timing of each target LED bead according to the starting shooting moment of the rolling shutter camera, the pixel position information of multiple target LED beads, and the rolling shutter parameters of the rolling shutter camera, may include:

[0072] Step S131, for the i-th row of pixels on the imaging plane of the rolling shutter camera, determine the i-th group of beads corresponding to the i-th row of pixels according to the pixel position information of each target LED bead, where the i-th group of beads includes multiple target LED beads projected onto the i-th row of pixels, 1 ≤ i ≤ N, and N is the total pixel height of the imaging plane;

[0073] Step S132, when i is 1, determine the starting moment when the rolling shutter camera waits to start scanning the first group of beads corresponding to the first row of pixels according to the starting shooting moment of the rolling shutter camera, the rolling shutter duration, and the shutter duration of the rolling shutter camera;

[0074] Step S133, when i is greater than 1, determine the starting moment when the rolling shutter camera waits to start scanning the i-th group of beads corresponding to the i-th row of pixels according to the pixel height of the i-th row of pixels, the rolling shutter parameters, the total pixel height of the imaging plane, and the starting moment when the rolling shutter camera waits to start scanning the first group of beads corresponding to the first row of pixels.

[0075] Among them, the i-th row of pixels on the imaging plane can be understood as the i-th row of pixels in the sensor array of the rolling shutter camera. The total pixel height of the imaging plane represents the total number of rows of sensor units in the sensor array of the rolling shutter camera. One sensor unit can represent one pixel. Knowing the pixel positions of each target LED bead is equivalent to knowing which row of pixels (i.e., sensor units) on the imaging plane each target LED bead is projected onto. Thus, the corresponding relationship between each target LED bead and each pixel in the imaging plane can be obtained, and thus any group of beads corresponding to any row of pixels can be obtained. This group of beads can be located in different rows and columns on the screen, as long as they are all projected onto the same row of pixels on the imaging plane, they are a group of beads corresponding to that row of pixels.

[0076] It should be understood that the acquisition frame rate of the rolling shutter camera is fixed and the starting shooting moment is known. It is possible to calculate the starting moment when the rolling shutter camera starts scanning the first group of light beads corresponding to the first row of pixels during the display of any frame of the frustum image on the LED screen. For example, assume that the starting shooting moment of the rolling shutter camera is 0 ms. During the first shooting cycle within the display of the frustum image in the first frame, the starting moment when the rolling shutter camera starts scanning the first group of light beads can be 0 ms. If the rolling shutter duration of the rolling shutter camera is 10 ms and the shutter duration is 5 ms, then the shooting cycle of the rolling shutter camera is the rolling shutter duration + the shutter duration, which is 15 ms (this means that the time taken for the rolling shutter camera to complete the shooting of one frame of the image is 15 ms). During the second shooting cycle within the display of the frustum image in the second frame, the starting moment when the rolling shutter camera starts scanning the first group of light beads can be 15 ms, that is, shooting is performed every 15 ms, and so on. Based on the shooting cycle corresponding to the frame sequence of the current frame of the frustum image, the starting moment when the rolling shutter camera starts scanning the first group of light beads corresponding to the first row of pixels within the shooting cycle where the current frame of the frustum image is located can be obtained. That is to say, between two adjacent frames, the time difference between the starting moments of scanning the same group of light beads is at least the rolling shutter duration + the shutter duration to ensure the complete shooting process.

[0077] As described above, the LED beads of the LED screen box are evenly arranged at equal intervals. Assuming that all the target LED beads of the LED screen display the inner cone image at the 0th moment, and the first group of LED beads starts to be scanned by the rolling shutter camera at the 0th moment, then after knowing the rolling shutter duration t_roll of the rolling shutter camera, the start time t_shutter_start_i of the ith group of LED beads corresponding to the ith row of pixels being scanned can be expressed as: t_shutter_start_i = [t_roll / (N - 1)]×(i - 1). For example, when the rolling shutter duration is 3 ms, there are 3 rows of pixels in total. The start time of the first group of LED beads being scanned is 0, the start time of the second group of LED beads corresponding to the second row of pixels being scanned is [3 / (3 - 1)]×(2 - 1) = 1.5 ms, and the start time of the third group of LED beads corresponding to the third row of pixels being scanned is [3 / (3 - 1)]×(3 - 1) = 3 ms; if the start time of the rolling shutter camera starting to scan the first group of LED beads is expressed as t_shutter_start_0 (i.e., not 0), then the start time of the ith group of LED beads being scanned by the rolling shutter camera can be calculated as: t_shutter_start_i = t_shutter_start_0 + [t_roll / (N - 1)]×(i - 1). In this formula, "i" represents the pixel height of the ith row of pixels, N is the total pixel height of the imaging plane, t_roll is the rolling shutter parameter (i.e., the rolling shutter duration), and t_shutter_start_0 represents the start time of the first group of LED beads corresponding to the first row of pixels being scanned; among them, the start time of the ith group of LED beads can be synchronized with the clock signal of the LED screen, that is, it can be corresponding to the screen timestamp of the LED screen, so as to facilitate the LED screen display and camera shooting under the unified clock.

[0078] In step S14, according to the scanning timings of the respective target LED beads, the shutter duration of the rolling shutter camera, and the continuous display duration of a single LED bead, the display timings of the multiple target LED beads for displaying the inner cone image are determined, where the display timings characterize the respective start display times and end display times of the respective target LED beads.

[0079] In practical applications, given the start time when the rolling shutter camera begins to scan each target LED bead, the end time when each target LED bead finishes scanning can be expressed as the start time of each target LED bead plus the shutter duration t_exp of the rolling shutter camera. For example, based on the start time of the i-th group of beads being scanned as t_shutter_start_i, the end time when the i-th group of beads finishes scanning is t_shutter_start_i + t_exp. This represents that the scanning time of the i-th group of beads is in the time range from the start time t_shutter_start_i to the end time t_shutter_start_i + t_exp. Thus, the duration t_led_a_frame for the content of the frustum view to be continuously displayed on each row of beads only needs to satisfy: t_led_a_frame ≥ t_exp; and as Figure 7 shown, if the display time of all target LED beads is the same, that is, the start time t_led_start when N groups of beads (i.e., bead 1, bead 2,..., bead i,..., bead N) start to display A frames of the picture (frustum view) is the same, in order for the rolling shutter camera to capture a complete frustum view, the display duration t_led_a_frame of each frame of the picture should at least satisfy: t_led_a_frame ≥ t_exp + t_roll.

[0080] In practical applications, if t_led_a_frame = t_exp, the start display time of the i-th group of LED beads can be the start time t_shutter_start_i when the rolling shutter camera starts scanning, and the end display time can be the start time t_shutter_start_i + the continuous display duration t_led_a_frame, which is equal to t_shutter_start_i + t_exp. For example, taking the rolling shutter duration as 10 ms and the shutter duration as 5 ms as an example, with the starting scanning time being 0 ms, if it is calculated that the start times when the rolling shutter camera starts scanning the first group of LED beads in multiple shooting cycles are 0 ms, 15 ms, 30 ms, 45 ms... respectively, and the start times when the rolling shutter camera starts scanning the second group of LED beads are 0.1 ms, 15.1 ms, 30.1 ms, 45.1 ms... respectively, if t_led_a_frame = t_exp = 5 ms, the display timings of the first group of LED beads can include: 0 ms - 5 ms, 15 ms - 20 ms, 30 ms - 35 ms, 45 ms - 50 ms...; the display timings of the second group of LED beads can include: 0.1 ms - 5.1 ms, 15.1 ms - 20.1 ms, 30.1 ms - 35.1 ms, 45.1 ms - 50.1 ms...; and so on for other groups of LED beads. That is, the display timings of each target LED bead within the inner viewing cone are obtained, and then the content of each frame such as the first frame, the second frame, the third frame, the fourth frame, etc. of each group of LED beads can be controlled to be displayed in sequence according to this display timing.

[0081] If t_led_a_frame - t_exp > 0, due to error factors such as camera pose tracking error and LED bead positioning error, in this case, the redundant time difference between t_led_a_frame and t_exp can be used to enhance the fault tolerance of the scanning time of the LED beads, that is, the display time of each group of LED beads can be made longer than the scanning time of each group of LED beads. For example, the start display time of each group of LED beads can be advanced relative to the start time when each group of LED beads is scanned, and the advanced time can be set as needed, as long as it can ensure that the end display time is later than the end time when each group of LED beads is scanned. Thus, in a possible implementation manner, step S14 above, determining the display timing when the multiple target LED beads display the inner viewing cone image according to the scanned timings of the respective target LED beads, the shutter duration of the rolling shutter camera, and the continuous display duration of a single LED bead, includes:

[0082] Step S141, taking the difference between the shutter duration of the rolling shutter camera and the continuous display duration of a single LED bead as the redundant duration;

[0083] Step S142: For the i-th group of lamp beads corresponding to the i-th row of pixels, determine the start display time of the i-th group of lamp beads according to the start time when the rolling shutter camera is about to start scanning the i-th group of lamp beads and the redundancy duration, where the start display time of the i-th group of lamp beads is earlier than the start time when it is about to be scanned by the rolling shutter camera.

[0084] Step S143: Use the sum of the start display time of the i-th group of lamp beads and the continuous display duration of a single LED lamp bead as the end display time of the i-th group of lamp beads.

[0085] In step S141, the redundancy duration can be expressed as t_redudancy = t_led_a_frame - t_exp. Then, in step S142, the start display time of the i-th group of lamp beads corresponding to the i-th row of pixels can be optimized to: t_led_start_i_better = t_shutter_start_i - t_redudancy / 2 =

[0086] t_shutter_start_0 + [t_roll / (N - 1)]×(i - 1) - t_redudancy / 2, which is equivalent to advancing the start display time of the i-th group of LED beads by 0.5 times the redundancy duration relative to the start time when the i-th group of LED beads is scanned. Of course, it can also be advanced to 0.4 or 0.6 times the redundancy duration, and the embodiments of the present disclosure do not limit this; furthermore, in step S143, the end display time of the i-th group of LED beads corresponding to the i-th row of pixels can be expressed as: t_led_end_i_better = t_led_start_i_better + t_led_a_frame, that is, the i-th group of LED beads starts to display from t_led_start_i_better to t_led_end_i_better when displaying the inner cone image, and the rolling shutter camera starts to scan the i-th group of LED beads at t_shutter_start_i and ends the scan at t_shutter_start_i + t_exp. After the i-th group of LED beads finishes displaying the color of the current frame, it can continue to display the color required for the next frame of the inner cone image. For example, taking the rolling shutter duration as 10 ms and the shutter duration t_exp as 5 ms as an example, with the starting scan time being 0 ms, if it is calculated that the start times when the rolling shutter camera starts to scan the first group of LED beads in multiple shooting cycles are 0 ms, 15 ms, 30 ms, 45 ms..., and the start times when it starts to scan the second group of LED beads are 0.1 ms, 15.1 ms, 30.1 ms, 45.1 ms..., if the continuous display duration t_led_a_frame of a single LED bead is 10 ms, it means the redundancy duration is t_redudancy = t_led_a_frame - t_exp = 5 ms, and t_redudancy / 2 = 2.5 ms. Then, if the LED screen also starts to display from 0 ms, the display timings of the first group of LED beads can include: 0 ms - 10 ms, 12.5 ms - 22.5 ms, 27.5 ms - 37.5 ms, 42.5 ms - 52.5 ms..., and the display timings of the second group of LED beads can include: 0.1 ms - 10.1 ms, 12.6 ms - 22.6 ms, 27.6 ms - 37.6 ms, 42.6 ms - 52.6 ms..., and so on for other groups of LED beads. That is, the display timings of each target LED bead within the inner cone range are obtained, and then each group of LED beads can be controlled to display the content of each frame such as the first frame, the second frame, the third frame, and the fourth frame in sequence according to this display timing.

[0087] Exemplarily, Figure 8 shows the display timings of each group of LED beads determined by using the method of the embodiments of the present disclosure. Compared with Figure 7In this regard, the start display times of each group of lamp beads are different and related to the positions of each group of lamp beads. The lower the lamp beads are, the later their start display times are. This can make the display times of each group of lamp beads match the progressive scan timing of the rolling shutter camera. Or rather, the display follows the progressive scan timing of the rolling shutter camera, so that the rolling shutter camera is not troubled by the rolling shutter duration, and the display acquisition ratio of the lamp beads in the LED screen can also be increased, which is equivalent to improving the utilization rate of the LED lamp beads.

[0088] In step S15, according to the display timing when multiple target LED lamp beads display the inner frustum view, control the multiple target LED lamp beads in the LED screen to perform color display.

[0089] In practical applications, after calculating the display timing (which can also be called the display phase) when multiple target LED lamp beads display the inner frustum view, the display timing can be sent to the playback control device (or the screen controller) in the virtual shooting system, so as to use the playback control device to control the color display of each target LED lamp bead in the LED screen according to the display timing, that is, to dynamically light up each target LED lamp bead, or to control each target LED lamp bead to display the required content within its respective display time. It should be understood that the embodiments of the present disclosure do not limit the control method of the LED lamp beads, as long as the display is performed according to the specified display timing, and the embodiments of the present disclosure do not limit this.

[0090] It should be understood that if the pose of the rolling shutter camera does not change during the shooting process, the respective target LED lamp beads within the inner frustum range of the rolling shutter camera are also fixed. In this case, the color display of each target LED lamp bead can be controlled according to the determined display timing. If the pose of the rolling shutter camera changes during the shooting process, multiple target LED lamp beads within the inner frustum range of the rolling shutter camera in the current pose can be re-determined according to the above steps S11 to S14, and the display timing of each target LED lamp bead can be determined, and the color display of the newly determined multiple target LED lamp beads can be controlled according to the newly determined display timing of each target LED lamp bead.

[0091] It should be understood that the inner frustum range of the rolling shutter camera may be smaller than the LED screen. For each target LED lamp bead within the inner frustum range, it can be controlled according to the display timing. For other LED lamp beads outside the inner frustum range, a certain outer frustum view can be fixedly displayed at the same time. This outer frustum view will not be captured by the rolling shutter camera and mainly serves the purpose of lighting. Therefore, the content and display method of this outer frustum view are not limited. Also, when controlling the multiple target LED lamp beads in the LED screen to perform color display, the rolling shutter camera can be controlled to shoot the LED screen and the actors, scenery, etc. in front of the LED screen at the same time, so as to realize virtual shooting with a combination of virtual and real.

[0092] In practical applications, the need for multi-background shooting may arise in virtual shooting. For example, the need for single-camera green screen shooting, which is a typical scenario in virtual shooting. To ensure flexibility in the later stage, it is necessary to capture both the content background and the green screen background. In this way, the LED screen needs to alternately display the virtual content background and the green screen background. After shooting, the entire video can be divided into two videos, one with the content background and the other with the green screen background, by means of odd-even frame splitting. Thus, the inner cone picture to be displayed can be any one of at least two pictures to be alternately displayed (for example, it can be the virtual content background picture). Therefore, the display timing of any one picture can be determined first, and then the other pictures can be displayed sequentially based on the continuous display duration of a single LED lamp bead, that is, the display timings of various other pictures can be obtained. Thus, the above method of controlling the color display of multiple target LED lamp beads in the LED screen according to the display timing of the inner cone picture displayed by the multiple target LED lamp beads includes:

[0093] Determine the target display timing for the multiple target LED lamp beads to alternately display at least two pictures according to the display timing of any one picture displayed by the multiple target LED lamp beads; control the color display of each target LED lamp bead in the LED screen according to the target display timing for the multiple target LED lamp beads to alternately display at least two pictures.

[0094] Exemplarily, assume that the two kinds of images alternately displayed on the LED screen in the virtual shooting requirement are divided into the first kind of image (such as an image with a content background) and the second kind of image (such as a green screen background image). The display timing of any one of the images can be calculated first. For example, if the calculated display timing of the first group of lamp beads for displaying the first kind of image includes: 0ms - 10ms, 12.5ms - 22.5ms, 27.5ms - 37.5ms, 42.5ms - 52.5ms..., and the display timing of the second group of lamp beads includes: 0.1ms - 10.1ms, 12.6ms - 22.6ms, 27.6ms - 37.6ms, 42.6ms - 52.6ms..., then based on the display order of the two kinds of images (such as the first kind of image is displayed first and the second kind of image is displayed later), the target display timing for each group of lamp beads to alternately display the two kinds of images can be determined. For example, the first kind of image and the second kind of image are alternately displayed according to the above display timing. For example, the first group of lamp beads displays the first kind of image from 0 to 10ms, the second kind of image from 12.5ms to 22.5ms, the first kind of image from 27.5ms to 37.5ms, the second kind of image from 42.5ms to 52.5ms, and so on. The second group of lamp beads displays the first kind of image from 0.1ms to 10.1ms, the second kind of image from 12.6ms to 22.6ms, the first kind of image from 27.6ms to 37.6ms, the second kind of image from 42.6ms to 52.6ms, and so on; of course, the second kind of image can also be displayed first and then the first kind of image, which depends on the user's needs, and the embodiments of the present disclosure do not limit this. It should be understood that if the pose of the rolling shutter camera is fixed, the display control of each target LED lamp bead can be performed according to the initially determined target display timing. If the pose of the rolling shutter camera changes, multiple target LED lamp beads within the internal viewing cone range of the rolling shutter camera in the current pose can be re-determined, and the target display timing for each target LED lamp bead to alternately display the first kind of image and the second kind of image can be re-determined, and the color display of the latest determined multiple target LED lamp beads can be controlled according to the latest determined target display timing. The embodiments of the present disclosure do not limit this.

[0095] According to the display control method of the embodiments of the present disclosure, the pose information of the rolling shutter camera can be obtained to determine the in-view cone screen within the in-view cone range and the pixel positions of multiple target LED beads. Then, according to the pixel positions of each target LED bead and the rolling shutter parameters of the rolling shutter camera, the scanned timing of each target LED bead scanned by the rolling shutter camera can be obtained. This scanned timing is related to the positions of each target LED bead. The closer the target LED bead is to the bottom, the later the start time when it is scanned by the rolling shutter camera. Then, by combining the continuous display duration of a single LED bead, the display timing of each target LED bead adapted to the line-by-line scanning characteristic of the rolling shutter camera can be obtained. Then, according to this display timing, each target LED bead is controlled to perform color display, which is equivalent to matching the display of the in-view cone screen on the screen with the working timing of the line-by-line scanning exposure of the rolling shutter camera. Thus, the rolling shutter camera can be applied to virtual shooting scenarios with higher frame rate picture refresh requirements (i.e., shorter continuous display duration of LED beads), without being troubled by the rolling shutter duration (i.e., the rolling time), and greatly improving the display acquisition ratio of the LED beads in the LED screen, that is, improving the utilization rate of the LED screen.

[0096] Based on the above steps S11 to S15 of the embodiments of the present disclosure, Figure 9 The block diagram of another virtual shooting system proposed by an embodiment of the present disclosure is shown, as Figure 9As shown in the figure, the system includes: a motion capture module, which is used to capture the camera position during shooting, obtain the camera pose information, and send it to the Unreal rendering cluster and the phase calculation module; a screen calibration module, which is used to calibrate the position of the LED screen and send it to the Unreal rendering cluster and the phase calculation module; a camera calibration module, which is used to calibrate other camera information (such as camera internal parameters, rolling shutter time) and send it to the Unreal rendering cluster and the phase calculation module; among them, the screen calibration module and the camera calibration module can be calibrated offline; the Unreal rendering cluster is used to render the on-screen image according to the camera information and the screen information (such as the screen position of the LED screen), and the rendered image passes through the playback control module and is sent to the LED rendering module; the playback control module is used to convert the rendered image into a format adapted to the LED screen display, and the LED rendering module is used to drive the LED screen to display the rendered image in the converted format (that is, to realize the on-screen display of the image); at the same time, the phase calculation module is used to calculate the time phase information of the physical lamp beads on the LED screen being scanned according to the camera information and the screen information (that is, the target LED lamp beads and the display timing of the target LED lamp beads), and the phase information is transmitted through the phase transmission module, received by the phase receiving module and implemented by the phase implementation module. Among them, the phase receiving module, the phase implementation module and the LED rendering module can be located on the same device, so that the phase information can be received and implemented by the LED rendering module, and the target LED lamp beads can be dynamically lit in cooperation with the phase information; the rolling shutter camera synchronously shoots 100fps images. If it is applied to the above single-camera green screen shooting scenario, the 100fps images captured by the camera can also be split into odd and even parts to obtain the background virtual shooting video and the green screen virtual shooting video.

[0097] In practical applications, when using multiple rolling shutter cameras for shooting, the starting shooting moment of each rolling shutter camera among the multiple rolling shutter cameras can also be fine-tuned according to the display timing of the lamp beads. Especially in the scenario of alternately displaying multiple background images, by adjusting the starting shooting moment of each rolling shutter camera, each rolling shutter camera can be mainly responsible for shooting the video of a certain background image to optimize the shooting experience. Thus, the LED screen displays the specified content at the specified time phase according to the determined display timing, and at the same time, each rolling shutter camera can perform synchronous shooting according to the determined starting shooting moment of each of them to complete the virtual shooting.

[0098] In the traditional virtual shooting technology based on a rolling shutter camera, due to the influence of the rolling shutter characteristics, the high refresh rate characteristics of the LED screen cannot be fully utilized, making the rolling shutter camera inapplicable in advanced application scenarios such as multi-camera shooting and multi-scene shooting. However, the above-mentioned display control method for the LED screen proposed in the embodiments of the present disclosure can, based on the camera pose and screen model with high-precision motion compensation, estimate the exposure phase of the LED screen beads (i.e., the scanned timing) in real time, and thereby plan the display timing of each target LED bead to match the scanning timing of the rolling shutter camera for display, thus overcoming the influence of the rolling shutter time. Ideally, it can achieve the same high-speed acquisition ability as the global shutter camera, enabling the rolling shutter camera to also meet the usage requirements in advanced application scenarios such as multi-camera shooting and multi-scene shooting.

[0099] The display control method for the LED screen proposed in the embodiments of the present disclosure can calculate the positions of each bead in the camera image based on the real-time spatial information of the camera and the screen, and then determine the exposure time of each bead by combining the rolling shutter time and the shutter time. Considering some systematic errors in positioning and making some time redundancy to arrange the lighting time of each bead can ensure that the LED screen meets the requirements of high-frame-rate green insertion shooting. It can dynamically and precisely arrange the display time of the bead content according to the time when the bead is scanned by the rolling shutter camera, achieve high-speed acquisition of the rolling shutter camera shooting the LED screen, realize single-camera green insertion rolling shutter camera shooting, and realize the calibration of the rolling shutter duration of the rolling shutter camera using the LED screen.

[0100] Figure 10 The block diagram of a display control device for an LED screen according to an embodiment of the present disclosure is shown as Figure 10 As shown, the device includes:

[0101] An acquisition module 101, configured to acquire the pose information of the rolling shutter camera during the virtual shooting process;

[0102] A first determination module 102, configured to determine, according to the pose information of the rolling shutter camera and the screen model of the LED screen, the in-view cone image within the in-view cone range of the rolling shutter camera to be displayed on the LED screen and the pixel position information of a plurality of target LED beads within the in-view cone range of the rolling shutter camera in the LED screen, where the pixel position information includes the pixel position where the target LED bead is projected onto the imaging plane of the rolling shutter camera;

[0103] A second determination module 103, configured to determine the scanning sequence of each target LED bead according to the starting shooting moment of the rolling shutter camera, the pixel position information of the multiple target LED beads, and the rolling shutter parameter of the rolling shutter camera; wherein, the rolling shutter parameter represents the rolling shutter duration of the rolling shutter camera, and the scanning sequence includes the starting moment when each target LED bead starts to be scanned by the rolling shutter camera;

[0104] A third determination module 104, configured to determine the display sequence of the multiple target LED beads for displaying the inner cone picture according to the scanning sequence of each target LED bead, the shutter duration of the rolling shutter camera, and the continuous display duration of a single LED bead, wherein the display sequence represents the starting display moment and the ending display moment of each target LED bead;

[0105] A display control module 105, configured to control the multiple target LED beads in the LED screen to perform color display according to the display sequence of the multiple target LED beads for displaying the inner cone picture.

[0106] In a possible implementation manner, the determining the scanning sequence of each target LED bead according to the starting shooting moment of the rolling shutter camera, the pixel position information of the multiple target LED beads, and the rolling shutter parameter of the rolling shutter camera includes: for the i-th row of pixels on the imaging plane of the rolling shutter camera, determining the i-th group of beads corresponding to the i-th row of pixels according to the pixel position information of the multiple target LED beads, where the i-th group of beads includes one or more target LED beads projected onto the i-th row of pixels, 1 ≤ i ≤ N, and N is the total pixel height of the imaging plane; when i is 1, determining the starting moment when the rolling shutter camera waits to start scanning the first group of beads corresponding to the first row of pixels according to the starting shooting moment of the rolling shutter camera, the rolling shutter duration, and the shutter duration of the rolling shutter camera; when i is greater than 1, determining the starting moment when the rolling shutter camera waits to start scanning the i-th group of beads according to the pixel height of the i-th row of pixels, the rolling shutter parameter, the total pixel height of the imaging plane, and the starting moment when the rolling shutter camera waits to start scanning the first group of beads corresponding to the first row of pixels.

[0107] In a possible implementation, determining the display timing for the plurality of target LED beads to display the endoscope cone view according to the scanned timing of each target LED bead, the shutter duration of the rolling shutter camera, and the continuous display duration of a single LED bead includes: using the difference between the shutter duration of the rolling shutter camera and the continuous display duration of a single LED bead as the redundant duration; for the i-th group of beads corresponding to the i-th row of pixels, determining the start display time of the i-th group of beads according to the start time when the rolling shutter camera is to start scanning the i-th group of beads and the redundant duration, where the start display time of the i-th group of beads is earlier than the start time when it is to be started to be scanned by the rolling shutter camera; using the sum of the start display time of the i-th group of beads and the continuous display duration of a single LED bead as the end display time of the i-th group of beads.

[0108] In a possible implementation, the rolling duration represents the time difference between the start exposure time of the topmost row of pixels and the start exposure time of the bottommost row of pixels of the rolling shutter camera. The calibration process of the rolling parameters of the rolling shutter camera includes: during the process of controlling the LED screen to alternately display the first image frame and the second image frame and continuously adjusting the display phase of the LED beads in the LED screen, using the rolling shutter camera to capture the LED screen at a preset acquisition frame rate to obtain an acquisition video; where the picture contents of the first image frame and the second image frame are different; by analyzing the video picture of the acquisition video, obtaining the first moment when the first mixed picture just becomes the picture content of the second image frame, and the second moment when the picture content of the second image frame just does not leak out of the second mixed picture; where the first mixed picture includes the picture content of the first image frame in the top region of the video picture and the picture content of the second image frame in other regions, and the second mixed picture includes the picture content of the first image frame in the bottom region of the video picture and the picture content of the second image frame in other regions; determining the rolling parameters of the rolling shutter camera according to the first moment, the second moment, the single-frame display duration of the second image frame, and the shutter duration of the rolling shutter camera.

[0109] In a possible implementation, determining the in-view cone image to be displayed on the LED screen and the pixel position information of a plurality of target LED beads within the in-view cone range of the rolling shutter camera in the LED screen according to the pose information of the rolling shutter camera and the screen model of the LED screen includes: determining the relative pose between the virtual camera in the virtual shooting environment and the screen model and the in-view cone image to be displayed on the LED screen according to the pose information of the rolling shutter camera, where the virtual camera has the same pose and internal parameters as the rolling shutter camera; determining the in-view cone area within the in-view cone range of the virtual camera in the screen model based on the relative pose, and taking the plurality of LED beads within the in-view cone area as target LED beads; projecting each target LED bead in the screen model onto the imaging plane of the virtual camera based on the spatial positions of the respective target LED beads in the screen model to obtain the pixel position information of the plurality of target LED beads within the in-view cone range of the rolling shutter camera.

[0110] In a possible implementation, the in-view cone image to be displayed is any one of at least two images to be alternately displayed. Among them, controlling the color display of the plurality of target LED beads in the LED screen according to the display timing of displaying the in-view cone image by the plurality of target LED beads includes: determining the target display timing for the plurality of target LED beads to alternately display at least two images according to the display timing of displaying any one image by the plurality of target LED beads; controlling the color display of each target LED bead in the LED screen according to the target display timing for the plurality of target LED beads to alternately display at least two images.

[0111] According to the display control device of an embodiment of the present disclosure, by acquiring the pose information of the rolling shutter camera, the in-cone view within the in-cone range and the pixel positions of multiple target LED beads can be determined. Then, according to the pixel positions of the respective target LED beads and the rolling shutter parameters of the rolling shutter camera, the scanning timing of each target LED bead scanned by the rolling shutter camera can be obtained. This scanning timing is related to the positions of the respective target LED beads. The closer the target LED bead is to the bottom, the later the start time when it is scanned by the rolling shutter camera. Then, by combining the continuous display duration of a single LED bead, the display timing of each target LED bead adapted to the line-by-line scanning characteristic of the rolling shutter camera can be obtained. Then, according to this display timing, each target LED bead is controlled to perform color display, which is equivalent to matching the display of the in-cone view on the screen with the working timing of the line-by-line scanning exposure of the rolling shutter camera. Thus, the rolling shutter camera can be applied to virtual shooting scenarios with higher frame rate requirements for screen refreshing (i.e., shorter continuous display duration of LED beads), without being troubled by the rolling shutter duration (i.e., the rolling time), and greatly improving the display acquisition ratio of the LED beads in the LED screen, that is, improving the utilization rate of the LED screen.

[0112] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the methods described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here.

[0113] The embodiments of the present disclosure also provide an electronic device, including a memory, a processor, and a computer program stored on the memory. The processor executes the computer program to implement the steps of the above method.

[0114] The embodiments of the present disclosure also provide a non-volatile computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above method are implemented.

[0115] The embodiments of the present disclosure also provide a computer program product, including a computer program, or a non-volatile computer-readable storage medium carrying the computer program. When the computer program is executed by a processor, the steps of the above method are implemented.

[0116] Figure 11 FIG. shows a block diagram of an electronic device 1900 according to an embodiment of the present disclosure. For example, the electronic device 1900 can be provided as a server or a terminal device. Refer to Figure 11, The electronic device 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by a memory 1932 for storing instructions executable by the processing component 1922, such as application programs. The application programs stored in the memory 1932 may include one or more modules each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0117] The electronic device 1900 may also include a power component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output interface 1958 (I / O interface). The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server TM , Mac OS X TM , Unix TM , Linux TM , FreeBSD TM or the like.

[0118] In an exemplary embodiment, a non-transitory computer-readable storage medium is also provided, such as the memory 1932 including computer program instructions, and the above computer program instructions can be executed by the processing component 1922 of the electronic device 1900 to complete the above method.

[0119] A computer-readable storage medium may be a tangible device that can hold and store programs / instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD), memory stick, floppy disk, mechanically encoded devices, such as punch cards or raised structures in grooves having instructions stored thereon, and any suitable combination of the above. The computer-readable storage medium used herein is not construed as an instantaneous signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., optical pulses through an optical fiber cable), or electrical signals transmitted through wires.

[0120] The computer programs (or computer-readable program instructions) described herein can be downloaded to various computing / processing devices from a computer-readable storage medium or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include copper transmission cables, optical fiber transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in each computing / processing device.

[0121] The computer programs (or computer program instructions) for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk, C++, etc., and conventional procedural programming languages such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, executed as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, by using the state information of the computer-readable program instructions to customize an electronic circuit, such as a programmable logic circuit, a field-programmable gate array (FPGA), or a programmable logic array (PLA), the electronic circuit can execute the computer-readable program instructions to implement various aspects of the present disclosure.

[0122] Aspects of the present disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0123] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that the instructions, when executed by the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in one or more boxes of the flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that causes a computer, a programmable data processing apparatus, and / or other devices to function in a particular manner, such that the computer-readable medium storing the instructions comprises a manufacture including instructions for implementing various aspects of the functions / acts specified in one or more boxes of the flowchart and / or block diagram.

[0124] The computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other devices to produce a computer-implemented process such that the instructions executed on the computer, other programmable data processing apparatus, or other devices implement the functions / acts specified in one or more boxes of the flowchart and / or block diagram.

[0125] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, a segment of code, or a portion of an instruction, and the module, segment of code, or portion of an instruction may include one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two consecutive blocks may in fact be executed substantially in parallel, or they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented by special-purpose hardware-based systems that perform the specified functions or acts, or by combinations of special-purpose hardware and computer instructions.

[0126] The embodiments of the present disclosure have been described above. The above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The choice of terms used herein is intended to best explain the principles of the embodiments, the practical application, or improvements made to the technology in the marketplace, or to enable other ordinary skilled persons in the art to understand the embodiments disclosed herein.

Claims

1. A display control method for an LED screen, characterized in that, Applied to a virtual shooting scenario of shooting the LED screen using a rolling shutter camera, the method includes: During virtual shooting, obtain the pose information of the rolling shutter camera; According to the pose information of the rolling shutter camera and the screen model of the LED screen, determine the in-view cone image within the in-view cone range of the rolling shutter camera to be displayed on the LED screen and the pixel position information of a plurality of target LED beads within the in-view cone range of the rolling shutter camera in the LED screen, where the pixel position information includes the pixel positions where the target LED beads are projected onto the imaging plane of the rolling shutter camera; According to the starting shooting time of the rolling shutter camera, the pixel position information of the plurality of target LED beads, and the rolling shutter parameter of the rolling shutter camera, determine the scanning sequence of each target LED bead; where the rolling shutter parameter represents the rolling shutter duration of the rolling shutter camera, and the scanning sequence includes the starting time when each target LED bead starts to be scanned by the rolling shutter camera; According to the scanning sequence of each target LED bead, the shutter duration of the rolling shutter camera, and the continuous display duration of a single LED bead, determine the display sequence of the plurality of target LED beads for displaying the in-view cone image, where the display sequence represents the starting display time and the ending display time of each target LED bead; According to the display sequence of the plurality of target LED beads for displaying the in-view cone image, control the plurality of target LED beads in the LED screen to perform color display.

2. The method according to claim 1, wherein The determining the scanning sequence of each target LED bead according to the starting shooting time of the rolling shutter camera, the pixel position information of the plurality of target LED beads, and the rolling shutter parameter of the rolling shutter camera includes: For the i-th row of pixels on the imaging plane of the rolling shutter camera, according to the pixel position information of the plurality of target LED beads, determine the i-th group of beads corresponding to the i-th row of pixels, where the i-th group of beads includes a plurality of target LED beads projected onto the i-th row of pixels, 1 ≤ i ≤ N, and N is the total pixel height of the imaging plane; When i is 1, according to the starting shooting time of the rolling shutter camera, the rolling shutter duration, and the shutter duration of the rolling shutter camera, determine the starting time when the rolling shutter camera is to start scanning the first group of beads corresponding to the first row of pixels; When i is greater than 1, according to the pixel height of the i-th row of pixels, the rolling shutter parameter, the total pixel height of the imaging plane, and the starting time when the rolling shutter camera is to start scanning the first group of beads corresponding to the first row of pixels, determine the starting time when the rolling shutter camera is to start scanning the i-th group of beads.

3. The method according to claim 1 or 2, characterized in that, The determining the display sequence of the plurality of target LED beads for displaying the in-view cone image according to the scanning sequence of each target LED bead, the shutter duration of the rolling shutter camera, and the continuous display duration of a single LED bead includes: Taking the difference between the shutter duration of the rolling shutter camera and the continuous display duration of a single LED bead as the redundant duration; For the i-th group of light beads corresponding to the i-th row of pixels, determine the start display time of the i-th group of light beads according to the start time when the rolling shutter camera is about to start scanning the i-th group of light beads and the redundant duration, where the start display time of the i-th group of light beads is earlier than the start time when it is about to be scanned by the rolling shutter camera; Use the sum of the start display time of the i-th group of light beads and the continuous display duration of the single LED light bead as the end display time of the i-th group of light beads.

4. The method according to claim 1, wherein The rolling shutter duration represents the time difference between the start exposure time of the topmost row of pixels and the start exposure time of the bottommost row of pixels of the rolling shutter camera, where the calibration process of the rolling shutter parameters of the rolling shutter camera includes: During the process of controlling the LED screen to alternately display the first image frame and the second image frame and continuously adjusting the display phase of the LED light beads in the LED screen, use the rolling shutter camera to capture the LED screen at a preset acquisition frame rate to obtain an acquisition video; where the picture contents of the first image frame and the second image frame are different; By analyzing the video picture of the acquisition video, obtain the first moment when the first mixed picture exactly becomes the picture content of the second image frame, and the second moment when the picture content of the second image frame exactly does not show the second mixed picture; where the first mixed picture includes the picture content of the first image frame in the top area of the video picture and the picture content of the second image frame in other areas, and the second mixed picture includes the picture content of the first image frame in the bottom area of the video picture and the picture content of the second image frame in other areas; Determine the rolling shutter parameters of the rolling shutter camera according to the first moment, the second moment, the single-frame display duration of the second image frame, and the shutter duration of the rolling shutter camera.

5. The method according to claim 1, characterized in that, The determining the inner cone picture within the inner cone range of the rolling shutter camera and the pixel position information of multiple target LED light beads within the inner cone range of the LED screen to be displayed on the LED screen according to the pose information of the rolling shutter camera and the screen model of the LED screen includes: According to the pose information of the rolling shutter camera, determine the relative pose between the virtual camera in the virtual shooting environment and the screen model and the inner cone picture to be displayed on the LED screen, where the pose and internal parameters of the virtual camera are the same as those of the rolling shutter camera; Based on the relative pose, determine the inner cone area within the inner cone range of the virtual camera in the screen model, and use the multiple LED light beads within the inner cone area as target LED light beads; Based on the spatial positions of the respective target LED light beads in the screen model, project the respective target LED light beads in the screen model onto the imaging plane of the virtual camera to obtain the pixel position information of the multiple target LED light beads within the inner cone range of the rolling shutter camera.

6. The method according to claim 1, wherein The endoscope cone screen to be displayed is any one of at least two screens to be alternately displayed. Among them, the display timing for displaying the endoscope cone screen according to the multiple target LED lamp beads, and controlling the multiple target LED lamp beads in the LED screen to perform color display includes: Determining the target display timing for the multiple target LED lamp beads to alternately display at least two screens according to the display timing for the multiple target LED lamp beads to display any one screen; Controlling each target LED lamp bead in the LED screen to perform color display according to the target display timing for the multiple target LED lamp beads to alternately display at least two screens.

7. A display control device for an LED screen, characterized in that, Applied to a virtual shooting scenario where a rolling shutter camera is used to shoot the LED screen, the device includes: An acquisition module, configured to acquire the pose information of the rolling shutter camera during virtual shooting; A first determination module, configured to determine, according to the pose information of the rolling shutter camera and the screen model of the LED screen, the endoscope cone screen within the endoscope cone range of the rolling shutter camera to be displayed on the LED screen and the pixel position information of the multiple target LED lamp beads within the endoscope cone range of the rolling shutter camera in the LED screen, where the pixel position information includes the pixel position where the target LED lamp bead is projected onto the imaging plane of the rolling shutter camera; A second determination module, configured to determine the scanning timing of each target LED lamp bead according to the starting shooting moment of the rolling shutter camera, the pixel position information of the multiple target LED lamp beads, and the rolling shutter parameter of the rolling shutter camera; where the rolling shutter parameter represents the rolling shutter duration of the rolling shutter camera, and the scanning timing includes the starting moment when each target LED lamp bead starts to be scanned by the rolling shutter camera; A third determination module, configured to determine the display timing for the multiple target LED lamp beads to display the endoscope cone screen according to the scanning timing of each target LED lamp bead, the shutter duration of the rolling shutter camera, and the continuous display duration of a single LED lamp bead, where the display timing represents the starting display moment and the ending display moment of each target LED lamp bead; A display control module, configured to control the multiple target LED lamp beads in the LED screen to perform color display according to the display timing for the multiple target LED lamp beads to display the endoscope cone screen.

8. An electronic device, comprising a memory, a processor, and a computer program stored on the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 6.

9. A non-volatile computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, or a non-volatile computer-readable storage medium carrying the computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the method according to any one of claims 1 to 6.

Images