A method, apparatus, and system for real-time generation of special effects in virtual filming

Abstract

This disclosure relates to a method, apparatus, and system for real-time generation of special effects in virtual shooting. The method includes: during virtual shooting, displaying a first frame captured by an image acquisition device on a display device of a terminal device; the first frame includes a virtual scene displayed on the screen and a real scene in front of the screen; in response to an operation of adding special effects to a target object in the real scene, adding a target special effect corresponding to the target object in real time to the first frame to generate a second frame; and displaying the second frame on the display device of the terminal device. This disclosure allows for real-time addition of target special effects to the frame captured by the image acquisition device, effectively improving the efficiency of special effects generation while significantly reducing the cost of special effects production, resulting in greater economic efficiency.

Description

Technical Field

[0001] This disclosure relates to the field of virtual shooting technology, and in particular to a method, apparatus, and system for real-time generation of special effects in virtual shooting. Background Technology

[0002] Virtual filming refers to rendering a virtual scene in real time on a screen such as a light-emitting diode (LED), with actors and other entities serving as the real scene. By simultaneously filming the real and virtual scenes with a camera, the effect of placing the actors in a virtual environment is achieved. This technology combines real-time rendering of virtual images with actor performances, bringing a completely new experience to film and television production.

[0003] Films and television dramas with themes such as fantasy, magic, and science fiction often include special effects such as "magic spells." The related technology involves merging virtual scenes with the real scenes of the actors' performances during the virtual shooting process. After the virtual shooting is completed, the special effects such as "magic spells" are then added in post-production based on the opinions of the director and other personnel and the script content. This method usually requires multiple communications and adjustments by the director and other personnel during the post-production process, which is relatively inefficient. Summary of the Invention

[0004] In view of this, this disclosure proposes a method, apparatus, system, electronic device, storage medium, and computer program product for real-time generation of special effects in virtual shooting.

[0005] According to one aspect of this disclosure, a method for real-time generation of special effects in virtual shooting is provided, the method comprising:

[0006] During the virtual shooting process, the first image captured by the image acquisition device is displayed on the display device of the terminal device; the first image includes the virtual scene displayed on the screen and the real scene in front of the screen;

[0007] In response to the operation of adding special effects to the target object in the real scene, the target special effects corresponding to the target object are added in real time to the first screen to generate the second screen;

[0008] The second screen is displayed on the display device of the terminal equipment.

[0009] In one possible implementation, the step of responding to the operation of adding special effects to a target object in the real scene, adding a target special effect corresponding to the target object in real time to the first screen, and generating a second screen, includes:

[0010] In response to the operation of adding special effects to the target object in the real scene, the pose information of the target special effects and the pose information of the image acquisition device when the first frame was captured are obtained;

[0011] Based on the pose information of the image acquisition device and the pose information of the target effect when the first scene was captured, a first relative pose relationship is determined.

[0012] Based on the first relative pose relationship, determine the target position in the first frame where the target effect is to be added;

[0013] Based on the target location, the target effect is added to the first screen to generate the second screen.

[0014] In one possible implementation, the step of obtaining the pose information of the target effect in response to the operation of adding an effect to a target object in the real scene includes:

[0015] In response to the operation of adding special effects to a target object in the real scene, a second relative pose relationship between the target special effects and the target object is determined;

[0016] Based on the pose information of the target object and the second relative pose relationship, the pose information of the target effect is determined.

[0017] In one possible implementation, the method further includes: generating an effects layer image in response to an operation of adding effects to a target object in the real scene, wherein the effects layer image includes the target effects;

[0018] The step of adding the target effect to the first image based on the target position to generate the second image includes: placing the effect layer image on top of the first image, and aligning the effect layer image with the first image based on the target position to generate the second image.

[0019] In one possible implementation, the pose information of the image acquisition device and the pose information of the target object when the first frame is captured are obtained by tracking through a motion capture device.

[0020] In one possible implementation, the method further includes:

[0021] In response to the first recording operation, one or more of the following are recorded: the pose information of the image acquisition device, the pose information of the target object, and the pose information of the target special effect when the first frame is captured.

[0022] In one possible implementation, the method further includes:

[0023] In response to the second recording operation, the second screen is recorded.

[0024] In one possible implementation, before placing the special effects layer image on top of the first image, the method further includes:

[0025] Obtain a first mapping relationship between the color space used by the image acquisition device for shooting and the color space used for special effects rendering, and a second mapping relationship between the color space used for creating special effects and the color space used for rendering special effects;

[0026] Based on the first mapping relationship, the first image is color-converted;

[0027] Based on the second mapping relationship, the colors of the special effects layer are converted.

[0028] In one possible implementation, prior to recording the second frame, the method further includes:

[0029] Obtain a third mapping relationship between the color space used for special effects rendering and the color space used for image capture by the image acquisition device;

[0030] Based on the third mapping relationship, the second image undergoes color conversion.

[0031] According to another aspect of this disclosure, a device for real-time generation of special effects in virtual shooting is provided, the device comprising:

[0032] The display module is used to display the first image captured by the image acquisition device on the display device of the terminal device during the virtual shooting process; the first image includes the virtual scene displayed on the screen and the real scene in front of the screen.

[0033] The special effects generation module is used to respond to the operation of adding special effects to the target object in the real scene, and to add the target special effects corresponding to the target object in real time in the first screen to generate the second screen;

[0034] The display module is also used to display the second screen on the display device of the terminal equipment.

[0035] According to another aspect of this disclosure, a real-time special effects generation system for virtual shooting is provided, comprising a display screen for displaying a virtual scene; an image acquisition device for capturing a first frame during virtual shooting; the first frame including the virtual scene displayed on the display screen and a real scene in front of the display screen; and a terminal device for displaying a second frame through the display device of the terminal device, the second frame being obtained by adding target special effects corresponding to the target object in real time to the first frame in response to an operation of adding special effects to a target object in the real scene.

[0036] According to another aspect of this disclosure, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement the above-described method when executing instructions stored in the memory.

[0037] According to another aspect of this disclosure, a non-volatile computer-readable storage medium is provided that stores computer program instructions thereon, wherein the computer program instructions, when executed by a processor, implement the above-described method.

[0038] According to another aspect of this disclosure, a computer program product is provided, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the above-described method.

[0039] Through various aspects of this disclosure, during virtual filming, a first frame captured by an image acquisition device is displayed on the display device of a terminal device. This first frame includes a virtual scene displayed on the screen and a real scene in front of the screen. In response to an operation adding special effects to a target object in the real scene, a target special effect corresponding to the target object is added in real time to the first frame, generating a second frame. This second frame is then displayed on the display device of the terminal device. Thus, during virtual filming, target special effects are added in real time to the frame captured by the image acquisition device, and the before-and-after images can be shown to directors, producers, and special effects artists. This allows these personnel to instantly view the combination of the added special effects and the actors' performances on the virtual filming set, and to adjust the special effects during the virtual filming process to meet specific needs, improving efficiency. Simultaneously, the generated special effects are added to the frame captured by the image acquisition device without needing to be rendered onto the screen displaying the virtual scene, significantly reducing the requirements for rendering performance and effectively lowering the cost of special effects production, resulting in higher economic efficiency.

[0040] Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0041] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of this disclosure together with the specification and serve to explain the principles of this disclosure.

[0042] Figure 1 This diagram illustrates a structural diagram of a real-time special effects generation system for virtual filming according to an embodiment of the present disclosure;

[0043] Figure 2 This diagram illustrates a scene where special effects are generated in real time during virtual shooting according to an embodiment of the present disclosure;

[0044] Figure 3 A flowchart illustrating a method for real-time generation of special effects in virtual shooting according to an embodiment of the present disclosure is shown.

[0045] Figure 4 A schematic diagram illustrating a color conversion according to an embodiment of the present disclosure is shown;

[0046] Figure 5 A flowchart illustrating a method for generating a second screen according to an embodiment of the present disclosure is shown;

[0047] Figure 6 This diagram illustrates data recorded during a virtual shooting process according to an embodiment of the present disclosure;

[0048] Figure 7 A schematic diagram illustrating the generation of a second screen according to an embodiment of the present disclosure is shown;

[0049] Figure 8 This diagram shows a structural diagram of a real-time special effects generation apparatus for virtual shooting according to an embodiment of the present disclosure;

[0050] Figure 9 A block diagram of an electronic device 1900 according to an embodiment of the present disclosure is shown. Detailed Implementation

[0051] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0052] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this disclosure include a particular feature, structure, or characteristic described in connection with that embodiment. Therefore, the terms "exemplary," "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in various parts of this specification, do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including, but not limited to," unless otherwise specifically emphasized.

[0053] In this disclosure, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A alone, A and B simultaneously, and B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0054] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0055] To address the issue of low efficiency in creating special effects such as "spells" in post-production based on the opinions of directors and script content during virtual shooting, special effects can be composited in real-time during virtual shooting to improve efficiency. One related technology uses Unreal Engine with the RenderStream plugin and Disguise (a virtual shooting platform) to launch two Unreal Engine instances to render special effects and virtual scenes separately. This allows for real-time display of special effects and virtual scenes on the screen during virtual shooting, enabling composite recording. However, this method requires the Disguise central control unit and multiple rendering machines to work together, making it cumbersome. Furthermore, special effects and virtual scenes need to be rendered separately using two Unreal Engine instances to avoid frame drops.

[0056] This disclosure proposes a real-time special effects generation scheme for virtual filming (detailed description below). During virtual filming, target special effects can be added in real-time to the footage captured by an image acquisition device (the footage includes both the virtual scene displayed on the screen and the real scene in front of the screen). The before-and-after images with the added special effects can be shown to the director, producer, and special effects artists, allowing them to instantly view the integration of the special effects with the actors' performances on set. Adjustments to the special effects can be made during virtual filming to meet specific needs, improving efficiency. Furthermore, the generated special effects are added to the footage captured by the image acquisition device without needing to be rendered onto the screen displaying the virtual scene, significantly reducing rendering performance requirements and effectively lowering the cost of special effects production, resulting in greater economic efficiency.

[0057] Figure 1 This diagram illustrates a structural diagram of a real-time special effects generation system for virtual filming according to an embodiment of the present disclosure; as shown. Figure 1 As shown, it includes:

[0058] The display screen is used to display virtual scenes; depending on the actual shooting needs, the display screen can be configured as a flat screen, curved screen, tri-fold screen, or irregularly shaped screen with a multi-faceted three-dimensional structure, etc.

[0059] An image acquisition device is used for real-time shooting during virtual shooting. During virtual shooting, a virtual scene is displayed on a screen in real time, and a real scene can be set up in front of the screen. Actors can perform in the real scene. The image acquisition device can simultaneously capture the virtual scene displayed on the screen and the real scene in front of the screen. For example, during virtual shooting, the image acquisition device can shoot in real time at a preset frame rate and can transmit the captured images to special effects production tools in real time through an SDI interface or the like.

[0060] For example, the system can be configured with a special effects creation tool, which is used to add desired special effects in real time to the footage captured by the image acquisition device, and then render the image with added effects through a special effects rendering engine. For example, the special effects creation tool can acquire the real-time footage captured by the image acquisition device through an SDI interface. As an example, the special effects creation tool can have a display panel and special effects creation functions, allowing directors, producers, special effects artists, etc., to perform related operations such as target object selection, adding special effects, previewing, and adjusting using the special effects creation tool. As an example, a special effects rendering engine can also be configured in the system, which has image rendering capabilities and can render the image with added effects.

[0061] The terminal device's display unit is used to display images captured by the image acquisition device, and can also display images rendered by the special effects rendering engine after the addition of special effects. For example, the terminal device can include various types of terminals such as laptops, tablets, desktop computers, smartphones, smartwatches, and smart TVs. For example, the display unit of the terminal device can be a display panel corresponding to the special effects production tool. The terminal device can have one or more display units; for example, it can also include monitors for directors, producers, etc. The images after the addition of special effects can be displayed on multiple display units, allowing different staff to preview and monitor the generated effects in real time.

[0062] The aforementioned special effects creation tools and rendering engines can be configured on terminal devices equipped with the aforementioned display devices, or on other terminal devices, or on a server. The special effects creation tools and rendering engines can be configured on the same device or different devices. The functions of the special effects creation tools and rendering engines can also be integrated into the same software.

[0063] For example, the system may further include: a virtual scene rendering engine and a motion capture device; wherein, the virtual scene rendering engine is used to render a virtual scene in real time and output the rendered virtual scene to a display screen for display. The motion capture device is used to capture the real-time pose information of the image acquisition device and the target object in the real scene, wherein the pose information includes position information and attitude information; the motion capture device, also known as a motion capture system, is a system for capturing information about changes in the motion state of an object. For example, the motion capture device can be OptiTrack, Mosys, etc.; as an example, the motion capture device may include multiple motion capture trackers, which can be installed on the image acquisition device and the target object to track the real-time pose of the image acquisition device and the target object at every moment.

[0064] For example, since the visual effects rendering engine generates an image with added effects that is displayed on the terminal device's display for directors, producers, and visual effects artists to view, rather than on a screen, and since the terminal device's display is typically much smaller than a screen, the rendering performance requirements are relatively low. Therefore, an engine with lower rendering performance can be used as the visual effects rendering engine. Alternatively, a single engine can serve as both the visual effects rendering engine and the virtual scene rendering engine, meeting the needs for real-time effects generation and thus reducing the cost of visual effects generation. As an example, the visual effects rendering engine and the virtual scene rendering engine can be implemented on the same Unreal Engine, eliminating the need for two Unreal Engines, effectively reducing the cost of visual effects production and making it more economical.

[0065] It should be noted that the above system is only an example. The components in the system can be integrated and configured, or the system can be configured with more or fewer components, without limitation.

[0066] For example, time synchronization can be achieved between various components in the system. For instance, a synchronization signal can be generated by a synchronization signal generator to achieve time synchronization between motion capture devices, image acquisition devices, display screens, virtual scene rendering engines, display devices, special effects production tools, special effects rendering engines, etc.

[0067] Figure 2 This diagram illustrates a scene where special effects are generated in real-time during virtual shooting according to an embodiment of the present disclosure, such as... Figure 2 As shown, the above-mentioned Figure 1 The system shown in the diagram, during virtual filming, has the target object (such as an actor) positioned in front of a display screen, and an image acquisition device (such as a camera) framing the shot to capture images of both the virtual scene displayed on the screen and the real scene in front of the screen. The captured images can be displayed on the terminal device's display. Simultaneously, the captured images can be transmitted to special effects production tools and special effects rendering engines, and raw footage can also be recorded, meaning the images captured by the image acquisition device can be saved for post-production use.

[0068] The motion capture device tracks the pose information of the image acquisition device and the target object in real time, and transmits the tracking data (such as the pose information of the image acquisition device and the target object) to the special effects production tool. At the same time, it transmits the tracking data (such as the pose information of the image acquisition device) to the virtual scene rendering engine.

[0069] Special effects production tools respond to the operations of directors, producers, and special effects artists who add special effects to target objects in real-world scenes captured by image acquisition devices. They add target special effects corresponding to the target objects in real time to the scene, and then render the scene with added effects using a special effects rendering engine. This achieves real-time special effects generation, and the scene with added effects is displayed on a terminal device for preview and monitoring. In some examples, the special effects rendering engine can align the relative position of the special effects layer with the scene captured by the image acquisition device based on tracking data, thereby adding target special effects corresponding to the target objects in real time and generating the scene with added effects. In other examples, the special effects rendering engine can perform color conversion on the scene captured by the image acquisition device, the special effects layer, and the scene with added effects to eliminate color differences. In still other examples, the special effects rendering engine can respond to recording operations to record the scene with added effects and / or tracking data, where the recorded tracking data can be used for post-production.

[0070] The virtual scene rendering engine renders images of the virtual scene in real time based on tracking data and displays the rendered images on the display screen. Specifically, the virtual scene rendering engine can provide the image acquisition device model in the virtual scene rendering engine with the pose information of the image acquisition device in three-dimensional space. Relying on technologies such as real-time rendering, the virtual scene rendering engine can synchronize the movement and composition of the image acquisition device model with the image acquisition device, so that the rendered virtual scene conforms to the perspective of the image acquisition device.

[0071] Figure 3 This diagram illustrates a flowchart of a method for real-time generation of special effects in virtual shooting according to an embodiment of the present disclosure. Exemplarily, this method can be derived from the above... Figure 1 The special effects shown are generated in real-time by some or all components of the system, such as... Figure 3 As shown, the method includes:

[0072] Step 301: During the virtual shooting process, the first image captured by the image acquisition device is displayed on the display device of the terminal device; the first image includes the virtual scene displayed on the screen and the real scene in front of the screen.

[0073] Among them, the image acquisition device can be Figure 1 The image acquisition device in the middle, the screen can be Figure 1 The virtual scene can be displayed on the screen in the middle. Figure 1 The virtual scene generated by the virtual scene rendering engine.

[0074] During virtual shooting, each frame captured by the image acquisition device can be displayed in real time on the display device of the terminal device; for example, the first frame can be any frame captured by the image acquisition device during the virtual shooting process. The virtual shooting can be implemented using conventional virtual shooting technology.

[0075] Step 302: In response to the operation of adding special effects to the target object in the real scene, add the target special effects corresponding to the target object in real time in the first screen to generate the second screen.

[0076] The second screen includes a virtual scene displayed on the screen, a real scene in front of the screen, and target special effects. For example, a terminal device or other device with the aforementioned display device can be configured with special effects creation tools and a special effects rendering engine. These tools or engines can add target special effects corresponding to the target object in real time within the first screen. As an example, in response to a user's operation to add effects to a target object in the real scene, the special effects creation tool adds the target special effects corresponding to the target object in real time within the first screen, and the special effects rendering engine then renders and generates the second screen. For example, the frame rate and timestamp of the special effects rendering engine can be synchronized with those of the image acquisition device to ensure that the screen rendered by the special effects rendering engine with added effects has the same frame rate and timestamp as the original screen captured by the image acquisition device. It is understood that during a virtual shooting process, the frame rate of the image acquisition device usually remains constant. Therefore, after synchronization, the frame rate of the special effects rendering engine remains constant, and effects can be directly added to other screens subsequently.

[0077] In this context, the target object refers to entities such as actors, props, and sets in a virtual shooting scene, or it can be a component of these entities. For example, the target object could be an actor's eyes, hands, feet, etc. As an example, when shooting films and television dramas of the genres of fantasy, magic, and science fiction, the special effects could be "magic." For instance, if the target object is an actor, the special effects could be "ice" appearing around the actor's body; if the target object is an actor's eyes, the special effects could be "lasers" emitted from the eyes; or if the target object is an actor's hands, the special effects could be "fireballs" with offensive capabilities appearing in the hands; or if the target object is a stone, the special effects could be "dazzling light" emitted from the stone.

[0078] For example, adding special effects to a target object in the real-world scene can include selecting the target object and determining the target special effect to be added. As an example, the display panel of the special effects creation tool can show a first screen, function buttons, and prompts. Special effects artists can use the function buttons to select the target object in the first screen to which they want to add special effects, based on the prompts. They can also create or adjust the desired target special effect in real time based on the prompts, or select the desired target effect from a pre-built special effects material library. The special effects rendering engine and special effects creation tool respond to the artist's actions by adding the selected or created target special effect to the first screen in real time, generating a second screen with the target special effect.

[0079] For example, special effects artists can choose to add special effects to a frame or a video segment captured by an image acquisition device as needed. Specifically, when special effects artists need to add special effects to a video segment including the first frame, the special effects rendering engine and special effects creation tools, in response to the artist's operation of adding effects to a target object in the real scene, can add target special effects corresponding to the target object to each frame of the video segment, thereby generating a video with the target special effects. Special effects artists can select a segment of video to add effects by choosing the start and end times.

[0080] In one possible implementation, the method further includes: generating an effects layer image in response to an operation of adding effects to a target object in the real scene, wherein the effects layer image includes the target effects.

[0081] The resolution and size of the special effects layer can be the same as the first image. As an example, a special effects creation tool can be used to generate the special effects layer. Responding to the selection of a target object by the special effects creator, the tool determines the target position of the target effect to be added in the first image based on the target object's position in the first image. Responding to the creation of the target effect by the special effects creator, the tool determines the target effect in the special effects layer and uses the aforementioned target position as the position of the target effect within the special effects layer. Furthermore, the special effects layer can be placed on top of the first image, and the first image and the special effects layer can be aligned using the target position as a reference. That is, the target position of the target effect to be added in the first image is aligned with the position of the target effect in the special effects layer, thereby overlapping the special effects layer and the first image to generate a second image.

[0082] Adding target effects by generating an effects layer allows for the addition of effects without altering the original first screen, making the process more convenient and flexible. Those skilled in the art will understand that effects can also be added without an effects layer; for example, effects can be added directly to the first screen by processing it directly. This application does not impose any limitations on this approach.

[0083] Because different devices may use different color spaces, color differences may occur in the output images. For example, the color space used by special effects creation tools may differ from the color space used by image capture devices. This results in a color difference between the special effects layer generated by the special effects creation tool and the first image captured by the image capture device. Furthermore, the special effects rendering engine composites the special effects layer with the first image to create a second image, and the color space used by the special effects rendering engine may also differ from the color spaces of the special effects creation tool and the image capture device. Therefore, to eliminate the color difference between the target special effects and other content in the generated second image, this color difference can be eliminated before generating the second image. In one possible implementation, before placing the special effects layer image on top of the first image, the method further includes: obtaining a first mapping relationship between the color space used by the image acquisition device for shooting and the color space used for special effects rendering, and a second mapping relationship between the color space used for creating special effects and the color space used for rendering special effects; performing color conversion on the first image based on the first mapping relationship; and performing color conversion on the special effects layer image based on the second mapping relationship.

[0084] The color space can be RGB, CMYK, Lab, YCbcr, etc., and the mapping relationship between different color spaces can be represented by a transformation matrix. For example, the color space used to create special effects can be the color space used by the special effects creation tool or the color space used by the special effects rendering engine. As an example, the color space used to create special effects can be the color space used by the special effects creation tool, and the color space used for special effects rendering can be the color space used by the special effects rendering engine. The color calibration mapping relationship LUT (Look-Up Table) of the image acquisition device and the color calibration mapping relationship look-up table of the special effects creation tool can be obtained or pre-calibrated from the device manufacturer. This look-up table contains the mapping relationship between the color space used by the corresponding device and other color spaces. In this way, color conversion is performed on both the first image and the special effects layer image, thereby uniformly converting the color space used by the image acquisition device for capturing the first image and the color space used for creating the special effects layer image into the color space used for special effects rendering, eliminating the color difference between the first image and the special effects layer image.

[0085] Step 303: Display the second screen on the display device of the terminal equipment.

[0086] For example, the first and second images can be displayed simultaneously on the display device of the terminal device, so that the director, producer, special effects artist, etc. can better view and compare the effects before and after adding special effects.

[0087] As an example, the display device of the terminal device can be a display panel corresponding to the special effects production tool, and the second screen can be displayed on the display panel so that the special effects production staff can preview the generated special effects; as another example, the display device of the terminal device can be the director's monitor screen, and the second screen is displayed on the director's monitor screen so that the director can monitor the generated special effects in real time.

[0088] Since the color space used by the terminal device's display may differ from the color space used for special effects rendering, one possible implementation is to perform color conversion on the second image before displaying it. For example, a mapping relationship between the color space used by the terminal device's display and the color space used for special effects rendering can be obtained, and the color of the second image rendered with special effects can be converted based on this mapping relationship. The color-converted second image is then displayed on the terminal device's display. In this way, there is no color difference between the first and second images displayed on the terminal device's display, allowing directors, producers, special effects artists, etc., to better compare the effects before and after adding special effects.

[0089] For example, when there are multiple display devices in the terminal equipment, the color spaces used by each display device may also differ. Therefore, the second image can be color-converted based on the mapping relationship between each display device and the color space used for special effects rendering. The color-converted second image is then displayed on the corresponding display device. For instance, when the second image needs to be displayed simultaneously on the director's monitor and the display panel corresponding to the special effects production tools, the second image can be color-converted and displayed on the display panel based on the mapping relationship between the color space used by the display panel and the color space used for special effects rendering. At the same time, the second image can also be color-converted and displayed on the monitor based on the mapping relationship between the color space used by the director's monitor and the color space used for special effects rendering. This allows the second image to be displayed synchronously on the display panel and the monitor, making it convenient for different personnel such as the director, producer, and special effects producer to view.

[0090] In one possible implementation, the method further includes: recording the second frame in response to a second recording operation.

[0091] For example, special effects artists can trigger a second recording operation via a recording function button displayed on the terminal device's display. As an example, the display panel corresponding to the special effects production tool can display a recording function button. Special effects artists can click this button to trigger the recording of the second frame with a single click, which can then be recorded and stored by the terminal device. The recorded second frame includes the original image captured by the image acquisition device and the added special effects. Thus, during virtual shooting, the frame with added special effects can be obtained in real time. For example, in a scenario where special effects are added in real time to a video captured by an image acquisition device, the terminal device responds to the recording operation by recording the video.

[0092] Furthermore, based on the timestamp, the first frame of the original video captured by the image acquisition device can be replaced with the recorded second frame, thus obtaining a complete video with added effects.

[0093] Considering the potential difference between the color space used for special effects rendering and the color space used by the image acquisition device, to ensure that the recorded second frame is identical to the first frame captured by the image acquisition device, a color conversion can be performed on the second frame during recording. In one possible implementation, before recording the second frame, the method further includes: obtaining a third mapping relationship between the color space used for special effects rendering and the color space used by the image acquisition device; and performing a color conversion on the second frame based on the third mapping relationship. This color conversion converts the color space used for special effects rendering to the color space used by the image acquisition device, ensuring that the converted second frame has no color difference from the frame captured by the image acquisition device, thus maintaining consistency in the display effect of each frame recorded in the complete video with added special effects. It is understood that the third mapping relationship is reversible with the first mapping relationship.

[0094] In this embodiment of the disclosure, during virtual filming, a first frame captured by an image acquisition device is displayed on the display device of a terminal device. The first frame includes a virtual scene displayed on the screen and a real scene in front of the screen. In response to the operation of adding special effects to a target object in the real scene, a target special effect corresponding to the target object is added in real time to the first frame, generating a second frame. The second frame is then displayed on the display device of the terminal device. Thus, during virtual filming, target special effects are added in real time to the frame captured by the image acquisition device, and the frames before and after adding the target special effects can be shown to the director, producer, and special effects artists. This allows these personnel to instantly view the combination of the added special effects and the actors' performances on the virtual filming set, and to adjust the special effects during the virtual filming process to meet requirements, improving efficiency. Simultaneously, the generated special effects are added to the frame captured by the image acquisition device without needing to be rendered onto the screen displaying the virtual scene, greatly reducing the requirements for rendering performance and effectively reducing the cost of special effects production, resulting in higher economic efficiency.

[0095] In some examples, as described above, considering that different devices may use different color spaces, during the generation of the target effect, this embodiment of the disclosure may also perform color conversion on the first image captured by the image acquisition device, the effect layer image, and the second image after adding the target effect, to eliminate color differences caused by different devices using different color spaces. For example, Figure 4 A schematic diagram illustrating a color conversion according to an embodiment of the present disclosure is shown, such as... Figure 4 As shown, the color space A used by the image acquisition device and the color space B used by the special effects production tools are first uniformly converted to the color space C used by the special effects rendering engine. That is, the conversion matrix M is based on the transformation matrix M from the color space A used by the image acquisition device to the color space C used by the special effects rendering engine. ac (i.e., the first mapping relationship) performs color conversion on the first image, based on the conversion matrix M from the color space B used by the special effects production tools to the color space C used by the special effects rendering engine. bc (That is, the second mapping relationship) performs color conversion on the special effects layer image. Then, the first image after color conversion is aligned with the special effects layer image, and the special effects rendering engine renders the second image. Afterwards, when previewing and monitoring the second image through the display panel corresponding to the special effects creation tool, the color space for monitoring is consistent with the color space for previewing, both being the color space D used by the display panel. The color space C used by the special effects rendering engine can be converted to the color space D used by the display panel, i.e., the conversion matrix M from the color space C used by the special effects rendering engine to the color space D used by the display panel. cdThe second frame is then color-converted and displayed on the display panel. When recording the second frame, the recorded color space is consistent with color space A used by the image acquisition device. This means the color space C used by the effects rendering engine can be converted to color space A used by the image acquisition device; that is, a conversion matrix is ​​used from color space C used by the effects rendering engine to color space A used by the image acquisition device. (That is, the third mapping relationship) performs color conversion on the second image (that is, the original image without color conversion) generated by the special effects rendering engine, whereby... With M ac These are inverse matrices, used to record the second image after color conversion, ensuring that the recorded image with added effects maintains the same color space as the original image captured by the image acquisition device. By unifying the color spaces of the first image and the effects layer, the second image is rendered. After rendering the second image, its color space is adjusted to match the color spaces used for previewing, monitoring, and recording, eliminating color differences during previewing, monitoring, and recording.

[0096] The following is an exemplary description of the specific process of adding target effects corresponding to the target object in real time to the first screen and generating the second screen in response to the operation of adding special effects to the target object in the real scene in step 302 above.

[0097] Figure 5 A flowchart illustrating a method for generating a second screen according to an embodiment of the present disclosure is shown, as follows: Figure 5 As shown, it includes the following steps:

[0098] Step 501: In response to the operation of adding special effects to the target object in the real scene, obtain the pose information of the target special effects and the pose information of the image acquisition device when the first frame was captured.

[0099] The pose information of the image acquisition device can include the position and orientation information of the image acquisition device at each moment. For example, the position information of the image acquisition device can indicate the position of the lens optical center in a preset coordinate system (such as the world coordinate system), and the orientation information of the image acquisition device can indicate the shooting direction of the lens in the preset coordinate system. The pose information of the target effect can include the position and orientation information of the target effect in the real scene at each moment. For example, the position information of the target effect can indicate the position of the target effect in the real scene (in a preset coordinate system), and the orientation information of the target effect can indicate the direction of the target effect in the real scene (in a preset coordinate system).

[0100] In one possible implementation, the step of obtaining the pose information of the target effect in response to the operation of adding an effect to the target object in the real scene may include: determining a second relative pose relationship between the target effect and the target object in response to the operation of adding an effect to the target object in the real scene; and determining the pose information of the target effect based on the pose information of the target object and the second relative pose relationship.

[0101] It is understandable that a target effect can be in the same pose as the target object. The corresponding second relative pose relationship between the two indicates that there is no difference in their poses. For example, if the target object is an actor's hand and the target effect is a "fireball" in the actor's hand, then at the same moment, the poses of the "fireball" and the "hand" are consistent. Alternatively, the target effect may be at a certain distance from the target object, and the pose information of the two may differ. The corresponding second relative pose relationship between the two indicates the pose difference between them. The magnitude of the difference indicated by the second relative pose relationship can be set according to the requirements. For example, if the target object is an actor's foot and the target effect is a "glowing footprint left by the actor in the previous step" when walking, then at the same moment, the poses of the "glowing footprint left by the previous step" and the "foot" are different. The second relative pose relationship between the "glowing footprint left by the previous step" and the "foot" can be set to a difference of 0.5 meters along the direction of foot movement.

[0102] Since the target object is an entity in a real scene, its pose information is easier to obtain. Therefore, after obtaining the pose information of the target object, the pose relationship of the target feature can be further determined by combining it with the second relative pose relationship. For example, the second relative pose relationship is relatively fixed. For instance, when displaying a "fireball" on an actor's hand, the second relative pose relationship between the target effect "fireball" and the target object "hand" remains unchanged, thus showing the effect of always having a "fireball" on the hand. Another example is that when an actor walks normally, the speed is usually uniform, and the second relative pose relationship between the "glowing footprint left by the previous step" and the "foot" remains unchanged. In this way, after presetting the second relative pose relationship between the target effect and the target object, as the pose of the target object changes, the latest pose information of the target effect can be determined based on the latest pose information of the target object and the second relative pose relationship.

[0103] In one possible implementation, the pose information of the image acquisition device and the pose information of the target object when the first frame is captured are obtained by tracking through a motion capture device.

[0104] For example, one or more motion capture trackers can be pre-installed on both the image acquisition device and the target object. The location and number of motion capture trackers can be configured as needed. For instance, if the target object is an actor, a motion capture tracker can be installed on the actor's back; if the target object is an actor's hands, feet, limbs, or other body parts, motion capture trackers can be installed on the corresponding body parts; or if the target object is a leaf in a real scene, a motion capture tracker can be installed on the leaf. The motion capture device can capture the pose information of each motion capture tracker in a preset coordinate system (such as the world coordinate system) in real time. This pose information can then be used as the pose information of the entity (i.e., the image acquisition device or the target object) on which the corresponding motion capture tracker is installed. Furthermore, after capturing the pose information of the target object in real time, the pose information of the target effect can be determined based on the second relative pose relationship between the target effect and the target object.

[0105] Considering that the motion capture tracker is fixedly mounted on the image acquisition device, the mounting position of the motion capture tracker may deviate from the optical center of the lens of the image acquisition device. Therefore, there may be an offset between the pose of the image acquisition device and the pose of the motion capture tracker mounted on the image acquisition device. Node offset information is used to indicate this offset. For example, the node offset information can be pre-calculated, and then, after the pose information of the motion capture tracker is captured in real time, the pose information of the image acquisition device with higher accuracy can be obtained based on the pose information of the motion capture tracker and the node offset information.

[0106] The node offset information can be pre-calculated using methods such as hand-eye calibration algorithms, for example, the Tsai-Lenz algorithm, the algorithm proposed by Horaud, and the algorithm proposed by Park.

[0107] As an example, firstly, the motion capture tracker is fixedly mounted on an image acquisition device. The image acquisition device is moved to N (N is a positive integer) different locations within a virtual shooting area. At each location, the image acquisition device captures an image including a calibration plate. The pose of the motion capture tracker at the N locations and the N images captured by the image acquisition device are recorded. The calibration plate can be any fixed-spaced patterned array plate, containing feature points with known spatial positions. Then, the feature points are extracted from the N images captured by the image acquisition device. Using the coordinates of these feature points in the images and their spatial positions, existing camera calibration algorithms are employed to calculate the intrinsic parameters and extrinsic parameters of the image acquisition device at the N locations. The intrinsic parameters of the image acquisition device may include focal length, lens optical center coordinates, distortion coefficients, etc., while the extrinsic parameters may include rotation and translation matrices. Furthermore, the node offset information is calculated iteratively using the formula AX = XB, where X represents the node offset matrix (i.e., node offset information), A represents the pose transformation matrix of the image acquisition device corresponding to any two position points among the N position points, and B represents the pose transformation matrix of the motion capture tracker corresponding to any two position points among the N position points; for example, AX = XB can be transformed into: (1) Where: R ai , t ai R represents the rotation transformation matrix and translation transformation matrix of the image acquisition device at different positions, respectively; bi , t bi I1 and I2 represent the rotation and translation transformation matrices of the motion capture tracker between different positions, respectively; I9 and I3 represent the 9x9 and 3x3 identity matrices, respectively, 0 9*3 Represents a 9×3 zero matrix, 0 9*1 Represents a 9×1 zero matrix. Represents matrix t bi The transpose of , T3 represents a 3x3 pose transformation matrix; vec(R x ),t x These represent the rotation offset vector and translation offset vector in the node offset matrix, respectively; Represents the Kronecker product of two matrices; (2) R a X+t a =R b X+t b Where X represents the node offset matrix, R a and t a R represents the rotation matrix and translation matrix of the image acquisition device, respectively. b and t bLet X and Y represent the rotation and translation matrices of the motion capture tracker, respectively. Finally, using the residuals of equations (1) and (2) above as objectives, the solution of the node offset matrix X is obtained through least squares iterative optimization, thereby improving the accuracy and stability of the node offset matrix solution.

[0108] In one possible implementation, the method further includes: in response to a first recording operation, recording one or more of the following: pose information of the image acquisition device, pose information of the target object, and pose information of the target special effect when the first frame is captured. The recorded pose information can be used to adjust the target special effect later.

[0109] For example, special effects artists can trigger the first recording operation by using the recording function button displayed on the terminal device's display. As an example, the display panel of the special effects production tool can display a function button for recording tracking information, as well as a tracking information selection button. Special effects artists can use this selection button to select the desired tracking data to record, i.e., select the pose information of the image acquisition device, the pose information of the target object, or the pose information of the target special effect when recording the first frame. Then, by clicking the recording function button, the corresponding tracking data is recorded and stored, for example, through the terminal device. In this way, relevant tracking data can be recorded during virtual shooting; and based on this tracking data, the target special effect can be quickly adjusted as needed in post-production, further improving the efficiency of post-production special effects adjustments. For example, during virtual filming, a "fireball" effect is added to the actor's hands. The position and pose information of the "fireball" can be recorded. In post-production, the position and pose information of the "fireball" can be used to directly replace the "fireball" in the hands with an "ice ball," thereby achieving rapid adjustment of the target effect. Compared with the frame-by-frame matching method in traditional technology, this greatly improves the efficiency and quality of post-production special effects.

[0110] For example, Figure 6 This diagram illustrates data recorded during a virtual shooting process according to an embodiment of the present disclosure, such as... Figure 6 As shown, during virtual shooting, the tracking data and the timestamps of the footage are aligned. Then, the first frame, the effects layer frame, the pose information of the image acquisition device, and the pose information of the target effects can be recorded. Based on the recorded data, rapid post-production effects adjustments can be made, automatically generating the adjusted composite image or video. In this way, by recording tracking data (i.e., the pose information of the image acquisition device and the target effects) and footage (i.e., the first frame and the effects layer frame captured by the image acquisition device) during virtual shooting, rapid adjustments to the target effects can be made in post-production, greatly improving the efficiency of post-production effects work for relevant personnel.

[0111] It should be noted that post-production staff can also import new special effects using special effects creation tools as needed, and make post-production adjustments to the target special effects generated during the virtual shooting process by inserting and modifying them frame by frame.

[0112] Step 502: Determine the first relative pose relationship based on the pose information of the image acquisition device and the pose information of the target effect when the first frame is captured.

[0113] Since there is a certain distance between the position of the target effect in the real scene and the position of the image acquisition device, there is a difference in their pose information. By using the pose information of the image acquisition device and the pose information of the target effect when the first scene was captured, the first relative pose relationship between the two can be calculated. This first relative pose relationship represents the difference between the pose of the target effect in the real scene and the pose of the image acquisition device.

[0114] Step 503: Based on the first relative pose relationship, determine the target position where the target effect is added to the first image.

[0115] During virtual shooting, a 3D virtual scene is constructed in the rendering engine, along with models such as a display screen model and an image acquisition device model. The display screen model is synchronized with the content displayed on the screen, and the image acquisition device model is synchronized with the field of view of the image acquisition device. For example, a target effect model can be constructed and placed in the aforementioned 3D virtual scene. The relative pose relationship between the target effect model and the image acquisition device model is synchronized with the relative pose relationship between the target effect and the image acquisition device. That is, after the target effect model is placed in the 3D virtual scene, when the relative pose between the image acquisition device and the target effect changes, the relative pose between the target effect model and the image acquisition device model also changes synchronously. Thus, based on the first relative pose relationship, the relative pose relationship between the target effect model and the image acquisition device model in the 3D virtual scene can be determined. Furthermore, combined with the pose of the image acquisition device model in the 3D virtual scene, the pose of the target effect model placed in the 3D virtual scene can be determined.

[0116] Since the first frame is captured by the image acquisition device, the perspective composition when the image acquisition device captures the first frame is the same as the perspective composition of the image acquisition device model. According to the aforementioned method, when the image acquisition device captures the first frame, the pose of the target effect model placed in the three-dimensional virtual scene can be determined based on the first relative pose relationship. The target effect model is placed according to this pose. Then, the position of the target effect model in the perspective composition of the image acquisition device model is the target position where the target effect needs to be added to the first frame.

[0117] Step 504: Based on the target location, add the target effect to the first screen to generate the second screen.

[0118] The second screen has the same size and resolution as the first screen.

[0119] For example, the special effects rendering engine renders a special effects layer image in the perspective composition of the image acquisition device model based on the pose of the target special effects model placed in the three-dimensional virtual scene and the pose of the image acquisition device model. The special effects layer image includes the target special effects. Then, the special effects layer image can be aligned with the first image to generate the second image.

[0120] For example, the special effects layer can be positioned on top of the first image, and the special effects layer can be aligned with the first image based on the target position to generate the second image. That is, the special effects layer can be placed on top of the first image, and the relative positions of the two images can be adjusted until the position of the target effect in the special effects layer overlaps with the target position in the first image. For example, Figure 7 This diagram illustrates a method for generating a second screen according to an embodiment of the present disclosure, such as... Figure 7 As shown, the special effects rendering engine places the special effects layer on top of the first image and aligns the special effects layer with the first image based on the target position to generate the second image. Then, the director, producer, special effects artist, etc. can preview, monitor, and record the second image as needed. For example, the director can preview or monitor the second image through the display panel or monitor screen corresponding to the special effects production tool, and the special effects artist can trigger the recording of the second image through the display panel corresponding to the special effects production tool.

[0121] In this embodiment, a first relative pose relationship is determined based on the pose information of the image acquisition device and the pose information of the target effect when the first frame is captured; then, the target position for adding the target effect to the first frame is determined; finally, based on the target position, the target effect is added to the first frame to generate the second frame. In this way, the target effect is aligned with the first frame captured by the image acquisition device based on the target position, thereby adding the target effect corresponding to the target object to the accurate position in the first frame and generating the second frame with the added effect. As an example, a motion capture device is used to obtain the pose information of the image acquisition device and the target object, and then the first relative pose relationship between the image acquisition device and the target effect is calculated. Based on the first relative pose relationship, the target position is determined, achieving accurate positioning of the target effect in the first frame. In the generated second frame, the target effect matches the target object, thereby ensuring the accurate display of the target effect in the real scene.

[0122] Based on the same inventive concept of the above method embodiments, the present disclosure also provides a device for real-time generation of special effects in virtual shooting, which can be used to execute the technical solutions described in the above method embodiments.

[0123] Figure 8 This diagram illustrates a structural diagram of a real-time special effects generation apparatus for virtual filming according to an embodiment of the present disclosure, such as... Figure 8 As shown, the device includes: a display module 801, used to display a first image captured by an image acquisition device on the display device of a terminal device during virtual shooting; the first image includes a virtual scene displayed on the screen and a real scene in front of the screen; a special effects generation module 802, used to add target special effects corresponding to the target object in real time to the first image in response to the operation of adding special effects to the target object in the real scene, and generate a second image; the display module 801 is also used to display the second image on the display device of the terminal device.

[0124] In this embodiment of the disclosure, during virtual filming, a first frame captured by an image acquisition device is displayed on the display device of a terminal device. The first frame includes a virtual scene displayed on the screen and a real scene in front of the screen. In response to the operation of adding special effects to a target object in the real scene, a target special effect corresponding to the target object is added in real time to the first frame, generating a second frame. The second frame is then displayed on the display device of the terminal device. Thus, during virtual filming, target special effects are added in real time to the frame captured by the image acquisition device, and the before-and-after frames can be shown to the director, producer, and special effects artists. This allows these personnel to instantly view the combination of the added special effects and the actors' performances on the virtual filming set, and to adjust the special effects during the virtual filming process to meet requirements, improving efficiency. Simultaneously, the generated special effects are added to the frame captured by the image acquisition device without needing to be rendered onto the screen displaying the virtual scene, greatly reducing the requirements for rendering performance and effectively lowering the cost of special effects production, resulting in higher economic efficiency.

[0125] In one possible implementation, the special effects generation module 802 is further configured to: in response to the operation of adding special effects to a target object in the real scene, acquire the pose information of the target special effects and the pose information of the image acquisition device when capturing the first frame; determine a first relative pose relationship based on the pose information of the image acquisition device when capturing the first frame and the pose information of the target special effects; determine the target position in the first frame where the target special effects are to be added based on the first relative pose relationship; and add the target special effects to the first frame based on the target position to generate the second frame.

[0126] In one possible implementation, the special effects generation module 802 is further configured to: in response to the operation of adding special effects to a target object in the real scene, determine a second relative pose relationship between the target special effects and the target object; and determine the pose information of the target special effects based on the pose information of the target object and the second relative pose relationship.

[0127] In one possible implementation, the special effects generation module 802 is further configured to: generate a special effects layer image in response to an operation of adding special effects to a target object in the real scene, wherein the special effects layer image includes the target special effects; configure the special effects layer image on top of the first image, and align the special effects layer image with the first image based on the target position to generate the second image.

[0128] In one possible implementation, the pose information of the image acquisition device and the pose information of the target object when the first frame is captured are obtained by tracking through a motion capture device.

[0129] In one possible implementation, the special effects generation module 802 is further configured to: in response to a first recording operation, record one or more of the following: the pose information of the image acquisition device, the pose information of the target object, and the pose information of the target special effects when the first frame is captured.

[0130] In one possible implementation, the special effects generation module 802 is further configured to: record the second frame in response to the second recording operation.

[0131] In one possible implementation, the special effects generation module 802 is further configured to: obtain a first mapping relationship between the color space used by the image acquisition device for shooting and the color space used for special effects rendering, and a second mapping relationship between the color space used for creating special effects and the color space used for special effects rendering; perform color conversion on the first image based on the first mapping relationship; and perform color conversion on the special effects layer image based on the second mapping relationship.

[0132] In one possible implementation, the special effects generation module 802 is further configured to: obtain a third mapping relationship between the color space used for special effects rendering and the color space used for shooting by the image acquisition device; and perform color conversion on the second image based on the third mapping relationship.

[0133] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0134] This disclosure also proposes a computer-readable storage medium storing computer program instructions that, when executed by a processor, implement the above-described method. The computer-readable storage medium can be volatile or non-volatile.

[0135] This disclosure also proposes an electronic device, including: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0136] This disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the above-described method.

[0137] Figure 9 A block diagram of an electronic device 1900 according to an embodiment of the present disclosure is shown. For example, the electronic device 1900 may be provided as a server or a terminal device. (Refer to...) Figure 9 The electronic device 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by memory 1932 for storing instructions, such as application programs, that can be executed by the processing component 1922. The application programs stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1922 is configured to execute instructions to perform the methods described above.

[0138] Electronic device 1900 may also include a power supply component 1926 configured to perform power management of electronic device 1900, a wired or wireless network interface 1950 configured to connect electronic device 1900 to a network, and an input / output interface 1958 (I / O interface). Electronic device 1900 can operate on an operating system, such as Windows Server, stored in memory 1932. TM Mac OS X TM Unix TM Linux TM FreeBSD TM Or similar.

[0139] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by a processing component 1922 of an electronic device 1900 to perform the above-described method.

[0140] This disclosure can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this disclosure.

[0141] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0142] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0143] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0144] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0145] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0146] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0147] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0148] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for real-time generation of special effects in virtual shooting, characterized in that, The method includes: During the virtual shooting process, the first image captured by the image acquisition device is displayed on the display device of the terminal device; the first image includes the virtual scene displayed on the screen and the real scene in front of the screen; In response to the operation of personnel on the virtual shooting set to add special effects to the target object in the real scene, the target special effects corresponding to the target object are added in real time in the first frame to generate the second frame; The second image is displayed on the display device of the terminal equipment for viewing by the personnel at the virtual shooting location.

2. The method according to claim 1, characterized in that, The operation of adding special effects to a target object in the real scene in response to personnel on a virtual shooting location includes adding target special effects corresponding to the target object in real time to the first frame and generating a second frame, including: In response to the operation of adding special effects to the target object in the real scene, the pose information of the target special effects and the pose information of the image acquisition device when the first frame was captured are obtained; Based on the pose information of the image acquisition device and the pose information of the target effect when the first scene was captured, a first relative pose relationship is determined. Based on the first relative pose relationship, determine the target position in the first frame where the target effect is to be added; Based on the target location, the target effect is added to the first screen to generate the second screen.

3. The method according to claim 2, characterized in that, The step of responding to the operation of adding special effects to a target object in the real scene, and obtaining the pose information of the target special effects, includes: In response to the operation of adding special effects to a target object in the real scene, a second relative pose relationship between the target special effects and the target object is determined; Based on the pose information of the target object and the second relative pose relationship, the pose information of the target effect is determined.

4. The method according to claim 2, characterized in that, The method further includes: In response to the operation of adding special effects to the target object in the real scene, a special effects layer image is generated, wherein the special effects layer image includes the target special effects; The step of adding the target effect to the first image based on the target location to generate the second image includes: The special effects layer is placed on top of the first image, and the special effects layer is aligned with the first image based on the target position to generate the second image.

5. The method according to claim 3, characterized in that, The pose information of the image acquisition device and the pose information of the target object when the first frame is captured are obtained by tracking through a motion capture device.

6. The method according to claim 3, characterized in that, The method further includes: In response to the first recording operation, one or more of the following are recorded: the pose information of the image acquisition device, the pose information of the target object, and the pose information of the target special effect when the first frame is captured.

7. The method according to claim 1, characterized in that, The method further includes: In response to the second recording operation, the second screen is recorded.

8. The method according to claim 4, characterized in that, Before placing the special effects layer image on top of the first image, the method further includes: Obtain a first mapping relationship between the color space used by the image acquisition device for shooting and the color space used for special effects rendering, and a second mapping relationship between the color space used for creating special effects and the color space used for rendering special effects; Based on the first mapping relationship, the first image is color-converted; Based on the second mapping relationship, the colors of the special effects layer are converted.

9. The method according to claim 7, characterized in that, Before recording the second frame, the process also includes: Obtain a third mapping relationship between the color space used for special effects rendering and the color space used for image capture by the image acquisition device; Based on the third mapping relationship, the second image undergoes color conversion.

10. A system for real-time generation of special effects in virtual filming, characterized in that, The system includes: A display screen is used to display virtual scenes; An image acquisition device is used to capture a first frame during virtual shooting; the first frame includes a virtual scene displayed on a display screen and a real scene in front of the display screen. A terminal device is used to display a second screen through its display device for viewing by personnel at a virtual shooting location; the second screen is obtained by adding target effects corresponding to the target object in real time to the first screen in response to the personnel's operation of adding special effects to the target object in the real scene.

11. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to implement the method of any one of claims 1 to 9 when executing instructions stored in the memory.

12. A non-volatile computer-readable storage medium storing computer program instructions thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 9.

Images