Special effect real-time generation method, device and system in virtual shooting

Abstract

The invention relates to a special effect real-time generation method, device and system in virtual shooting, and the method comprises the steps: displaying a first image shot by an image collection device in a display device of terminal equipment in a virtual shooting process; the first picture comprises a virtual scene displayed in a screen and a real scene in front of the screen; in response to an operation of adding a special effect to a target object in the real scene, adding a target special effect corresponding to the target object in the first picture in real time, and generating a second picture; and displaying the second picture in a display device of the terminal equipment. According to the method and the device, the target special effect is added in real time in the picture shot by the image acquisition device, the special effect generation efficiency can be effectively improved, meanwhile, the special effect production cost is effectively reduced, and the economical efficiency is higher.

Description

Technical Field

[0001] The present disclosure relates to the technical field of virtual shooting, and in particular to a method, device and system for real-time generation of special effects in virtual shooting. Background Art

[0002] Virtual shooting refers to rendering virtual scenes in real time on screens such as light-emitting diodes (LEDs), with actors and other entities as real scenes. In this way, the real scene and the virtual scene are shot simultaneously by the camera, achieving the effect of placing the actors in a virtual environment. This technology combines real-time rendering of virtual images with actor performances, bringing a new experience to film and television creation.

[0003] Special effects such as "spells" are usually added to TV dramas and movies with themes such as fairy tales, magic, and science fiction. In the relevant technology, the virtual scene is integrated with the real scene of the actors' performance during the virtual shooting process. After the virtual shooting is completed, the "spells" and other special effects are post-produced according to the opinions of the director and other personnel and the content of the script. This method usually requires the director and other personnel to communicate and adjust many times during the post-production process, which is inefficient. Summary of the invention

[0004] In view of this, the present disclosure proposes a method, device, system, electronic device, storage medium and computer program product for real-time generation of special effects in virtual shooting.

[0005] According to one aspect of the present disclosure, a method for real-time generation of special effects in virtual shooting is provided, the method comprising:

[0006] During the virtual shooting process, a first picture shot by the image acquisition device is displayed on a display device of the terminal device; the first picture includes a virtual scene displayed on the screen and a real scene in front of the screen;

[0007] In response to an operation of adding special effects to a target object in the real scene, adding a target special effect corresponding to the target object in real time to the first picture to generate a second picture;

[0008] The second screen is displayed on the display device of the terminal equipment.

[0009] In a possible implementation, in response to the operation of adding special effects to the target object in the real scene, adding a target special effect corresponding to the target object in real time in the first picture to generate a second picture includes:

[0010] In response to an operation of adding special effects to a target object in the real scene, acquiring position information of the target special effects and position information of the image acquisition device when shooting the first picture;

[0011] Determining a first relative posture relationship based on the posture information of the image acquisition device when shooting the first picture and the posture information of the target special effect;

[0012] Based on the first relative position relationship, determining a target position where the target special effect is added to the first picture;

[0013] Based on the target position, the target special effect is added to the first picture to generate the second picture.

[0014] In a possible implementation, in response to the operation of adding special effects to a target object in the real scene, obtaining the position information of the target special effects includes:

[0015] In response to an operation of adding a special effect to a target object in the real scene, determining a second relative position relationship between the target special effect and the target object;

[0016] Based on the pose information of the target object and the second relative pose relationship, the pose information of the target special effect is determined.

[0017] In a possible implementation, the method further includes: in response to an operation of adding a special effect to a target object in the real scene, generating a special effect layer picture, wherein the special effect layer picture includes the target special effect;

[0018] The step of adding the target special effect to the first picture based on the target position to generate the second picture includes: configuring the special effect layer picture on the first picture, and aligning the special effect layer picture with the first picture based on the target position to generate the second picture.

[0019] In a possible implementation, the position information of the image acquisition device and the position information of the target object when the first picture is captured are tracked by a motion capture device.

[0020] In a possible implementation, the method further includes:

[0021] In response to the first recording operation, one or more of the posture information of the image acquisition device, the posture information of the target object, and the posture information of the target special effect when shooting the first picture are recorded.

[0022] In a possible implementation, the method further includes:

[0023] In response to a second recording operation, the second picture is recorded.

[0024] In a possible implementation manner, before configuring the special effect layer picture on the first picture, the method further includes:

[0025] Acquire a first mapping relationship between a color space used by the image acquisition device for shooting and a color space used for special effects rendering, and a second mapping relationship between a color space used for producing special effects and a color space used for the special effects rendering;

[0026] Based on the first mapping relationship, performing color conversion on the first picture;

[0027] Based on the second mapping relationship, the color of the special effect layer picture is converted.

[0028] In a possible implementation manner, before recording the second picture, the method further includes:

[0029] Acquire a third mapping relationship between a color space used for special effects rendering and a color space used for shooting by the image acquisition device;

[0030] Based on the third mapping relationship, color conversion is performed on the second picture.

[0031] According to another aspect of the present disclosure, a device for real-time generation of special effects in virtual shooting is provided, the device comprising:

[0032] A display module is used to display a first picture taken by the image acquisition device in a display device of a terminal device during virtual shooting; the first picture includes a virtual scene displayed on the screen and a real scene in front of the screen;

[0033] A special effect generation module, configured to, in response to an operation of adding a special effect to a target object in the real scene, add a target special effect corresponding to the target object in real time in the first picture to generate a second picture;

[0034] The display module is also used to display the second picture on the display device of the terminal equipment.

[0035] According to another aspect of the present disclosure, a system for real-time generation of special effects in virtual shooting is provided, comprising a display screen for displaying a virtual scene; an image acquisition device for capturing a first picture during virtual shooting; the first picture comprises the virtual scene displayed in the display screen and a real scene in front of the display screen; and a terminal device for displaying a second picture through a display device of the terminal device, wherein the second picture is obtained by adding a target special effect corresponding to the target object in the real scene in real time to the first picture in response to an operation of adding a special effect to the target object in the real scene.

[0036] According to another aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0037] According to another aspect of the present disclosure, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored, wherein the computer program instructions implement the above method when executed by a processor.

[0038] According to another aspect of the present disclosure, a computer program product is provided, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0039] Through various aspects of the present disclosure, during the virtual shooting process, the first picture shot by the image acquisition device is displayed in the display device of the terminal device; the first picture includes the virtual scene displayed in the screen and the real scene in front of the screen; in response to the operation of adding special effects to the target object in the real scene, the target special effects corresponding to the target object are added to the first picture in real time to generate a second picture; the second picture is displayed in the display device of the terminal device. In this way, during the virtual shooting process, the target special effects are added in real time to the picture shot by the image acquisition device, and the pictures before and after adding the target special effects can be shown to the director, producer, special effects production personnel, etc., so that these personnel can instantly view the combined effect of the added special effects and the actor's performance at the scene of the virtual shooting, and can adjust the special effects during the virtual shooting process to meet the needs, thereby improving efficiency. At the same time, the generated special effects are added to the picture shot by the image acquisition device without being rendered to the screen displaying the virtual scene, which greatly reduces the requirements for rendering performance, effectively reduces the cost of special effects production, and is more economical.

[0040] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0042] Figure 1 A structural diagram of a real-time special effect generation system in virtual shooting according to an embodiment of the present disclosure is shown;

[0043] Figure 2 A schematic diagram showing a scene of real-time generation of special effects in virtual shooting according to an embodiment of the present disclosure;

[0044] Figure 3 A flowchart showing a method for real-time generation of special effects in virtual shooting according to an embodiment of the present disclosure is shown;

[0045] Figure 4 A schematic diagram showing a color conversion according to an embodiment of the present disclosure;

[0046] Figure 5 A flow chart of a method for generating a second screen according to an embodiment of the present disclosure is shown;

[0047] Figure 6 A schematic diagram showing data recorded during a virtual shooting process according to an embodiment of the present disclosure;

[0048] Figure 7 A schematic diagram showing a method of generating a second picture according to an embodiment of the present disclosure is shown;

[0049] Figure 8 A structural diagram showing a device for real-time generation of special effects in virtual shooting according to an embodiment of the present disclosure is shown;

[0050] Fig. 9 A block diagram of an electronic device 1900 according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0051] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0052] References to "one embodiment" or "some embodiments" etc. described in this specification mean that one or more embodiments of the present disclosure include specific features, structures or characteristics described in conjunction with the embodiment. Thus, the phrases "exemplary", "in one embodiment", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0053] In the present disclosure, "at least one" means one or more, and "plurality" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: including the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or plural.

[0054] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the main purpose of the present disclosure.

[0055] In order to solve the problem of low efficiency of special effects generation caused by the production of special effects such as "magic" based on the opinions of directors and other personnel and the content of the script in the later stage of virtual shooting, the efficiency of special effects generation can be improved by synthesizing special effects in real time during virtual shooting. In the related technology, Unreal Engine is used to install the RenderStream plug-in, and Disguise (a virtual shooting production platform) is used to start two Unreal Engines to render special effects and virtual scenes respectively, so that special effects and virtual scenes can be displayed on the screen in real time during virtual shooting to achieve synthetic recording; however, this method requires the Disguise central control and multiple rendering machines to work together, which is cumbersome to operate; and special effects and virtual scenes need to be rendered separately using two Unreal Engines to avoid frame drops.

[0056] In the disclosed embodiment, a real-time special effect generation scheme for virtual shooting is proposed (see below for detailed description), which can add target special effects in real time to the picture captured by the image acquisition device (the picture includes the virtual scene displayed on the screen and the real scene in front of the screen) during the virtual shooting process, and can show the pictures before and after adding the target special effects to the director, producer, special effects production personnel, etc., so that these personnel can instantly view the combination of special effects and actor performances at the scene of virtual shooting, and can adjust the special effects during the virtual shooting process to meet the needs, thereby improving efficiency. At the same time, the generated special effects are added to the picture captured by the image acquisition device without rendering to the screen displaying the virtual scene, which greatly reduces the requirements for rendering performance, effectively reduces the cost of special effects production, and is more economical.

[0057] Figure 1 FIG. 4 shows a structural diagram of a real-time special effect generation system in virtual shooting according to an embodiment of the present disclosure; Figure 1 As shown, including:

[0058] The display screen is used to display the virtual scene. According to the actual shooting requirements, the display screen can be configured as a flat screen, a curved screen, a tri-fold screen, or a special-shaped screen with a multi-faceted three-dimensional structure, etc.

[0059] An image acquisition device is used to perform real-time shooting during a virtual shooting process, wherein, during the virtual shooting process, a display screen displays a virtual scene in real time, a real scene can be arranged in front of the display screen, actors can perform in the real scene, and the image acquisition device can simultaneously capture the virtual scene displayed on the display screen and the real scene in front of the display screen; illustratively, during the virtual shooting process, the image acquisition device can shoot in real time at a preset frame rate, and can transmit the captured images to a special effects production tool in real time via an SDI interface, etc.

[0060] Exemplarily, the system may be configured with a special effects production tool, which is used to add desired special effects in real time to the images captured by the image acquisition device, and then render the images after adding special effects through the special effects rendering engine; Exemplarily, the special effects production tool can obtain the images captured by the image acquisition device in real time through the SDI interface. As an example, the special effects production tool may have a display panel and special effects production functions, and directors, producers, special effects production personnel, etc. can use the special effects production tool to perform target object selection, special effects addition, preview, adjustment and other related operations. As an example, a special effects rendering engine can also be configured in the system, and the special effects rendering engine has an image rendering function, which can render and generate images after adding special effects.

[0061] The display device of the terminal device is used to display the picture taken by the image acquisition device, and can also be used to display the picture with special effects added generated by the special effects rendering engine. Exemplarily, the terminal device may include various types of terminals such as laptops, tablet computers, desktop computers, smart phones, smart watches, smart TVs, etc. Exemplarily, the display device of the terminal device may be a display panel corresponding to the special effects production tool. The number of display devices of the terminal device may be one or more, for example, it may also include: a monitoring screen viewed by a director, producer, etc.; the picture with special effects added can be displayed on multiple display devices, so that different staff members can preview and monitor the generation effect of special effects in real time.

[0062] The above-mentioned special effect production tool and special effect rendering engine can be configured in a terminal device having the above-mentioned display device, or can be configured in other terminal devices, or can be configured on a server. The special effect production tool and the special effect rendering engine can be configured on the same device, or on different devices. The functions of the special effect production tool and the special effect rendering engine can also be integrated into the same software for implementation.

[0063] Exemplarily, the system may also include: a virtual scene rendering engine and a motion capture device; wherein the virtual scene rendering engine is used to render the virtual scene in real time and output the rendered virtual scene to a display screen for display. The motion capture device is used to capture the real-time posture information of the image acquisition device and the target object in the real scene, wherein the posture information includes position information and posture information; the motion capture device, which may also be called a motion capture system, is a system that captures information about changes in the motion state of an object, for example, the motion capture device may be OptiTrack, Mosys, etc.; as an example, the motion capture device may include a plurality of motion capture trackers, which may be installed on the image acquisition device and the target object, so as to track the real-time posture of the image acquisition device and the target object at each moment.

[0064] For example, since the images with special effects added generated by the special effects rendering engine are displayed on the display device of the terminal device for viewing by the director, producer, special effects production personnel, etc., without being displayed on the display screen, the size of the display device of the terminal device is usually much smaller than the display screen, and the requirements for rendering performance are relatively low. Therefore, an engine with lower rendering performance can be used as a special effects rendering engine, or one engine can be used as both a special effects rendering engine and a virtual scene rendering engine to meet the needs of real-time special effects generation, thereby reducing the cost of special effects generation. As an example, the special effects rendering engine and the virtual scene rendering engine can be implemented based on the same Unreal Engine without the need to configure two Unreal Engines, thereby effectively reducing the cost of special effects production and being more economical.

[0065] It should be noted that the above system is only an example, and the components in the system can be integrated and configured, or the above system can be configured with more or fewer components, which is not limited.

[0066] Exemplarily, time synchronization can be performed between components in the system. For example, a synchronization signal can be generated by a synchronization signal generator, etc., thereby achieving time synchronization between a motion capture device, an image acquisition device, a display screen, a virtual scene rendering engine, a display device, a special effects production tool, a special effects rendering engine, etc.

[0067] Figure 2 A schematic diagram of a real-time generation scene of special effects in virtual shooting according to an embodiment of the present disclosure is shown as follows: Figure 2 As shown, using the above Figure 1 In the system shown in , during the virtual shooting process, the target object (such as an actor) is located in front of the display screen, and the image acquisition device (such as a camera) is used to frame the scene, thereby capturing a picture including the virtual scene displayed on the display screen and the real scene in front of the display screen. The captured picture can be displayed on the display device of the terminal device. At the same time, the captured picture can be transmitted to the special effects production tool and the special effects rendering engine, and the original picture can be recorded, that is, the picture captured by the image acquisition device can be saved for post-production.

[0068] The motion capture device tracks the posture information of the image acquisition device and the target object in real time, and transmits the tracking data (such as the posture information of the image acquisition device and the posture information of the target object) to the special effects production tool. At the same time, the tracking data (such as the posture information of the image acquisition device) is transmitted to the virtual scene rendering engine.

[0069] The special effects production tool responds to the operation of adding special effects to the target object in the real scene of the picture taken by the image acquisition device by the director, producer, special effects production personnel, etc., and adds the target special effects corresponding to the target object to the picture in real time, and renders the picture after adding the special effects through the special effects rendering engine, thereby realizing real-time generation of special effects, and displays the picture after adding the special effects on the display device of the terminal device so that these personnel can preview and monitor; in some examples, the special effects rendering engine can align the special effects layer picture with the picture taken by the image acquisition device based on the tracking data, so as to realize the real-time addition of the target special effects corresponding to the target object to the picture, and generate the picture after adding the special effects; in other examples, the special effects rendering engine can perform color conversion on the picture taken by the image acquisition device, the special effects layer picture, the picture after adding the special effects, etc. to eliminate color difference; in other examples, the special effects rendering engine can record the picture after adding the special effects and / or the tracking data in response to the recording operation, wherein the recorded tracking data can be used for post-production.

[0070] The virtual scene rendering engine renders the image of the virtual scene in real time based on the tracking data, and displays the rendered image on the display screen. Specifically, the virtual scene rendering engine can provide the image acquisition device model in the virtual scene rendering engine with the position information of the image acquisition device in three-dimensional space. The virtual scene rendering engine relies on real-time rendering and other technologies to synchronize the movement and composition of the image acquisition device model with the image acquisition device, so that the rendered virtual scene conforms to the perspective of the image acquisition device.

[0071] Figure 3 A flowchart of a method for real-time generation of special effects in virtual shooting according to an embodiment of the present disclosure is shown. For example, the method can be performed by the above Figure 1 Some or all of the components in the real-time special effects generation system shown in are executed, such as Figure 3 As shown, the method includes:

[0072] Step 301: During the virtual shooting process, a first picture shot by an image acquisition device is displayed on a display device of a terminal device; the first picture includes a virtual scene displayed on the screen and a real scene in front of the screen.

[0073] The image acquisition device may be Figure 1 In the image acquisition device, the screen can be Figure 1 The virtual scene can be Figure 1 The virtual scene is rendered by the virtual scene rendering engine.

[0074] During the virtual shooting process, each frame of the image captured by the image acquisition device can be displayed in real time on the display device of the terminal device; illustratively, the first image can be any frame of the image captured by the image acquisition device during the virtual shooting process. The virtual shooting can be implemented using conventional virtual shooting technology.

[0075] Step 302: In response to the operation of adding special effects to the target object in the real scene, a target special effect corresponding to the target object is added to the first picture in real time to generate a second picture.

[0076] Wherein, the second picture includes a virtual scene displayed on the screen, a real scene in front of the screen, and a target special effect. Exemplarily, a terminal device or other device having the above-mentioned display device may be configured with a special effect production tool and a special effect rendering engine, wherein the special effect production tool or the special effect rendering engine can add a target special effect corresponding to the target object in real time in the first picture. As an example, the special effect production tool responds to the user's operation of adding special effects to the target object in the real scene, adds a target special effect corresponding to the target object in real time in the first picture, and then the special effect rendering engine renders and generates the second picture. Exemplarily, the special effect rendering engine can be synchronized with the frame rate and timestamp of the image acquisition device to ensure that the picture after adding special effects generated by the special effect rendering engine is the same as the frame rate and timestamp corresponding to the original picture taken by the image acquisition device. It can be understood that during a virtual shooting process, the frame rate of the image acquisition device usually remains unchanged. Therefore, after a synchronization, the frame rate of the special effect rendering engine remains unchanged, and special effects can be directly added to other pictures later.

[0077] The target object refers to entities such as actors, props, and scenery in the real scene of the virtual shooting, and can also be a component of these entities. For example, the target object can be the eyes, hands, feet, etc. of the actors. As an example, when shooting movies and TV series with themes such as fairy tales, magic, and science fiction, the special effects can be "magic spells". For example, if the target object is the actor, the special effect can be "freezing" around the actor's body; for another example, if the target object is the actor's eyes, the special effect can be "laser" emitted by the eyes, or if the target object is the actor's hand, the special effect can be a "fireball" with attack skills appearing in the hand; for another example, if the target object is a stone, the special effect can be "dazzling light" emitted by the stone.

[0078] Exemplarily, the operation of adding special effects to the target object in the real scene may include the operation of selecting the target object and the operation of determining the target special effects to be added. As an example, the display panel corresponding to the special effects production tool may display the first picture, function buttons and prompt information, and the special effects production personnel and the like may select the target object to be added with special effects in the first picture according to the prompt information by using the function button for selecting the target object; the special effects production personnel and the like may also produce or adjust the target special effects to be added in real time according to the prompt information, or may select the target special effects to be added in the pre-built special effects material library. In response to the operation of the special effects production personnel, the special effects rendering engine and the special effects production tool add the target special effects selected or produced by the special effects production personnel to the first picture in real time, and generate a second picture with the target special effects.

[0079] For example, special effects producers can choose to add special effects to a frame or a video captured by the image acquisition device as needed; wherein, when special effects producers need to add special effects to a video including a first frame, the special effects rendering engine and the special effects production tool respond to the special effects producers' operation of adding special effects to the target object in the real scene, and can add target special effects corresponding to the target object in each frame of the video, thereby generating a video with the target special effects. Special effects producers can select a video to add special effects by selecting the start and end time in the video.

[0080] In a possible implementation, the method further includes: generating a special effect layer picture in response to an operation of adding a special effect to a target object in the real scene, wherein the special effect layer picture includes the target special effect.

[0081] The resolution and size of the special effects layer screen may be the same as those of the first screen. As an example, a special effects production tool may be used to generate a special effects layer screen. The special effects production tool responds to the operation of the special effects production personnel selecting the target object, and determines the target position corresponding to the target special effect to be added in the first screen based on the position of the target object in the first screen. The special effects production tool responds to the operation of the special effects production personnel determining the target special effect to be added, determines the target special effect in the special effects layer screen, and uses the above target position as the position of the target special effect in the special effects layer screen. Furthermore, the special effects layer screen may be configured on the first screen, and the first screen and the special effects layer screen may be aligned with the target position as a reference, that is, the target position corresponding to the target special effect to be added in the first screen is aligned with the position of the target special effect in the special effects layer screen, so that the special effects layer screen overlaps with the first screen to generate a second screen.

[0082] By adding the target special effects by generating the special effects layer screen, the original first screen may not be changed, making the special effects addition more convenient and flexible. Those skilled in the art should understand that special effects may not be added through the special effects layer screen, for example, special effects may be added directly to the first screen by directly processing the first screen, and this application does not limit this.

[0083] Since different devices may use different color spaces, the images output by different devices may have color differences. For example, the color space used by the special effects production tool may be different from the color space used by the image acquisition device. In this way, there is a color difference between the special effects layer image generated by the special effects production tool and the first image captured by the image acquisition device. At the same time, the special effects rendering engine will synthesize the special effects layer image and the first image to render a second image, and the color space used by the special effects rendering engine may also be different from the color space of the special effects production tool and the image acquisition device. Therefore, in order to eliminate the color difference between the target special effects and other content in the generated second image, the color difference between the target special effects and other content in the image can be eliminated before the second image is generated. In a possible implementation, before configuring the special effects layer picture on the first picture, it also includes: obtaining a first mapping relationship between the color space used by the image acquisition device for shooting and the color space used for special effects rendering, and a second mapping relationship between the color space used for making special effects and the color space used for special effects rendering; based on the first mapping relationship, performing color conversion on the first picture; based on the second mapping relationship, performing color conversion on the special effects layer picture.

[0084] Among them, the color space can be RGB color space, CMYK color space, Lab color space, YCbcr color space, etc., and the mapping relationship between different color spaces can be represented by a conversion matrix. Exemplarily, the color space used for making special effects can be the color space used by the special effects making tool or the color space used by the special effects rendering engine to make special effects; as an example, the color space used for making special effects can be the color space used by the special effects making tool, and the color space used for special effects rendering can be the color space used for rendering by the special effects rendering engine. The color calibration mapping relationship LUT (Look-Up Table) of the image acquisition device and the color calibration mapping relationship lookup table of the special effects making tool can be obtained in advance from the manufacturer of the device or pre-calibrated in advance, and the lookup table contains the mapping relationship between the color space used by the corresponding device and other color spaces. In this way, the first picture and the special effects layer picture are both color converted, so that the color space used by the image acquisition device corresponding to the first picture and the color space used by the special effects making corresponding to the special effects layer picture are uniformly converted to the color space used for special effects rendering, eliminating the color difference between the first picture and the special effects layer picture.

[0085] Step 303: Display the second picture on the display device of the terminal equipment.

[0086] Exemplarily, the first picture and the second picture can be displayed on the display device of the terminal device at the same time, so that the director, producer, special effects production personnel, etc. can better view and compare the effects before and after adding special effects.

[0087] As an example, the display device of the terminal device may be a display panel corresponding to a special effects production tool, and the second picture may be displayed in the display panel so that the special effects production personnel can preview the special effects generation effect; as another example, the display device of the terminal device may be a director's monitoring screen, and the second picture is displayed on the director's monitoring screen so that the director can monitor the special effects generation effect in real time.

[0088] Since the color space used by the display device of the terminal device for display may be different from the color space used for special effect rendering, in a possible implementation, the second picture may be color converted before the second picture is displayed. Exemplarily, the mapping relationship between the color space used by the display device of the terminal device for display and the color space used for special effect rendering may be obtained, and the second picture rendered by the special effect may be color converted based on the mapping relationship, and the second picture after color conversion may be displayed on the display device of the terminal device; in this way, there is no color difference between the first picture and the second picture displayed on the display device of the terminal device, so that the director, producer, special effect producer, etc. can better compare the effects before and after the special effect is added.

[0089] Exemplarily, when there are multiple display devices in the terminal equipment, the color spaces used for display by each display device may also be different. In this case, the second picture can be color converted based on the mapping relationship between each display device and the color space used for special effects rendering, and the second picture after color conversion is displayed on the corresponding display device. For example, when it is necessary to simultaneously display the second picture on the director's monitoring screen and the display panel corresponding to the special effects production tool, the second picture can be color converted based on the mapping relationship between the color space used for display of the display panel and the color space used for special effects rendering, and then displayed on the display panel. At the same time, based on the mapping relationship between the color space used for display of the director's monitoring screen and the color space used for special effects rendering, the second picture can be color converted and then displayed on the monitoring screen, so that the second picture is displayed synchronously on the display panel and the monitoring screen, so as to facilitate viewing by different personnel such as directors, producers, and special effects producers.

[0090] In a possible implementation manner, the method further includes: recording the second picture in response to a second recording operation.

[0091] Exemplarily, special effects producers and the like can trigger the second recording operation through the function button for recording displayed in the display device of the terminal device; as an example, the display panel corresponding to the special effects production tool can display a function button for recording the picture, and the special effects producers and the like can trigger the recording of the second picture by clicking the button, and the second picture can be recorded and stored by the terminal device; the recorded second picture contains the original picture content captured by the image acquisition device and the added special effects, so that in the process of virtual shooting, the picture with the added special effects can be obtained in real time. Exemplarily, in a scenario where special effects are added to a video captured by the image acquisition device in real time, the terminal device records the video in response to the operation of recording the picture.

[0092] Furthermore, based on the timestamp, the first frame in the original video shot by the recorded image acquisition device can be replaced with the recorded second frame, thereby obtaining a complete video with special effects added.

[0093] Considering that the color space used for special effects rendering may be different from the color space used for shooting by the image acquisition device, in order to ensure that there is no deviation between the recorded second picture and the first picture shot by the image acquisition device, the second picture may be color converted when recording the second picture. In a possible implementation, before recording the second picture, it also includes: obtaining a third mapping relationship between the color space used for special effects rendering and the color space used for shooting by the image acquisition device; based on the third mapping relationship, performing color conversion on the second picture. In this way, the second picture is color converted, so that the color space used for special effects rendering corresponding to the second picture is converted into the color space used for shooting by the image acquisition device. There is no color difference between the second picture after color conversion and the picture shot by the image acquisition device, so that the display effect of each picture recorded in the complete video with special effects added is consistent. It can be understood that the third mapping relationship is reversible with the first mapping relationship.

[0094] In the disclosed embodiment, during the virtual shooting process, the first picture shot by the image acquisition device is displayed in the display device of the terminal device; the first picture includes the virtual scene displayed in the screen and the real scene in front of the screen; in response to the operation of adding special effects to the target object in the real scene, the target special effects corresponding to the target object are added in real time to the first picture to generate a second picture; the second picture is displayed in the display device of the terminal device. In this way, during the virtual shooting process, the target special effects are added in real time to the picture shot by the image acquisition device, and the pictures before and after the addition of the target special effects can be shown to the director, producer, special effects production personnel, etc., so that these personnel can instantly view the combined effect of the added special effects and the actor's performance at the scene of the virtual shooting, and can adjust the special effects during the virtual shooting process to meet the needs, thereby improving efficiency. At the same time, the generated special effects are added to the picture shot by the image acquisition device without being rendered to the screen displaying the virtual scene, which greatly reduces the requirements for rendering performance, effectively reduces the cost of special effects production, and is more economical.

[0095] In some examples, as described above, considering that the color spaces used in different devices may be different, in the process of generating the target special effects, the embodiments of the present disclosure may also perform color conversion on the first picture taken by the image acquisition device, the special effect layer picture, the second picture after adding the target special effects, etc., to eliminate the color difference caused by the different color spaces used by different devices. For example, Figure 4 A schematic diagram of color conversion according to an embodiment of the present disclosure is shown as follows: Figure 4 As shown, first, the color space A used by the image acquisition device for shooting and the color space B used by the special effect production tool are uniformly converted into the color space C used by the special effect rendering engine for rendering, that is, the conversion matrix M from the color space A used by the image acquisition device for shooting to the color space C used by the special effect rendering engine for rendering is ac (i.e., the first mapping relationship) performs color conversion on the first picture, based on the conversion matrix M from the color space B used by the special effects production tool to the color space C used by the special effects rendering engine bc (i.e., the second mapping relationship) performs color conversion on the special effect layer screen, and then aligns the first screen after color conversion with the special effect layer screen, and the special effect rendering engine renders the second screen. Afterwards, when the second screen is previewed and monitored through the display panel corresponding to the special effect production tool, the color space corresponding to the monitoring is consistent with the color space corresponding to the preview, both of which are the color space D used for display on the display panel. The color space C used for rendering by the special effect rendering engine can be converted to the color space D used for display on the display panel, that is, the conversion matrix M based on the color space C used for rendering by the special effect rendering engine to the color space D used for display on the display panel cdThe second picture is displayed on the display panel after color conversion. When recording the second picture, the color space corresponding to the recording is consistent with the color space A used by the image acquisition device for shooting. The color space C used by the special effects rendering engine for rendering can be converted to the color space A used by the image acquisition device for shooting, that is, based on the conversion matrix from the color space C used by the special effects rendering engine for rendering to the color space A used by the image acquisition device for shooting (i.e., the third mapping relationship) performs color conversion on the second picture rendered by the special effects rendering engine (i.e., the original picture rendered by the special effects rendering engine without color conversion), wherein: With M ac The inverse matrices are used to record the second picture after color conversion, so that the color space corresponding to the recorded picture with special effects added is consistent with the color space corresponding to the original picture taken by the image acquisition device. In this way, the color space corresponding to the first picture and the special effect layer picture is unified, and the second picture is rendered. After the second picture is rendered, the color space corresponding to the second picture is adjusted to the color space corresponding to preview, monitoring, recording, etc., so as to eliminate the color difference in preview, monitoring, and recording.

[0096] The following is an exemplary description of the specific process of generating the second picture by adding a target special effect corresponding to the target object in the real scene in the above step 302 in real time.

[0097] Figure 5 A flow chart of a method for generating a second screen according to an embodiment of the present disclosure is shown as follows: Figure 5 As shown, the following steps are included:

[0098] Step 501: In response to an operation of adding special effects to a target object in the real scene, obtaining the position information of the target special effects and the position information of the image acquisition device when shooting the first picture.

[0099] The pose information of the image acquisition device may include the position information and pose information of the image acquisition device at each moment. As an example, the position information of the image acquisition device may indicate the position of the optical center of the lens of the image acquisition device in a preset coordinate system (such as a world coordinate system), and the pose information of the image acquisition device may indicate the shooting direction of the lens of the image acquisition device in the preset coordinate system. The pose information of the target special effect may include the position information and pose information of the target special effect corresponding to the real scene at each moment. As an example, the position information of the target special effect may indicate the position of the target special effect corresponding to the real scene (under the preset coordinate system), and the pose information of the target special effect may indicate the direction of the target special effect corresponding to the real scene (under the preset coordinate system).

[0100] In a possible implementation, in this step, in response to the operation of adding special effects to the target object in the real scene, obtaining the pose information of the target special effect; may include: in response to the operation of adding special effects to the target object in the real scene, determining a second relative pose relationship between the target special effect and the target object; based on the pose information of the target object and the second relative pose relationship, determining the pose information of the target special effect.

[0101] It can be understood that the target special effect can be in the same posture as the target object, and the corresponding second relative posture relationship between the two indicates that there is no difference in the posture of the two. For example, the target object is the actor's hand, and the target special effect is the "fireball" in the actor's hand. At the same time, the posture of the "fireball" and the "hand" remain consistent; the target special effect may also be at a certain distance from the target object, and there is a difference in the posture information of the two. The corresponding second relative posture relationship between the two indicates the difference in posture between the two, wherein the size of the difference indicated by the second relative posture relationship can be set according to needs. For example, the target object is the actor's foot, and the target special effect is the "glowing footprints left by the previous step" when the actor walks. At the same time, there is a difference in posture between the "glowing footprints left by the previous step" and the "foot". The second relative posture relationship between the "glowing footprints left by the previous step" and the "foot" can be set to a difference of 0.5 meters between their positions along the direction of the footsteps.

[0102] Since the target object is an entity in the real scene, its posture information is easier to obtain. Therefore, after obtaining the posture information of the target object, the posture relationship of the target feature can be further determined in combination with the second relative posture relationship. Exemplarily, the second relative posture relationship is relatively fixed. For example, when a "fireball" is displayed on the hand of an actor, the second relative posture relationship between the target special effect "fireball" and the target object "hand" remains unchanged, thereby showing the effect of always having a "fireball" on the hand; for another example, when an actor walks normally, it is usually at a uniform speed, and the second relative posture relationship between the "glowing footprints left by the previous step" and the "foot" remains unchanged; in this way, after the second relative posture relationship between the target special effect and the target object is preset, as the posture of the target object changes, based on the latest posture information of the target object and the second relative posture relationship, the latest posture information of the target special effect can be determined.

[0103] In a possible implementation, the position information of the image acquisition device and the position information of the target object when the first picture is captured are tracked by a motion capture device.

[0104] Exemplarily, one or more motion capture trackers can be pre-installed on both the image acquisition device and the target object, wherein the location and number of the motion capture trackers can be configured as required; for example, if the target object is an actor, the motion capture tracker can be installed on the back of the actor's body; for another example, if the target object is the actor's hands, feet, limbs and other body parts, the motion capture tracker can be installed on the corresponding body parts; for another example, the target object can be a leaf in a real scene, and the motion capture tracker can be installed on the leaf. The motion capture device can capture the pose information of each motion capture tracker in a preset coordinate system (such as the world coordinate system) in real time, and the pose information can be used as the pose information of the entity (i.e., the image acquisition device or the target object) on which the corresponding motion capture tracker is installed. Furthermore, after the pose information of the target object is captured in real time, the pose information of the target special effect can be determined based on the second relative pose relationship between the target special effect and the target object.

[0105] Considering that the motion capture tracker is fixedly mounted on the image acquisition device, the mounting position of the motion capture tracker may deviate from the optical center of the lens of the image acquisition device. Therefore, there may be an offset between the pose of the image acquisition device and the pose of the motion capture tracker mounted on the image acquisition device, and the node offset information is used to indicate the offset. Exemplarily, the node offset information can be calculated in advance, and then after the pose information of the motion capture tracker is captured in real time, the pose information of the image acquisition device with higher accuracy can be obtained based on the pose information of the motion capture tracker and the node offset information.

[0106] The node offset information may be pre-calculated by using a hand-eye calibration algorithm or the like, for example, the Tsai-Lenz algorithm, the algorithm proposed by Horaud, the algorithm proposed by Park, and the like.

[0107] As an example, first, the motion capture tracker is fixedly installed on the image acquisition device, and the image acquisition device is moved to N (N is a positive integer) different positions in the virtual shooting site. The image acquisition device takes a picture containing a calibration plate at each position, and records the position of the motion capture tracker at the N positions and the N pictures taken by the image acquisition device, wherein the calibration plate can be any fixed-spacing pattern array plate, and the calibration plate includes feature points with known spatial positions. Then, the feature points in the N pictures taken by the image acquisition device are extracted, and the coordinates of these feature points in the pictures and the spatial positions of the feature points are used, and the existing camera calibration algorithm is used to calculate the internal parameters of the image acquisition device and the external parameters of the image acquisition device at the N positions, wherein the internal parameters of the image acquisition device may include focal length, lens optical center coordinates, distortion coefficient, etc., and the external parameters of the image acquisition device may include rotation matrix and translation matrix. Then, the node offset information is calculated by iterative method using the formula AX=XB, where X represents the node offset matrix (i.e., the node offset information), A represents the pose transformation matrix of the image acquisition device corresponding to any two positions among the N positions, and B represents the pose transformation matrix of the motion capture tracker corresponding to any two positions among the N positions; exemplarily, AX=XB can be transformed into: (1) Where: R ai , t ai Respectively represent the rotation transformation matrix and translation transformation matrix of the image acquisition device at different positions; R bi , t bi They represent the rotation transformation matrix and translation transformation matrix of the motion capture tracker at different positions; I9 and I3 represent the 9x9 and 3x3 unit matrices respectively, and 0 9*3 represents a 9×3 zero matrix, 0 9*1 represents a 9×1 zero matrix, Represents the matrix t bi The transpose of T3 represents the 3x3 pose transformation matrix; vec(R x ), t x Respectively represent the rotation offset vector and translation offset vector in the node offset matrix; represents the Kronecker product of two matrices; (2) R a X+t a =R b X+t b , where X represents the node offset matrix, R a and t a Respectively represent the rotation matrix and translation matrix of the image acquisition device, R b and t bThey represent the rotation matrix and translation matrix of the motion capture tracker respectively. Finally, the residuals of equations (1) and (2) are taken as the targets, and the solution of the node offset matrix X is obtained by iterative optimization using the least squares method, thereby improving the accuracy and stability of the node offset matrix solution.

[0108] In a possible implementation, the method further includes: in response to the first recording operation, recording one or more of the posture information of the image acquisition device, the posture information of the target object, and the posture information of the target special effect when shooting the first picture. The recorded posture information can be used to adjust the target special effect later.

[0109] Exemplarily, special effects producers and the like can trigger the first recording operation through the function button for recording displayed in the display device of the terminal device; as an example, the display panel corresponding to the special effects production tool can display the function button for recording tracking information and the selection button for tracking information, and the special effects producers and the like can select the tracking data to be recorded through the selection button, that is, select the posture information of the image acquisition device when recording the first picture, the posture information of the target object, or the posture information of the target special effect, and then click the recording function button to trigger the recording of the corresponding tracking data, and store the tracking data, for example, record and store the tracking data through the terminal device. In this way, relevant tracking data can be recorded during the virtual shooting process; and then based on the tracking data, the target special effects can be quickly adjusted as needed in the later stage, further improving the efficiency of the later special effects adjustment. For example, during the virtual shooting process, a "fireball" is added to the actor's hand as a special effect, and the position information of the "fireball" can be recorded. In post-production, the position information of the "fireball" can be used to directly use the "ice ball" to replace the "fireball" on the hand, so as to quickly adjust the target special effects. Compared with the frame-by-frame matching method in traditional technology, the efficiency and quality of post-production special effects production are greatly improved.

[0110] For example, Figure 6 A schematic diagram showing recording data during a virtual shooting process according to an embodiment of the present disclosure is shown as follows: Figure 6 As shown, during the virtual shooting process, the tracking data and the timestamps of the pictures are aligned, and then the first picture, the special effect layer picture, the position information of the image acquisition device, and the position information of the target special effect can be recorded, and then the special effects can be quickly adjusted based on the recorded data, and the adjusted special effects composite picture or video can be automatically generated. In this way, by recording the tracking data (i.e., the position information of the image acquisition device, the position information of the target special effect) and the picture (i.e., the first picture and the special effect layer picture taken by the image acquisition device) during the virtual shooting process, the target special effect can be quickly adjusted in the later stage, which greatly improves the efficiency of the relevant staff in the later special effects production.

[0111] It should be noted that post-production staff can also use special effects production tools to import new special effects according to needs, and make post-production adjustments to the target special effects generated during the virtual shooting process by inserting and modifying them frame by frame.

[0112] Step 502: Determine a first relative posture relationship based on the posture information of the image acquisition device when shooting the first picture and the posture information of the target special effect.

[0113] Since there is a certain distance between the position corresponding to the target special effect in the real scene and the position of the image acquisition device, there is a difference in the posture information of the two. The first relative posture relationship between the two can be calculated by using the posture information of the image acquisition device when shooting the first picture and the posture information of the target special effect obtained as mentioned above. The first relative posture relationship indicates the difference between the posture corresponding to the target special effect in the real scene and the posture of the image acquisition device.

[0114] Step 503: Based on the first relative posture relationship, determine the target position where the target special effect is added to the first picture.

[0115] During the virtual shooting process, a three-dimensional virtual scene is constructed in the rendering engine, and models such as a display screen model and an image acquisition device model are constructed; wherein the display screen model is synchronized with the content displayed on the display screen, and the image acquisition device model is synchronized with the field of view composition of the image acquisition device; illustratively, a target special effect model can be constructed and placed in the above-mentioned three-dimensional virtual scene, and the relative posture relationship between the target special effect model and the image acquisition device model is synchronized with the relative posture relationship between the target special effect and the image acquisition device, that is, after the target special effect model is placed in the three-dimensional virtual scene, when the relative posture of the image acquisition device and the target special effect changes, the relative posture between the target special effect model and the image acquisition device model will also change synchronously; in this way, based on the first relative posture relationship, the relative posture relationship between the target special effect model and the image acquisition device model in the three-dimensional virtual scene can be determined, and then combined with the posture of the image acquisition device model in the three-dimensional virtual scene, the posture of the target special effect model placed in the three-dimensional virtual scene can be determined.

[0116] Since the first picture is a picture taken by the image acquisition device, the perspective composition when the image acquisition device takes the first picture is the same as the perspective composition of the image acquisition device model. According to the aforementioned method, when the image acquisition device takes the first picture, the posture of the target special effects model placed in the three-dimensional virtual scene can be determined based on the first relative posture relationship, and the target special effects model is placed according to the posture. Then, the position of the target special effects model in the perspective composition of the image acquisition device model is the target position where the target special effects need to be added to the first picture.

[0117] Step 504: Based on the target position, add the target special effect to the first picture to generate the second picture.

[0118] The second picture has the same size and resolution as the first picture.

[0119] Exemplarily, the special effects rendering engine renders a special effects layer picture in the perspective composition of the image acquisition device model based on the posture of the target special effects model placed in the three-dimensional virtual scene and the posture of the image acquisition device model, and the special effects layer picture includes the target special effects; the special effects layer picture can then be aligned with the first picture to generate the second picture.

[0120] For example, the special effect layer screen can be placed on the first screen, and the special effect layer screen can be aligned with the first screen based on the target position to generate the second screen. That is, the special effect layer screen can be placed on the first screen, and the relative positions of the two screens can be adjusted until the position of the target special effect in the special effect layer screen overlaps with the target position in the first screen. For example, Figure 7 A schematic diagram of generating a second picture according to an embodiment of the present disclosure is shown as follows: Figure 7 As shown, the special effects rendering engine configures the special effects layer picture on the first picture, and aligns the special effects layer picture with the first picture based on the target position to generate a second picture; then, the director, producer, special effects production personnel, etc. can preview, monitor, record, etc. the second picture as needed. For example, the director can preview or monitor the second picture on the display panel or monitoring screen corresponding to the special effects production tool, and the special effects production personnel can trigger the recording of the second picture on the display panel corresponding to the special effects production tool.

[0121] In the disclosed embodiment, based on the posture information of the image acquisition device and the posture information of the target special effect when shooting the first picture, a first relative posture relationship is determined; then the target position of the target special effect added to the first picture is determined; finally, based on the target position, the target special effect is added to the first picture to generate the second picture. In this way, the target special effect is relatively aligned with the first picture shot by the image acquisition device based on the target position, so that the target special effect corresponding to the target object is added to the accurate position of the first picture, and the second picture after the special effect is added is generated. As an example, a motion capture device is used to obtain the posture information of the image acquisition device and the target object, and then the first relative posture relationship between the image acquisition device and the target special effect is calculated. The target position is determined based on the first relative posture relationship, and the target special effect is accurately positioned in the first picture. In the generated second picture, the target special effect matches the target object, thereby ensuring the accurate display of the target special effect in the real scene.

[0122] Based on the same inventive concept of the above method embodiments, the embodiments of the present disclosure also provide a device for real-time generation of special effects in virtual shooting, which can be used to execute the technical solution described in the above method embodiments.

[0123] Figure 8 A structural diagram of a device for real-time generation of special effects in virtual shooting according to an embodiment of the present disclosure is shown as follows: Figure 8 As shown, the device includes: a display module 801, which is used to display a first picture taken by an image acquisition device in a display device of a terminal device during a virtual shooting process; the first picture includes a virtual scene displayed in the screen and a real scene in front of the screen; a special effect generation module 802, which is used to respond to an operation of adding special effects to a target object in the real scene, and to add a target special effect corresponding to the target object in real time in the first picture to generate a second picture; the display module 801 is also used to display the second picture in the display device of the terminal device.

[0124] In the disclosed embodiment, during the virtual shooting process, the first picture shot by the image acquisition device is displayed in the display device of the terminal device; the first picture includes the virtual scene displayed in the screen and the real scene in front of the screen; in response to the operation of adding special effects to the target object in the real scene, the target special effects corresponding to the target object are added in real time to the first picture to generate a second picture; the second picture is displayed in the display device of the terminal device. In this way, during the virtual shooting process, the target special effects are added in real time to the picture shot by the image acquisition device, and the pictures before and after adding the target special effects can be shown to the director, producer, special effects production personnel, etc., so that these personnel can instantly view the combined effect of the added special effects and the actor's performance at the scene of the virtual shooting, and can adjust the special effects during the virtual shooting process to meet the needs, thereby improving efficiency. At the same time, the generated special effects are added to the picture shot by the image acquisition device without rendering to the screen displaying the virtual scene, which greatly reduces the requirements for rendering performance, effectively reduces the cost of special effects production, and is more economical.

[0125] In a possible implementation, the special effects generation module 802 is further used to: in response to an operation of adding special effects to a target object in the real scene, obtain the posture information of the target special effect and the posture information of the image acquisition device when shooting the first picture; determine a first relative posture relationship based on the posture information of the image acquisition device when shooting the first picture and the posture information of the target special effect; determine a target position of the target special effect added to the first picture based on the first relative posture relationship; and add the target special effect to the first picture based on the target position to generate the second picture.

[0126] In a possible implementation, the special effects generation module 802 is further used to: determine a second relative posture relationship between the target special effect and the target object in response to an operation of adding special effects to the target object in the real scene; and determine the posture information of the target special effect based on the posture information of the target object and the second relative posture relationship.

[0127] In one possible implementation, the special effect generation module 802 is further used to: generate a special effect layer screen in response to an operation of adding special effects to a target object in the real scene, wherein the special effect layer screen includes the target special effect; configure the special effect layer screen on the first screen, and align the special effect layer screen with the first screen based on the target position to generate the second screen.

[0128] In a possible implementation, the position information of the image acquisition device and the position information of the target object when the first picture is captured are tracked by a motion capture device.

[0129] In a possible implementation, the special effects generation module 802 is further used to: in response to a first recording operation, record one or more of the posture information of the image acquisition device when shooting the first picture, the posture information of the target object, and the posture information of the target special effect.

[0130] In a possible implementation, the special effect generation module 802 is further configured to: record the second picture in response to a second recording operation.

[0131] In a possible implementation, the special effects generation module 802 is further used to: obtain a first mapping relationship between the color space used for shooting by the image acquisition device and the color space used for special effects rendering, and a second mapping relationship between the color space used for producing special effects and the color space used for special effects rendering; based on the first mapping relationship, perform color conversion on the first picture; based on the second mapping relationship, perform color conversion on the special effects layer picture.

[0132] In a possible implementation, the special effect generation module 802 is further used to: obtain a third mapping relationship between the color space used for special effect rendering and the color space used for shooting by the image acquisition device; and perform color conversion on the second picture based on the third mapping relationship.

[0133] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0134] The embodiment of the present disclosure also provides a computer-readable storage medium on which computer program instructions are stored, and the computer program instructions implement the above method when executed by a processor. The computer-readable storage medium can be a volatile or non-volatile computer-readable storage medium.

[0135] An embodiment of the present disclosure further proposes an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0136] The embodiments of the present disclosure also provide a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0137] Fig. 9 1 is a block diagram of an electronic device 1900 according to an embodiment of the present disclosure. For example, the electronic device 1900 may be provided as a server or a terminal device. Fig. 9 , the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0138] The electronic device 1900 may also include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output interface 1958 (I / O interface). The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server 2003. TM , Mac OS X TM , Unix TM ,Linux TM , FreeBSD TM or similar.

[0139] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the electronic device 1900 to perform the above method.

[0140] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0141] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0142] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0143] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0144] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0145] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0146] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0147] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0148] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for real-time generation of special effects in virtual shooting, characterized in that: The method comprises: During the virtual shooting process, a first picture shot by the image acquisition device is displayed on a display device of the terminal device; the first picture includes a virtual scene displayed on the screen and a real scene in front of the screen; In response to an operation of adding special effects to a target object in the real scene, adding a target special effect corresponding to the target object in real time to the first picture to generate a second picture; The second screen is displayed on the display device of the terminal equipment.

2. The method according to claim 1, characterized in that In response to the operation of adding special effects to the target object in the real scene, adding a target special effect corresponding to the target object in real time in the first picture to generate a second picture includes: In response to an operation of adding special effects to a target object in the real scene, acquiring position information of the target special effects and position information of the image acquisition device when shooting the first picture; Determining a first relative posture relationship based on the posture information of the image acquisition device when shooting the first picture and the posture information of the target special effect; Based on the first relative position relationship, determining a target position where the target special effect is added to the first picture; Based on the target position, the target special effect is added to the first picture to generate the second picture.

3. The method according to claim 2, characterized in that The step of obtaining the position information of the target special effect in response to the operation of adding the special effect to the target object in the real scene comprises: In response to an operation of adding a special effect to a target object in the real scene, determining a second relative position relationship between the target special effect and the target object; Based on the pose information of the target object and the second relative pose relationship, the pose information of the target special effect is determined.

4. The method according to claim 2, characterized in that: The method further comprises: In response to an operation of adding a special effect to a target object in the real scene, generating a special effect layer picture, wherein the special effect layer picture includes the target special effect; The adding the target special effect to the first picture based on the target position to generate the second picture includes: The special effect layer picture is arranged on the first picture, and the special effect layer picture is aligned with the first picture based on the target position to generate the second picture.

5. The method according to claim 3, characterized in that: The position information of the image acquisition device and the position information of the target object when the first picture is captured are tracked by a motion capture device.

6. The method according to claim 3, characterized in that The method further comprises: In response to the first recording operation, one or more of the posture information of the image acquisition device, the posture information of the target object, and the posture information of the target special effect when shooting the first picture are recorded.

7. The method according to claim 1, characterized in that The method further comprises: In response to a second recording operation, the second picture is recorded.

8. The method according to claim 4, characterized in that Before configuring the special effect layer picture on the first picture, the method further includes: Acquire a first mapping relationship between a color space used by the image acquisition device for shooting and a color space used for special effects rendering, and a second mapping relationship between a color space used for producing special effects and a color space used for the special effects rendering; Based on the first mapping relationship, performing color conversion on the first picture; Based on the second mapping relationship, the color of the special effect layer picture is converted.

9. The method according to claim 7, characterized in that: Before recording the second picture, the method further includes: Acquire a third mapping relationship between a color space used for special effects rendering and a color space used for shooting by the image acquisition device; Based on the third mapping relationship, color conversion is performed on the second picture.

10. A real-time special effects generation system for virtual shooting, characterized in that: The system comprises: A display screen, used for displaying a virtual scene; The image acquisition device is used to capture a first picture during virtual shooting; the first picture includes a virtual scene displayed on the display screen and a real scene in front of the display screen; The terminal device is used to display a second picture through the display device of the terminal device, wherein the second picture is obtained by adding a target special effect corresponding to the target object in the real scene in real time in the first picture in response to an operation of adding a special effect to the target object in the real scene.

11. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to implement the method described in any one of claims 1 to 9 when executing the instructions stored in the memory.

12. A non-volatile computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images