Lighting controlling method in virtual film production, device, equipment and storage medium

Abstract

The application provides a light control method and device in virtual production, wherein the virtual production is used for fusing a real scene and a virtual scene to generate a target video; the method comprises the following steps: taking a physical lamp in the real scene as a reference, creating a virtual lamp corresponding to the physical lamp in the virtual scene; obtaining a target audio applied to the target video and playing the target audio; in the process of playing the target audio, controlling the virtual lamp to generate a first light effect corresponding to the currently played target audio in the virtual scene, and synchronously controlling the physical lamp to generate a second light effect corresponding to the currently played target audio in the real scene; wherein the first light effect and the second light effect are used for virtual production to generate the target video containing the first light effect and the second light effect. Through the application, synchronous control of the virtual lamp and the physical lamp can be realized, and the efficiency of the virtual production is improved.

Description

Technical Field

[0001] This application relates to the field of virtual production, and more particularly to a lighting control method, apparatus, device, computer-readable storage medium, and computer program product in virtual production. Background Technology

[0002] Virtual production is a novel production method that integrates virtual computer graphics with the performances of real actors, allowing for real-time visualization of special effects on set. In virtual production, filmmakers can avoid green screens and weeks or even months of post-production; all key decision-makers collaborate on-site to generate the final footage in real time, with the ability to make changes at any time.

[0003] The effectiveness of virtual production largely depends on lighting effects. When shooting against a background of LED (light-emitting diode) screens, it's often necessary to synchronize virtual lighting, physical lighting, and on-site sound. In this approach, sound engineers, lighting technicians, and virtual lighting operators typically activate the lighting and sound effects in unison according to the director's instructions. However, this method makes it difficult for operators of physical, sound, and virtual lighting to maintain perfect synchronization, resulting in poor synchronization and low shooting efficiency. Summary of the Invention

[0004] This application provides a lighting control method, apparatus, device, computer-readable storage medium, and computer program product for virtual film production, which can realize synchronous control of virtual and physical lighting fixtures and improve the efficiency of virtual film production.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] This application provides a lighting control method in virtual video production, wherein the virtual video production is used to generate a target video by combining real-world and virtual scenes, including:

[0007] Using the physical lighting fixtures in the real scene as a reference, create virtual lighting fixtures in the virtual scene that correspond to the physical lighting fixtures;

[0008] Obtain the target audio to be applied to the target video, and play the target audio;

[0009] During the playback of the target audio, the virtual lights are controlled to produce a first lighting effect in the virtual scene corresponding to the currently playing target audio, and the physical lights are simultaneously controlled to produce a second lighting effect in the real scene corresponding to the currently playing target audio.

[0010] The first lighting effect and the second lighting effect are used for the virtual video production to generate the target video containing the first lighting effect and the second lighting effect.

[0011] This application provides a lighting control device for virtual film production, which is used to merge real-world and virtual scenes to generate a target video, including:

[0012] A creation module is used to create virtual lighting fixtures in the virtual scene that correspond to the physical lighting fixtures in the real scene, using the physical lighting fixtures in the real scene as a reference.

[0013] The acquisition module is used to acquire target audio applied to the target video and play the target audio;

[0014] The control module is used to control the virtual lights to generate a first lighting effect in the virtual scene corresponding to the currently played target audio during the playback of the target audio, and to simultaneously control the physical lights to generate a second lighting effect in the real scene corresponding to the currently played target audio; wherein the first lighting effect and the second lighting effect are used for virtual video production to generate the target video containing the first lighting effect and the second lighting effect.

[0015] In the above scheme, the control module is also used to acquire the audio attributes of the target audio in real time during the playback of the target audio, and the audio attributes include at least one of sound frequency, audio rhythm, and audio content;

[0016] In the virtual scene, the virtual lights are controlled to produce a first lighting effect corresponding to the audio attributes of the currently playing target audio.

[0017] In the above scheme, the control module is further configured to obtain the mapping relationship between the lighting effect of each virtual lamp and the audio attribute when the number of virtual lamps is multiple;

[0018] Based on the audio attributes of the currently playing target audio and the mapping relationship, each of the virtual lights is controlled to produce corresponding lighting effects to form the first lighting effect.

[0019] In the above scheme, the control module is further configured to acquire the virtual lighting effect of the virtual lamps corresponding to the target video when the number of virtual lamps is multiple;

[0020] The virtual lighting effect includes: at each playback time point of the target audio, the first lighting effect in the virtual scene, wherein the first lighting effect is formed by the coordinated lighting effects of each of the virtual lamps;

[0021] Based on the virtual lighting effects, a mapping relationship is established between the lighting effects of each virtual lamp and the audio attributes at each playback time point.

[0022] In the above scheme, the control module is also used to acquire the audio rhythm of the target audio being played in real time during the playback of the target audio when the audio attribute includes audio rhythm;

[0023] Based on the audio rhythm of the currently playing target audio, the virtual lights are controlled in real time to produce lighting effects corresponding to the audio rhythm;

[0024] Different audio rhythms correspond to different lighting effects.

[0025] In the above scheme, the control module is also used to acquire the audio content of the target audio being played in real time during the playback of the target audio when the audio attribute includes audio content;

[0026] The acquired audio content is categorized to obtain the content type of the audio content;

[0027] Control the virtual lights to produce a first lighting effect corresponding to the content type of the audio content;

[0028] Different content types correspond to different lighting effects.

[0029] In the above scheme, the control module is also used to acquire the sound frequency of the target audio being played in real time during the playback of the target audio when the audio attribute includes sound frequency;

[0030] Determine the frequency range to which the acquired sound frequency belongs;

[0031] Control the virtual lighting fixtures to produce a first lighting effect corresponding to the frequency range to which the sound frequency belongs;

[0032] Different frequency ranges correspond to different lighting effects.

[0033] In the above scheme, the control module is further configured to obtain the attribute value of at least one target attribute of the virtual lamp corresponding to the first lighting effect;

[0034] The attribute values ​​of each of the target attributes are assigned to the corresponding target attributes of the virtual lamp, so that the virtual lamp produces the first lighting effect.

[0035] In the above scheme, the control module is further configured to obtain the correspondence between the audio tracks and the virtual lights when the number of virtual lights is multiple and the target audio includes audio data corresponding to at least two audio tracks;

[0036] Based on the correspondence, the virtual lights corresponding to each of the audio tracks are determined;

[0037] In the virtual scene, the virtual lights corresponding to each audio track are controlled to generate virtual lighting effects corresponding to the audio data of the audio track, so as to form the first lighting effect;

[0038] The audio data of different audio tracks correspond to different virtual lighting effects, and the first lighting effect is formed by the coordinated virtual lighting effects of each virtual lamp.

[0039] In the above scheme, the control module is also used to establish a communication connection with the physical lighting fixtures in the real scene;

[0040] Based on the communication connection, a control command is sent to the physical lamp, which instructs the physical lamp to produce a second lighting effect corresponding to the currently playing target audio.

[0041] In the above scheme, the control module is further configured to obtain the value of the lighting parameter of the physical lamp corresponding to the first lighting effect, and the value of the lighting parameter is used by the physical lamp to generate the second lighting effect;

[0042] The step of sending control commands to the physical lighting fixture based on the communication connection includes:

[0043] Based on the communication connection, a control command carrying the value of the lighting parameters is sent to the physical luminaire so that the physical luminaire applies the value of the lighting parameters.

[0044] In the above scheme, the control command carries lighting effect indication information for indicating the second lighting effect, and the control module is further used to send a control command carrying the lighting effect indication information to the physical lamp based on the communication connection;

[0045] The control command is used by the physical luminaire to determine the value of the lighting parameter corresponding to the second lighting effect based on the lighting effect indication information, and to apply the value of the lighting parameter to produce the second lighting effect.

[0046] In the above scheme, the creation module is also used to obtain the values ​​of the lighting parameters of the physical lighting fixtures in the real scene, as well as the association between the physical lighting fixtures and the virtual lighting fixtures;

[0047] Based on the values ​​of the physical lighting fixtures' lighting parameters and the associated relationships, the values ​​of the virtual lighting fixtures' lighting parameters in the virtual scene are determined.

[0048] The determined values ​​of the lighting parameters are assigned to the lighting parameters of the virtual lighting fixtures in the virtual scene.

[0049] This application provides an electronic device, including:

[0050] Memory, used to store executable instructions;

[0051] A processor, when executing executable instructions stored in the memory, implements the method provided in the embodiments of this application.

[0052] This application provides a computer-readable storage medium storing executable instructions for inducing a processor to execute and implement the method provided in this application.

[0053] This application provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements the lighting control method in virtual filmmaking provided in this application.

[0054] The embodiments of this application have the following beneficial effects:

[0055] By applying the embodiments of this application, firstly, virtual lighting fixtures are created with reference to physical lighting fixtures, which ensures the rationality and practicality of the virtual lighting fixtures. Then, target audio for the target video is selected and played. During the playback of the target audio, not only are the virtual lighting fixtures controlled to produce a first lighting effect corresponding to the played target audio in the virtual scene, but the physical lighting fixtures are also simultaneously controlled to produce a second lighting effect corresponding to the currently played target audio in the real scene. In this way, the synchronous control of virtual lighting fixtures and physical lighting fixtures can be achieved, improving the efficiency of virtual production. Attached Figure Description

[0056] Figure 1 This is a schematic diagram of the architecture of the lighting control system 100 in virtual filmmaking provided in this application embodiment;

[0057] Figure 2 This is a schematic diagram of the structure of the electronic device 500 for lighting control in virtual filmmaking provided in this application embodiment;

[0058] Figure 3 This is a flowchart illustrating the lighting control method in virtual filmmaking provided in the embodiments of this application;

[0059] Figure 4 This is a flowchart of the method for creating virtual lighting fixtures provided in the embodiments of this application;

[0060] Figure 5This is a flowchart illustrating how the lighting effects of a virtual lamp are generated, as provided in the embodiments of this application.

[0061] Figure 6 This is a schematic diagram illustrating the determination of lighting effects based on audio attributes, provided in an embodiment of this application.

[0062] Figure 7 This is a schematic diagram of the method for establishing the mapping relationship between lighting effects and audio attributes provided in the embodiments of this application;

[0063] Figure 8 This is a flowchart of a method for determining lighting effects based on audio rhythm provided in an embodiment of this application;

[0064] Figure 9 This is a flowchart of a method for determining lighting effects based on audio content, provided in an embodiment of this application.

[0065] Figure 10 This is a flowchart of a method for determining lighting effects based on sound frequency, provided in an embodiment of this application.

[0066] Figure 11 This is a flowchart of a method for determining the lighting effect of a virtual lamp based on target attributes, provided in an embodiment of this application.

[0067] Figure 12 This is a flowchart of a method for determining the lighting effect of virtual lamps based on an audio track, provided in an embodiment of this application.

[0068] Figure 13 This is a flowchart illustrating the method for generating the lighting effect of a physical lamp provided in this application embodiment;

[0069] Figure 14 This is a schematic diagram of device connections in a virtual production scene provided in an embodiment of this application;

[0070] Figure 15 This is a flowchart of a method for generating lighting effects based on lighting parameters using physical lighting fixtures, as provided in an embodiment of this application.

[0071] Figure 16 This is a flowchart of a method for synchronizing light and sound in a virtual scene provided in an embodiment of this application;

[0072] Figure 17 This is a schematic diagram of a method for creating virtual lighting fixtures in a virtual scene provided in an embodiment of this application;

[0073] Figure 18 This is a schematic diagram of a lighting fixture installation method for a physical lighting fixture provided in an embodiment of this application;

[0074] Figure 19 This is a schematic diagram of a lighting and sound synchronization control method in virtual filmmaking provided in an embodiment of this application. Detailed Implementation

[0075] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0076] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0077] If the application documents contain similar descriptions such as "first / second", the following explanation shall be added: In the following description, the terms "first / second / third" are used only to distinguish similar objects and do not represent a specific order of objects. It is understood that "first / second / third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0078] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0079] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0080] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0081] 1) Virtual Production: Virtual production is a broad term referring to a range of computer-aided production and visual filmmaking methods. Virtual production combines virtual and augmented reality with CGI and game engine technologies, allowing producers to see scenes unfold before them as if these scenes were actually composited and filmed on location.

[0082] 2) Virtual Studio: A common virtual studio is a photography studio that uses a green screen combined with on-site lighting, wire work, and special props, along with post-production special effects. Current new studios combine LED screens, motion capture technology, and camera tracking, which can make shooting in the studio more diverse and convenient.

[0083] 3) LED Curtain Wall: A large LED curtain wall is used to display virtual content in the virtual production shooting set. Front Set Design: Actual props are placed in front of the LED screen. On-Set Camera: On-set cameras in the virtual production capture the combined image of the LED screen and the front set design. Physical Lighting: Real-world lighting fixtures used in the virtual shooting studio to illuminate characters and sets; types generally include modeling lights, ambient lights, and special effects lights. Virtual Scene: A digital scene created in the game engine based on the artist's requirements or a real-world location. Virtual Lighting: A lighting system in the game engine that illuminates the virtual scene, primarily providing lighting that meets artistic requirements or approximates the effect of a real scene. Real-World Scene (Set Design): Real props or scenes built in the virtual production.

[0084] 4) Digital Multiplex (DMX): In the film and television media industry, DMX is a widely used data communication standard for controlling on-site lighting, sound effects, fireworks, animation broadcasts, and other content. When a DMX plugin is integrated into a virtual engine (UE), users can control various audio-visual effects of on-site events through UE. Application scenarios of UE combined with DMX mainly include: allowing users to preview various audio-visual effects of on-site events within UE; controlling various on-site lighting fixtures, equipment, and smoke devices compliant with the DMX protocol through UE; and triggering some real-time on-site effects and animations through UE.

[0085] DMX512: The DMX512 protocol is a data dimming protocol that provides a standard for communication between lighting controllers and lighting equipment. This protocol provides a good standard for controlling lighting equipment using digital signals.

[0086] 5) Art-Net: Art-Net is an Ethernet protocol based on Transmission Control Protocol (TCP) / Internet Protocol (IP). Its purpose is to allow the transmission of large amounts of DMX512 data over a wide area using standard networking technologies. Art-Net is a buy-out licensed communication protocol used to transmit DMX512-A lighting control protocols and Remote Device Management (RDM) protocols via User Datagram Protocol (UDP). It is used for communication between "nodes" (e.g., intelligent lighting devices) and "servers" (lighting consoles or general-purpose computers running lighting control software).

[0087] Based on the above explanation of the nouns and terms used in the embodiments of this application, the lighting control system in the virtual filmmaking provided in the embodiments of this application is described below. See also Figure 1 , Figure 1 This is a schematic diagram of the architecture of the lighting control system 100 in the virtual production provided in this application embodiment. In order to support an exemplary application, the lighting control system in the virtual production includes an LED curtain wall 600, an image acquisition device 700, a terminal 400 with a virtual engine deployed, physical lighting fixtures 800 in the real scene, and peripheral audio playback devices 900 (peripheral speakers). These devices are interconnected through a network 300, which can be a wide area network or a local area network, or a combination of both, and uses wireless or wired links to realize data transmission.

[0088] Terminal 400 is used to create virtual lighting fixtures in a virtual scene that correspond to the physical lighting fixtures in a real scene, using physical lighting fixtures in a real scene as a reference; acquire target audio for a target video and play the target audio; during the playback of the target audio, control the virtual lighting fixtures to generate a first lighting effect in the virtual scene corresponding to the currently played target audio, and simultaneously control the physical lighting fixtures to generate a second lighting effect in the real scene corresponding to the currently played target audio; wherein, the first lighting effect and the second lighting effect are used for virtual production to generate the target video containing the first lighting effect and the second lighting effect.

[0089] Server 200 is used to receive scene rendering requests sent by the terminal, render the virtual scene in the virtual production, and then return the rendered virtual scene to the terminal.

[0090] LED Curtain Wall 600, the LED curtain wall includes at least one LED screen, used to display the lighting effects corresponding to virtual lighting fixtures created in the virtual scene, as well as the scene content of the virtual scene.

[0091] Image acquisition device 700 is used to simultaneously capture the fused image of at least one LED screen and the scene set in front of the screen (there are actual props placed in front of the LED screen), and to acquire the lighting effects of virtual lights in the virtual scene and the lighting effects of physical lights in the real scene in real time. The data is transmitted to terminal 400 via target data cable, and target video is generated and saved in the virtual engine in terminal 400.

[0092] The physical light fixture 800 is connected to the terminal via a target data cable, receives control signals sent by the terminal, and produces a second lighting effect that corresponds to the target audio in the real scene.

[0093] Audio playback device 900, used to play target audio.

[0094] In some embodiments, server 200 may be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Terminal 400 may be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. Terminals and servers can be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment.

[0095] The embodiments of this application can be implemented with the help of cloud technology, which refers to a hosting technology that unifies a series of resources such as hardware, software, and network within a wide area network or local area network to realize the computation, storage, processing, and sharing of data.

[0096] Cloud technology is a general term encompassing network technology, information technology, integration technology, management platform technology, and application technology based on the cloud computing business model. It can form resource pools, providing flexible and convenient on-demand access. Cloud computing technology will become a crucial support. The backend services of technical network systems require substantial computing and storage resources. With the rapid development and application of the internet industry, every item may have its own identification mark in the future, requiring transmission to backend systems for logical processing. Data at different levels will be processed separately, and various industry data will require robust system support, which can be achieved through cloud computing.

[0097] See Figure 2 , Figure 2 This is a schematic diagram of the electronic device 500 for lighting control in virtual filmmaking provided in this embodiment of the application. In practical applications, the electronic device 500 can be... Figure 1 The server or terminal shown is exemplified by electronic device 500. Figure 1 Taking the domain name resolution node shown as an example, the electronic device for lighting control in virtual filmmaking according to the embodiments of this application will be described. The electronic device 500 provided in the embodiments of this application includes: at least one processor 510, a memory 550, at least one network interface 520, and a user interface 530. The various components in the electronic device 500 are coupled together through a bus system 540. It is understood that the bus system 540 is used to realize the connection and communication between these components. In addition to a data bus, the bus system 540 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 2 The general labeled all buses as Bus System 540.

[0098] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0099] User interface 530 includes one or more output devices 531 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 530 also includes one or more input devices 532, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0100] The memory 550 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 550 may optionally include one or more storage devices physically located away from the processor 510.

[0101] The memory 550 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 550 described in this application embodiment is intended to include any suitable type of memory.

[0102] In some embodiments, memory 550 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0103] Operating system 551 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0104] The network communication module 552 is used to reach other computing devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

[0105] Presentation module 553 is used to enable the presentation of information (e.g., user interface for operating peripheral devices and displaying content and information) via one or more output devices 531 (e.g., display screen, speaker, etc.) associated with user interface 530.

[0106] The input processing module 554 is used to detect and translate one or more user inputs or interactions from one or more input devices 532.

[0107] In some embodiments, the lighting control device in the virtual filmmaking process provided in this application can be implemented in software. Figure 2 A lighting control device 555 for a virtual filmmaking process stored in memory 550 is shown. It can be software in the form of programs and plug-ins, including the following software modules: creation module 5551, acquisition module 5552, and control module 5553. These modules are logical and can therefore be arbitrarily combined or further split according to the functions they implement. The functions of each module will be described below.

[0108] In other embodiments, the lighting control device in the virtual fabrication provided in this application can be implemented using a combination of hardware and software. As an example, the lighting control device in the virtual fabrication provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the lighting control in the virtual fabrication provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0109] Based on the above description of the lighting control system and electronic equipment in virtual filmmaking provided in the embodiments of this application, the lighting control method in virtual filmmaking provided in the embodiments of this application is described below. In some embodiments, the lighting control method in virtual filmmaking provided in the embodiments of this application can be implemented by a server or a terminal alone, or by a server and a terminal working together. The lighting control method in virtual filmmaking provided in the embodiments of this application is described below using a terminal implementation as an example.

[0110] See Figure 3 , Figure 3This is a flowchart illustrating the lighting control method in virtual video production provided in this application embodiment. Virtual video production is used to generate a target video by combining real-world and virtual scenes. The lighting control method in virtual video production provided in this application embodiment includes:

[0111] In step 101, the terminal uses the physical lighting fixtures in the real scene as a reference to create virtual lighting fixtures that correspond to the physical lighting fixtures in the virtual scene.

[0112] In practice, the terminal is equipped with a game engine (such as Unreal Engine). During the virtual production process, physical lighting fixtures are placed in the real-world scene, and virtual lighting in the virtual scene is rendered within the game engine. The virtual scene is a digital scene created in the game engine based on the artist's requirements or the real-world scene. To provide lighting effects that meet artistic requirements or closely resemble the effects of the real scene, a lighting system that illuminates the virtual scene is built in the game engine. The lighting system includes virtual lighting fixtures, which can be one or more in number and of various types, such as spotlights, soft lights, diffused lights, beam lights, moving head lights, follow spots, etc. When there are multiple virtual lighting fixtures, their arrangement can be configured differently according to the actual situation.

[0113] It's important to note that virtual lighting fixtures in the game engine can be created based on real physical lighting fixtures. This means the type of virtual lighting fixture, its parameters, and their values ​​all match those of the real physical fixture. Alternatively, they can be created by modifying the parameters of a physical lighting fixture. This allows for more diverse lighting effects. Some lighting effects are simply unavailable based on the physical fixture's parameter values. In such cases, the parameter values ​​can be adjusted based on the physical fixture to achieve lighting effects that are impossible to obtain with a physical fixture. Lighting fixtures and electronic devices supporting the game engine can connect via a network, ensuring they are on the same network. Lighting fixtures can be connected to each other, and to electronic devices, via DMX cables. The game engine supports receiving user-uploaded lighting fixture files in a common device type format, parsing these files, and creating corresponding virtual lighting fixtures. Among them, physical lamps conforming to the DMX512 protocol can be monochrome lamps (occupying 1 channel, one 512 controller can support up to 512 monochrome lamps), RGBW single-pixel lamps (occupying 4 channels, one DMX512 controller can support up to 128 RGBW single-pixel lamps), and RGB single-pixel lamps (occupying 3 channels, one DMX512 controller can support up to 512 / 3=170 RGB single-pixel lamps).

[0114] For example, virtual lighting fixtures can be created using Unreal Engine (UE) combined with DMX. This involves adding a DMX plugin to UE, receiving user-imported GDTF format lighting fixture data (the GDTF file contains information about specific lighting fixture types), parsing the data, and creating various types of virtual lighting fixtures. The process of creating virtual lighting fixtures in Unreal Engine 4 (UE4) is as follows: First, create a new Unreal Engine project, search for DMX plugins in the plugin library, open all of them, and restart the engine. Then, import the GDTF file containing information about specific lighting fixture types into Unreal Engine. Next, create a DMX lighting library, and in the Fixture Types tab of the DMX lighting library, create the lighting fixture types defined in the lighting file. Finally, perform the corresponding lighting fixture settings to obtain the virtual lighting fixtures in the virtual scene.

[0115] In some embodiments, see Figure 4 , Figure 4 This is a flowchart of the method for creating virtual lighting fixtures provided in the embodiments of this application, based on... Figure 3 Step 101 can be implemented by steps 1011 to 1013.

[0116] Step 1011: The terminal obtains the values ​​of the lighting parameters of the physical lighting fixtures in the real scene, as well as the relationship between the physical lighting fixtures and the virtual lighting fixtures.

[0117] In practical implementation, the lighting parameters of physical lighting fixtures can include physical parameters such as light direction, intensity, color temperature, and texture. The relationship between physical and virtual lighting fixtures can be that the lighting parameters of the virtual fixtures are the same as those of the physical fixtures, and the values ​​of each parameter are also the same; or the lighting parameters of the virtual fixtures are different from those of the physical fixtures. It should be noted that the relationship between virtual and physical lighting fixtures can be a user-defined rule.

[0118] For example, taking the color temperature of the lamp as an example, suppose the color temperature of the physical lamp of the target type is adjustable, with an adjustable range of 1-20, but the virtual lamp of the same type in the virtual scene needs a color temperature of 25. In this case, the adjustable range of the color temperature of the virtual lamp can be set to 1-25, so that more lighting effects can be set.

[0119] Step 1012: Based on the values ​​of the lighting parameters of the physical lighting fixtures and their relationships, determine the values ​​of the lighting parameters of the virtual lighting fixtures in the virtual scene.

[0120] In actual implementation, the terminal determines the lighting parameters and values ​​of virtual lighting fixtures in the virtual scene by using the lighting parameters of physical lighting fixtures (i.e., the actual existing parameters) and the preset association between physical lighting fixtures and virtual lighting fixtures (the association includes setting the lighting parameters exactly according to the lighting parameters of physical lighting fixtures and having the same parameter values, as well as adjusting them with reference to the lighting parameters of physical lighting fixtures).

[0121] Step 1013: Assign the determined values ​​of the lighting parameters to the lighting parameters of the virtual lighting fixtures in the virtual scene.

[0122] In actual implementation, the parameter values ​​of the virtual lighting fixtures, which are determined by referring to the lighting fixture parameters of the physical lighting fixtures, are assigned to the virtual lighting fixtures created in Unreal Engine.

[0123] The above-described method for creating virtual lighting fixtures in virtual scenes can not only create virtual lighting fixtures with the same lighting attributes as physical lighting fixtures, ensuring the accuracy of virtual lighting fixtures, but also create virtual lighting fixtures with different lighting attributes than physical lighting fixtures. In this way, the diversity of virtual lighting fixtures can be improved, and lighting effects that conform to more and richer virtual scenes can be generated based on a rich variety of virtual lighting fixtures.

[0124] In step 102, the target audio applied to the target video is obtained and the target audio is played.

[0125] In practice, the target audio is the audio from the target video corresponding to the current virtual production. The selection of the target audio can be based on the application scenario of the virtual production. The user selects a suitable target video and imports it into the electronic device used for virtual production. Since the electronic device is equipped with the Unreal Engine required for virtual production, the Unreal Engine parses the target audio and can then play it through its integrated audio player or by controlling an external audio playback device. It should be noted that the virtual engine has requirements for the target audio file format. For example, taking Unreal Engine (UE) as an example, the UE engine requires the target audio file to be in WaveformAudio File Format (wav) format. Therefore, audio files in other formats need to be converted to obtain a wav format target audio file.

[0126] In step 103, during the playback of the target audio, the virtual lights are controlled to produce a first lighting effect in the virtual scene that corresponds to the currently playing target audio.

[0127] In actual implementation, the actual operation of virtual production often requires the synchronization of sound and lighting effects. Therefore, while the terminal is playing the target audio (or sound) through the audio player (or audio playback device), it simultaneously controls the virtual lights in the virtual scene to produce the first lighting effect corresponding to the currently playing target audio. This first lighting effect can be regarded as the first lighting effect of the virtual lights that are finally displayed in the target video corresponding to the current virtual production.

[0128] In some embodiments, see Figure 5 , Figure 5 This is a flowchart illustrating the method for generating the lighting effects of a virtual lamp provided in this application embodiment, based on... Figure 3 Step 103 can be achieved through steps 1031-1032.

[0129] Step 1031: During the playback of the target audio, the terminal acquires the audio attributes of the target audio in real time. The audio attributes include at least one of the following: sound frequency, audio rhythm, and audio content.

[0130] In practice, since the target audio has a certain playback duration, its audio attributes differ at each playback time segment or point in time. To generate lighting effects corresponding to the sub-audio at each playback time segment or point in time, the audio attributes can be acquired in real time. These audio attributes include at least one of the following: sound frequency, audio rhythm, and audio content. Based on these audio attributes, corresponding lighting effects are generated.

[0131] Step 1032: In the virtual scene, control the virtual lights to produce a first lighting effect corresponding to the audio attributes of the currently playing target audio.

[0132] In practice, the terminal controls the virtual lights to produce lighting effects that correspond to the audio attributes of the currently playing target audio, serving as the first lighting effect in the virtual scene presented during the playback of the target video.

[0133] For example, taking a piece of music as the target audio, during the playback of the music, the corresponding lighting effects can be determined according to the rhythm, frequency, and content of the music. For example, when the rhythm (audio rhythm) is slow, the corresponding lighting effect is a warm color with yellow as the main tone; when the rhythm is cheerful, the corresponding lighting effect is a bright color with blue as the main tone. The type of audio rhythm, whether it is cheerful or slow, can be determined according to the actual situation.

[0134] In some embodiments, see Figure 6 , Figure 6 This is a schematic diagram of generating lighting effects based on audio attributes provided in an embodiment of this application. Step 1032 can be implemented through steps 201-202:

[0135] Step 201: When there are multiple virtual lights, obtain the mapping relationship between the lighting effects and audio attributes of each virtual light.

[0136] In practical implementation, when there are multiple virtual lights in a virtual scene, the virtual lights can be grouped based on the mapping relationship between the lighting effects of the virtual lights and the audio attributes of the target audio. The audio attributes of the target audio include at least one of sound frequency, audio rhythm, and audio content. For multiple virtual lights, they can be divided into multiple groups. In some groups, the virtual lights generate lighting effects based on sound frequency; in others, they generate lighting effects based on audio rhythm; in still others, they generate lighting effects based on any combination of sound frequency, audio rhythm, and audio content.

[0137] For example, the number of virtual lights is set to 50, divided into 5 groups of 10 each. The first group has a lighting effect of yellow light with different color differences, determined by the audio content. The second group has a lighting effect of red light with different color differences, determined by the audio rhythm. The third group has a lighting effect of white light, determined by the sound frequency.

[0138] In some embodiments, see Figure 7 , Figure 7 This is a schematic diagram of the method for establishing the mapping relationship between lighting effects and audio attributes provided in the embodiments of this application. Before obtaining the mapping relationship between the lighting effects and audio attributes of each virtual lamp, the terminal can obtain the mapping relationship between lighting effects and audio attributes through steps 301-302.

[0139] Step 301: The terminal acquires the virtual lighting effects of the virtual lights corresponding to the target video.

[0140] The virtual lighting effect includes: a first lighting effect in the virtual scene at each playback time point of the target audio, wherein the first lighting effect is formed by the coordinated lighting effects of each virtual lamp.

[0141] Step 302: Based on the virtual lighting effects, establish the mapping relationship between the lighting effects of each virtual light fixture and the audio attributes at each playback time point.

[0142] In practical implementation, each playback time point of the target audio is acquired. These playback time points can be considered key playback time points, meaning that at these playback time points, the audio attributes may change. For example, the sound frequency may change from below a frequency threshold to above a frequency threshold. For instance, if the frequency threshold is 40 kHz, the sound frequency may jump directly from 20 kHz to 50 kHz at the current playback time point; or the audio content may switch from one content type to another (such as a piece of music switching from narration to lyrics); or the pitch of the audio rhythm may switch from one pitch to another (such as switching from D major to G major). In other words, for the target audio, a time point sequence T = {t0, t1, ..., tn} can be acquired, including the start playback time point, at least one intermediate playback time point, and the end playback time point. At each playback time point, a mapping relationship between the lighting effects of the virtual lights and the audio attributes is established.

[0143] For example, taking a musical piece as the target audio, the time sequence corresponding to the musical piece is T={t0, t1, ..., tn}. At time t, the audio content in the audio attribute changes from narration to singing. At time t2, the audio rhythm changes from D major to G major. At time t3, the sound frequency drops from 50kHz to 25kHz. Then, a mapping relationship R between the lighting effect and the audio attribute can be established. Between t0 and t1, the virtual lighting effect is warm light. Between t1 and t2, the virtual lighting effect is bright light. After t2, it switches to fireworks effect, etc.

[0144] Step 202: Based on the audio attributes and mapping relationship of the currently playing target audio, control each virtual lamp to produce corresponding lighting effects to form the first lighting effect.

[0145] In actual implementation, the corresponding lighting effects of each virtual lamp can be determined by at least one of the audio attributes, such as sound frequency, audio rhythm, and audio content, as well as the mapping relationship, and the first lighting effect can be formed by the collaboration of the corresponding lighting effects.

[0146] In some embodiments, see Figure 8 , Figure 8 This is a flowchart of a method for determining lighting effects based on audio rhythm, provided in an embodiment of this application. Figure 6 Step 202 in the figure can be achieved through steps 2021a-2022a.

[0147] Step 2021a: When the audio attribute includes audio rhythm, the audio rhythm of the target audio is acquired in real time during the playback of the target audio.

[0148] In practice, the target audio includes multiple audio rhythms, which characterize the rhythmic state of the target audio. The terminal performs rhythm detection on the target audio in real time, obtaining multiple audio rhythms. Based on the relationship between the rhythm frequency and the rhythm frequency threshold, the audio rhythms can be categorized into different types, such as soothing, upbeat, and cheerful rhythms.

[0149] Step 2022a: Based on the audio rhythm of the currently playing target audio, control the virtual lights in real time to produce lighting effects corresponding to the audio rhythm, where different audio rhythms correspond to different lighting effects.

[0150] In actual implementation, the terminal determines the lighting parameters of each virtual lamp corresponding to multiple audio rhythms based on the audio rhythm of the currently playing target audio, and adjusts the values ​​of the lighting parameters of the virtual lamps in real time based on each lighting parameter, controlling the virtual lamps to produce lighting effects corresponding to the audio rhythm.

[0151] In some embodiments, see Figure 9 , Figure 9 This is a flowchart of a method for determining lighting effects based on audio content, provided in an embodiment of this application. Figure 6 Step 202 in the figure can be achieved through steps 2021b-2023b.

[0152] Step 2021b: When the audio attribute includes audio content, the audio content of the target audio is acquired in real time during the playback of the target audio.

[0153] In practice, the audio content of the target audio is acquired during playback. Taking a musical piece as an example, the audio content can include lyrics, narration, etc. The lyrics of a piece of music can also be divided into Chinese lyrics, English lyrics, rap parts, etc.

[0154] Step 2022b: Classify the acquired audio content to obtain the content type of the audio content.

[0155] In practice, audio content can be categorized to determine the content type of each part of the audio content.

[0156] For example, if the target audio is a piece of music, the lyrics in the music can be categorized into different content types such as Chinese, English, rap, and chorus.

[0157] Step 2023b: Control the virtual lights to produce a first lighting effect corresponding to the content type of the audio content.

[0158] In actual implementation, the terminal determines the virtual lights corresponding to each content type based on the correspondence between content types and virtual lights. Each virtual light is distinguished by a virtual light fixture identifier. Then, based on the correspondence between audio content and lighting effects, the terminal obtains the lighting effects corresponding to the audio content of the target content type and controls the virtual lights corresponding to the target content type to produce the corresponding lighting effects.

[0159] For example, taking a piece of music as the target audio, the correspondence between the content type of the music and the virtual lights is {"Chinese lyrics": "lights 1, lights 3", "rap": "lights 1, lights 2, lights 3", "English lyrics": "lights 1"}, and the correspondence between the content type and the lighting effects is {"Chinese lyrics": "lighting effect 1", "rap": "lighting effect 2", "English lyrics": "lighting effect 3"}. When the music plays the "Chinese lyrics" part, "lights 1, lights 3" displays "lighting effect 1"; when the music plays the "rap" part, "lights 1, lights 2, lights 3" displays "lighting effect 2", and so on.

[0160] In some embodiments, see Figure 10 , Figure 10 This is a flowchart of a method for determining lighting effects based on sound frequency, provided in an embodiment of this application. Figure 6 Step 202 in the figure can be achieved through steps 2021c-2023c.

[0161] Step 2021c: When the audio attribute includes sound frequency, the terminal acquires the sound frequency of the target audio being played in real time during the playback of the target audio.

[0162] In practice, during the playback of the target audio, the sound frequency (a type of audio attribute) of the target audio is acquired. A frequency threshold corresponding to the sound frequency is pre-set, and the sound frequency is divided using this threshold to obtain at least two frequency ranges. Multiple frequency thresholds can be used; one threshold divides the sound into two frequency ranges, and two frequency thresholds can divide the sound frequency into three frequency ranges.

[0163] For example, taking a musical piece as the target audio, the frequency thresholds for the sound frequency are set to 8kHz and 15kHz, and the frequency range can include less than 5kHz, 5kHz to 15kHz, and greater than 15kHz.

[0164] Step 2022c: Determine the frequency range to which the acquired sound frequency belongs.

[0165] In actual implementation, during the playback of the target audio, the sound frequency of the target audio is acquired in real time, and the frequency range to which the sound frequency belongs is determined.

[0166] Step 2023c: Control the virtual lights to produce a first lighting effect corresponding to the frequency range to which the sound frequency belongs, wherein different frequency ranges correspond to different lighting effects.

[0167] In actual implementation, different frequency ranges correspond to different lighting effects. During the playback of the target audio, the sound frequency of the target audio is acquired in real time, and the target frequency range to which the sound frequency belongs is determined. After determining the target frequency range, the virtual lights are controlled to produce lighting effects corresponding to the target frequency range. It should be noted that since the sound frequency is dynamically changing, the first lighting effect of the virtual lights in the target video can be formed by the coordinated lighting effects corresponding to the above-mentioned sound frequencies.

[0168] Continuing with the previous example, when the target audio's sound frequency is high and greater than 15kHz, the virtual lights emitting green light are turned on and then gradually dim after a 1-second delay; when the target audio's sound frequency is low and lower than 8kHz, the virtual lights emitting red light are turned on and then gradually dim after a 1-second delay; when the target audio's sound frequency is between 8kHz and 15kHz, the virtual lights emitting yellow light are turned on and then gradually dim after a 1-second delay.

[0169] In some embodiments, see Figure 11 , Figure 11 This is a flowchart of a method for determining the lighting effect of a virtual lamp based on target attributes, provided in an embodiment of this application. Figure 3 Step 103 can be achieved through steps 1031b-1032b.

[0170] Step 1031b: The terminal obtains the attribute value of at least one target attribute of the first lighting effect corresponding to the virtual lamp.

[0171] In practice, before virtual production begins, the user pre-sets the lighting effects in the target video. This mainly includes setting the parameters of the virtual lights in the virtual scene, determining the various attributes of each virtual light based on the first lighting effect, and saving this configuration information related to the virtual lights in a GDTF file. When virtual production begins, the Unreal Engine on the terminal is started, receives the lighting file imported by the user, parses it, and obtains the configuration information of each virtual light in the virtual scene, such as the color temperature, brightness, and illumination time of the virtual lights matching the first lighting effect during audio playback.

[0172] Step 1032b: Assign the attribute values ​​of each target attribute to the corresponding target attribute of the virtual lamp so that the virtual lamp produces the first lighting effect.

[0173] In actual implementation, the terminal groups the target attributes based on the lamp identifier of the virtual lamp, and determines the attribute value of the target attribute corresponding to each virtual lamp based on the lamp identifier. Then, it assigns each attribute value to the corresponding target attribute of the virtual lamp in the virtual scene. In this way, the virtual lamp can produce the first lighting effect based on these attribute values.

[0174] In some embodiments, see Figure 12 , Figure 12 This is a flowchart illustrating a method for determining the lighting effects of virtual lamps based on audio tracks, as provided in an embodiment of this application. Figure 3 Step 103 can be achieved through steps 1031c-1033c.

[0175] Step 1031c: When there are multiple virtual lights and the target audio includes audio data corresponding to at least two audio tracks, obtain the correspondence between the audio tracks and the virtual lights.

[0176] In practice, the target audio contains audio data corresponding to at least one audio track. The parallel "tracks" seen in the sequencer software are called audio tracks. The audio data of each track is an independent sound unit, and each track can be indicated by a track identifier; different tracks correspond to different track identifiers. When there are at least two virtual lights, and the target audio includes audio data corresponding to at least two audio tracks, the correspondence between each track and the virtual lights can be predetermined, i.e., the virtual light corresponding to each track can be determined in advance.

[0177] For example, a piece of music may include two different audio tracks: vocals and piano accompaniment, labeled A1 and A2 respectively. Five virtual lights L are created in the virtual scene, numbered L1, L2, L3, L4, and L5. A pre-established correspondence can be established as A1->{L1, L3} and A2->{L2, L4, L5}. That is, during virtual production, the lighting effects of virtual lights L1 and L3 can be generated based on the audio data (voice) of the corresponding A1 audio track, and the lighting effects of virtual lights L2, L4, and L5 can be generated based on the audio data (piano accompaniment) of the corresponding A2 audio track.

[0178] Step 1032c: Based on the correspondence, determine the virtual lights corresponding to each audio track.

[0179] In practice, the game engine can determine the virtual lights corresponding to each audio track by analyzing the correspondence between the virtual lights and the audio tracks in step 1031c.

[0180] Continuing from the previous example, audio track A1 corresponds to virtual lights L1 and L3, and audio track A2 corresponds to virtual lights L2, L4, and L5.

[0181] Step 1033c: In the virtual scene, control the virtual lights corresponding to each audio track to generate virtual lighting effects corresponding to the audio data of the audio track, so as to form the first lighting effect.

[0182] Among them, the audio data of different audio tracks correspond to different virtual lighting effects, and the first lighting effect is formed by the coordinated virtual lighting effects of each virtual lamp.

[0183] In practical implementation, the game engine controls the virtual lights corresponding to each audio track to generate virtual lighting effects corresponding to the audio track data. That is, when the target audio includes audio data from at least two audio tracks, a lighting effect can be generated for each audio track, meaning the number of lighting effects is the same as the number of audio tracks. The first lighting effect in the virtual scene generated in the final target video is formed by the coordinated lighting effects of the virtual lights corresponding to each audio track. The coordinated lighting effects of the virtual lights for each audio track can be displayed independently, mixed with at least two lighting effects, or displayed periodically, etc. This application embodiment does not limit the coordinated lighting effect method. It should be noted that the generation method of the lighting effects of the virtual lights corresponding to each audio track can be as mentioned above, determined according to the audio attributes of the audio data on each audio track. Audio attributes include at least one of sound frequency, audio rhythm, and audio content.

[0184] Continuing the previous example, audio track A1 corresponds to virtual lights L1 and L3, and audio track A2 corresponds to virtual lights L2, L4, and L5. The lighting effects produced by virtual lights L1 and L3 on audio track A1 are S1 and S2, and the lighting effects produced by virtual lights L2, L4, and L5 on audio track A2 are S2, S4, and S5. Therefore, the first lighting effect in the virtual scene presented in the target video can be either these five lighting effects displayed independently or a mixture of at least two of these five lighting effects.

[0185] In step 104, the physical lighting fixtures are synchronously controlled to produce a second lighting effect in the real scene that corresponds to the target audio being played.

[0186] The first lighting effect and the second lighting effect are used for virtual video production to generate the target video that includes the first lighting effect and the second lighting effect.

[0187] In practice, the physical lighting fixtures deployed during the virtual production process (there can be one or more physical lighting fixtures) are located on the same local area network as the electronic devices (terminals) of the Unreal Engine. The physical lighting fixtures are connected to each other via target data cables. The purpose of these target data cables is to transmit data conforming to the target protocol. The physical lighting fixtures are connected to the electronic devices of the Unreal Engine via target data cables so that the physical lighting fixtures can receive data sent by the virtual engine through the electronic devices. This data can be control signals or lighting parameters for the physical lighting fixtures.

[0188] For example, taking DMX lighting fixtures as physical lighting in virtual production, see [link to relevant documentation]. Figure 1 DMX lights are used to represent lights that conform to the DMX512 protocol. DMX is a digital communication network standard, which is usually used to control stage lighting and effects. The Unreal Engine deployed in the electronic device is Unreal Engine 4 (UE4, used to generate and demonstrate lighting effects). The UE outputs standard DMX512 data control signals through the electronic device and sends them to the physical lights in the real scene to drive the lights that conform to the DMX512 protocol (DMX lights) to produce lighting effects corresponding to the target audio (i.e., a second lighting effect that is different from the first lighting effect of the virtual lights).

[0189] It should be noted that the method of controlling the lighting effects of virtual lights based on the target audio of the aforementioned terminal is also applicable to physical lights in real-world scenarios.

[0190] In actual implementation, the terminal also analyzes the spectrum of the target audio in real time based on the current playback time of the target audio to obtain corresponding data. The spectrum can be represented by multiple curves with different amplitudes and frequencies. When analyzing the spectrum, it can be decomposed into multiple different amplitude curves according to actual needs. For example, when the sampling is 10, it means that the spectrum is decomposed into 10 different curves. As time goes by, the value of each amplitude curve also changes, so at the same point in time, ten different amplitudes will be obtained. Then, the obtained multiple data are combined into data conforming to the DMX512 protocol and sent to physical lights that conform to the DMX512 protocol. The physical lights present corresponding lighting effects according to the received data, such as the physical lights changing brightness or color with the rhythm of the sound (this depends on which data is assigned to which channels of the lights when combining the DMX512 protocol data). At the same time, the obtained data is also assigned to the brightness, color, and other attributes of the virtual lights according to the preset allocation strategy, and the virtual lights will change brightness or color with the rhythm of the sound. Here, the allocation strategy is used to determine the correspondence between amplitude and lamp channel: the lighting effect generated by the virtual lamp based on the data depends on the above allocation strategy (the correspondence between amplitude and lamp channel), that is, which channel of the lamp is assigned to the ten amplitudes obtained above. For example, the first channel is the horizontal axis rotation of the lamp (the lamp can "shake" left and right), the second channel is the vertical axis rotation (the lamp can "nod" up and down), the third channel is brightness, and the fourth channel is hue, etc.

[0191] For example, taking a piece of music as the target audio, the first five seconds of the music can be used to make the lights "nod" according to the rhythm of the music. You can choose to assign a certain data or the data with the largest amplitude to the upper and lower axis rotation of the lights, and add a correction value (a suitable value obtained from multiple experiments) to make the lights achieve the effect of sweeping light up and down.

[0192] In some embodiments, see Figure 13 , Figure 13 This is a flowchart illustrating the method for generating the lighting effect of a physical lighting fixture according to an embodiment of this application, based on... Figure 3 Step 104 can be achieved through steps 1041a-1042a.

[0193] Step 1041: The terminal establishes a communication connection with the physical lighting fixtures in the real scene.

[0194] In actual implementation, during the virtual production process, the terminal (electronic device) deployed with Unreal Engine, or the electronic device deployed with the demonstration lighting effects design application, is on the same network as the physical lighting fixtures in the real scene, and establishes a communication connection between the terminal and the physical lighting fixtures through the target communication protocol.

[0195] For example, see Figure 14 , Figure 14 This is a schematic diagram of device connection in a virtual production scenario provided in the embodiments of this application. In the virtual production application scenario, the devices mainly include: a terminal with Unreal Engine installed, various types of physical lighting fixtures, a router, at least one LED screen, etc. All devices are on the same network and communicate with each other through a target data line.

[0196] Step 1042a: Based on the communication connection, a control command is sent to the physical lighting fixture. The control command is used to instruct the physical lighting fixture to produce a second lighting effect corresponding to the currently playing target audio.

[0197] In practice, the virtual engine analyzes the target audio and determines the corresponding lighting effect (i.e., the second lighting effect) for the physical lights based on the aforementioned method. In other words, the lighting effect required by the physical lights can be determined by the audio attributes of the target audio, or by analyzing the correspondence between the audio data of each track of the target audio and the physical lights. That is, the method used in Unreal Engine to control virtual lights to produce the first lighting effect can also be applied to control physical lights to produce the second lighting effect. During the playback of the target audio, Unreal Engine controls the virtual lights in the virtual scene to produce the first lighting effect, and simultaneously sends control signals to the physical lights based on the established connection. At this time, the physical lights' own controllers automatically adjust the physical lights' parameters based on the received control signals, thereby enabling the physical lights to produce the second lighting effect.

[0198] In some embodiments, when the control command carries lighting effect indication information for indicating the second lighting effect, the terminal can also send the control command carrying the lighting effect indication information to the physical luminaire based on the communication connection; wherein, the control command is used by the physical luminaire to determine the value of the lighting parameter corresponding to the second lighting effect based on the lighting effect indication information, and apply the value of the lighting parameter to generate the second lighting effect.

[0199] In practical implementation, terminals supporting the virtual engine can send control commands carrying lighting effect indication information only to physical luminaires. Each physical luminaire receives the control commands, parses the lighting effect indication information, obtains the lighting effect corresponding to itself, and then its own controller adjusts the values ​​of the luminaire parameters and applies the adjusted values ​​to generate the second lighting effect. The lighting indication information includes the lighting effect corresponding to at least one physical luminaire. The format of the lighting indication information can be {"luminaire identifier": "lighting effect identifier"}.

[0200] For example, the lighting indication information is {"Light fixture 1": "Red light for 3 seconds followed by blue light for 5 seconds", "Light fixture 2": "Yellow light for 4 seconds followed by green light for 8 seconds"}. In the real scene, the physical light fixture corresponding to "Light fixture 1" parses the lighting indication information to obtain the lighting effect of "Red light for 3 seconds followed by blue light for 5 seconds", and its own controller controls the physical light fixture to present the lighting effect of "Red light for 3 seconds followed by blue light for 5 seconds".

[0201] In some embodiments, see Figure 15 , Figure 15 This is a flowchart illustrating a method for generating lighting effects based on lighting parameters using physical lighting fixtures, as provided in this application embodiment. Figure 3 Step 104 can be achieved through steps 1041b-1043b.

[0202] Step 1041b: The terminal establishes a communication connection with the physical lighting fixtures in the real scene.

[0203] In actual implementation, the terminal deploying Unreal Engine and the physical lighting fixtures in the real scene are on the same network and communicate with each other through a target data cable. That is, being on the same network is a prerequisite for communication. After the network connection is established, the physical lighting fixtures and the terminal deploying Unreal Engine are connected again through the target data cable.

[0204] For example, taking the lighting fixtures used in virtual production as LED lighting fixtures that conform to DMX512 (lighting fixtures that comply with the DMX512 protocol), each LED lighting fixture and the terminal where UE4 is deployed are on the same network (on the same network), and the terminal where UE4 is located is connected to the LEDs via DMX lines. In this way, the terminal where UE4 is located can communicate with the LED lighting fixtures.

[0205] Step 1042b: Obtain the values ​​of the lighting parameters of the physical luminaire corresponding to the second lighting effect. The values ​​of the lighting parameters are used by the physical luminaire to generate the second lighting effect.

[0206] In actual implementation, the Unreal Engine deployed in the terminal receives pre-set configuration information for the lighting parameters of physical lighting fixtures in the real-world scene. It then parses this configuration file to determine the lighting parameters of the physical lighting fixtures corresponding to the second lighting effect (the lighting effect that the physical lighting fixtures need to generate). The configuration information for the lighting parameters is identified by the fixture's identifier; that is, one physical lighting fixture corresponds to one set of lighting parameter configuration information. During the playback of the target audio, the lighting parameters corresponding to the physical lighting fixtures are sent to the physical lighting fixtures in real time.

[0207] Step 1043b: Based on the communication connection, a control command carrying the values ​​of the lighting parameters is sent to the physical luminaire so that the physical luminaire applies the values ​​of the lighting parameters.

[0208] In actual implementation, the terminal can directly send the lighting parameters of the physical lighting fixtures required in the target video to the physical lighting fixtures. After the controller of the physical lighting fixtures obtains the corresponding lighting parameters, it applies the values ​​of the lighting parameters and generates the second lighting effect.

[0209] By applying the embodiments of this application, during the playback of the target audio, while controlling the virtual lights to generate a first lighting effect corresponding to the played target audio in the virtual scene, it is also possible to simultaneously control the physical lights to generate a second lighting effect corresponding to the currently played target audio in the real scene. In this way, not only can the sound playback, the response of virtual lights and physical lights be controlled by only one person, but also the effect of three-way synchronous response can be achieved. This can effectively reduce the degree of human intervention, and can significantly reduce the errors caused by the cooperation between speakers, lights and Unreal Engine. It can also greatly reduce the influence of external factors, thereby significantly improving the shooting efficiency of virtual production.

[0210] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0211] Virtual production combines virtual reality and augmented reality with CGI and game engine technologies, allowing production staff to see scenes unfold before them as if these scenes were actually composited and filmed on location. In related virtual production techniques, sound engineers, lighting technicians, and Unreal Engine operators typically activate lighting and sound effects synchronously according to the director's instructions. This method makes it difficult for the lighting, sound, and Unreal Engine operators to achieve complete synchronization, and it is greatly affected by external conditions, such as communication interruptions or interference from other personnel. Furthermore, if any one party makes a mistake, the entire take must be scrapped, significantly impacting filming efficiency.

[0212] Based on this, this application provides a lighting control method for virtual production, which also synchronizes sound, virtual lighting, and physical lighting in virtual production. Using the DMX512 transmission protocol, sound, virtual lighting, and physical lighting in virtual production can be operated by only one person and can be synchronized. Since only one person needs to operate physical lighting, sound, and virtual lighting, the possibility of external influences is greatly reduced, and complete unification of physical lighting, sound, and virtual lighting can be achieved. In other words, this method allows sound, virtual lighting, and physical lighting in virtual production to be operated by only one person and synchronized.

[0213] Next, the lighting control method in the virtual filmmaking process provided in the embodiments of this application will be described from the product side. For practical implementation, please refer to... Figure 1 , Figure 1The image shows various equipment and props in a virtual film production shooting set (virtual studio), including LED screens, in-screen sets, on-set cameras, and physical lighting fixtures. The virtual studio in virtual production combines LED screens, motion capture technology, and camera tracking, making studio shooting methods richer and more convenient. At least one large LED screen is set up in the virtual studio to display virtual content, while actual props are placed in front of the LED screen. On-set cameras in virtual production simultaneously capture the merged image of the LED screen and the in-screen sets. Real-world lighting fixtures are used in the virtual studio to illuminate characters and sets; the types of lighting generally include modeling lights, ambient lights, and special effects lights. Simultaneously, digital scenes (i.e., virtual scenes in virtual production) are created in the Unreal Engine based on the artist's needs or real-world scenes within the electronic devices. A lighting system is also set up in the Unreal Engine to illuminate these virtual scenes, primarily to provide lighting that meets artistic requirements or approximates the effects of real scenes. It should be noted that the prerequisite for using Unreal Engine to control both physical and virtual lights is that the physical lights have a network module that supports the Art-Net or sAcn protocols. If the physical lights do not have a network module, a lighting console that supports sending Art-Net or sAcn protocol data over the network can be used to control the virtual lights, sound playback, and physical lights within Unreal Engine simultaneously.

[0214] See Figure 16 , Figure 16 This is a flowchart of a lighting and sound synchronization method in a virtual scene provided in this application embodiment. In the diagram, an electronic device (or terminal) equipped with Unreal Engine receives target audio (typically including at least one audio track) uploaded by a user, suitable for the current virtual production. Unreal Engine parses the audio track data and applies it to the virtual lights. Simultaneously, the audio track data is combined into data conforming to the DMX512 format and sent to the physical lighting fixtures in the real scene (set setting) to control the physical lighting fixtures to produce lighting effects (physical lighting) corresponding to the currently playing target audio. This not only allows for simultaneous control of sound playback, virtual lighting, and physical lighting by a single person, achieving a three-way synchronized response, but also significantly reduces errors caused by the coordination between speakers, lights, and the Unreal Engine, and greatly reduces the influence of external factors, thereby significantly improving the shooting efficiency of this type of shot.

[0215] Next, the lighting control method in virtual filmmaking provided in the embodiments of this application will be described from a technical perspective.

[0216] First, let's explain the control process for controlling the lighting effects of virtual lights in a virtual scene using Unreal Engine. (See [link to relevant documentation]). Figure 17 , Figure 17This is a schematic diagram illustrating the method for creating virtual lighting fixtures in a virtual scene provided in this application embodiment. First, step 1 is executed: create a new Unreal Engine project, search for DMX in Plugins, open all plugins, and restart the engine. The DMX plugins include lighting fixture functionality. Then, step 2 is executed: import the lighting fixture file into Unreal Engine. The lighting fixture file's data format is a GDTF file. The GDTF file contains information specific to the lighting fixture type, such as mode name, attribute name, and default values. The GDTF file can be used to automatically configure DMX lighting fixture types, making project setup faster and easier. Next, step 3 is executed: create a new DMX library. Then, step 4 is executed: create a new lighting fixture type (New Fixture Type) and name it under the Fixture Types tab in the DMX Library. Finally, step 5 is executed: in the newly created lighting fixture type (Fixture Type) sub-window's Fixture Setting interface, the DMX... Select the GDTF file you just imported in the Import menu, and proceed to step 6. Multiple modes will be displayed in the Modes sub-window. Next, proceed to step 7. In the Fixture Path (also known as Fixture Patch) tab, first create a new Add Fixture, selecting the Fixture Type you just created. Create as many Fixture Types as needed for different types of virtual fixtures. Continue to step 8. In Fixture Mode, select the target mode, i.e., in the Fixture Patch sub-window, under Fixture Patch, select the desired Mode in Active Mode, and proceed to step 9. In Active Mode, you can set the Manual Starting Address. It should be noted that the Address configured here must be consistent with the Address under the DMX tab of the physical fixture. In practical applications, if the Address configured for the virtual fixture is inconsistent with the Address of the physical fixture, an additional DMX physical fixture needs to be configured, and Unreal Engine will send the same instructions to both fixtures simultaneously. Finally, in Unreal Engine's Project Settings, set the DMX plugin and select Protocol. The name is Art-Net. (It should be noted that the choice between Art-Net and sAcn here is determined by the lighting fixture settings or the protocols supported by the physical lighting fixture. When the electronic device running Unreal Engine and the physical lighting fixture are in the same local area network environment, if the sAcn protocol is used, configure the electronic device running Unreal Engine with a standard IP address, such as 192.168.XX. If the Art-Net protocol is used, configure the electronic device running Unreal Engine with an IP address such as 2.0.0.)(The IP address should be in the X format). Select Network Interface Card IP Address as the target IP. This target IP is the IP address of the electronic device running Unreal Engine and the physical lighting fixture, both located on the same local area network and conforming to the communication protocol rules used by the lighting fixture.

[0217] Secondly, the method for setting up physical lighting fixtures in real-world scenes during virtual production is explained; see [link to relevant documentation]. Figure 18 , Figure 18 This is a schematic diagram of the lighting fixture setup method provided in this application embodiment. Step 1: Connect the power cord and network cable to the lighting fixture (ensuring the physical lighting fixture and the electronic device running Unreal Engine are on the same local area network); Step 2: Configure the network, start the lighting fixture, press the Menu button, select Network Setting, set the IP type to DHCP, enter the target IP address, and use the target IP address in the Unreal Engine electronic device; Step 3: Set DMX, setting the address and mode respectively. On the Unreal Engine electronic device, launch a browser and enter the target IP address. The DMX setting menu is configured as follows: First, set the DMX address (1, 11, or 21, etc.), which mainly depends on which Sky panel. Then, set the starting position of the channel, corresponding to the channel occupied by the specific lighting fixture in Universe 1 of UE4. Next, set the DMX mode to P4 (Mode 4); then set the Art-Net protocol, choosing "Art-Net Only"; finally, set "Art-Net Settings" to {Universe "1", Enter}. In the Fixture Setting menu above, set the SpecialModes to "Calibrated RGBW", i.e., RGBW mode.

[0218] Finally, after describing the above-described layout and configuration of virtual and physical lighting fixtures, the lighting and sound synchronization control method in virtual filmmaking according to the embodiments of this application is explained. (See also...) Figure 19 , Figure 19This is a schematic diagram of a lighting and sound synchronization control method in virtual production provided in this application embodiment. First, step 1 is executed: import the target audio (sound file, generally in WAV format) into Unreal Engine. Then, step 2 is executed: create an Actor and add an audio component. Step 3 is executed: set the target audio and play it through a speaker device. The target audio is selected from multiple imported sound files. Subsequently, the speakers, physical lights, and virtual lights respond synchronously. That is, during the playback of the target audio, the virtual lights display their corresponding virtual lighting effects, and the physical lights display their corresponding physical lighting effects. The specific synchronization process includes: during the playback of the target audio through the speaker device, step 4 is executed: analyze the audio tracks in real time according to the playback time of the target audio and obtain data corresponding to each audio track (the amount of data depends on the degree of subdivision of the analyzed sound; the audio track can be represented by multiple curves with different amplitudes and frequencies. When analyzing the target audio, an audio track can be decomposed into multiple audio tracks according to actual needs). Different curves, such as a sampling of 10, represent decomposing a musical spectrum into 10 different curves. As time progresses, the amplitude of each curve changes, resulting in ten different amplitude values ​​at the same time. Step 5: Combine the obtained audio data corresponding to each track into data conforming to the DMX512 protocol and send it to the physical lighting fixtures. Step 6: The physical lighting fixtures display corresponding lighting effects based on the received audio data, such as changes in brightness or color with the rhythm of the sound (depending on which data was given to which channels of the lights during the DMX512 protocol data combination). Step 8: Set the obtained data to the virtual lighting fixtures in real time. Step 9: The virtual lighting fixtures present corresponding lighting effects in real time based on the obtained data, i.e., assign brightness, color, and other attributes to the virtual lighting fixtures as needed. The virtual lighting fixtures will change brightness or color with the rhythm of the sound, or control the stage moving head lights to produce a head-shaking or nodding effect.

[0219] It should be noted that during virtual production, lighting can be controlled not only in the ways mentioned above, but other objects in Unreal Engine can also change according to the rhythm of the lighting, such as moving up, down, left, or right, rotating, and changing the brightness and color of materials. In the future, it can also be combined with a lighting console, which may make the methods even richer.

[0220] By applying the embodiments of this application, the response of sound playback, virtual lighting, and physical lighting can be controlled by only one person, achieving the effect of synchronous response of three parties. This method can greatly reduce the errors caused by the cooperation between speakers, lights, and Unreal Engine, and can also greatly reduce the influence of external factors, thereby greatly improving the shooting efficiency of this type of shot.

[0221] The following description continues to illustrate the exemplary structure of the lighting control device 555 in the virtual filmmaking provided in the embodiments of this application as a software module. In some embodiments, such as Figure 2 As shown, the software module in the lighting control device 555 stored in the virtual filmmaking memory 550 may include:

[0222] The creation module 5551 is used to create virtual lighting fixtures in the virtual scene that correspond to the physical lighting fixtures in the real scene, with reference to the physical lighting fixtures in the real scene.

[0223] The acquisition module 5552 is used to acquire target audio applied to the target video and play the target audio;

[0224] The control module 5553 is used to control the virtual lights to generate a first lighting effect in the virtual scene corresponding to the currently played target audio during the playback of the target audio, and to simultaneously control the physical lights to generate a second lighting effect in the real scene corresponding to the currently played target audio; wherein the first lighting effect and the second lighting effect are used for virtual video production to generate the target video containing the first lighting effect and the second lighting effect.

[0225] In some embodiments, the control module is further configured to acquire, in real time, the audio attributes of the target audio during playback, the audio attributes including at least one of sound frequency, audio rhythm, and audio content; and in the virtual scene, control the virtual lights to generate a first lighting effect corresponding to the audio attributes of the currently played target audio.

[0226] In some embodiments, the control module is further configured to, when there are multiple virtual lamps, obtain the mapping relationship between the lighting effects of each virtual lamp and the audio attributes; and, based on the audio attributes of the currently playing target audio and the mapping relationship, control each virtual lamp to produce a corresponding lighting effect to form the first lighting effect.

[0227] In some embodiments, the control module is further configured to, when the number of virtual lights is multiple, acquire the virtual lighting effects of the virtual lights corresponding to the target video; wherein the virtual lighting effects include: the first lighting effect in the virtual scene at each playback time point of the target audio, the first lighting effect being formed collaboratively by the lighting effects of each of the virtual lights; and based on the virtual lighting effects, establishing a mapping relationship between the lighting effects of each of the virtual lights and the audio attributes at each playback time point.

[0228] In some embodiments, the control module is further configured to, when the audio attribute includes audio rhythm, acquire the audio rhythm of the target audio being played in real time during the playback of the target audio; and, based on the audio rhythm of the currently playing target audio, control the virtual lamps to produce lighting effects corresponding to the audio rhythm in real time; wherein different audio rhythms correspond to different lighting effects.

[0229] In some embodiments, the control module is further configured to, when the audio attribute includes audio content, acquire the audio content of the target audio being played in real time during the playback of the target audio; classify the acquired audio content to obtain the content type of the audio content; and control the virtual lamp to generate a first lighting effect corresponding to the content type of the audio content; wherein different content types correspond to different lighting effects.

[0230] In some embodiments, the control module is further configured to, when the audio attribute includes sound frequency, acquire the sound frequency of the target audio being played in real time during the playback of the target audio; determine the frequency range to which the acquired sound frequency belongs; and control the virtual lamp to produce a first lighting effect corresponding to the frequency range to which the sound frequency belongs; wherein different frequency ranges correspond to different lighting effects.

[0231] In some embodiments, the control module is further configured to obtain the attribute value of at least one target attribute of the virtual lamp corresponding to the first lighting effect; and assign the attribute value of each target attribute to the corresponding target attribute of the virtual lamp, so that the virtual lamp produces the first lighting effect.

[0232] In some embodiments, the control module is further configured to: acquire the correspondence between audio tracks and virtual lights when the number of virtual lights is multiple and the target audio includes audio data corresponding to at least two audio tracks; determine the virtual lights corresponding to each audio track based on the correspondence; and control the virtual lights corresponding to each audio track in the virtual scene to generate virtual lighting effects corresponding to the audio data of the audio track, thereby forming the first lighting effect; wherein the audio data of different audio tracks correspond to different virtual lighting effects, and the first lighting effect is formed collaboratively by the virtual lighting effects of each virtual light.

[0233] In some embodiments, the control module is further configured to establish a communication connection with the physical lighting fixture in the real scene; based on the communication connection, send control commands to the physical lighting fixture, the control commands being used to instruct the physical lighting fixture to produce a second lighting effect corresponding to the currently playing target audio.

[0234] In some embodiments, the control module is further configured to acquire the values ​​of the lighting parameters of the physical luminaire corresponding to the first lighting effect, the values ​​of the lighting parameters being used by the physical luminaire to generate the second lighting effect; the step of sending control commands to the physical luminaire based on the communication connection includes:

[0235] Based on the communication connection, a control command carrying the value of the lighting parameters is sent to the physical luminaire so that the physical luminaire applies the value of the lighting parameters.

[0236] In some embodiments, the control command carries lighting effect indication information for indicating the second lighting effect, and the control module is further configured to send a control command carrying the lighting effect indication information to the physical luminaire based on the communication connection; wherein, the control command is used by the physical luminaire to determine the value of the lighting parameter corresponding to the second lighting effect based on the lighting effect indication information, and apply the value of the lighting parameter to generate the second lighting effect.

[0237] In some embodiments, the creation module is further configured to obtain the values ​​of the lighting parameters of the physical lighting fixtures in the real scene, and the association relationship between the physical lighting fixtures and the virtual lighting fixtures; determine the values ​​of the lighting parameters of the virtual lighting fixtures in the virtual scene based on the values ​​of the lighting parameters of the physical lighting fixtures and the association relationship; and assign the determined values ​​of the lighting parameters to the lighting parameters of the virtual lighting fixtures in the virtual scene.

[0238] This application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the lighting control method in virtual filmmaking described above in this application embodiment.

[0239] This application provides a computer-readable storage medium storing executable instructions. When these executable instructions are executed by a processor, they cause the processor to execute the lighting control method in virtual filmmaking provided in this application. For example, ... Figure 3 The lighting control method in virtual production is shown.

[0240] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0241] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0242] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborating files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0243] As an example, executable instructions can be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0244] In summary, the embodiments of this application enable one person to control the sound playback, virtual lighting, and physical lighting responses, achieving a three-way synchronous response. This method can significantly reduce errors caused by the coordination between speakers, lights, and the Unreal Engine, and can also greatly reduce the influence of external factors, thereby significantly improving the shooting efficiency of this type of shot.

[0245] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A lighting control method in virtual filmmaking, characterized in that, The virtual production method is used to generate a target video by combining real-world and virtual scenes. The method includes: Using the physical lighting fixtures in the real scene as a reference, create virtual lighting fixtures in the virtual scene that correspond to the physical lighting fixtures; Obtain the target audio to be applied to the target video, and play the target audio; During the playback of the target audio, the spectrum of the target audio is analyzed in real time according to the current playback time of the target audio, and the spectrum is decomposed into multiple different amplitude curves. Multiple different amplitudes at the playback time point are determined from the multiple different amplitude curves. Using a preset allocation strategy, the multiple different amplitudes are allocated to the channels of the lamps. The channels include at least one of the lamps' horizontal axis rotation, vertical axis rotation, brightness, and hue. The lamps include the physical lamps and the virtual lamps. Based on the aforementioned allocation strategy, the multiple amplitude data are sent to the virtual lights, causing the virtual lights to change brightness or color in response to the sound rhythm. This controls the virtual lights to produce a first lighting effect in the virtual scene corresponding to the currently playing target audio. Simultaneously... The multiple different amplitudes are combined into data that conforms to the digital multiplexing protocol and sent to the physical lamp. The data is used to synchronously control the physical lamp to make a corresponding response, so as to generate a second lighting effect in the real scene that corresponds to the target audio being played. The first lighting effect and the second lighting effect are used for the virtual video production to generate the target video containing the first lighting effect and the second lighting effect.

2. The method as described in claim 1, characterized in that, The control of the virtual lights to generate a first lighting effect in the virtual scene corresponding to the currently playing target audio includes: During the playback of the target audio, the audio attributes of the target audio are acquired in real time, and the audio attributes include at least one of sound frequency, audio rhythm, and audio content; In the virtual scene, the virtual lights are controlled to produce a first lighting effect corresponding to the audio attributes of the currently playing target audio.

3. The method as described in claim 2, characterized in that, The control of the virtual lighting fixtures to generate a first lighting effect corresponding to the audio attributes of the currently playing target audio includes: When there are multiple virtual lamps, obtain the mapping relationship between the lighting effects of each virtual lamp and the audio attributes; Based on the audio attributes of the currently playing target audio and the mapping relationship, each of the virtual lights is controlled to produce corresponding lighting effects to form the first lighting effect.

4. The method as described in claim 3, characterized in that, When there are multiple virtual lights, before obtaining the mapping relationship between the lighting effects of each virtual light and the audio attributes, the method further includes: Obtain the virtual lighting effect of the virtual lamps corresponding to the target video; The virtual lighting effect includes: at each playback time point of the target audio, the first lighting effect in the virtual scene, wherein the first lighting effect is formed by the coordinated lighting effects of each of the virtual lamps; Based on the virtual lighting effects, a mapping relationship is established between the lighting effects of each virtual lamp and the audio attributes at each playback time point.

5. The method as described in claim 2, characterized in that, The control of the virtual lighting fixtures to generate a first lighting effect corresponding to the audio attributes of the currently playing target audio includes: When the audio attribute includes audio rhythm, the audio rhythm of the target audio being played is acquired in real time during the playback of the target audio. Based on the audio rhythm of the currently playing target audio, the virtual lights are controlled in real time to produce lighting effects corresponding to the audio rhythm; Different audio rhythms correspond to different lighting effects.

6. The method as described in claim 2, characterized in that, The control of the virtual lighting fixtures to generate a first lighting effect corresponding to the audio attributes of the currently playing target audio includes: When the audio attribute includes audio content, the audio content of the target audio being played is obtained in real time during the playback of the target audio. The acquired audio content is categorized to obtain the content type of the audio content; Control the virtual lights to produce a first lighting effect corresponding to the content type of the audio content; Different content types correspond to different lighting effects.

7. The method as described in claim 2, characterized in that, The control of the virtual lighting fixtures to generate a first lighting effect corresponding to the audio attributes of the currently playing target audio includes: When the audio attribute includes sound frequency, the sound frequency of the target audio being played is acquired in real time during the playback of the target audio. Determine the frequency range to which the acquired sound frequency belongs; Control the virtual lighting fixtures to produce a first lighting effect corresponding to the frequency range to which the sound frequency belongs; Different frequency ranges correspond to different lighting effects.

8. The method as described in claim 1, characterized in that, The control of the virtual lights to generate a first lighting effect in the virtual scene corresponding to the currently playing target audio includes: Obtain the attribute value of at least one target attribute of the virtual lamp corresponding to the first lighting effect; The attribute values ​​of each of the target attributes are assigned to the corresponding target attributes of the virtual lamp, so that the virtual lamp produces the first lighting effect.

9. The method as described in claim 1, characterized in that, When the number of virtual lights is multiple and the target audio includes audio data corresponding to at least two audio tracks, controlling the virtual lights to generate a first lighting effect in the virtual scene corresponding to the currently playing target audio includes: Obtain the correspondence between audio tracks and virtual lights; Based on the correspondence, the virtual lights corresponding to each of the audio tracks are determined; In the virtual scene, the virtual lights corresponding to each audio track are controlled to generate virtual lighting effects corresponding to the audio data of the audio track, so as to form the first lighting effect; The audio data of different audio tracks correspond to different virtual lighting effects, and the first lighting effect is formed by the combined virtual lighting effects of each virtual lamp.

10. The method as described in claim 1, characterized in that, The synchronous control of the physical lighting fixtures to produce a second lighting effect in the real scene corresponding to the currently playing target audio includes: Establish a communication connection with the physical lighting fixtures in the real-world scenario; Based on the communication connection, a control command is sent to the physical lamp, which instructs the physical lamp to produce a second lighting effect corresponding to the currently playing target audio.

11. The method as described in claim 10, characterized in that, Before sending control commands to the physical lighting fixture, the method further includes: Obtain the values ​​of the lighting parameters of the physical luminaire corresponding to the first lighting effect; the values ​​of the lighting parameters are used by the physical luminaire to generate the second lighting effect. The step of sending control commands to the physical lighting fixture based on the communication connection includes: Based on the communication connection, a control command carrying the value of the lighting parameters is sent to the physical luminaire so that the physical luminaire applies the value of the lighting parameters.

12. The method as described in claim 10, characterized in that, The control command carries lighting effect indication information for indicating the second lighting effect. The step of sending the control command to the physical luminaire based on the communication connection includes: Based on the communication connection, a control command carrying the lighting effect indication information is sent to the physical lamp; The control command is used by the physical luminaire to determine the value of the lighting parameter corresponding to the second lighting effect based on the lighting effect indication information, and to apply the value of the lighting parameter to produce the second lighting effect.

13. The method as described in claim 1, characterized in that, The step of creating virtual lighting fixtures in the virtual scene corresponding to the physical lighting fixtures in the real scene, using the physical lighting fixtures in the real scene as a reference, includes: Obtain the values ​​of the lighting parameters of the physical lighting fixtures in the real scene, as well as the association between the physical lighting fixtures and the virtual lighting fixtures; Based on the values ​​of the physical lighting fixtures' lighting parameters and the associated relationships, the values ​​of the virtual lighting fixtures' lighting parameters in the virtual scene are determined. The determined values ​​of the lighting parameters are assigned to the lighting parameters of the virtual lighting fixtures in the virtual scene.

14. A lighting control device for virtual film production, characterized in that, The virtual production method is used to generate a target video by combining real-world and virtual scenes, and the device includes: A creation module is used to create virtual lighting fixtures in the virtual scene that correspond to the physical lighting fixtures in the real scene, using the physical lighting fixtures in the real scene as a reference. The acquisition module is used to acquire target audio applied to the target video and play the target audio; The control module is used to analyze the spectrum of the target audio in real time according to the current playback time of the target audio during the playback of the target audio, decompose the spectrum into multiple different amplitude curves, and determine multiple different amplitudes at the playback time point from the multiple different amplitude curves. Using a preset allocation strategy, the multiple different amplitudes are allocated to the channels of the lamps. The channels include at least one of the lamps' horizontal axis rotation, vertical axis rotation, brightness, and hue. The lamps include the physical lamps and the virtual lamps. Based on the aforementioned allocation strategy, the multiple amplitude data are sent to the virtual lights, causing the virtual lights to change brightness or color in response to the sound rhythm. This controls the virtual lights to produce a first lighting effect in the virtual scene corresponding to the currently playing target audio. Simultaneously... The multiple different amplitudes are combined into data that conforms to the digital multiplexing protocol and sent to the physical lamp. The data is used to synchronously control the physical lamp to make a corresponding response, so as to generate a second lighting effect in the real scene that corresponds to the target audio being played. The first lighting effect and the second lighting effect are used for the virtual video production to generate the target video containing the first lighting effect and the second lighting effect.

15. The apparatus according to claim 14, characterized in that, The control module is also used to acquire the audio attributes of the target audio in real time during the playback of the target audio, and the audio attributes include at least one of sound frequency, audio rhythm, and audio content; In the virtual scene, the virtual lights are controlled to produce a first lighting effect corresponding to the audio attributes of the currently playing target audio.

16. The apparatus according to claim 15, characterized in that, The control module is also used to obtain the mapping relationship between the lighting effect of each virtual lamp and the audio attribute when there are multiple virtual lamps; Based on the audio attributes of the currently playing target audio and the mapping relationship, each of the virtual lights is controlled to produce corresponding lighting effects to form the first lighting effect.

17. The apparatus according to claim 16, characterized in that, The control module is also used to acquire the virtual lighting effect of the virtual lamps corresponding to the target video; The virtual lighting effect includes: at each playback time point of the target audio, the first lighting effect in the virtual scene, wherein the first lighting effect is formed by the coordinated lighting effects of each of the virtual lamps; Based on the virtual lighting effects, a mapping relationship is established between the lighting effects of each virtual lamp and the audio attributes at each playback time point.

18. The apparatus according to claim 15, characterized in that, The control module is also used to acquire the audio rhythm of the target audio being played in real time during the playback of the target audio when the audio attribute includes audio rhythm. Based on the audio rhythm of the currently playing target audio, the virtual lights are controlled in real time to produce lighting effects corresponding to the audio rhythm; Different audio rhythms correspond to different lighting effects.

19. The apparatus according to claim 15, characterized in that, The control module is also used to acquire the audio content of the target audio being played in real time during the playback of the target audio when the audio attribute includes audio content; The acquired audio content is categorized to obtain the content type of the audio content; Control the virtual lights to produce a first lighting effect corresponding to the content type of the audio content; Different content types correspond to different lighting effects.

20. The apparatus according to claim 15, characterized in that, The control module is also used to acquire the sound frequency of the target audio being played in real time during the playback of the target audio when the audio attribute includes sound frequency; Determine the frequency range to which the acquired sound frequency belongs; Control the virtual lighting fixtures to produce a first lighting effect corresponding to the frequency range to which the sound frequency belongs; Different frequency ranges correspond to different lighting effects.

21. An electronic device, characterized in that, The electronic device includes: Memory, used to store executable instructions; A processor, when executing executable instructions stored in the memory, implements the lighting control method in virtual filmmaking as described in any one of claims 1 to 13.

22. A computer-readable storage medium storing executable instructions, characterized in that, When the executable instructions are executed by the processor, they implement the lighting control method in virtual filmmaking as described in any one of claims 1 to 13.

23. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by the processor, they implement the lighting control method in virtual filmmaking as described in any one of claims 1 to 13.

Images