Methods and devices for aligning virtual and real light directions, electronic devices, storage media and program products

Abstract

This disclosure relates to a method and apparatus for aligning the directions of virtual and real lighting, electronic devices, storage media, and program products. The method includes: acquiring physical light source information, the physical light source signal including the lighting direction vectors of multiple physical light sources; acquiring virtual light source information, the virtual light source information including the lighting direction vector of at least one virtual light source with directionality in a virtual scene displayed on a real screen; determining a target physical light source from the multiple physical light sources that has the same lighting direction as each of the at least one virtual light source, based on the physical light source information and the virtual light source information; and illuminating the target physical light source to align the lighting effect in the virtual scene displayed on the real screen with the lighting effect in the foreground area in front of the real screen. Therefore, the alignment of the light and shadow effects of virtual and real lighting in virtual shooting can be automatically achieved, resulting in better lighting direction alignment more efficiently and reducing the difficulty of virtual lighting direction alignment.

Description

Technical Field

[0001] This disclosure relates to the field of virtual photography, and more particularly to a method and apparatus for aligning virtual and real light directions, electronic devices, storage media, and program products. Background Technology

[0002] Virtual filming based on LED screens refers to a process where actors perform in front of an LED screen wall, where a pre-made virtual scene is played on the LED screen. Cameras film the actors and the LED screen, and the director's monitor displays the visual effect of the real actors' performance blending with the virtual scene in real time (previously this could only be done in post-production).

[0003] In LED screen-based virtual shooting projects, aligning the lighting in the actual foreground shooting area with the lighting in the virtual scene on the LED screen is a crucial prerequisite for achieving realistic and vivid shooting results. Especially when the lighting in the virtual scene has a certain directionality, it's necessary to ensure the actual foreground lighting illuminates in the same direction to maintain consistency in the lighting effects between the virtual and actual shooting scenes.

[0004] The large number and complex layout of lights required in the virtual scene during filming make it difficult to align the direction of the physical lights on set. It's challenging to physically install corresponding real-world lights to achieve consistent lighting effects between the virtual and real environments. Furthermore, most current methods for aligning virtual and real lights in virtual filming rely on manual adjustment, which is inefficient, difficult, and produces subjective results. Summary of the Invention

[0005] In view of this, this disclosure proposes a method and apparatus for aligning virtual and real light directions, an electronic device, a storage medium and a program product, which can reduce the difficulty of light alignment in real space and can achieve better virtual and real light direction alignment effect more efficiently.

[0006] According to one aspect of this disclosure, a method for aligning virtual and real lighting directions is provided, comprising: acquiring physical light source information, wherein the physical light source information includes lighting direction vectors of a plurality of physical light sources, the lighting direction vectors representing the lighting direction of the light sources, the plurality of physical light sources facing a foreground area in front of a real screen; acquiring virtual light source information, wherein the virtual light source information includes lighting direction vectors of at least one virtual light source having directionality in a virtual scene displayed on the real screen; determining, based on the physical light source information and the virtual light source information, a target physical light source having the same lighting direction as each of the at least one virtual light source from the plurality of physical light sources; and illuminating the target physical light source to align the lighting effects in the virtual scene displayed on the real screen with the lighting effects in the foreground area in front of the real screen.

[0007] In one possible implementation, the physical light source signal includes the lighting direction vectors of the plurality of physical light sources in the screen coordinate system of the real screen; the virtual light source information includes the lighting direction vectors of at least one directional virtual light source in the virtual scene in the virtual world coordinate system of the virtual scene; wherein, determining the target physical light source with the same lighting direction as each of the at least one virtual light source from the plurality of physical light sources based on the physical light source information and the virtual light source information includes: transforming the physical light source information and the virtual light source information to the same coordinate system based on the transformation relationship between the virtual world coordinate system and the screen coordinate system to obtain target physical light source information and target virtual light source information located in the same coordinate system; and determining the target physical light source with the same lighting direction as each of the at least one virtual light source from the plurality of physical light sources based on the target physical light source information and the target virtual light source information.

[0008] In one possible implementation, determining the target physical light source with the same lighting direction as each of the at least one virtual light source from the plurality of physical light sources, based on the target physical light source information and the target virtual light source information, includes: for any virtual light source among the at least one virtual light source, calculating the angle between the lighting direction vector of the virtual light source and the lighting direction vector of each physical light source, based on the target physical light source information and the target virtual light source information; and determining the physical light source with an angle less than a specified degree as the target physical light source with the same lighting direction as the virtual light source.

[0009] In one possible implementation, determining the target physical light source from the plurality of physical light sources that has the same lighting direction as each of the at least one virtual light source includes: for any virtual light source, selecting the physical light source from the plurality of physical light sources whose lighting direction differs from the lighting direction of the virtual light source by less than a preset threshold as the target physical light source corresponding to that virtual light source, wherein the magnitude of the preset threshold is negatively correlated with the magnitude of the setting density of the plurality of physical light sources.

[0010] In one possible implementation, obtaining virtual light source information includes: obtaining the position and angle of at least one directional virtual light source in the virtual scene through Unreal Engine, and determining the lighting direction vector of the at least one virtual light source based on the position and angle of the at least one virtual light source.

[0011] In one possible implementation, the plurality of physical light sources include uniformly distributed physical light sources mounted above the real screen; the plurality of physical light sources are directed toward the central region of the foreground area in front of the real screen.

[0012] In one possible implementation, the method further includes: when the lighting direction vector of any virtual light source in the virtual scene changes, determining a target physical light source with the same lighting direction as the virtual light source after the change, based on the changed lighting direction vector of the virtual light source and the physical light source information, so as to turn on the target physical light source with the same lighting direction as the virtual light source after the change.

[0013] According to another aspect of this disclosure, a virtual-real lighting direction alignment device is provided, comprising: a first acquisition module, configured to acquire physical light source information, the physical light source signal including lighting direction vectors of a plurality of physical light sources, the lighting direction vectors representing the lighting direction of the light sources, the plurality of physical light sources facing a foreground area in front of a real screen; a second acquisition module, configured to acquire virtual light source information, the virtual light source information including lighting direction vectors of at least one virtual light source having directionality in a virtual scene displayed on the real screen; a determination module, configured to determine, based on the physical light source information and the virtual light source information, a target physical light source having the same lighting direction as each of the at least one virtual light source; and a control module, configured to illuminate the target physical light source by turning on the target physical light source, so that the lighting effect in the virtual scene displayed on the real screen is aligned with the lighting effect in the foreground area in front of the real screen.

[0014] According to another aspect of this disclosure, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement the above-described method when executing instructions stored in the memory.

[0015] According to another aspect of this disclosure, a non-volatile computer-readable storage medium is provided that stores computer program instructions thereon, wherein the computer program instructions, when executed by a processor, implement the above-described method.

[0016] According to another aspect of this disclosure, a computer program product is provided, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the above-described method.

[0017] According to various aspects of this disclosure, by acquiring the lighting direction vectors of multiple physical light sources and the lighting direction vectors of directional virtual light sources in a virtual scene, a target physical light source with the same lighting direction as each virtual light source can be determined from multiple physical light sources. Then, by turning on the target physical light source, the alignment of the light and shadow effects of virtual and real lights in virtual shooting can be automatically achieved, which can achieve a better lighting direction alignment effect more efficiently and reduce the difficulty of virtual light direction alignment.

[0018] Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0019] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of this disclosure together with the specification and serve to explain the principles of this disclosure.

[0020] Figure 1 A flowchart illustrating a method for aligning the directions of virtual and real lights according to an embodiment of the present disclosure is shown.

[0021] Figure 2 A schematic diagram of a shooting scene according to an embodiment of the present disclosure is shown.

[0022] Figure 3 A block diagram is shown of a virtual and real light direction alignment device according to an embodiment of the present disclosure.

[0023] Figure 4 A block diagram of an electronic device 1900 according to an embodiment of the present disclosure is shown. Detailed Implementation

[0024] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0025] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0026] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more of a plurality of elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C. In the description of this disclosure, "multiple" means two or more, unless otherwise explicitly specified.

[0027] It should be understood that the terms “comprising” and “including” used in the specification and claims of this disclosure indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0028] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0029] The virtual and real light direction alignment method of this disclosure can be deployed on various terminal devices through software or hardware modifications. The terminal devices involved in this disclosure can refer to devices with wireless and / or wired connection functions. Wireless connection means that they can connect to other devices via Wi-Fi, Bluetooth, or other wireless connection methods. The terminal devices involved in this disclosure can also communicate with other devices via wired connection functions. The terminal devices involved in this disclosure can be touchscreen, non-touchscreen, or screenless. Touchscreen devices can be controlled by clicking or swiping on the display screen using fingers, styluses, etc. Non-touchscreen devices can connect to input devices such as mice, keyboards, and touch panels to control the terminal device. Screenless devices can be, for example, screenless Bluetooth speakers. For example, the terminal devices in this application can include, but are not limited to, user equipment (UE), mobile devices, user terminals, terminals, handheld devices, tablet computers, laptops, PDAs, computing devices, etc.

[0030] The virtual-real lighting direction alignment method of this disclosure can also be deployed on a server. This server can be located in the cloud or locally, and can be a physical device or a virtual device, such as a virtual machine or container. It has wireless communication capabilities, which can be configured in the server's chip (system) or other components. It can refer to a device with wireless connectivity, meaning it can connect to other servers or terminal devices via Wi-Fi, Bluetooth, or other wireless connection methods. The server involved in this disclosure can also have wired communication capabilities. For example, the server in this disclosure can be located in the cloud, communicating with terminal devices, receiving physical and virtual light source information sent by the terminal devices, and using the virtual-real lighting direction alignment method deployed on the server, generating a target physical light source with the same lighting direction as the virtual light source in the virtual scene based on the physical and virtual light source information. This target physical light source is then returned to the terminal device, allowing the terminal device to control the activation of the target physical light source for lighting.

[0031] Figure 1 A flowchart illustrating a method for aligning the directions of real and virtual lights according to an embodiment of the present disclosure is shown. Figure 1 As shown, the method includes steps S11 to S14.

[0032] In step S11, physical light source information is obtained. The physical light source signal includes the lighting direction vectors of multiple physical light sources. The lighting direction vectors represent the lighting direction of the light sources. The multiple physical light sources face the foreground area in front of the real screen.

[0033] The physical light source can be a point light source, that is, a light source that can produce directionality. It should be understood that those skilled in the art can set the type, quantity, and installation location of the physical light sources according to actual needs, and this disclosure does not limit this. For example, multiple physical light sources may include uniformly distributed physical light sources installed above a real screen. For example, Figure 2 The diagram shown illustrates a filming location, such as... Figure 2 As shown, 12 evenly distributed physical light sources (D1, D2, ..., D12) can be installed above the actual screen (i.e., the arc-shaped LED screen) as the light source for real hard light. The physical light sources can be installed in a ring or at least 270° of point light sources above the actual screen; this embodiment does not limit this. Depending on actual needs, the physical light sources can also be installed in other locations besides above the screen, and can be evenly or unevenly distributed. The lighting direction can also be set according to actual needs; this application does not limit this.

[0034] The real screen, also known as the screen used to display the virtual scene, could be, for example, a... Figure 2 The circular LED screen shown; the foreground area in front of the actual screen can be understood as the filming area, that is, the area where the actors perform and the props are set up, for example. Figure 2 The foreground area is in front of the LED screen; therefore, the physical light source should be oriented towards this foreground area, that is, the lighting direction of the physical light source should be towards this foreground area. Considering that the area where the actors perform and the props are set up is usually in the center of the foreground area (that is, in the central area in front of the real screen), multiple physical light sources can be oriented towards the center of the foreground area in front of the real screen to meet the lighting needs in the real shooting location, so that the foreground area can obtain hard light from any direction from multiple surrounding directions (such as 360 degrees).

[0035] It should be understood that, given that the position and orientation of physical light sources in the physical space of the shooting location are known, for example, the lighting direction vector of each physical light source in the screen coordinate system can be obtained by establishing a screen coordinate system of the real screen (e.g., a coordinate system established with a vertex or center point of the real screen as the origin). Alternatively, the lighting direction vector of each physical light source in the world coordinate system can be obtained by establishing a world coordinate system of the shooting location. The lighting direction vector can be visualized as a line connecting the position of the physical light source to the center area of ​​the foreground, which is equivalent to the direction of light emitted by the physical light source.

[0036] In step S12, virtual light source information is obtained, which includes the lighting direction vector of at least one virtual light source with directionality in the virtual scene displayed on the real screen.

[0037] The virtual scene can be referred to as a virtual asset. It should be understood that those skilled in the art can design the virtual scene currently displayed on the real screen according to the background image required for actual shooting. At least one directional virtual light source can be set in the virtual scene, such as a virtual light source to simulate the sunlight rising from the east, or a virtual light source to simulate indoor lighting or candlelight. These virtual light sources are directional and will cause other objects in the virtual scene to produce light and shadow effects.

[0038] Obtaining virtual light source information may include: acquiring the position and angle of at least one directional virtual light source in the virtual scene using Unreal Engine, and determining the lighting direction vector of at least one virtual light source based on the position and angle of the at least one virtual light source. It should be understood that Unreal Engine is used for virtual scene construction and visual effects production; therefore, virtual scenes can be constructed using Unreal Engine. Consequently, the position (i.e., the virtual light source's location within the virtual scene) and angle (i.e., the virtual light source's orientation within the virtual scene) of at least one directional virtual light source in the virtual scene can be obtained using relevant plugins provided with Unreal Engine. Based on the position and angle of the virtual light source, the lighting direction vector of the virtual light source in the virtual world coordinate system of the virtual scene can be determined. The lighting direction vector of the virtual light source can represent the lighting direction of the virtual light source. The virtual world coordinate system can be understood as the world coordinate system of Unreal Engine (UE). Of course, other methods can also be used to obtain virtual light source information, depending on the technology used to construct the virtual scene; this disclosure does not limit such methods.

[0039] It should be understood that at least one virtual light source in the virtual light source information can be all the virtual light sources in the virtual scene. That is, the target physical light source can be found for all virtual light sources. Users can also select a portion of virtual light sources as needed to determine the corresponding target physical light source.

[0040] In step S13, based on the physical light source information and the virtual light source information, a target physical light source is determined from multiple physical light sources that has the same lighting direction as each virtual light source in at least one virtual light source.

[0041] In one possible implementation, determining the target physical light source from a plurality of physical light sources that has the same lighting direction as each of the at least one virtual light source may include: for any virtual light source, selecting the physical light source from the plurality of physical light sources whose lighting direction differs from that of the virtual light source by less than a preset threshold as the target physical light source corresponding to that virtual light source, wherein the magnitude of the preset threshold is negatively correlated with the magnitude of the setting density of the plurality of physical light sources.

[0042] The difference between the lighting direction of the physical light source and the lighting direction of the virtual light source can be quantified, for example, as the angle or distance between the lighting direction vector of the physical light source and the lighting direction vector of the virtual light source. In other words, among multiple physical light sources, the physical light source whose angle between the lighting direction vector of the physical light source and the lighting direction vector of the virtual light source is less than a preset threshold can be identified as the corresponding target physical light source. Alternatively, among multiple physical light sources, the physical light source whose distance between the lighting direction vector of the physical light source and the lighting direction vector of the virtual light source is less than a preset threshold can be identified as the corresponding target physical light source. This embodiment of the present disclosure does not limit this.

[0043] The preset threshold is negatively correlated with the density of multiple physical light sources. This can be understood as follows: the denser the physical light sources are (i.e., the closer they are to each other), the closer the lighting directions of adjacent physical light sources are, and the smaller the preset threshold can be. Consequently, the alignment effect is more precise.

[0044] As described above, the physical light source signal includes the lighting direction vectors of multiple physical light sources, and the virtual light source information includes the lighting direction vector of at least one virtual light source with directionality in the virtual scene displayed on the real screen. Therefore, determining the target physical light source from multiple physical light sources with the same lighting direction as each of the at least one virtual light source, based on the physical light source information and the virtual light source information, can include: for any virtual light source, calculating the angle between the lighting direction vector of the virtual light source and the lighting direction vectors of each physical light source, based on the physical light source information and the virtual light source information, and determining the physical light source with an angle less than a specified degree as the target physical light source with the same lighting direction as the virtual light source; wherein, the specified degree can be set custom-defined, for example, it can be set to 10°, and this embodiment of the present disclosure does not limit this. The specified degree can be negatively correlated with the setting density of physical light sources, that is, the denser the physical light sources are set (i.e., the closer the physical light sources are), the closer the lighting directions of adjacent physical light sources are, and the smaller the specified degree can be.

[0045] For example, the lighting direction vector of a virtual light source can be calculated and compared with the above. Figure 2 The angle between each of the 12 physical light sources (D1, D2, ..., D12) is determined as follows: if the angle between physical light source D2 and the lighting direction vector of the virtual light source is less than a specified degree, then physical light source D2 can be determined as the target physical light source with the same lighting direction as the virtual light source. If the angle between physical light sources D2 and D3 and the lighting direction vector of the virtual light source is less than a specified angle, then physical light sources D2 and D3 can both be determined as the target physical light sources with the same lighting direction as the virtual light source.

[0046] The above method can be understood as physical light source information and virtual light source information being in a unified coordinate system. However, considering that in reality, physical light source information and virtual light source information are usually established in different coordinate systems, for example, physical light source signals include the lighting direction vectors of multiple physical light sources in the screen coordinate system of the real screen, while virtual light source information includes the lighting direction vector of at least one directional virtual light source in the virtual scene in the virtual world coordinate system of the virtual scene, in order to more accurately determine the physical light source with the same lighting direction, in one possible implementation, step S13 above, based on the physical light source information and virtual light source information, determines the target physical light source from multiple physical light sources that has the same lighting direction as each of the at least one virtual light source, including:

[0047] Step S131: Based on the transformation relationship between the virtual world coordinate system and the screen coordinate system, the physical light source information and the virtual light source information are transformed to the same coordinate system to obtain the target physical light source information and the target virtual light source information located in the same coordinate system.

[0048] Step S132: Based on the target physical light source information and the target virtual light source information, determine the target physical light source from multiple physical light sources that has the same lighting direction as each virtual light source in at least one virtual light source.

[0049] Specifically, the transformation relationship between the virtual world coordinate system and the screen coordinate system can be obtained in advance through spatial calibration technology. This transformation relationship can be represented as a transformation matrix. In virtual shooting based on an LED screen, spatial calibration technology, through a certain calculation process, obtains the relative transformation relationship between the LED screen coordinate system, the world coordinate system, and the camera coordinate system. That is, in the field of virtual shooting, spatial calibration technology can obtain the relative positional relationship between the real camera and the real screen. Since the pose of the virtual camera in the virtual world coordinate system of the virtual scene is known, and the poses of the real camera and the virtual camera are consistent, the transformation relationship between the screen coordinate system of the real screen and the virtual world coordinate system can be obtained. This disclosure does not limit the process of determining the above transformation relationship.

[0050] In step S131, based on the transformation relationship between the virtual world coordinate system and the screen coordinate system, the physical light source information and the virtual light source information are transformed to the same coordinate system to obtain the target physical light source information and the target virtual light source information located in the same coordinate system, including:

[0051] Based on the transformation relationship between the virtual world coordinate system and the screen coordinate system, the virtual light source information is transformed into the screen coordinate system to obtain the target physical light source information and the target virtual light source information located in the same coordinate system; or...

[0052] Based on the transformation relationship between the virtual world coordinate system and the screen coordinate system, the physical light source information is transformed into the virtual world coordinate system to obtain the target physical light source information and the target virtual light source information located in the same coordinate system.

[0053] In other words, the lighting direction vector of a virtual light source in the virtual world coordinate system can be converted to the screen coordinate system, or the lighting direction vector of a physical light source in the screen coordinate system can be converted to the virtual world coordinate system, so as to obtain the target physical light source information and the target virtual light source information located in the same coordinate system.

[0054] Furthermore, after transforming to the same coordinate system, for example, the angle between the lighting direction vector of each virtual light source and the lighting direction vector of each physical light source can be calculated. When the angle is less than a specified degree, the lighting direction of the physical light source is considered to be consistent with that of the virtual light source. That is, step S132 above, which determines the target physical light source from multiple physical light sources that has the same lighting direction as each virtual light source in at least one virtual light source, based on the target physical light source information and the target virtual light source information, may include:

[0055] For any virtual light source among at least one virtual light source, the angle between the lighting direction vector of the virtual light source and the lighting direction vector of each physical light source is calculated based on the target physical light source information and the target virtual light source information.

[0056] Physical light sources with an included angle less than a specified degree are designated as target physical light sources with the same lighting direction as the virtual light source.

[0057] As mentioned above, the specified degree can be customized, and this embodiment of the disclosure does not impose any limitations on this. Furthermore, each virtual light source can identify at least one target physical light source with the same lighting direction.

[0058] It should be understood that the above-described method of determining the target physical light source with the same lighting direction as the virtual light source by calculating the angle between two lighting direction vectors is one possible implementation method provided by the embodiments of this disclosure.

[0059] In fact, those skilled in the art can also use similarity calculation methods known in the art to determine the target physical light source with the same lighting direction as the virtual light source. For example, by calculating the cosine similarity between the lighting direction vector of the virtual light source and the lighting direction vector of each physical light source, the physical light source with a cosine similarity greater than a specified threshold can be determined as the target physical light source with the same lighting direction as the virtual light source. This disclosure does not limit this aspect.

[0060] It should be understood that for each virtual light source in a virtual scene, at least one target physical light source can be identified that has the same lighting direction as each virtual light source. These target physical light sources can then be used to illuminate the real foreground area with the same direction as in the virtual scene, thus achieving consistency between the virtual and real light source effects, i.e., alignment of virtual lighting effects.

[0061] In step S14, the target physical light source is turned on to illuminate the scene so that the lighting effects in the virtual scene displayed on the real screen are aligned with the lighting effects in the foreground area in front of the real screen.

[0062] This involves activating the target physical light source for illumination, which means activating a physical light source with the same illumination direction as each virtual light source in the virtual scene. Activating the target physical light source also means controlling it to be on (powered on) and controlling all other physical light sources to be off (powered off). It should be understood that those skilled in the art can use known lighting controllers to control the on / off state of each physical light source, and this disclosure does not limit this approach. Since the target physical light source and the virtual light sources have the same illumination direction, activating the target physical light source allows the lighting effects in the virtual scene displayed on the real screen to be aligned with the lighting effects in the foreground area in front of the real screen.

[0063] In practical applications, the lighting direction of the virtual light source in the virtual scene may change during the shooting process. For example, the virtual sun (i.e., the virtual light source) in the virtual scene may change from east to west within a few minutes. In this case, the physical light source of the target that needs to be aligned also changes accordingly. Therefore, in one possible implementation, the method may further include:

[0064] When the lighting direction vector of any virtual light source in the virtual scene changes, a target physical light source with the same lighting direction as the virtual light source after the change and the physical light source information are determined from multiple physical light sources, so as to turn on the target physical light source with the same lighting direction as the virtual light source after the change.

[0065] The specific implementation of step S13 above can be used to determine the target physical light source with the same lighting direction as the changed virtual light source from multiple physical light sources. For example, the target physical light source with the same lighting direction as the changed virtual light source can be determined based on the angle between the lighting direction vector of the changed virtual light source and the lighting direction vectors of each physical light source. Then, the target physical light source with the same lighting direction as the changed virtual light source can be turned on, and the target physical light source with the same lighting direction as before the change can be turned off. This allows the opening and closing of each physical light source set in the actual shooting location to automatically change with the change of the lighting direction vector of the virtual light source, meeting the alignment requirements of virtual and real light directions. Compared to manual alignment, this reduces the difficulty of aligning virtual light directions. This method is more suitable for situations where the virtual scene remains unchanged, but only a few virtual light sources change. By re-determining the target physical light source for the changed virtual light source, real-time alignment of virtual and real light can be ensured in the shooting process, with a natural transition, without interrupting the shooting.

[0066] In another application scenario, the virtual scene may switch from one scene to another, such as from indoors to outdoors, or from daytime to nighttime. The type, number, and direction of the light source may change. In this case, steps S11 to S14 can be re-executed for the switched virtual scene.

[0067] According to the method of this embodiment, by obtaining the lighting direction vectors of multiple physical light sources and the lighting direction vectors of directional virtual light sources in the virtual scene, a target physical light source with the same lighting direction as each virtual light source can be determined from multiple physical light sources. Then, by turning on the target physical light source, the alignment of the light and shadow effects of virtual and real lights in virtual shooting can be automatically achieved, which can achieve a better lighting direction alignment effect more efficiently and reduce the difficulty of virtual light direction alignment.

[0068] According to the method of this embodiment, it is equivalent to proposing a physical light source selection scheme based on spatial calibration results. By transforming the lighting direction vector of the virtual light source in the virtual scene, the number and position of the physical light source corresponding to the direction are matched, and then the selected target physical light source is turned on for lighting, which can better achieve the alignment of the light and shadow effects of virtual and real lights.

[0069] According to the method of this embodiment, it is equivalent to proposing a scheme for aligning the lighting direction of the virtual scene with the on-site lighting. When the virtual scene on the LED screen changes the virtual lighting under different shooting requirements, the number and position range of physical light sources that should correspond to the direction of the virtual light source in the virtual scene are determined according to the light emission direction of the virtual light source in the virtual scene. The lighting direction of the actual physical light source on-site is automatically controlled to align with it, which can better achieve the effect of lighting direction alignment and meet the requirements of aligning the virtual and real lighting effects in virtual shooting.

[0070] Figure 3 A block diagram of a virtual and real light direction alignment device according to an embodiment of the present disclosure is shown, such as Figure 3 As shown, the device includes:

[0071] The first acquisition module 301 is used to acquire physical light source information. The physical light source signal includes the lighting direction vectors of multiple physical light sources. The lighting direction vectors represent the lighting direction of the light source. The multiple physical light sources face the foreground area in front of the real screen.

[0072] The second acquisition module 302 is used to acquire virtual light source information, the virtual light source information including the lighting direction vector of at least one virtual light source with directionality in the virtual scene displayed on the real screen;

[0073] The determining module 303 is used to determine, based on the physical light source information and the virtual light source information, a target physical light source that has the same lighting direction as each of the at least one virtual light source from the plurality of physical light sources;

[0074] The control module 304 is used to illuminate the target physical light source so that the lighting effects in the virtual scene displayed on the real screen are aligned with the lighting effects in the foreground area in front of the real screen.

[0075] In one possible implementation, the physical light source signal includes the lighting direction vectors of the plurality of physical light sources in the screen coordinate system of the real screen; the virtual light source information includes the lighting direction vectors of at least one directional virtual light source in the virtual scene in the virtual world coordinate system of the virtual scene; wherein, determining the target physical light source with the same lighting direction as each of the at least one virtual light source from the plurality of physical light sources based on the physical light source information and the virtual light source information includes: transforming the physical light source information and the virtual light source information to the same coordinate system based on the transformation relationship between the virtual world coordinate system and the screen coordinate system to obtain target physical light source information and target virtual light source information located in the same coordinate system; and determining the target physical light source with the same lighting direction as each of the at least one virtual light source from the plurality of physical light sources based on the target physical light source information and the target virtual light source information.

[0076] In one possible implementation, determining the target physical light source with the same lighting direction as each of the at least one virtual light source from the plurality of physical light sources, based on the target physical light source information and the target virtual light source information, includes: for any virtual light source among the at least one virtual light source, calculating the angle between the lighting direction vector of the virtual light source and the lighting direction vector of each physical light source, based on the target physical light source information and the target virtual light source information; and determining the physical light source with an angle less than a specified degree as the target physical light source with the same lighting direction as the virtual light source.

[0077] In one possible implementation, determining the target physical light source from the plurality of physical light sources that has the same lighting direction as each of the at least one virtual light source includes: for any virtual light source, selecting the physical light source from the plurality of physical light sources whose lighting direction differs from the lighting direction of the virtual light source by less than a preset threshold as the target physical light source corresponding to that virtual light source, wherein the magnitude of the preset threshold is negatively correlated with the magnitude of the setting density of the plurality of physical light sources.

[0078] In one possible implementation, obtaining virtual light source information includes: obtaining the position and angle of at least one directional virtual light source in the virtual scene through Unreal Engine, and determining the lighting direction vector of the at least one virtual light source based on the position and angle of the at least one virtual light source.

[0079] In one possible implementation, the plurality of physical light sources include uniformly distributed physical light sources mounted above the real screen; the plurality of physical light sources are directed toward the central region of the foreground area in front of the real screen.

[0080] In one possible implementation, the device further includes a light source change module, configured to, when the lighting direction vector of any virtual light source in the virtual scene changes, determine a target physical light source from the plurality of physical light sources that has the same lighting direction as the virtual light source after the change, based on the changed lighting direction vector of the virtual light source and the physical light source information, so as to turn on the target physical light source that has the same lighting direction as the virtual light source after the change.

[0081] According to the apparatus of this embodiment, by acquiring the lighting direction vectors of multiple physical light sources and the lighting direction vectors of directional virtual light sources in the virtual scene, a target physical light source with the same lighting direction as each virtual light source can be determined from multiple physical light sources. Then, by turning on the target physical light source, the alignment of the light and shadow effects of virtual and real lights in virtual shooting can be automatically achieved, which can achieve a better lighting direction alignment effect more efficiently and reduce the difficulty of virtual light direction alignment.

[0082] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0083] This disclosure also proposes a computer-readable storage medium storing computer program instructions that, when executed by a processor, implement the above-described method. The computer-readable storage medium can be volatile or non-volatile.

[0084] This disclosure also proposes an electronic device, including: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0085] This disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the above-described method.

[0086] Figure 4 A block diagram of an electronic device 1900 according to an embodiment of the present disclosure is shown. For example, the electronic device 1900 may be provided as a server or a terminal device. (Refer to...) Figure 4The electronic device 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by memory 1932 for storing instructions, such as application programs, that can be executed by the processing component 1922. The application programs stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1922 is configured to execute instructions to perform the methods described above.

[0087] Electronic device 1900 may also include a power supply component 1926 configured to perform power management of electronic device 1900, a wired or wireless network interface 1950 configured to connect electronic device 1900 to a network, and an input / output interface 1958 (I / O interface). Electronic device 1900 can operate on an operating system, such as Windows Server, stored in memory 1932. TM Mac OS X TM Unix TM Linux TM FreeBSD TM Or similar.

[0088] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by a processing component 1922 of an electronic device 1900 to perform the above-described method.

[0089] This disclosure can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this disclosure.

[0090] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0091] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0092] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0093] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0094] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0095] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0096] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0097] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for aligning the directions of virtual and real lights, characterized in that, include: The physical light source information is obtained. The physical light source signal includes the lighting direction vectors of multiple physical light sources. The lighting direction vectors represent the lighting direction of the light source. The multiple physical light sources face the center area of ​​the foreground region in front of the real screen. The multiple physical light sources include uniformly or non-uniformly distributed physical light sources installed above the real screen. The position and orientation of the physical light sources are fixed and known. The lighting direction vector is visualized as a line connecting the position of the light source to the center area of ​​the foreground region. Obtain virtual light source information, which includes the lighting direction vector of at least one virtual light source with directionality in the virtual scene displayed on the real screen; Based on the physical light source information and the virtual light source information, a target physical light source with the same lighting direction as each of the at least one virtual light source is determined from the plurality of physical light sources; By turning on the target physical light source for illumination, the lighting effects in the virtual scene displayed on the real screen are aligned with the lighting effects in the foreground area in front of the real screen, wherein all physical light sources other than the target physical light source are in an off state.

2. The method according to claim 1, characterized in that, The physical light source signal includes the lighting direction vectors of the plurality of physical light sources in the screen coordinate system of the real screen; the virtual light source information includes the lighting direction vector of at least one directional virtual light source in the virtual scene in the virtual world coordinate system of the virtual scene. The step of determining, based on the physical light source information and the virtual light source information, a target physical light source with the same lighting direction as each of the at least one virtual light source from the plurality of physical light sources includes: Based on the transformation relationship between the virtual world coordinate system and the screen coordinate system, the physical light source information and the virtual light source information are transformed to the same coordinate system to obtain the target physical light source information and the target virtual light source information located in the same coordinate system. Based on the target physical light source information and the target virtual light source information, a target physical light source with the same lighting direction as each of the at least one virtual light source is determined from the plurality of physical light sources.

3. The method according to claim 2, characterized in that, The step of determining, based on the target physical light source information and the target virtual light source information, a target physical light source with the same lighting direction as each of the at least one virtual light source from the plurality of physical light sources includes: For any virtual light source among the at least one virtual light source, the angle between the lighting direction vector of the virtual light source and the lighting direction vector of each physical light source is calculated based on the target physical light source information and the target virtual light source information. Physical light sources with an included angle less than a specified degree are identified as target physical light sources with the same lighting direction as the virtual light source.

4. The method according to claim 2, characterized in that, The step of determining the target physical light source from the plurality of physical light sources that has the same lighting direction as each of the at least one virtual light source includes: For any virtual light source, among the plurality of physical light sources, the physical light source whose lighting direction differs from that of the virtual light source by less than a preset threshold is selected as the target physical light source corresponding to that virtual light source. The preset threshold value is negatively correlated with the density of the plurality of physical light sources.

5. The method according to claim 1, characterized in that, The acquisition of virtual light source information includes: The position and angle of at least one directional virtual light source in the virtual scene are obtained using Unreal Engine, and the lighting direction vector of the at least one virtual light source is determined based on the position and angle of the at least one virtual light source.

6. The method according to claim 1, characterized in that, The method further includes: When the lighting direction vector of any virtual light source in the virtual scene changes, a target physical light source with the same lighting direction as the virtual light source after the change and the physical light source information are determined from the plurality of physical light sources, so as to turn on the target physical light source with the same lighting direction as the virtual light source after the change.

7. A device for aligning the direction of virtual and real lights, characterized in that, include: The first acquisition module is used to acquire physical light source information. The physical light source signal includes the lighting direction vectors of multiple physical light sources. The lighting direction vectors represent the lighting direction of the light sources. The multiple physical light sources face the center area of ​​the foreground region in front of the real screen. The multiple physical light sources include uniformly or non-uniformly distributed physical light sources installed above the real screen. The positions and orientations of the physical light sources are fixed and known. The lighting direction vector is visualized as a line connecting the position of the light source to the center area of ​​the foreground region. The second acquisition module is used to acquire virtual light source information, which includes the lighting direction vector of at least one virtual light source with directionality in the virtual scene displayed on the real screen. The determining module is used to determine, based on the physical light source information and the virtual light source information, a target physical light source that has the same lighting direction as each of the at least one virtual light source from the plurality of physical light sources; The control module is used to illuminate the virtual scene displayed on the real screen by turning on the target physical light source, so that the lighting effects are aligned with the lighting effects of the foreground area in front of the real screen, wherein all physical light sources other than the target physical light source are off.

8. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to implement the method of any one of claims 1 to 6 when executing instructions stored in the memory.

9. A non-volatile computer-readable storage medium storing computer program instructions thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 6.

10. A computer program product comprising computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is executed in a processor of an electronic device, the processor in the electronic device performs the method described in any one of claims 1 to 6.

Images