Virtual and real light direction alignment method and device, electronic equipment, storage medium and program product

Abstract

The invention relates to a virtual and real light direction alignment method and device, electronic equipment, a storage medium and a program product, and the method comprises the steps: obtaining physical light source information, a physical light source signal comprising lighting direction vectors of a plurality of physical light sources; acquiring virtual light source information, wherein the virtual light source information comprises a lighting direction vector of at least one virtual light source with directivity in a virtual scene displayed on a real screen; according to the physical light source information and the virtual light source information, a target physical light source with the same lighting direction as each virtual light source in the at least one virtual light source is determined from the multiple physical light sources; and lighting by turning on the target physical light source, so that the light effect in the virtual scene displayed on the real screen is aligned with the light effect of the foreground area in front of the real screen. Therefore, alignment of light and shadow effects of virtual light and real light in virtual shooting can be automatically achieved, a good light direction alignment effect can be achieved more efficiently, and the difficulty of virtual light direction alignment is reduced.

Description

Technical Field

[0001] The present disclosure relates to the field of virtual shooting, and in particular to a method and device for aligning virtual and real lighting directions, an electronic device, a storage medium and a program product. Background Art

[0002] Virtual shooting based on LED screens means that when actors perform in front of an LED screen wall, a pre-made virtual scene will be played on the LED, the camera will shoot the actors and the LED screen, and the director's monitor will present in real time the visual effect of the real actor's performance integrated with the virtual scene (which previously had to be completed in post-production).

[0003] In virtual shooting projects based on LED screens, aligning the lights in the actual foreground shooting area of ​​the shooting scene with the lights in the virtual scene on the LED screen of the shooting scene is one of the important prerequisites for achieving a realistic and vivid shooting effect. In particular, when the lights in the virtual scene have a certain direction, the actual lights in the foreground need to illuminate in the same direction to maintain the consistency of the light and shadow effects of the virtual scene and the actual shooting scene.

[0004] However, the number of lights in the virtual scene required during the shooting process is large and the layout is complex, which makes it difficult to align the lighting direction of the physical lights on the actual shooting scene. It is difficult to set up real lights in the physical space that correspond to those in the virtual scene to achieve the consistency of light and shadow effects in the virtual and real environment. In addition, most of the virtual and real light alignment in virtual shooting currently adopts the method of manually adjusting the light direction, which is inefficient, difficult, and the effect is also relatively subjective. Summary of the invention

[0005] In view of this, the present disclosure proposes a method and device for aligning the directions of virtual and real lights, an electronic device, a storage medium and a program product, which can reduce the difficulty of aligning lights in real space and can more efficiently achieve better virtual and real light direction alignment effects.

[0006] According to one aspect of the present disclosure, a method for aligning virtual and real lighting directions is provided, including: obtaining physical light source information, the physical light source signal including lighting direction vectors of multiple physical light sources, the lighting direction vectors representing the lighting directions of the light sources, and the multiple physical light sources facing a foreground area in front of a real screen; obtaining virtual light source information, the virtual light source information including the lighting direction vector of at least one virtual light source with directionality in a virtual scene displayed on the real screen; determining, from the multiple physical light sources, a target physical light source having the same lighting direction as each virtual light source in the at least one virtual light source based on the physical light source information and the virtual light source information; and lighting by turning on the target physical light source so that the lighting effect in the virtual scene displayed on the real screen is aligned with the lighting effect of the foreground area in front of the real screen.

[0007] In one possible implementation, the physical light source signal includes the lighting direction vectors of the multiple physical light sources in the screen coordinate system of the real screen; the virtual light source information includes the lighting direction vector of at least one directional virtual light source in the virtual scene in the virtual world coordinate system of the virtual scene; wherein, determining, from the multiple physical light sources, a target physical light source having the same lighting direction as each virtual light source in the at least one virtual light source based on the physical light source information and the virtual light source information comprises: based on the conversion relationship between the virtual world coordinate system and the screen coordinate system, converting the physical light source information and the virtual light source information to the same coordinate system to obtain target physical light source information and target virtual light source information located in the same coordinate system; and determining, from the multiple physical light sources, a target physical light source having the same lighting direction as each virtual light source in the at least one virtual light source based on the target physical light source information and the target virtual light source information.

[0008] In one possible implementation, the target physical light source having the same lighting direction as that of each virtual light source in the at least one virtual light source is determined from the multiple physical light sources based on the target physical light source information and the target virtual light source information, including: for any virtual light source in the at least one virtual light source, respectively calculating the angle between the lighting direction vector of the virtual light source and the lighting direction vectors of each physical light source based on the target physical light source information and the target virtual light source information; and determining the physical light source having an angle less than a specified degree as the target physical light source having the same lighting direction as the virtual light source.

[0009] In one possible implementation, determining a target physical light source from the multiple physical light sources having the same lighting direction as that of each virtual light source in the at least one virtual light source includes: for any virtual light source, taking a physical light source among the multiple physical light sources, whose lighting direction differs from that of the virtual light source by less than a preset threshold, as the target physical light source corresponding to the virtual light source, wherein the size of the preset threshold is negatively correlated with the size of the setting density of the multiple physical light sources.

[0010] In one possible implementation, obtaining virtual light source information includes: obtaining the position and angle of at least one directional virtual light source in the virtual scene through the Unreal Engine, and determining the lighting direction vector of the at least one virtual light source based on the position and angle of the at least one virtual light source.

[0011] In a possible implementation, the multiple physical light sources include uniformly distributed physical light sources installed above the real screen; the multiple physical light sources are oriented toward a central area of ​​a foreground area in front of the real screen.

[0012] In one possible implementation, the method further includes: when the lighting direction vector of any virtual light source in the virtual scene changes, based on the lighting direction vector of the virtual light source after the change and the physical light source information, determining a target physical light source with the same lighting direction as the changed virtual light source from the multiple physical light sources, so as to turn on the target physical light source with the same lighting direction as the changed virtual light source.

[0013] According to another aspect of the present disclosure, a device for aligning virtual and real lighting directions is provided, including: a first acquisition module, used to acquire physical light source information, the physical light source signal includes lighting direction vectors of multiple physical light sources, the lighting direction vectors represent the lighting directions of the light sources, and the multiple physical light sources are directed toward a foreground area in front of a real screen; a second acquisition module, used to acquire virtual light source information, the virtual light source information includes the lighting direction vector of at least one virtual light source with directionality in a virtual scene displayed on the real screen; a determination module, used to determine, from the multiple physical light sources, a target physical light source having the same lighting direction as each virtual light source in the at least one virtual light source based on the physical light source information and the virtual light source information; and a control module, used to turn on the target physical light source for lighting, so that the lighting effect in the virtual scene displayed on the real screen is aligned with the lighting effect in the foreground area in front of the real screen.

[0014] According to another aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0015] According to another aspect of the present disclosure, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored, wherein the computer program instructions implement the above method when executed by a processor.

[0016] According to another aspect of the present disclosure, a computer program product is provided, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0017] According to various aspects of the present disclosure, by obtaining the lighting direction vectors of multiple physical light sources and the lighting direction vectors of directional virtual light sources in the virtual scene, a target physical light source with the same lighting direction as each virtual light source can be determined from the multiple physical light sources, and then by turning on the target physical light source, the alignment of light and shadow effects of virtual and real lights in virtual shooting can be automatically achieved, and better lighting direction alignment effects can be achieved more efficiently, reducing the difficulty of virtual light direction alignment.

[0018] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0020] Figure 1 A flowchart of a method for aligning virtual and real light directions according to an embodiment of the present disclosure is shown.

[0021] Figure 2 A schematic diagram of a shooting scene according to an embodiment of the present disclosure is shown.

[0022] Figure 3 A block diagram of a device for aligning the directions of virtual and real lights according to an embodiment of the present disclosure is shown.

[0023] Figure 4 A block diagram of an electronic device 1900 according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0024] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0025] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0026] The term "and / or" herein is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can represent any one or more elements selected from the set consisting of A, B, and C. In the description of the present disclosure, "plurality" means two or more, unless otherwise clearly and specifically defined.

[0027] It should be understood that the terms “include” and “comprising” used in the specification and claims of the present disclosure indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections thereof.

[0028] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0029] The virtual and real light direction alignment method of the embodiment of the present disclosure can be deployed on various terminal devices through software or hardware modification. The terminal device involved in the embodiment of the present disclosure may refer to a device with a wireless connection function and / or a wired connection function. The wireless connection function refers to the ability to connect to other devices through wireless connection methods such as wifi and Bluetooth. The terminal device involved in the embodiment of the present disclosure may also communicate with other devices through a wired connection function. The terminal device involved in the embodiment of the present disclosure may be a touch screen, a non-touch screen, or a screenless device. The touch screen device can be controlled by clicking and sliding on the display screen with a finger or a stylus. The non-touch screen device can be connected to an input device such as a mouse, a keyboard, a touch panel, and the terminal device is controlled through the input device. For example, the device without a screen may be a Bluetooth speaker without a screen. For example, the terminal device of the present application may include but is not limited to a user equipment (UE), a mobile device, a user terminal, a terminal, a handheld device, a tablet computer, a laptop, a PDA, a computing device, etc.

[0030] The virtual-real light direction alignment method of the embodiment of the present disclosure can also be deployed on a server. The server can be located in the cloud or locally. It can be a physical device or a virtual device, such as a virtual machine, a container, etc., and has a wireless communication function, wherein the wireless communication function can be set in the chip (system) or other parts or components of the server. It can refer to a device with a wireless connection function. The wireless connection function means that it can be connected to other servers or terminal devices through wireless connection methods such as Wi-Fi and Bluetooth. The server involved in the embodiment of the present disclosure can also have the function of communicating through a wired connection. For example, the server of the embodiment of the present disclosure can be located in the cloud, communicate with the terminal device, receive the physical light source information and virtual light source information sent by the terminal device, and use the virtual-real light direction alignment method deployed on the server to generate a target physical light source with the same lighting direction as the virtual light source in the virtual scene based on the physical light source information and the virtual light source information, and return it to the terminal device so that the target physical light source can be turned on for lighting through the terminal device control.

[0031] Figure 1 A flowchart of a method for aligning virtual and real light directions according to an embodiment of the present disclosure is shown. Figure 1 As shown, the method includes: step S11 to step S14.

[0032] In step S11, physical light source information is obtained, wherein the physical light source signal includes lighting direction vectors of multiple physical light sources, wherein the lighting direction vectors represent the lighting directions of the light sources, and the multiple physical light sources are oriented toward a foreground area in front of a real screen.

[0033] The physical light source may be a point light source, that is, a light source that can generate directionality. It should be understood that those skilled in the art can set the type, quantity, installation location, etc. of the physical light source according to actual needs, and the embodiments of the present disclosure are not limited to this. For example, the multiple physical light sources may include uniformly distributed physical light sources installed above the real screen. For example, Figure 2 A schematic diagram of a shooting scene is shown, such as Figure 2 As shown, 12 evenly distributed physical light sources (i.e., D1, D2, ..., D12) can be installed above the real screen (i.e., the arc-shaped LED screen) as the luminous light sources of the real hard light, wherein the physical light sources can be installed above the real screen in a circle, or at least 270° of point light sources, which is not limited in the embodiments of the present disclosure. According to actual needs, the physical light sources can also be installed at other positions except above the screen, and can be evenly or unevenly distributed, and the lighting direction can be set according to actual needs, which is not limited in the present application.

[0034] The real screen is also the screen used to display the virtual scene, for example, it can be Figure 2 The arc-shaped LED screen shown in FIG. 1 is a real screen; the foreground area in front of the real screen can be understood as the shooting area, that is, the area where the actors perform and the props are arranged, such as Figure 2 The foreground area before the LED screen in the middle; therefore, the physical light source should be directed toward the foreground area, that is, the lighting direction of the physical light source should be directed toward the foreground area. Considering that the area where actors perform and props are arranged is usually in the central area of ​​the foreground area (that is, the central area before the real screen), multiple physical light sources can be directed toward the central area of ​​the foreground area before the real screen to meet the lighting requirements in the real shooting location, so that the foreground area can obtain hard light from any direction in multiple surrounding directions (such as 360 degrees).

[0035] It should be understood that based on the known position and orientation of the physical light source in the physical space of the shooting location, for example, the lighting direction vector of each physical light source in the screen coordinate system can be obtained by establishing a screen coordinate system of the real screen (for example, a coordinate system established with a vertex or center point of the real screen as the origin). Alternatively, the lighting direction vector of each physical light source in the world coordinate system can be obtained by establishing a world coordinate system of the shooting location, wherein the lighting direction vector can be visualized as a line from the position of the physical light source to the center area of ​​the foreground area, which is equivalent to the light direction emitted by the physical light source.

[0036] In step S12, virtual light source information is acquired, where the virtual light source information includes a lighting direction vector of at least one virtual light source having directionality in a virtual scene displayed on a real screen.

[0037] Among them, the virtual scene can be called a virtual asset. It should be understood that those skilled in the art can design the virtual scene currently displayed on the real screen according to the needs of the background picture required for the actual shooting. At least one directional virtual light source can be set in the virtual scene, for example, a virtual light source for simulating the sunlight rising from the east, and a virtual light source for simulating indoor lighting and candlelight. These virtual light sources are directional and will cause other objects in the virtual scene to produce light and shadow effects.

[0038] Wherein, obtaining virtual light source information may include: obtaining the position and angle of at least one virtual light source with directionality in the virtual scene through the Unreal Engine, and determining the lighting direction vector of at least one virtual light source based on the position and angle of at least one virtual light source. It should be understood that the Unreal Engine is used for virtual scene construction and visual effects production, so the virtual scene can be built through the Unreal Engine, and thus the position (that is, the position of the virtual light source in the virtual scene) and angle (that is, the direction of the virtual light source in the virtual scene) of at least one virtual light source with directionality in the virtual scene can be obtained through the relevant plug-ins of the Unreal Engine, and then the lighting direction vector of the virtual light source in the virtual world coordinate system of the virtual scene can be determined based on the position and angle of the virtual light source, and the lighting direction vector of the virtual light source can represent the lighting direction of the virtual light source, wherein the virtual world coordinate system can be understood as the world coordinate system of the Unreal Engine (UE). Of course, other methods can also be used to obtain virtual light source information, which can depend on the technology used to build the virtual scene, and the embodiments of the present disclosure are not limited to this.

[0039] It should be understood that at least one virtual light source in the virtual light source information may be all virtual light sources in the virtual scene, that is, the target physical light source may be searched for all virtual light sources, and the user may also select a part of the virtual light sources as needed to determine the corresponding target physical light source.

[0040] In step S13, a target physical light source having the same lighting direction as each virtual light source in at least one virtual light source is determined from a plurality of physical light sources according to the physical light source information and the virtual light source information.

[0041] In one possible implementation, determining a target physical light source from a plurality of physical light sources having the same lighting direction as that of each virtual light source in at least one virtual light source may include: for any virtual light source, taking a physical light source among the plurality of physical light sources, whose lighting direction differs from that of the virtual light source by less than a preset threshold, as the target physical light source corresponding to the virtual light source, wherein the size of the preset threshold is negatively correlated with the size of the setting density of the plurality of physical light sources.

[0042] Among them, the difference between the lighting direction of the physical light source and the lighting direction of the virtual light source can be quantified as the angle, distance, etc. between the lighting direction vector of the physical light source and the lighting direction vector of the virtual light source, for example. That is, the physical light source among multiple physical light sources, whose angle between the lighting direction vector and the lighting direction vector of the virtual light source is less than a preset threshold, can be determined as the corresponding target physical light source; and the physical light source among multiple physical light sources, whose distance between the lighting direction vector and the lighting direction vector of the virtual light source is less than a preset threshold, can be determined as the corresponding target physical light source. The embodiments of the present disclosure are not limited to this.

[0043] The negative correlation between the preset threshold and the density of the multiple physical light sources can be understood as the denser the physical light sources are (i.e., the closer the physical light sources are), the closer the lighting directions of the adjacent physical light sources are, and the smaller the preset threshold can be. Accordingly, the alignment effect is more accurate.

[0044] As described above, the physical light source signal includes the lighting direction vectors of multiple physical light sources, and the virtual light source information includes the lighting direction vector of at least one virtual light source with directionality in the virtual scene displayed on the real screen. Therefore, according to the physical light source information and the virtual light source information, determining the target physical light source with the same lighting direction as each virtual light source in at least one virtual light source from the multiple physical light sources may include: for any virtual light source, according to the physical light source information and the virtual light source information, respectively calculating the angle between the lighting direction vector of the virtual light source and the lighting direction vector of each physical light source, and determining the physical light source with an angle less than a specified degree as the target physical light source with the same lighting direction as the virtual light source; wherein the specified degree can be set by user, for example, it can be set to 10°, and the embodiment of the present disclosure does not limit this. The specified degree can be negatively correlated with the setting density of the physical light source, that is, the denser the physical light sources are set (that is, the closer the physical light sources are), the closer the lighting directions of adjacent physical light sources are, and the smaller the specified degree can be.

[0045] For example, the lighting direction vector of a virtual light source can be calculated Figure 2 The angle between each of the 12 physical light sources (D1, D2, ..., D12) in the virtual light source is: if the angle between the physical light source D2 and the lighting direction vector of the virtual light source is less than a specified degree, the physical light source D2 can be determined as a target physical light source with the same lighting direction as the virtual light source; if the angles between the physical light sources D2 and D3 and the lighting direction vector of the virtual light source are less than a specified angle, the physical light sources D2 and D3 can be determined as target physical light sources with the same lighting direction as the virtual light source.

[0046] The above method can be understood as that the physical light source information and the virtual light source information are in the same coordinate system, but considering that the physical light source information and the virtual light source information are usually established in different coordinate systems in actual situations, for example, the physical light source signal includes the lighting direction vectors of multiple physical light sources in the screen coordinate system of the real screen, and the virtual light source information includes the lighting direction vector of at least one directional virtual light source in the virtual scene in the virtual world coordinate system of the virtual scene. Therefore, in order to more accurately determine the physical light source with the same lighting direction, in a possible implementation method, the above step S13, according to the physical light source information and the virtual light source information, determines from multiple physical light sources a target physical light source with the same lighting direction as each virtual light source in at least one virtual light source, including:

[0047] Step S131, based on the conversion relationship between the virtual world coordinate system and the screen coordinate system, convert the physical light source information and the virtual light source information to the same coordinate system to obtain the target physical light source information and the target virtual light source information in the same coordinate system;

[0048] Step S132: determining, from a plurality of physical light sources, a target physical light source having the same lighting direction as each virtual light source in at least one virtual light source according to the target physical light source information and the target virtual light source information.

[0049] Among them, the conversion relationship between the virtual world coordinate system and the screen coordinate system can be obtained in advance through the space calibration technology, and the conversion relationship can be expressed as a conversion matrix, wherein the space calibration technology is in the virtual shooting based on the LED screen, and the relative transformation relationship between the LED screen coordinate system, the world coordinate system and the camera coordinate system is obtained through a certain calculation process, that is, in the field of virtual shooting, the relative position relationship between the real camera and the real screen can be obtained through the space calibration technology, and the position and posture of the virtual camera in the virtual world coordinate system of the virtual scene is known, and the position and posture of the real camera and the virtual camera are consistent, thereby the conversion relationship between the screen coordinate system of the real screen and the virtual world coordinate system can be obtained. The disclosed embodiment does not limit the determination process of the above-mentioned conversion relationship.

[0050] In step S131, based on the conversion relationship between the virtual world coordinate system and the screen coordinate system, the physical light source information and the virtual light source information are converted to the same coordinate system to obtain the target physical light source information and the target virtual light source information in the same coordinate system, including:

[0051] Based on the conversion relationship between the virtual world coordinate system and the screen coordinate system, the virtual light source information is converted to the screen coordinate system to obtain the target physical light source information and the target virtual light source information in the same coordinate system; or,

[0052] Based on the conversion relationship between the virtual world coordinate system and the screen coordinate system, the physical light source information is converted to the virtual world coordinate system to obtain the target physical light source information and the target virtual light source information in the same coordinate system.

[0053] That is, the lighting direction vector of the virtual light source in the virtual world coordinate system can be converted to the screen coordinate system, or the lighting direction vector of the physical light source in the screen coordinate system can be converted to the virtual world coordinate system to obtain the target physical light source information and the target virtual light source information in the same coordinate system.

[0054] Then, after conversion to the same coordinate system, for example, the angle between the lighting direction vector of each virtual light source and the lighting direction vector of each physical light source can be calculated, and when the angle is less than a specified degree, the lighting direction of the physical light source is deemed to be consistent with the virtual light source. That is, the above step S132, determining the target physical light source having the same lighting direction as each virtual light source in at least one virtual light source from multiple physical light sources according to the target physical light source information and the target virtual light source information, may include:

[0055] For any virtual light source of the at least one virtual light source, respectively calculate the angle between the lighting direction vector of the virtual light source and the lighting direction vector of each physical light source according to the target physical light source information and the target virtual light source information;

[0056] A physical light source whose included angle is less than a specified degree is determined as a target physical light source having the same lighting direction as the virtual light source.

[0057] As described above, the specified degree can be customized, and the embodiments of the present disclosure do not limit this. In addition, each virtual light source can determine at least one target physical light source with the same lighting direction.

[0058] It should be understood that determining the target physical light source having the same lighting direction as the virtual light source by calculating the angle between the two lighting direction vectors is a possible implementation method provided by the embodiment of the present disclosure.

[0059] In fact, those skilled in the art can also use similarity calculation methods known in the art to determine the target physical light source with the same lighting direction as the virtual light source. For example, by calculating the cosine similarity between the lighting direction vector of the virtual light source and the lighting direction vectors of each physical light source, the physical light source with a cosine similarity greater than a specified threshold can be determined as the target physical light source with the same lighting direction as the virtual light source. The embodiments of the present disclosure are not limited to this.

[0060] It should be understood that for each virtual light source in the virtual scene, at least one target physical light source can be determined that is consistent with the lighting direction of each virtual light source. These target physical light sources can then be used to illuminate the real foreground area in the same direction as in the virtual scene, thereby achieving consistency in the effects of virtual and real light sources, that is, alignment of virtual lighting effects.

[0061] In step S14, the target physical light source is turned on for lighting so that the lighting effect in the virtual scene displayed on the real screen is aligned with the lighting effect in the foreground area in front of the real screen.

[0062] Among them, turning on the target physical light source for lighting means turning on the physical light source with the same lighting direction as each virtual light source in the virtual scene. Turning on the target physical light source for lighting means controlling the target physical light source to be in a bright state (that is, a powered-on state), and controlling other physical light sources except the target physical light source to be in an off state (that is, a powered-off state). It should be understood that those skilled in the art can use a lighting controller known in the art to realize the control of the on and off of each physical light source, and the embodiments of the present disclosure are not limited to this. Since the lighting direction of the target physical light source is the same as that of the virtual light source, by turning on the target physical light source for lighting, the lighting effect in the virtual scene displayed on the real screen can be aligned with the lighting effect of the foreground area in front of the real screen.

[0063] In practical applications, the lighting direction of the virtual light source in the virtual scene may change during the shooting process. For example, the virtual sun (i.e., the virtual light source) in the virtual scene changes from the east to the west within a few minutes. In this case, the target physical light source to be aligned also changes accordingly. Therefore, in a possible implementation manner, the method may further include:

[0064] When the lighting direction vector of any virtual light source in the virtual scene changes, a target physical light source with the same lighting direction as the changed virtual light source is determined from multiple physical light sources based on the lighting direction vector of the changed virtual light source and the physical light source information, so as to turn on the target physical light source with the same lighting direction as the changed virtual light source.

[0065] Among them, the specific implementation method of the above-mentioned step S13 can be referred to to realize the determination of the target physical light source with the same lighting direction as the virtual light source after the change from multiple physical light sources. For example, the target physical light source with the same lighting direction as the virtual light source after the change can be determined according to the angle between the lighting direction vector of the virtual light source after the change and the lighting direction vector of each physical light source; then the target physical light source with the same lighting direction as the virtual light source after the change can be turned on, and the target physical light source with the same lighting direction as the virtual light source before the change can be turned off, so that the opening and closing of each physical light source set in the actual shooting site can automatically change with the change of the lighting direction vector of the virtual light source, meet the requirements of virtual and real light direction alignment, and reduce the difficulty of virtual light direction alignment compared to manual alignment. This method is more suitable for the situation where the virtual scene has not changed, and only some of the virtual light sources have changed. By re-determining the target physical light source for the changed virtual light source, it can be ensured that the virtual and real lights in the shooting procedure are aligned in real time and transition naturally without interrupting the shooting.

[0066] In another application scenario, the virtual scene may switch from one scene to another, such as from indoors to outdoors, from daytime to nighttime, and the type, quantity, and lighting direction of the light source may change. In this case, steps S11 to S14 may be re-executed for the switched virtual scene.

[0067] According to the method of the present embodiment, by obtaining the lighting direction vectors of multiple physical light sources and the lighting direction vectors of directional virtual light sources in the virtual scene, a target physical light source with the same lighting direction as each virtual light source can be determined from the multiple physical light sources, and then by turning on the target physical light source, the alignment of the light and shadow effects of the virtual and real lights in the virtual shooting can be automatically achieved, and a better lighting direction alignment effect can be achieved more efficiently, reducing the difficulty of virtual light direction alignment.

[0068] According to the method of the present embodiment, it is equivalent to proposing a physical light source selection scheme based on the spatial calibration results. By transforming the lighting direction vector of the virtual light source in the virtual scene, matching the number and position of the physical light sources corresponding to that direction, and then turning on the selected target physical light source for lighting, the alignment of the light and shadow effects of the virtual and real lights can be better achieved.

[0069] According to the method of the present embodiment, it is equivalent to proposing a solution for aligning the on-site lighting with the lighting direction in the virtual scene. When the virtual scene on the LED screen undergoes virtual lighting changes under different shooting requirements, the number and position range of physical light sources that should correspond to the direction of the virtual light source in the virtual scene are determined, and the lighting direction of the actual physical light source on the scene is automatically controlled to be aligned with it, which can better achieve the effect of aligning the lighting direction and meet the requirements of aligning the virtual and real lighting effects in virtual shooting.

[0070] Figure 3 A block diagram of a virtual and real light direction alignment device according to an embodiment of the present disclosure is shown as follows: Figure 3 As shown, the device comprises:

[0071] A first acquisition module 301 is used to acquire physical light source information, wherein the physical light source signal includes lighting direction vectors of multiple physical light sources, wherein the lighting direction vectors represent the lighting directions of the light sources, and the multiple physical light sources are oriented toward a foreground area in front of a real screen;

[0072] A second acquisition module 302 is used to acquire virtual light source information, where the virtual light source information includes a lighting direction vector of at least one virtual light source having directionality in the virtual scene displayed on the real screen;

[0073] A determination module 303 is used to determine, from the multiple physical light sources, a target physical light source having the same lighting direction as each virtual light source in the at least one virtual light source according to the physical light source information and the virtual light source information;

[0074] The control module 304 is used to turn on the target physical light source for lighting, so that the lighting effect in the virtual scene displayed on the real screen is aligned with the lighting effect in the foreground area in front of the real screen.

[0075] In one possible implementation, the physical light source signal includes the lighting direction vectors of the multiple physical light sources in the screen coordinate system of the real screen; the virtual light source information includes the lighting direction vector of at least one directional virtual light source in the virtual scene in the virtual world coordinate system of the virtual scene; wherein, determining, from the multiple physical light sources, a target physical light source having the same lighting direction as each virtual light source in the at least one virtual light source based on the physical light source information and the virtual light source information comprises: based on the conversion relationship between the virtual world coordinate system and the screen coordinate system, converting the physical light source information and the virtual light source information to the same coordinate system to obtain target physical light source information and target virtual light source information located in the same coordinate system; and determining, from the multiple physical light sources, a target physical light source having the same lighting direction as each virtual light source in the at least one virtual light source based on the target physical light source information and the target virtual light source information.

[0076] In one possible implementation, the target physical light source having the same lighting direction as that of each virtual light source in the at least one virtual light source is determined from the multiple physical light sources based on the target physical light source information and the target virtual light source information, including: for any virtual light source in the at least one virtual light source, respectively calculating the angle between the lighting direction vector of the virtual light source and the lighting direction vectors of each physical light source based on the target physical light source information and the target virtual light source information; and determining the physical light source having an angle less than a specified degree as the target physical light source having the same lighting direction as the virtual light source.

[0077] In one possible implementation, determining a target physical light source from the multiple physical light sources having the same lighting direction as that of each virtual light source in the at least one virtual light source includes: for any virtual light source, taking a physical light source among the multiple physical light sources, whose lighting direction differs from that of the virtual light source by less than a preset threshold, as the target physical light source corresponding to the virtual light source, wherein the size of the preset threshold is negatively correlated with the size of the setting density of the multiple physical light sources.

[0078] In one possible implementation, obtaining virtual light source information includes: obtaining the position and angle of at least one directional virtual light source in the virtual scene through the Unreal Engine, and determining the lighting direction vector of the at least one virtual light source based on the position and angle of the at least one virtual light source.

[0079] In a possible implementation, the multiple physical light sources include uniformly distributed physical light sources installed above the real screen; the multiple physical light sources are oriented toward a central area of ​​a foreground area in front of the real screen.

[0080] In one possible implementation, the device also includes: a light source change module, which is used to determine, when the lighting direction vector of any virtual light source in the virtual scene changes, a target physical light source with the same lighting direction as the changed virtual light source from the multiple physical light sources based on the changed lighting direction vector of the virtual light source and the physical light source information, so as to turn on the target physical light source with the same lighting direction as the changed virtual light source.

[0081] According to the device of the present embodiment, by acquiring the lighting direction vectors of multiple physical light sources and the lighting direction vectors of directional virtual light sources in the virtual scene, a target physical light source with the same lighting direction as each virtual light source can be determined from the multiple physical light sources, and then by turning on the target physical light source, the alignment of the light and shadow effects of the virtual and real lights in the virtual shooting can be automatically achieved, which can more efficiently achieve a better lighting direction alignment effect and reduce the difficulty of virtual lighting direction alignment.

[0082] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0083] The embodiment of the present disclosure also provides a computer-readable storage medium on which computer program instructions are stored, and the computer program instructions implement the above method when executed by a processor. The computer-readable storage medium can be a volatile or non-volatile computer-readable storage medium.

[0084] An embodiment of the present disclosure further proposes an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0085] The embodiments of the present disclosure also provide a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0086] Figure 4 1 is a block diagram of an electronic device 1900 according to an embodiment of the present disclosure. For example, the electronic device 1900 may be provided as a server or a terminal device. Figure 4, the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0087] The electronic device 1900 may also include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output interface 1958 (I / O interface). The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server 2003. TM , Mac OS X TM , Unix TM ,Linux TM , FreeBSD TM or similar.

[0088] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the electronic device 1900 to perform the above method.

[0089] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0090] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0091] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0092] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0093] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0094] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0095] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0096] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0097] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for aligning virtual and real light directions, characterized in that: include: Acquire physical light source information, where the physical light source signal includes lighting direction vectors of multiple physical light sources, where the lighting direction vectors represent lighting directions of the light sources, and the multiple physical light sources are oriented toward a foreground area in front of a real screen; Acquire virtual light source information, wherein the virtual light source information includes a lighting direction vector of at least one virtual light source having directionality in the virtual scene displayed on the real screen; According to the physical light source information and the virtual light source information, determining, from the multiple physical light sources, a target physical light source having the same lighting direction as each virtual light source in the at least one virtual light source; The target physical light source is turned on for lighting so that the lighting effect in the virtual scene displayed on the real screen is aligned with the lighting effect in the foreground area in front of the real screen.

2. The method according to claim 1, characterized in that The physical light source signal includes lighting direction vectors of the multiple physical light sources in the screen coordinate system of the real screen; the virtual light source information includes the lighting direction vector of at least one directional virtual light source in the virtual scene in the virtual world coordinate system of the virtual scene; The step of determining, from the plurality of physical light sources, a target physical light source having the same lighting direction as each virtual light source in the at least one virtual light source according to the physical light source information and the virtual light source information comprises: Based on the conversion relationship between the virtual world coordinate system and the screen coordinate system, the physical light source information and the virtual light source information are converted to the same coordinate system to obtain the target physical light source information and the target virtual light source information in the same coordinate system; According to the target physical light source information and the target virtual light source information, a target physical light source having the same lighting direction as that of each virtual light source in the at least one virtual light source is determined from the multiple physical light sources.

3. The method according to claim 2, characterized in that The step of determining, from the plurality of physical light sources, a target physical light source having the same lighting direction as each virtual light source in the at least one virtual light source according to the target physical light source information and the target virtual light source information, comprises: For any virtual light source among the at least one virtual light source, respectively calculate the angle between the lighting direction vector of the virtual light source and the lighting direction vector of each physical light source according to the target physical light source information and the target virtual light source information; A physical light source whose included angle is smaller than a specified degree is determined as a target physical light source having the same lighting direction as the virtual light source.

4. The method according to claim 2, characterized in that: The step of determining, from the plurality of physical light sources, a target physical light source having the same lighting direction as each virtual light source in the at least one virtual light source comprises: For any virtual light source, a physical light source among the multiple physical light sources, whose lighting direction differs from that of the virtual light source by less than a preset threshold, is used as a target physical light source corresponding to the virtual light source. The size of the preset threshold is negatively correlated with the size of the setting density of the multiple physical light sources.

5. The method according to claim 1, characterized in that The obtaining of virtual light source information includes: The position and angle of at least one directional virtual light source in the virtual scene are acquired through the Unreal Engine, and based on the position and angle of the at least one virtual light source, a lighting direction vector of the at least one virtual light source is determined.

6. The method according to any one of claims 1 to 5, characterized in that: The plurality of physical light sources include uniformly distributed physical light sources installed above the real screen; The multiple physical light sources are oriented toward a central area of ​​a foreground area in front of the real screen.

7. The method according to claim 1, characterized in that The method further comprises: When the lighting direction vector of any virtual light source in the virtual scene changes, a target physical light source having the same lighting direction as the changed virtual light source is determined from the multiple physical light sources based on the lighting direction vector of the changed virtual light source and the physical light source information, so as to turn on the target physical light source having the same lighting direction as the changed virtual light source.

8. A virtual and real light direction alignment device, characterized in that: include: A first acquisition module is used to acquire physical light source information, wherein the physical light source signal includes lighting direction vectors of multiple physical light sources, wherein the lighting direction vectors represent the lighting directions of the light sources, and the multiple physical light sources are oriented toward a foreground area in front of a real screen; A second acquisition module is used to acquire virtual light source information, wherein the virtual light source information includes a lighting direction vector of at least one virtual light source having directionality in the virtual scene displayed on the real screen; A determination module, configured to determine, from the plurality of physical light sources, a target physical light source having the same lighting direction as each virtual light source in the at least one virtual light source according to the physical light source information and the virtual light source information; The control module is used to turn on the target physical light source for lighting so that the lighting effect in the virtual scene displayed on the real screen is aligned with the lighting effect in the foreground area in front of the real screen.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to implement the method described in any one of claims 1 to 7 when executing the instructions stored in the memory.

10. A non-volatile computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

11. A computer program product, comprising a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code, wherein when the computer-readable code is executed in a processor of an electronic device, the processor in the electronic device executes the method described in any one of claims 1 to 7.

Images