Light virtual and real alignment method, system and device, program product and storage medium

Abstract

The invention provides a lamplight virtual and real alignment method, system and device, a program product and a storage medium. A virtual shooting system comprises a display screen, a lamp cluster, a lamp controller, a main control computer for deploying a main control end and a shooting device. The method comprises the following steps: displaying a shooting picture shot by shooting equipment in a user interface; the shooting picture comprises an actual scene picture shot by the shooting equipment and a virtual scene picture displayed by the display screen; after it is detected that a user selects a to-be-aligned range from the shot picture, the master control end is informed, so that under the condition that initial color values between two specified ranges in the to-be-aligned range are not matched, the master control end controls the lamp cluster through the lamp controller to carry out light adjustment, and the to-be-aligned range is aligned to the to-be-aligned range. The adjusted color values between the two specified ranges are matched.

Description

Technical Field

[0001] The present invention relates to the field of virtual photography technology, and in particular to a method, system, device, program product and storage medium for aligning virtual and real lighting. Background Art

[0002] The main principle of virtual filming is to digitize the background and render it in real time onto a set of large display screens. These display screens are arranged around the actual performance area, where physical props can be arranged, actors can perform in the filming area, and the display screens can adjust the display content in real time as needed to present a realistic virtual background that blends with the actual performance area.

[0003] Usually, a certain number of physical lamps are arranged in the actual scene, so the lights of the virtual and real scenes need to be aligned, that is, the lights in the actual scene need to be consistent with the lights in the virtual scene displayed on the display screen to ensure that the shooting image of the shooting device does not have a fragmented visual effect. At present, the method of aligning the virtual and real lights requires the lighting technician to make repeated adjustments, which is time-consuming and laborious, and is inefficient. Summary of the invention

[0004] To overcome the problems existing in the related art, the embodiments of this specification provide a method, system, device, program product and storage medium for aligning virtual and real lights.

[0005] According to a first aspect of an embodiment of this specification, a method for aligning virtual and real lights in a virtual shooting system is provided, wherein the virtual shooting system includes a display screen, a lighting cluster, a lighting controller, a main control machine for deploying a main control terminal, and a shooting device;

[0006] The main control machine can communicate with the lamp controller, and the lamp controller is connected to the lamp cluster;

[0007] The method is applied to an operation client deployed in a mobile device, and the operation client can communicate with the master control end; the method comprises:

[0008] Displaying a captured image captured by the camera in a user interface; the captured image includes an actual scene image captured by the camera and a virtual scene image displayed on the display screen;

[0009] After detecting that the user selects a range to be aligned from the captured image, the main control end is notified so that when the initial color values ​​between the two specified ranges in the range to be aligned do not match, the main control end controls the lamp cluster through the lamp controller to adjust the lights so that the adjusted color values ​​between the two specified ranges match; wherein the two specified ranges include: an actual specified range belonging to the actual scene image and a virtual specified range belonging to the virtual scene image.

[0010] According to a second aspect of an embodiment of this specification, a method for aligning virtual and real lights in a virtual shooting system is provided, wherein the virtual shooting system includes a display screen, a lighting cluster, a lighting controller, a main control machine for deploying a main control terminal, and a shooting device;

[0011] The main control machine can communicate with the lamp controller, and the lamp controller is connected to the lamp cluster;

[0012] The method is applied to the master control terminal, and the master control terminal can communicate with the operation client deployed in the mobile device; the method includes:

[0013] Receive the range to be aligned in the captured picture sent by the operating client; the captured picture includes: the actual scene picture captured by the capturing device and the virtual scene picture displayed by the display screen; the range to be aligned includes the following two specified ranges: an actual specified range belonging to the actual scene picture and a virtual specified range belonging to the virtual scene picture; the initial color values ​​between the two specified ranges do not match;

[0014] Based on the range to be aligned, the lamp cluster is controlled by the lamp controller to adjust the lighting so that the adjusted color values ​​between the two specified ranges match.

[0015] According to a third aspect of the embodiments of this specification, a virtual shooting system is provided, the virtual shooting system comprising a display screen, a lighting cluster, a lighting controller, a main control machine on which a main control terminal is deployed, a shooting device, and an operation client deployed in a mobile device; the main control machine can communicate with the lighting controller, and the lighting controller is connected to the lighting cluster; the main control terminal and the operation client deployed in the mobile device can communicate with each other;

[0016] The operation client is used to execute the steps of the method described in the first aspect; the main control end is used to execute the steps of the method described in the second aspect.

[0017] According to a fourth aspect of the embodiments of this specification, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the method embodiments described in the first or second aspect are implemented.

[0018] According to a fifth aspect of the embodiments of this specification, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method embodiments described in the first or second aspect are implemented.

[0019] According to a sixth aspect of the embodiments of this specification, a computer program product is provided, including a computer program, which, when executed by a processor, implements the steps of the method embodiments described in the first aspect or the second aspect.

[0020] The technical solutions provided by the embodiments of this specification may have the following beneficial effects:

[0021] In the embodiment of the specification, since the embodiment is designed to deploy an operation client in a mobile device, the operation client provides the user with a function of aligning the virtual and real lights. In this way, the user can hold the mobile device, can conveniently adjust the shooting device, can timely view the shooting picture taken by the shooting device displayed in the user interface next to the shooting device, and can check whether there is a range of unaligned virtual and real lights through the shooting picture. The operation client can detect the range to be aligned selected by the user in the shooting picture and notify the main control end. Due to the link design in this embodiment in which the main control machine communicates with the lamp controller and the lamp controller is connected to the lamp cluster, the main control end can control the lamp cluster to adjust the lights through the lamp controller when the initial color values ​​between the two specified ranges in the range to be aligned do not match, and the adjusted color values ​​between the two specified ranges after the lights are adjusted match. In this way, the user can conveniently view the shooting picture with the mobile device, and conveniently specify the range to be aligned through the operation client. Through the above-mentioned link design, the main control end can help the user automatically adjust the lights, significantly reduce user operations, and improve the efficiency of virtual and real light alignment.

[0022] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present specification. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1A is a schematic diagram of a virtual shooting scene according to an exemplary embodiment of the present specification.

[0024] Figure 1B It is a schematic diagram of a lighting control console according to an exemplary embodiment of the present specification.

[0025] Figure 2A is a schematic diagram of a virtual shooting scene according to an exemplary embodiment of the present specification.

[0026] Figure 2B This is a flowchart of a method for aligning virtual and real lights in a virtual shooting system according to an exemplary embodiment of the present specification.

[0027] Figure 2C It is a schematic diagram of a user interface shown in this specification according to an exemplary embodiment.

[0028] Figure 2D It is a schematic diagram of a shooting screen according to an exemplary embodiment of the present specification.

[0029] Figure 3 This is a flowchart of another method for aligning virtual and real lighting in a virtual shooting system according to an exemplary embodiment of the present specification.

[0030] Figure 4 It is a hardware structure diagram of a computer device where a lighting virtual-real alignment device in a virtual shooting system is located according to an exemplary embodiment of the present specification.

[0031] Figure 5 It is a block diagram of a device for aligning virtual and real lighting in a virtual shooting system according to an exemplary embodiment of the present specification.

[0032] Figure 6 It is a block diagram of another device for aligning virtual and real lighting in a virtual shooting system according to an exemplary embodiment of the present specification. DETAILED DESCRIPTION

[0033] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with this specification. Instead, they are merely examples of devices and methods consistent with some aspects of this specification as detailed in the appended claims.

[0034] The terms used in this specification are for the purpose of describing specific embodiments only and are not intended to limit this specification. The singular forms "a", "the" and "the" used in this specification and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0035] It should be understood that although the terms first, second, third, etc. may be used in this specification to describe various information, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of this specification, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0036] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this manual are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0037] Virtual filming is a virtual filming technology that uses display screens to build virtual backgrounds. It combines real-time rendering and LED (Light Emitting Diode) display technology to present a realistic virtual environment in real time on site, replacing traditional green screen or blue screen filming. The main principle of virtual filming is to digitize the background and render it in real time onto a set of large display screens. These display screens are arranged around the actual performance area, where physical props can be arranged, actors can perform in the filming area, and the display screens can adjust the display content in real time as needed to present a realistic virtual background that blends with the actual performance area.

[0038] like Figure 1A As shown, it is a schematic diagram of a virtual shooting scene shown in this specification according to an exemplary embodiment. The virtual shooting scene may include a virtual shooting system composed of one or more computer devices. As an example, the virtual shooting system may include a combination of one or more of the following devices: one or more main control machines 011, one or more rendering devices 021 (also called screen machines), one or more broadcast control processing devices 031, one or more display screens 040 (three display screens are shown in the figure: display screen 041, display screen 042 and display screen 043), and one or more shooting devices 051; wherein, the number of various types of equipment can be flexibly configured according to actual needs, and this embodiment does not limit this. In actual applications, the virtual shooting system may also include other devices, such as mobile terminals or network devices, etc., as needed, and this embodiment does not limit this.

[0039] Optionally, each host computer 011 may be connected to one or more rendering devices 021, and the specific connection method may be selected according to actual needs and device compatibility. As an example, a wired or wireless connection may be made through a local area network or the Internet, and a network transmission protocol may be used for communication. As an example, the host computer may send a variety of control instructions to the rendering device connected thereto, such as control instructions containing specific image information, etc.

[0040] Optionally, each rendering device 021 can be connected to one or more broadcast control processing devices 031; the specific connection method can be selected according to actual needs and device compatibility. As an example, it can include DP (DisplayPort, a digital display interface standard) connection, DP can be used to transmit high-quality audio and video signals. It can also be HDMI (High-Definition Multimedia Interface, a high-definition digital audio and video interface standard), HDMI can combine audio, video and control signals on a single cable for transmission. As an example, the rendering device 021 can send a variety of control instructions to the broadcast control processing device connected to it. For example, the rendering device can be used as an image signal source to send control instructions containing rendered images, and so on.

[0041] In actual applications, the broadcast control processing device 031 is optional and may not be configured in some scenarios. Optionally, each broadcast control processing device 031 can be connected to one or more display screens 040; the specific connection method can be selected according to actual needs and device compatibility. As an example, it can also include DP or HDMI connection, USB (Universal Serial Bus) or network connection, etc. The broadcast control processing device can be used to control and manage the display screen connected to it. As an example, the broadcast control processing device 031 can be used for data transmission and decoding, such as the broadcast control processing device 031 can receive signals from external sources (such as screen machines, computers, mobile terminals or media players, etc.) and decode them into a format suitable for display on the display screen; it can also be used for display control, such as overall control and scheduling of the display screen, including brightness adjustment, color correction, grayscale control, etc.; it can also be used for partition management of the display screen, and the display screen can be divided into multiple independent areas, each area can display different content.

[0042] Optionally, each host computer 011 can be connected to one or more shooting devices 051; the specific connection method can be selected according to actual needs and device compatibility. As an example, it can include wired connections such as HDMI or SDI (Serial Digital Interface, a digital video transmission standard), and can also include wireless connections such as Wi-Fi (Wireless Fidelity) or RF (Radio Frequency). The shooting device 051 can transmit the shooting data to the host computer.

[0043] Optionally, the display screen 040 may be an LED screen, an LCD screen, or other types, and may be a curved screen or a flat screen. It should be understood that those skilled in the art may customize the type, quantity, size, resolution, etc. of the display screen in the virtual shooting system according to actual needs, and this specification embodiment does not limit this. It should be understood that this specification embodiment does not limit the communication connection method between devices.

[0044] In the virtual shooting scene, the display screen can display the virtual scene created by the Unreal Engine, and the actual scene corresponding to the virtual scene will be arranged in front of the display screen. Usually, a certain number of physical lamps will be arranged in the actual scene, so the lights of the virtual and real scenes need to be aligned, that is, the lights in the actual scene need to be consistent with the lights in the virtual scene displayed on the display screen to ensure that the shooting image of the shooting device will not have a fragmented visual effect.

[0045] At present, the way to align the virtual and real lights is for the lighting engineer to check the pictures taken by the shooting equipment and adjust the lighting parameters of each physical lamp through the lighting console. Figure 1B The figure is a schematic diagram of a lighting control console according to an exemplary embodiment of the present specification. The lighting control console can be connected to a lighting controller. The lighting control console has adjustment buttons that can adjust the lighting parameters of each lighting fixture. The lighting technician needs to constantly adjust the actual lighting to achieve the virtual and real alignment of the lighting. However, due to the large number of lighting fixtures on the shooting site, each lighting fixture can adjust parameters such as color or intensity, and the lights will also affect each other, resulting in the lighting technician spending a long time to adjust the virtual and real alignment of the lighting, and it is generally difficult to achieve an ideal alignment effect, which affects the efficiency and quality of the virtual shooting on the scene.

[0046] In addition, due to the limitations of the shooting site, the lighting console and the shooting equipment are not arranged in the same position. For example, the lighting console may be arranged behind the display screen, while the shooting equipment is arranged in front of the display screen for shooting. The lighting engineer needs to arrange the shooting equipment in front of the display screen first, and then move to other equipment (such as the main control machine) that can obtain the shooting picture of the shooting equipment to view the shooting picture, and then move to the lighting console to adjust the lighting. After the adjustment, the lighting engineer needs to check the shooting picture at the main control machine to determine whether the lighting is aligned with the real and the virtual. This is repeated. It can be seen that the current method of aligning the real and the virtual with the lighting is inefficient, which brings great inconvenience to users.

[0047] Based on this, the embodiments of this specification provide a method for aligning the virtual and real lights in a virtual shooting system, which can efficiently and automatically complete the virtual and real light alignment.

[0048] like Figure 2A , which is a schematic diagram of a virtual shooting scene according to an exemplary embodiment of the present specification. Figure 2AA virtual shooting system including a display screen, a lighting cluster, a lighting controller, a main control machine on which a main control terminal is deployed, and a shooting device is shown; the main control machine can communicate with the lighting controller, and the lighting controller is connected to the lighting cluster; the method is applied to an operating client deployed in a mobile device, and the operating client and the main control terminal can communicate with each other.

[0049] like Figure 2B FIG. 1 is a flowchart of a method for aligning virtual and real lights according to an exemplary embodiment of the present specification. The method may include the following steps:

[0050] In step 202, the captured images captured by the capturing device are displayed in a user interface.

[0051] The captured image includes the actual scene image captured by the capturing device and the virtual scene image displayed on the display screen.

[0052] In step 204, after detecting that the user selects the range to be aligned from the captured image, the main control end is notified so that when the initial color values ​​between the two specified ranges in the range to be aligned do not match, the main control end controls the lamp cluster through the lamp controller to adjust the lights so that the adjusted color values ​​between the two specified ranges match.

[0053] The two designated ranges include: an actual designated range belonging to the actual scene picture and a virtual designated range belonging to the virtual scene picture.

[0054] As an example, the number of lamps in the lamp cluster can be arbitrary and can be flexibly set according to the actual virtual shooting scene, which is not limited in this embodiment.

[0055] As an example, the number of lighting controllers can be arbitrary and can be flexibly set according to the actual virtual shooting scene, which is not limited in this embodiment.

[0056] As for the connection relationship between the lamp controller and the lamp cluster, the two can optionally be a many-to-many relationship, a one-to-many relationship, or a one-to-one relationship, that is, one or more lamp controllers can be arranged, and each lamp controller can be connected to one or more lamps. Optionally, the specific connection method between the lamp controller and the lamp can be selected according to actual needs and the compatibility of the equipment. As an example, the lamp controller can have one or more interfaces, each of which can be connected to one or more lamps in series.

[0057] As an example, a connection method based on a Digital Multiplex (DMX) protocol may be used between the lamp controller and the lamp.

[0058] As an example, the host control machine can be any computer device, including but not limited to a personal computer or a laptop computer. As an example, in a virtual shooting scene, the host control machine can be set in an area in front of the display screen in the shooting scene, or can be set behind the display screen. Usually, the host control machine is used to control the aforementioned screen-mounting machine in the virtual shooting scene.

[0059] The host computer can communicate with the lamp controller, and the communication method can be configured according to actual needs; as an example, communication can be achieved through a wired connection, or wireless communication can be achieved by joining the same local area network. For example, the optional methods are as follows:

[0060] ① Taking the lighting controller as an example to support the DMX protocol, the main control machine and the lighting controller can be connected through a USB to DMX interface device, the output end of the USB to DMX device is connected to the DMX input end of the lighting controller, the main control end can send out DMX signals through the USB interface, and the DMX signals are transmitted to the lighting controller by the USB to DMX device.

[0061] ②The main control machine and the lamp controller can be connected to the same local area network via wired or wireless means, and the main control terminal can directly send signals to the lamp controller through the network.

[0062] Of course, other communication methods may also be used in actual applications, and this embodiment does not limit this.

[0063] The lighting control method for a virtual shooting system in the embodiments of this specification can be deployed on various mobile terminals through software or hardware modification. The mobile terminal involved in the embodiments of this specification can be a mobile terminal that can provide an interactive interface. For example, the mobile terminal can include but is not limited to handheld devices, tablet computers, PDAs, laptop computers, smart phones, wearable devices, etc. The mobile terminal can refer to a device with a wireless connection function and / or a wired connection function. The wireless connection function refers to the ability to connect to other devices through wireless connection methods such as Wi-Fi and Bluetooth. The mobile terminal involved in the embodiments of this specification can also communicate with other devices through a wired connection function. The mobile terminal involved in the embodiments of this specification can be a touch screen or a non-touch screen, which is not limited in the embodiments of this specification.

[0064] For example, the method for aligning the virtual and real lights of the embodiments of the present specification can be deployed in a mobile terminal through software, for example, it can be deployed in a mobile terminal in the form of an application (Application, APP). When a user expects to align the virtual and real lights, the above application can be opened to enter a user interface for aligning the virtual and real lights. The user interface can display a picture taken by a shooting device, and the user can select a range to be aligned for which the virtual and real lights need to be aligned in the picture. Thus, the application can execute subsequent steps to enable the user to select lights in the range to be aligned to achieve virtual and real alignment.

[0065] In some examples, the host computer may be deployed with an Unreal Engine, which provides a software development kit (SDK) that allows users to develop and insert custom plug-ins to expand the engine's functionality. By using the Unreal Engine's SDK, developers can add specific functions, tools, or workflows to a project to meet the needs of the project. Based on this, the host control end for virtual-real light alignment in this embodiment may be a plug-in program running in the Unreal Engine and developed based on the software development kit provided by the Unreal Engine.

[0066] In actual applications, the number of various devices in the virtual shooting system can be flexibly configured according to actual needs, and this embodiment does not limit this. In actual applications, the virtual shooting system can also include other devices, such as network devices such as routers, as needed, and this embodiment does not limit this.

[0067] In some examples, the main control end and the operation client can communicate with each other, and the specific communication method can be configured as needed; for example, the main control machine and the mobile device are connected to the same router to join the same local area network. The main control end can be configured with a network connection service, and the network control service can broadcast messages to each device connected to the router. The broadcast message carries relevant information representing the network connection service of the main control end; the operation client is also configured with a corresponding network connection service. When it receives a broadcast message sent by the router, it recognizes that the broadcast message is relevant information about the network connection service of the main control end through the information carried in the broadcast message. Then the network connection service of the operation client and the network connection service of the main control end establish a network connection, thereby realizing communication between the main control end and the operation client. For example, using a CS (Client / Server) architecture, the main control end can be a server, and the operation client can be a client. Both parties can use a communication connection based on the Transmission Control Protocol (TCP).

[0068] As an example, the shooting picture displayed on the user interface in step 202 can be obtained by the operating client in a variety of ways; for example, the shooting device can be connected to the main control machine by wire or wirelessly, and the shooting device can transmit the collected data to the main control machine, which is then transmitted to the operating client by the main control terminal deployed in the main control machine.

[0069] In some examples, the main control machine may be configured with a video capture card, and the video capture card is connected to the shooting device via a transmission line; the video capture card is used to receive the Raw data captured by the shooting device via the transmission line;

[0070] The acquiring of the captured image captured by the capturing device may include:

[0071] Receive the encoded data sent by the main control end that is obtained by encoding the Raw data, and obtain the shooting picture shot by the shooting device.

[0072] Among them, the transmission line can be configured as needed, for example, it can be an SDI transmission line, or an HDMI transmission line, etc., which is not limited in this embodiment. Among them, the video capture card configured by the main control machine corresponds to the transmission line. For example, if the main control machine is configured with an SDI video capture card, the main control machine and the shooting device are connected via an SDI transmission line. In this way, the shooting device can transmit the captured video stream signal to the main control machine in real time. In some examples, based on the virtual shooting scene of this embodiment, SDI can be selected, and the SDI interface can be transmitted over long distances and can ensure stable signal quality.

[0073] The manufacturer of the shooting device provides an SDK, so that the video capture card can obtain the Raw data captured by the shooting device based on the SDK of the shooting device; the video capture card of this embodiment can support different types of shooting devices.

[0074] The amount of Raw data is relatively large. The master control end of this embodiment can encode the Raw data to compress the Raw data. In practical applications, the encoding standard can be set according to actual needs. For example, standards such as H.265 (HEVC High Efficiency Video Coding) can be used, which is not limited in this embodiment. Therefore, when the master control end of this embodiment sends the encoded data to the operating client, it can be more efficient and faster.

[0075] In actual applications, taking a shooting device as an example, the shooting device can be located in front of the display screen and shoot towards the actual scene and the display screen. The actual scene can refer to the actual performance area that needs to be shot in the shooting scene. The actual shooting area is usually arranged with physical props, and the actual performance area is for actors to perform. The shooting range of the shooting device covers the display screen and the actual scene, so that the shooting picture can include the actual scene picture shot by the shooting device and the virtual scene picture displayed on the display screen.

[0076] Optionally, in actual applications, multiple shooting devices are set up, and different shooting devices are located at different positions. It is also optional to obtain different shooting pictures taken by multiple shooting devices. This embodiment does not limit this.

[0077] Since the present embodiment designs an operation client deployed in a mobile device, the operation client provides the user with a light virtual-real alignment function, so that the user can conveniently check the light virtual-real situation at the shooting scene by holding the mobile device. For example, the user can conveniently adjust the shooting device to start the light virtual-real alignment, and then the user can timely check the shooting picture taken by the shooting device displayed in the user interface next to the shooting device, and check whether there is a range of light virtual-real misalignment through the shooting picture.

[0078] As an example, the operation client may provide the user with a function of selecting a range to be aligned through a user interface. The selection function may be implemented in a variety of ways, which are not limited in this embodiment.

[0079] In some examples, the operating client can obtain the range to be aligned selected by the user by detecting the trigger position of the user in the shooting picture; for example, the user can select two specified ranges in the range to be aligned by clicking two different positions, double-clicking two different positions, or circling two different ranges in the shooting picture, and the client determines the two specified ranges in the range to be aligned by detecting the trigger position of the user.

[0080] The user may select one or more groups of ranges to be aligned, and this embodiment does not limit the number of ranges to be aligned.

[0081] In some other examples, the method may further include:

[0082] Displaying a selection control in the user interface, wherein the selection control is used for allowing a user to select the range to be aligned;

[0083] In response to detecting that the user triggers the selection control, the range to be aligned selected by the user is determined according to the position of the selection control in the captured image.

[0084] For example, the operating client can display a selection control in the user interface, and the user can operate the selection control, and the selection control is used to select the range to be aligned; as an example, the selection control can be a pair to respectively select the actual specified range in the actual scene screen and the virtual specified range in the virtual scene screen.

[0085] As an example, the selection control may be an operation box, and its shape may be any shape such as a rectangular box, an elliptical box, a circular box, etc. The selection control may be movable and scalable, and the user may move or scale the selection control to flexibly set the sizes of the two specified ranges.

[0086] As an example, when displaying a selection control in a user interface, the selection control may be displayed in a shooting screen according to a preset default position; the default position may be set according to actual needs, and this embodiment does not limit this.

[0087] As an example, the user interface may first display a pair of selection controls to allow the user to select the first set of ranges to be aligned; after the user selects the first set of ranges to be aligned, if necessary, a new pair of selection controls may be added to select the second set of ranges to be aligned, and the client may provide the function of adding selection controls. In this way, the user may be allowed to select any number of ranges to be aligned.

[0088] like Figure 2C FIG. 1 is a schematic diagram of a user interface according to an exemplary embodiment of the present specification, and the two pairs of selection controls shown in the figure are the pair of "V1 and R1" and the pair of "V2 and R2". It can be understood that the two pairs of selection controls correspond to two groups of ranges to be aligned. Optionally, in actual applications, when the user needs to select multiple groups of ranges to be aligned, different identifiers can be used to distinguish each group of ranges to be aligned. For example, Figure 2C The first group of ranges to be aligned / the first pair of selection controls uses the two labels "V1" and "R1" to represent the virtual specified range and the actual specified range respectively; while the second group of ranges to be aligned / the second pair of selection controls uses the two labels "V2" and "R2" to represent the virtual specified range and the actual specified range respectively; this makes it easy for users to check and distinguish each group of ranges to be aligned.

[0089] In a virtual shooting scene, the fusion of virtual and real is a key factor affecting the visual effect of virtual shooting. At the shooting scene, the display screens are arranged around the actual performance area. The actual objects arranged in the actual performance area (such as props such as tables and carpets) are usually matched with the virtual objects in the virtual scene screen displayed on the display screen; for example, assuming that a virtual palace scene is presented in the virtual scene screen, the actual performance area is usually arranged with physical pillars that are the same or similar to the virtual pillars of the virtual palace; assuming that a virtual forest scene is presented in the virtual scene screen, the actual performance area is usually arranged with physical plants that are the same or similar to the virtual plants in the virtual forest. In order to achieve the matching virtual objects and real objects of virtual and real fusion, the corresponding lighting effects in the shooting screen should also be matched. If the lighting effects of the virtual objects in the shooting screen are greatly different from the lighting effects of the matching real objects, it will lead to the lack of fusion of virtual and real, and the audience will feel that the virtual is virtual and the real is real, thus generating a sense of conflict and unreality.

[0090] Based on this, in order to facilitate user operation and guide the user to accurately select the range to be aligned, the display of the selection control in the user interface may include:

[0091] Respectively identifying the actual objects contained in the actual scene picture and the virtual objects contained in the virtual scene picture;

[0092] Determine, from the identified actual objects and virtual objects, a target actual object and a target virtual object whose types match and whose image positions in the captured image match;

[0093] Based on the image positions of the target actual object and the target virtual object in the shooting picture respectively, a selection control is displayed; wherein the selection control includes: a selection control representing the selection of the target actual object and a selection control representing the selection of the target virtual object.

[0094] In the embodiments of this specification, after acquiring the captured image, each actual object contained in the actual scene image and each virtual object contained in the virtual scene image can be identified. As an example, image recognition technology can be used for implementation, for example, an object detection algorithm in the field of image recognition can be used for implementation, and the object detection algorithm can find all the objects of interest in the image, and can also determine their categories and positions in the image.

[0095] As an example, the target detection algorithm can be used to first identify the various objects contained in the captured picture, and then identify the actual objects and virtual objects among the various objects contained in the captured picture; for example, in the captured picture, the virtual scene picture belongs to the background, and the actual scene picture belongs to the foreground. The image segmentation algorithm can be used to segment the virtual scene picture belonging to the background and the actual scene picture belonging to the foreground from the captured picture, and then the actual objects and virtual objects can be distinguished from the various objects contained in the captured picture.

[0096] Alternatively, a rendered image (i.e., a virtual scene image) displayed on a display screen may be obtained. For example, in a virtual shooting scene, a rendering device may serve as an image signal source, and the rendered image displayed on the display screen may be obtained through a main control computer. Virtual objects contained in the rendered image may be identified from the rendered image. The rendered image displayed on the display screen may be captured by a shooting device and included in the shooting picture as a virtual scene picture. Of course, the rendered image may not be completely consistent with the virtual scene picture in the shooting picture. The virtual objects identified from the rendered image may be used to assist in identifying actual objects and virtual objects in the shooting picture.

[0097] Next, the target actual object and the target virtual object whose types match and whose image positions in the captured image match can be determined from the actual objects and the virtual objects; wherein the type matching can be the same type or similar type, for example, in a virtual forest scene, the virtual objects are trees and the actual objects are weeds, and it can be determined that the types of the two match; in actual applications, the conditions for whether the type of the virtual object matches the type of the actual object can be set based on the actual scene; this is not limited in this embodiment.

[0098] In addition, the image positions of the virtual object and the actual object in the captured image can also be determined. Usually, in order to achieve matching virtual objects and actual objects for virtual-real fusion, the image positions in the captured image are usually not too far apart, and preset matching conditions representing the matching of image positions can be set.

[0099] For example, the shooting screen is a rectangle. In a virtual shooting scene, the length of the shooting screen is greater than the width. Taking the coordinate system with the length as the horizontal axis and the width as the vertical axis as an example, the difference between the coordinate ranges of the virtual object and the actual object on the horizontal axis is usually not too far apart. Based on this, the coordinate ranges of the virtual object and the actual object on the horizontal axis can be used to determine whether the image positions of the two in the shooting screen match. For example, the target detection algorithm will detect the rectangular frame surrounding the virtual object and the rectangular frame surrounding the physical object to represent the positions of the two in the shooting screen, and the position of the rectangular frame in the shooting screen can be represented by the coordinates of the diagonal vertices of a diagonal line of the rectangular frame, taking the coordinates of the diagonal vertices as the coordinates of the upper left corner and the lower right corner of the rectangular frame as an example; Figure 2D, is a schematic diagram of a shooting screen according to an exemplary embodiment of the present specification, wherein a rectangular frame of a virtual object located above the screen and a rectangular frame of an actual object located below the screen are shown, and the diagonal vertices of the same type of diagonal lines of the rectangular frame of the virtual object and the rectangular frame of the actual object are taken, and the vertex coordinates of the upper left corner are taken as an example, that is, Figure 2D Taking the midpoint O11 and the point O21 as an example, the difference between the horizontal coordinate value of the upper left corner of the rectangular frame of the virtual object and the horizontal coordinate value of the upper left corner of the rectangular frame of the physical object can be used to determine whether the virtual object and the actual object are too far apart in position to cause a position mismatch. For example, a threshold can be set, and if the absolute value of the difference is less than or equal to the set threshold, it can be considered that the position matches, otherwise it is considered that the position does not match.

[0100] Based on this, this embodiment can display a selection control for selecting a target real object and a selection control for selecting a target virtual object after identifying a matching target real object and a matching target virtual object, wherein there can be multiple pairs of matched target real objects and target virtual objects identified, and multiple pairs of selection controls can be displayed. Thus, when viewing the user interface, the user can see the range to be aligned pre-selected by the selection control, and if it meets the user's expectations, the user can quickly trigger it, reducing user operations.

[0101] In other examples, in order to facilitate user operation, after the user selects an object as a specified range, another corresponding object can be automatically matched; for example, if it is detected that the user selects a target actual object in the actual scene screen, the client can automatically match the target virtual object that matches the target actual object; vice versa, if it is detected that the user selects a target virtual object in the virtual scene screen, the client can automatically match the target actual object that matches the target virtual object. Optionally, the client can prompt the user with the automatically matched target virtual object / target actual object, and can also prompt the user to confirm whether to use it as another specified range. After detecting the user's confirmation information, the automatically matched target virtual object / target actual object is used as another specified range. Thus, this embodiment can reduce user operations and help users quickly and accurately select the range to be aligned.

[0102] In other examples, in order to facilitate user operation, the user can also perform trigger operations such as "clicking" or "double-clicking" on the shooting screen, and the client can obtain the user's actual trigger position in the actual scene screen and the virtual trigger position in the virtual scene screen. Since the various objects are identified from the shooting screen in the aforementioned embodiments, the corresponding actual object can be detected according to the actual trigger position, and the corresponding virtual object can be detected according to the virtual trigger position. The detected actual object and virtual object are then displayed to the user, and the user can be prompted to confirm. If user confirmation is detected, the detected actual object and virtual object can be used as the range to be aligned, thereby reducing user operations and helping users quickly and accurately select the range to be aligned.

[0103] In this embodiment, after obtaining the range to be aligned selected by the user, it is also possible to detect whether the user makes a wrong selection; for example, in this embodiment, before the step of notifying the main control end, the method may further include:

[0104] Identify whether the image positions of the actual designated range and the virtual designated range in the shooting picture match each other, and obtain a first recognition result;

[0105] Identify whether the image contents of the actual designated range and the virtual designated range in the captured image match each other, and obtain a second recognition result;

[0106] If the first recognition result is a mismatch or the second recognition result is a mismatch, a prompt message is outputted to prompt whether an error is selected in the range to be aligned.

[0107] As analyzed above, if the image positions of the actual specified range and the virtual specified range in the range to be aligned selected by the user are greatly different in the captured image, it is possible that the user has made a mistake; the way to determine whether the image positions match can refer to the aforementioned implementation method of whether the position of the virtual object matches the position of the actual object. For example, the actual specified range and the virtual specified range are both rectangular frames. The horizontal coordinate values ​​of the diagonal points of the same type in the two rectangular frames are taken, and whether the absolute value of the difference between the two horizontal coordinate values ​​is less than or equal to the set threshold is used to determine whether the image positions of the two specified ranges match.

[0108] Alternatively, the image contents of the actual designated range and the virtual designated range in the captured image do not match, such as a large difference in type, or the user may have selected the wrong one. For example, as described above, in order to determine whether the type of the virtual object matches the type of the actual object, the type of the actual object contained in the image content of the actual designated range can be identified, and the type of the virtual object contained in the image content of the virtual designated range can be identified. Based on whether the two types match, it is determined whether the two graphic contents match. Based on this, the present embodiment can output a prompt message to prompt whether the range to be aligned is selected incorrectly, so as to prompt the user.

[0109] In this embodiment, after the selected range to be aligned is obtained, it can be determined whether the initial color values ​​between the two specified ranges in the range to be aligned match. For example, the color values ​​of each pixel contained in the actual specified range can be obtained from the captured image, and the color statistics can be calculated based on the color values ​​of each pixel. One or more indicators can be determined as color statistics as needed. For example, taking the color average as an example, the color average of the actual specified range can be calculated; similarly, the color average of the virtual specified range is calculated. As an example, the color value can be represented by RGB (RED, Green, Blue, red, green, and blue) values. Of course, other color representation methods are also optional. In addition to the average value, the color value of the main color that accounts for the largest proportion of the number of pixels can also be used as the color statistics value, or other methods can be used to measure whether the initial color values ​​between the two specified ranges in the range to be aligned match. This embodiment does not limit this.

[0110] Optionally, in this embodiment, the color statistical values ​​of two specified ranges in the range to be aligned may also be displayed on the user interface, so that the user can view the specific difference between the color statistical values ​​of the two specified ranges.

[0111] Optionally, the present embodiment may also pre-set an error threshold. If the error threshold is less than the error threshold, the method may further include: in response to detecting that the initial color values ​​between the two specified ranges match, outputting a prompt message for prompting that the lights between the two specified ranges are aligned. The specific error threshold can be configured as needed, and the present embodiment does not limit this. Taking the color value as an RGB value as an example, the error threshold may include: a first error threshold corresponding to the R channel, a second error threshold corresponding to the G channel, and a third error threshold corresponding to the B channel. Whether the color values ​​between the two specified ranges match can be determined by judging the difference in the color values ​​of the three RGB channels and the size of the error threshold of the corresponding channel. If the difference in the color values ​​of the three channels is greater than the error threshold of the corresponding channel, the color values ​​are considered to match; if the difference in the color value of one of the channels is greater than or equal to the error threshold of the corresponding channel, the color values ​​are considered to not match.

[0112] Optionally, if the difference in color statistics between the two specified ranges is greater than or equal to the error threshold, the initial color values ​​between the two specified ranges do not match, and the light virtual-real alignment process may be initiated.

[0113] Optionally, in the above steps, the actions of "calculating the initial color value between the two specified ranges in the range to be aligned" and "determining whether the initial color values ​​between the two specified ranges match" can be specifically executed in a variety of ways; for example, both can be executed by the client, and the client notifies the main control end after obtaining the result that the initial color values ​​between the two specified ranges do not match. Alternatively, the client can notify the main control end after obtaining the range to be aligned selected by the user, and then the main control end executes these two actions. Alternatively, after the client executes the "calculation" action, it notifies the main control end of the "initial color value between the two specified ranges", and the main control end executes the "determination" action, and the main control end can notify the client of the result of determining "whether the initial color values ​​between the two specified ranges match". Optionally, in the case where the "determination" action is executed by the main control end, the "determination" action can also be executed on the client side.

[0114] In this embodiment, the operating client can notify the main control end so that when the initial color values ​​between the two specified ranges do not match, the main control end can control the lamp cluster through the lamp controller to adjust the lights so that the adjusted color values ​​between the two specified ranges match.

[0115] As an example, after the master generates the light adjustment parameters for the lamp cluster, it sends them to the lamp cluster through the lamp controller, so that after the lamp cluster performs light adjustment based on the light adjustment parameters, the adjusted color values ​​between the two specified ranges match.

[0116] Optionally, the master control end may be configured with a lighting adjustment algorithm, and the master control end may generate lighting adjustment parameters for the lamp cluster based on the color average of two specified ranges in the range to be aligned. For example, based on the current orientation, position and current lighting parameters of each lamp in the lamp cluster, lighting adjustment parameters that can match the adjusted color values ​​between the two specified ranges in the range to be aligned may be generated. Optionally, the user may select multiple groups of ranges to be aligned, and the lighting adjustment parameters generated by the lighting adjustment algorithm of the master control end may match the adjusted color values ​​between the two specified ranges in each group of ranges to be aligned selected by the user.

[0117] As an example, the lighting adjustment parameters include, but are not limited to, parameters such as the position of the lamp, the direction of the lamp, the brightness of the lamp, the color, and the lighting effect of the lamp. The lighting adjustment parameters of the lamp cluster may include the lighting adjustment parameters of each target lamp that needs to be adjusted in the lamp cluster. That is, the generated lighting adjustment parameters of the lamp cluster may be the lighting adjustment parameters of one or more target lamps in the lamp cluster. The master control end may send the lighting adjustment parameters of the lamp cluster to the lamp controller, which in turn sends them to the lamp cluster, so that the target lamps can be adjusted based on the corresponding lighting adjustment parameters.

[0118] As an example, the notifying the main control end may include: sending the image position and color statistics of the actual designated range in the shooting picture, and the image position and color statistics of the virtual designated range in the shooting picture to the main control end, so that the main control end generates light adjustment parameters for the lamp cluster based on the image position and color statistics of the actual designated range in the shooting picture, and the image position and color statistics of the virtual designated range in the shooting picture, and the light adjustment parameters are used to provide to the light cluster for light adjustment so that the adjusted color values ​​between the two designated ranges match.

[0119] As an example, the master control end can determine the color adjustment parameters for adjusting the actual color value of the actual specified range to the virtual color value based on the virtual color value of each pixel within the virtual specified range; a three-dimensional lighting model can be established to simulate the lighting effect of the lamp cluster. An optimization algorithm (such as gradient descent or genetic algorithm, etc.) can be used to find the best lighting adjustment parameters to minimize the color difference between the virtual color value and the actual color value. The scope of the actual area affected by each lamp can also be determined through the three-dimensional model of the scene and the lamp information. The brightness and color of the corresponding lamp can be adjusted according to the degree of influence.

[0120] Among them, the shooting picture is two-dimensional, and the main control end needs to generate the lighting adjustment parameters of the lamps in the three-dimensional actual shooting scene; therefore, this embodiment is designed to transmit the image position of the actual specified range in the shooting picture and the image position of the virtual specified range in the shooting picture, so that the main control end can determine the various lamps that affect the actual specified range from the two-dimensional image position and color value, and generate the lamp adjustment parameters.

[0121] As an example, a mapping between a two-dimensional shooting picture and a three-dimensional actual scene can be established. For example, the mapping relationship between image coordinates and world coordinates can be established using camera parameters (focal length, optical center position, etc.). Based on this relationship, the position of the actual specified range in the two-dimensional shooting picture mapped to the three-dimensional actual scene can be determined, and then the lamps corresponding to the actual specified range (i.e., the lamps that have an impact on the lighting of the actual specified range) can be determined based on the installation position of the lamps and the angle that can be illuminated. The three-dimensional structural information of the scene can be obtained by using three-dimensional reconstruction technology. Combined with the three-dimensional model of the scene and the three-dimensional position of the lamps, a three-dimensional lighting simulation is performed based on the lighting parameters of the lamps, and the actual color value of the actual specified range is simulated. The lighting parameters are optimized with the color difference between the virtual color value and the actual color value as the optimization goal to obtain the lighting adjustment parameters. Among them, the lighting adjustment parameters can be the target lighting parameters to be set to the lamps. Therefore, based on the calculated lamps corresponding to the actual specified range, the lighting impact of each lamp on the actual performance area can be simulated, and the lighting adjustment parameters of the lamps can be generated. Those skilled in the art can select a specific method for generating lighting adjustment parameters as needed.

[0122] As an example, the DMX lighting library in UE supports users to create lighting fixtures and manage and control DMX lighting fixtures through DMX protocols such as Art-Net (an Ethernet protocol based on the TCP / IP protocol family) or sACN (Streaming ACN, a lighting control protocol). In a virtual shooting scene, it is necessary to model the lighting fixture cluster to record the position and angle information of each fixture, and record the information of these real lighting fixtures into the DMX lighting library of UE.

[0123] Based on this, the master control end of this embodiment can load the information of each lamp in the lamp cluster configured in the DMX lighting library of the UE, such as position information or angle information.

[0124] In addition, according to the connection method between the lamp cluster and the lamp controller, the control method of the lamp controller can be configured in the master control end. For example, each lamp controller can have one or more connection ports (four connection ports or more are common), each connection port represents a different control domain, and can be distinguished by different identifiers, such as: 0, 1, 2, 3, etc.

[0125] Multiple lamps can be connected in series and connected to one of the connection ports. If the connection port represented by 0 is connected, the control domain needs to be specified as 0 when sending light adjustment parameters to a lamp in this string.

[0126] The lighting parameters supported by each lamp may include but are not limited to:

[0127] In light mode, parameters for adjusting light brightness or color may be included;

[0128] In the lighting effect mode, parameters for adjusting various real lighting effects may be included, such as lightning mode, strobe mode, gradient mode, etc.

[0129] Different modes determine the number of bytes of data that need to be sent to the lamp and the meaning of each byte.

[0130] You can also set the DMX address of the lamp, that is, the starting position of the lamp in a domain. As an example, each domain is 512 bytes.

[0131] In the master control end, you can specify the domain of the light to be controlled (such as 0, 1, 2, 3), the starting address of the light (such as the byte position in the domain), and specify different light adjustment parameters according to different modes (such as assigning values ​​to the bytes corresponding to the light). Then you can send DMX control signals to real lamps through the DMX signal controller.

[0132] For example, the lamp may be equipped with a driving device, and the movement of the driving device may be controlled as needed to adjust at least one of the position and angle of the lamp.

[0133] When it is necessary to align the virtual and real lights, you can start the main control terminal on the main control machine. The main control terminal can load the real lighting position, angle and other lighting information configured in the UEDMX lighting library.

[0134] Next, the shooting device may be started to capture pictures, which may include real scenes and virtual scenes.

[0135] The main control end can transmit the shooting images captured by the shooting device to the operation client, and the shooting images are displayed on the operation client.

[0136] The operator can select an actual designated range R and a virtual designated range V in the shooting picture as a set of ranges to be aligned on the user interface of the operation client. Of course, the user can also select multiple sets of ranges to be aligned.

[0137] The operating client can respectively calculate the color average values ​​of the actual specified range R and the virtual specified range V in each group of ranges to be aligned, and send the information of each group of ranges to be aligned to the main control end.

[0138] The information of each group of ranges to be aligned received by the master may be as follows:

[0139] For each group of ranges to be aligned, if the color average values ​​of the actual specified range R and the virtual specified range V, such as the difference values ​​of the three channels of R, G, and B, are within the predefined error, then the virtual and real alignment of the lights in the group of ranges to be aligned is completed, and a completion message of the group of ranges to be aligned can be sent to the operating client.

[0140] If the color values ​​of two specified ranges in a group of alignment ranges do not match, the loaded information of each lamp can be used to calculate which real lamps should be adjusted and the lighting parameters of the lamps that need to be adjusted, such as color and intensity, etc. Finally, the lighting adjustment parameters are sent to the lamp controller. The lamp controller sends the data to the lamp.

[0141] Next, the main control end may send a message indicating that the virtual-real alignment of the light is completed to the operation client. After receiving the message, the operation client may output a message indicating that the virtual-real alignment of the light is completed.

[0142] As can be seen from the above embodiments, the virtual shooting system includes a main control machine that can communicate with the shooting device and a lighting system controlled by the main control machine; the lighting system provides lighting for the actual scene; the client can obtain the overall picture shot by the shooting device, and prompt the user to select a set of corresponding specified ranges from the actual scene picture and the virtual scene picture respectively; a set of corresponding specified ranges is used to characterize the situation where the actual scene picture and the virtual scene picture have misaligned lights; the client can use the set of corresponding specified ranges selected by the user as reference data for adjusting the light output of the lighting system, and send it to the main control end. In this embodiment, a framework and interactive process for automatic light virtual-real alignment are proposed. The user can select one or more sets of ranges to be aligned from the virtual scene and the real scene, and the light can be automatically adjusted by an algorithm to make the colors of the ranges to be aligned consistent, thereby achieving the effect of overall light virtual-real alignment. Compared with manual adjustment, the efficiency of light virtual-real alignment is greatly improved, and the effect of light virtual-real alignment is improved.

[0143] Due to the link design in this embodiment in which the master control machine communicates with the lamp controller and the lamp controller is connected with the lamp cluster, the master control end can control the lamp cluster to adjust the light through the lamp controller when the initial color values ​​between the two specified ranges in the range to be aligned do not match, and the adjusted color values ​​between the two specified ranges match after the light adjustment. In this way, the user can conveniently view the shooting picture with a mobile device and conveniently specify the range to be aligned by operating the client. Through the above link design, the master control end can help the user automatically adjust the light, significantly reducing user operations and improving the efficiency of virtual-real light alignment.

[0144] like Figure 3As shown, another method for aligning virtual and real lights in a virtual shooting system is shown in this specification according to an exemplary embodiment. The virtual shooting system includes a display screen, a lighting cluster, a lighting controller, a main control machine for deploying a main control terminal, and a shooting device;

[0145] The main control machine can communicate with the lamp controller, and the lamp controller is connected to the lamp cluster;

[0146] The method is applied to the master control terminal, and the master control terminal can communicate with the operation client deployed in the mobile device; the method may include the following steps:

[0147] In step 302, a range to be aligned in a captured image sent by the operating client is received; the captured image includes: an actual scene image captured by the capturing device and a virtual scene image displayed by the display screen; the range to be aligned includes the following two specified ranges: an actual specified range belonging to the actual scene image and a virtual specified range belonging to the virtual scene image; initial color values ​​between the two specified ranges do not match;

[0148] In step 304, based on the range to be aligned, the lamp controller controls the lamp cluster to adjust the lighting so that the adjusted color values ​​between the two specified ranges match.

[0149] As an example, a light adjustment parameter for the lamp cluster is generated and sent to the lamp cluster through the lamp controller, so that after the lamp cluster performs light adjustment based on the light adjustment parameter, the adjusted color values ​​between the two specified ranges match.

[0150] In some examples, the receiving the to-be-aligned range in the captured image sent by the operating client includes:

[0151] receiving the image position and color statistics of the actual designated range in the shooting picture and the image position and color statistics of the virtual designated range in the shooting picture sent by the operation client;

[0152] The step of controlling the lamp cluster to adjust the lighting by the lamp controller based on the range to be aligned includes:

[0153] Based on the image position and color statistics of the actual designated range in the shooting picture and the image position and color statistics of the virtual designated range in the shooting picture, light adjustment parameters for the lamp cluster are generated, and the light adjustment parameters are used to provide to the light cluster for light adjustment so that the adjusted color values ​​between the two designated ranges match.

[0154] In some examples, the main control machine is equipped with a video capture card, and the video capture card is connected to the shooting device via a transmission line; the video capture card is used to receive the Raw data captured by the shooting device via the transmission line;

[0155] The method further comprises:

[0156] The Raw data is encoded to obtain encoded data and sent to the operating client, so that the operating client can obtain the shooting picture taken by the shooting device.

[0157] In some examples, the host control machine is deployed with an Unreal Engine, and the host control terminal includes: a plug-in program running in the Unreal Engine and developed based on a software development kit provided by the Unreal Engine.

[0158] Corresponding to the embodiment of the method for aligning virtual and real lights in the aforementioned virtual shooting system, the present specification also provides an embodiment of a device for aligning virtual and real lights in the virtual shooting system and a computer device used therein.

[0159] The embodiments of the virtual-real light alignment device in the virtual shooting system of this specification can be applied to computer devices, such as servers or terminal devices. The device embodiments can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor in which it is located reading the corresponding computer program instructions in the non-volatile memory into the memory and running them. From the hardware level, if Figure 4 The figure is a hardware structure diagram of the computer device where the virtual light alignment device in the virtual shooting system of this manual is located. Figure 4 In addition to the processor 410, network interface 420, memory 430, and non-volatile memory 440 shown, the computer device where the virtual-real light alignment device in the virtual shooting system in the embodiment is located may also include other hardware, usually according to the actual function of the computer device, which will not be described in detail.

[0160] like Figure 5 As shown, Figure 5 This is a block diagram of a lighting virtual-real alignment device in a virtual shooting system according to an exemplary embodiment of the present specification, wherein the virtual shooting system includes a display screen, a lighting cluster, a lighting controller, a main control machine for deploying a main control terminal, and a shooting device;

[0161] The main control machine can communicate with the lamp controller, and the lamp controller is connected to the lamp cluster;

[0162] The method is applied to an operation client deployed in a mobile device, and the operation client can communicate with the master control end; the device includes:

[0163] The display module 51 is used to display the shooting picture taken by the shooting device in the user interface; the shooting picture includes the actual scene picture taken by the shooting device and the virtual scene picture displayed on the display screen;

[0164] The notification module 52 is used to: after detecting that the user selects a range to be aligned from the captured image, notify the main control end so that when the initial color values ​​between two specified ranges in the range to be aligned do not match, after the main control end controls the lamp cluster through the lamp controller to adjust the lights, the adjusted color values ​​between the two specified ranges match; wherein the two specified ranges include: an actual specified range belonging to the actual scene image and a virtual specified range belonging to the virtual scene image.

[0165] like Figure 6 As shown, Figure 6 is a block diagram of another device for aligning virtual and real lights in a virtual shooting system according to an exemplary embodiment of the present specification, wherein the virtual shooting system includes a display screen, a lighting cluster, a lighting controller, a main control machine for deploying a main control terminal, and a shooting device;

[0166] The main control machine can communicate with the lamp controller, and the lamp controller is connected to the lamp cluster;

[0167] The method is applied to the master control end, and the master control end can communicate with the operation client deployed in the mobile device; the device includes:

[0168] The receiving module 61 is used to: receive the range to be aligned in the captured picture sent by the operating client; the captured picture includes: the actual scene picture captured by the shooting device and the virtual scene picture displayed by the display screen; the range to be aligned includes the following two specified ranges: an actual specified range belonging to the actual scene picture and a virtual specified range belonging to the virtual scene picture; the initial color values ​​between the two specified ranges do not match;

[0169] The adjustment module 62 is used to: based on the range to be aligned, control the lamp cluster to adjust the lighting through the lamp controller so that the adjusted color values ​​between the two specified ranges match.

[0170] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, which will not be repeated here.

[0171] Accordingly, the embodiment of the present specification also provides a virtual shooting system, which includes a display screen, a lamp cluster, a lamp controller, a main control machine on which a main control terminal is deployed, a shooting device, and an operation client deployed in a mobile device; the main control machine can communicate with the lamp controller, and the lamp controller is connected to the lamp cluster; the main control terminal and the operation client deployed in the mobile device can communicate with each other;

[0172] The operating client is used to execute the steps of the embodiment of the method for aligning the virtual and real lights in the virtual shooting system; the main control end is used to execute the steps of the embodiment of the method for aligning the virtual and real lights in the virtual shooting system.

[0173] Accordingly, the embodiments of the present specification also provide a computer program product, including a computer program, which, when executed by a processor, implements the steps of the embodiment of the method for aligning virtual and real lights in the aforementioned virtual shooting system.

[0174] Correspondingly, an embodiment of the present specification also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of an embodiment of a method for aligning virtual and real lights in a virtual shooting system are implemented.

[0175] Accordingly, the embodiments of the present specification further provide a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the embodiment of the method for aligning virtual and real lights in a virtual shooting system are implemented.

[0176] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this specification. A person of ordinary skill in the art can understand and implement it without paying creative labor.

[0177] The above embodiments can be applied to one or more computer devices, where the computer device is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions. The hardware of the computer device includes but is not limited to a microprocessor, an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP), an embedded device, etc.

[0178] The computer device may be any electronic product that can perform human-computer interaction with a user, such as a personal computer, a tablet computer, a smart phone, a personal digital assistant (PDA), a game console, an interactive network television (IPTV), a smart wearable device, etc.

[0179] The computer device may also include a network device and / or a user device, wherein the network device includes, but is not limited to, a single network server, a server group consisting of multiple network servers, or a cloud consisting of a large number of hosts or network servers based on cloud computing.

[0180] The network where the computer device is located includes but is not limited to the Internet, a wide area network, a metropolitan area network, a local area network, a virtual private network (VPN), etc.

[0181] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0182] The step division of the above methods is only for clear description. When implemented, they can be combined into one step or some steps can be split and decomposed into multiple steps. As long as they include the same logical relationship, they are all within the protection scope of this patent; adding insignificant modifications to the algorithm or process or introducing insignificant designs without changing the core design of the algorithm and process are all within the protection scope of this application.

[0183] Although this specification includes many specific implementation details, these should not be interpreted as limiting the scope of any invention or the scope of protection claimed, but are mainly used to describe the features of the specific embodiments of specific inventions. Certain features described in multiple embodiments in this specification may also be implemented in combination in a single embodiment. On the other hand, the various features described in a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. In addition, although features may work in certain combinations as described above and even initially claim protection, one or more features from the claimed combination may be removed from the combination in some cases, and the claimed combination may point to a sub-combination or a variation of a sub-combination.

[0184] The description of "specific examples" or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of this specification. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0185] Those skilled in the art will readily appreciate other embodiments of the specification after considering the specification and practicing the invention claimed herein. The specification is intended to cover any variations, uses or adaptations of the specification that follow the general principles of the specification and include common knowledge or customary techniques in the art that are not claimed in the specification. The specification and examples are to be considered exemplary only, and the true scope and spirit of the specification are indicated by the following claims.

[0186] It should be understood that the present description is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present description is limited only by the appended claims.

[0187] The above description is only a preferred embodiment of this specification and is not intended to limit this specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of this specification should be included in the scope of protection of this specification.

Claims

1. A method for aligning virtual and real lighting in a virtual shooting system, wherein the virtual shooting system includes a display screen, a lighting cluster, a lighting controller, a main control machine of a deployment main control terminal, and a shooting device; The main control machine can communicate with the lamp controller, and the lamp controller is connected to the lamp cluster; The method is applied to an operation client deployed in a mobile device, and the operation client can communicate with the master control end; the method comprises: Displaying a captured image captured by the camera in a user interface; the captured image includes an actual scene image captured by the camera and a virtual scene image displayed on the display screen; After detecting that the user selects a range to be aligned from the captured image, the main control end is notified so that when the initial color values ​​between the two specified ranges in the range to be aligned do not match, the main control end controls the lamp cluster through the lamp controller to adjust the lights so that the adjusted color values ​​between the two specified ranges match; wherein the two specified ranges include: an actual specified range belonging to the actual scene image and a virtual specified range belonging to the virtual scene image.

2. The method according to claim 1, characterized in that The method further comprises: Displaying a selection control in the user interface, wherein the selection control is used for allowing a user to select the range to be aligned; In response to detecting that the user triggers the selection control, the range to be aligned selected by the user is determined according to the position of the selection control in the captured image.

3. The method according to claim 2, wherein displaying a selection control in the user interface comprises: Respectively identifying the actual objects contained in the actual scene picture and the virtual objects contained in the virtual scene picture; Determine, from the identified actual objects and virtual objects, a target actual object and a target virtual object whose types match and whose image positions in the captured image match; Based on the image positions of the target actual object and the target virtual object in the shooting picture respectively, a selection control is displayed; wherein the selection control includes: a selection control representing the selection of the target actual object and a selection control representing the selection of the target virtual object.

4. The method according to claim 1, further comprising: In response to detecting that the initial color values ​​between the two specified ranges match, a prompt message is outputted to prompt that the virtual and real lights between the two specified ranges are aligned.

5. The method according to claim 1, before the step of notifying the master control end, the method further comprises: Identify whether the image positions of the actual designated range and the virtual designated range in the shooting picture match each other, and obtain a first recognition result; Identify whether the image contents of the actual designated range and the virtual designated range in the captured image match each other, and obtain a second recognition result; If the first recognition result is a mismatch or the second recognition result is a mismatch, a prompt message is outputted to prompt whether an error is selected in the range to be aligned.

6. The method according to claim 1, wherein notifying the master control terminal comprises: The image position and color statistics of the actual designated range in the shooting picture, and the image position and color statistics of the virtual designated range in the shooting picture are sent to the main control end, so that the main control end generates light adjustment parameters for the lamp cluster based on the image position and color statistics of the actual designated range in the shooting picture, and the image position and color statistics of the virtual designated range in the shooting picture. The light adjustment parameters are used to be provided to the light cluster for light adjustment so that the adjusted color values ​​between the two designated ranges match.

7. According to the method of claim 1, the main control computer is equipped with a video capture card, and the video capture card is connected to the shooting device via a transmission line; the video capture card is used to receive the Raw data captured by the shooting device via the transmission line; The picture taken by the shooting device is obtained in the following way: Receive the encoded data sent by the main control end that is obtained by encoding the Raw data, and obtain the shooting picture shot by the shooting device.

8. A method for aligning virtual and real lighting in a virtual shooting system, the virtual shooting system comprising a display screen, a lighting cluster, a lighting controller, a main control machine of a deployment main control terminal, and a shooting device; The main control machine can communicate with the lamp controller, and the lamp controller is connected to the lamp cluster; The method is applied to the master control terminal, and the master control terminal can communicate with the operation client deployed in the mobile device; the method includes: Receive the range to be aligned in the captured picture sent by the operating client; the captured picture includes: the actual scene picture captured by the capturing device and the virtual scene picture displayed by the display screen; the range to be aligned includes the following two specified ranges: an actual specified range belonging to the actual scene picture and a virtual specified range belonging to the virtual scene picture; the initial color values ​​between the two specified ranges do not match; Based on the range to be aligned, the lamp cluster is controlled by the lamp controller to adjust the lighting so that the adjusted color values ​​between the two specified ranges match.

9. According to the method of claim 8, the step of receiving the to-be-aligned range in the captured image sent by the operating client comprises: receiving the image position and color statistics of the actual designated range in the shooting picture and the image position and color statistics of the virtual designated range in the shooting picture sent by the operation client; The step of controlling the lamp cluster to adjust the lighting by the lamp controller based on the range to be aligned includes: Based on the image position and color statistics of the actual designated range in the shooting picture and the image position and color statistics of the virtual designated range in the shooting picture, light adjustment parameters for the lamp cluster are generated, and the light adjustment parameters are used to provide to the light cluster for light adjustment so that the adjusted color values ​​between the two designated ranges match.

10. The method according to claim 8, wherein the main control computer is provided with a video capture card, and the video capture card is connected to the shooting device via a transmission line; the video capture card is used to receive the Raw data captured by the shooting device via the transmission line; The method further comprises: The Raw data is encoded to obtain encoded data and sent to the operating client, so that the operating client can obtain the shooting picture taken by the shooting device.

11. The method according to claim 8, wherein the host control machine is deployed with an Unreal Engine, and the host control terminal comprises: A plug-in program running in the Unreal Engine and developed based on a software development kit provided by the Unreal Engine.

12. A virtual shooting system, comprising a display screen, a lighting cluster, a lighting controller, a main control machine on which a main control terminal is deployed, a shooting device, and an operation client deployed in a mobile device; the main control machine can communicate with the lighting controller, and the lighting controller is connected to the lighting cluster; the main control terminal can communicate with the operation client deployed in the mobile device; The operation client is used to execute the steps of the method described in any one of claims 1 to 7; the main control end is used to execute the steps of the method described in any one of claims 8 to 11.

13. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 11 are implemented.

14. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

15. A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method according to any one of claims 1 to 11.

Images