Light virtual-real alignment method, system, device, program product and storage medium

Abstract

The present specification provides a light virtual-real alignment method, system, device, program product and storage medium. The virtual shooting system includes a display screen, a lamp cluster, a lamp controller, a host computer for deployment of a master control end, and a shooting device. The method includes: displaying a shooting picture taken by the shooting device in a user interface; the shooting picture includes an actual scene picture taken by the shooting device and a virtual scene picture displayed by the display screen; after detecting that a user selects a range to be aligned from the shooting picture, the master control end is notified so that, in the case that the initial color values between two specified ranges in the range to be aligned are not matched, the master control end controls the lamp cluster to perform light adjustment through the lamp controller, and the adjusted color values between the two specified ranges are matched.

Description

TECHNICAL FIELD

[0001] The present specification relates to the technical field of virtual shooting, and in particular to a light real-virtual alignment method, system, device, program product and storage medium. BACKGROUND

[0002] The main principle of virtual shooting is to digitize the background and render it to a set of large display screens in real time. These display screens are arranged around the actual performance area, and physical props can be arranged in the actual performance area. The actors can perform in the shooting area, and the display screens can adjust the display content in real time as needed to present a realistic and integrated virtual background with the actual performance area.

[0003] Generally, a certain number of physical lamps are arranged in the actual scene, so the light of the virtual-real scene needs to be aligned, that is, the light in the actual scene needs to be consistent with the light in the virtual scene displayed by the display screen, so as to ensure that the shooting picture of the shooting device does not have a split visual effect. At present, the light real-virtual alignment method needs to be adjusted repeatedly by the lighting designer, which is time-consuming and laborious and has low efficiency. SUMMARY

[0004] To overcome the problems in the related art, the embodiments of the present specification provide a light real-virtual alignment method, system, device, program product and storage medium.

[0005] According to a first aspect of the embodiments of the present specification, a light real-virtual alignment method in a virtual shooting system is provided, the virtual shooting system comprising a display screen, a lamp cluster, a lamp controller, a host computer deploying a host control end, and a shooting device;

[0006] The host computer can communicate with the lamp controller, and the lamp controller is connected with the lamp cluster;

[0007] The method is applied to an operation client deployed in a mobile device, and the operation client can communicate with the host control end; the method comprises:

[0008] displaying a shooting picture taken by the shooting device in a user interface; the shooting picture comprises an actual scene picture taken by the shooting device and a virtual scene picture displayed by the display screen;

[0009] After detecting that a user selects a to-be-aligned range from the shooting picture, the host control end is notified so that, in the case that initial color values between two specified ranges in the to-be-aligned range do not match, the host control end controls the lamp cluster to perform light adjustment through the lamp controller, and the adjusted color values between the two specified ranges match; wherein the two specified ranges comprise an actual specified range belonging to the actual scene picture and a virtual specified range belonging to the virtual scene picture.

[0010] According to a second aspect of the embodiments of the present specification, a light virtual-real alignment method in a virtual shooting system is provided, the virtual shooting system comprising a display screen, a lamp cluster, a lamp controller, a host computer deployed in a host terminal, and a shooting device;

[0011] The host computer can communicate with the lamp controller, and the lamp controller is connected with the lamp cluster.

[0012] The method is applied to the host terminal, and the host terminal can communicate with an operation client deployed in a mobile device; the method comprises:

[0013] receiving a to-be-aligned range in a shooting picture sent by the operation client; the shooting picture comprises an actual scene picture shot by the shooting device and a virtual scene picture displayed by the display screen; the to-be-aligned range comprises two specified ranges, an actual specified range belonging to the actual scene picture and a virtual specified range belonging to the virtual scene picture; and initial color values between the two specified ranges are not matched;

[0014] Based on the to-be-aligned range, the lamp cluster is controlled by the lamp controller to perform light adjustment, so that the adjusted color values between the two specified ranges are matched.

[0015] According to a third aspect of the embodiments of the present specification, a virtual shooting system is provided, the virtual shooting system comprising a display screen, a lamp cluster, a lamp controller, a host computer deployed in a host terminal, a shooting device, and an operation client deployed in a mobile device; the host computer can communicate with the lamp controller, and the lamp controller is connected with the lamp cluster; the host terminal can communicate with the operation client deployed in the mobile device;

[0016] The operation client is configured to execute the steps of the method of the first aspect, and the host terminal is configured to execute the steps of the method of the second aspect.

[0017] According to a fourth aspect of the embodiments of the present specification, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method embodiments of the first aspect or the second aspect.

[0018] According to a fifth aspect of the embodiments of the present specification, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to implement the steps of the method embodiments of the first aspect or the second aspect.

[0019] According to a sixth aspect of the embodiments of the present specification, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the steps of the method embodiment of the preceding first aspect or second aspect.

[0020] The technical solutions provided by the embodiments of the present specification can include the following beneficial effects:

[0021] In the embodiments of the present specification, the operation client deployed in the mobile device is designed, and the light virtual-real alignment function is provided to the user through the operation client. In this way, the user can hold the mobile device, conveniently adjust the shooting device, and view the shooting picture taken by the shooting device displayed in the user interface in a timely manner beside the shooting device, and view whether there is a range of light virtual-real misalignment through the shooting picture. The operation client can detect the to-be-aligned range selected by the user in the shooting picture and notify the host. Due to the link design that the host communicates with the lamp controller and the lamp controller is connected with the lamp cluster in the embodiments, the host can control the lamp cluster to adjust the light in the case that the initial color values between the two specified ranges in the to-be-aligned range are not matched, and the adjusted color values between the two specified ranges after the light adjustment are matched. In this way, the user can hold the mobile device to conveniently view the shooting picture, conveniently specify the to-be-aligned range through the operation client, and automatically adjust the light through the link design described above, which significantly reduces the user operation and improves the light virtual-real alignment efficiency.

[0022] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and cannot limit the present specification. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1A is a schematic diagram of a virtual shooting scene according to an exemplary embodiment of the present specification.

[0024] Figure 1B is a schematic diagram of a light console according to an exemplary embodiment of the present specification.

[0025] Figure 2A is a schematic diagram of a virtual shooting scene according to an exemplary embodiment of the present specification.

[0026] Figure 2B is a flowchart of a light virtual-real alignment method in a virtual shooting system according to an exemplary embodiment of the present specification.

[0027] Figure 2C is a schematic diagram of a user interface according to an exemplary embodiment of the present specification.

[0028] Figure 2D is a schematic diagram of a photographed picture according to an exemplary embodiment of the present specification.

[0029] Figure 3 is a flow chart of a light virtual-real alignment method in another virtual photographing system according to an exemplary embodiment of the present specification.

[0030] Figure 4 is a hardware structure diagram of a computer device in which a light virtual-real alignment device in a virtual photographing system is located according to an exemplary embodiment of the present specification.

[0031] Figure 5 is a block diagram of a light virtual-real alignment device in a virtual photographing system according to an exemplary embodiment of the present specification.

[0032] Figure 6 is a block diagram of another light virtual-real alignment device in a virtual photographing system according to an exemplary embodiment of the present specification. DETAILED DESCRIPTION

[0033] The exemplary embodiments will be described in detail herein below with reference to the accompanying drawings. In the following description, the same drawings refer to the same elements or similar elements throughout the different drawings. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present specification. Instead, they are merely examples consistent with some aspects of the present specification as detailed in the appended claims.

[0034] The terminology used in the present specification is for the purpose of describing particular embodiments only and is not intended to limit the present specification. As used in the present specification and the appended claims, the singular forms “a,” “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term “and / or” as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0035] It will be understood that, although the terms first, second, third, etc. can be used herein to describe various information, these terms are not intended to denote a temporal or chronological order. Rather, these terms are used only to distinguish different sets of information from one another. For example, a first information can be termed a second information, and similarly, a second information can be termed a first information, without departing from the scope of the present specification. As used herein, the word “if’ can be interpreted to mean “when” or “upon” or “in response to determining” depending on the context.

[0036] The user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the specification are information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation portal for user to choose authorization or refusal.

[0037] Virtual filming is a virtual filming technology that uses display screens to construct virtual backgrounds. It combines real-time rendering and LED (Light Emitting Diode) display technology, and can present a realistic virtual environment in real time on site to replace traditional green screen or blue screen filming. The main principle of virtual filming is to digitize the background and render it in real time to a set of large display screens. These display screens are arranged around the actual performance area, and physical props can be placed in the actual performance area. The actor can perform in the shooting area, and the display screen can adjust the display content in real time as needed to present a realistic and integrated virtual background with the actual performance area.

[0038] As shown in Figure 1A Fig. 1 is a schematic diagram of a virtual filming scene according to an example embodiment of the specification, which can include a virtual filming system composed of one or more computer devices. As an example, the virtual filming system can include one or more combinations of the following devices: one or more master machines 011, one or more rendering devices 021 (also referred to as on-screen machines), one or more broadcast processing devices 031, one or more display screens 040 (three display screens are shown in the figure: display screen 041, display screen 042 and display screen 043), and one or more shooting devices 051. The number of each type of device can be flexibly configured according to actual needs, and this embodiment does not limit it. In actual application, other devices such as mobile terminals or network devices can also be included in the virtual filming system according to needs, and this embodiment does not limit it.

[0039] Optionally, each master machine 011 can be connected to one or more rendering devices 021. The specific connection method can be selected according to actual needs and device compatibility. As an example, wired or wireless connection can be made through a local area network or the Internet, and network transmission protocols can be used for communication. As an example, the master machine can send various control instructions to the rendering devices connected thereto, such as control instructions containing specific image information, etc.

[0040] Optionally, each rendering device 021 can be connected with one or more control processing devices 031; the specific connection manner can be selected according to actual requirements and compatibility of the devices. As an example, it can include a DP (DisplayPort, a digital display interface standard) connection, which can be used to transmit high-quality audio and video signals. It can also be an HDMI (High-Definition Multimedia Interface, a high-definition digital audio and video interface standard), which can combine audio, video and control signals for transmission on a cable. As an example, the rendering device 021 can send various control instructions to the control processing device connected thereto, such as the rendering device serving as an image signal source, sending control instructions containing rendered images, etc.

[0041] In actual applications, the control processing device 031 is optional and can also not be configured in some scenarios. Optionally, each control processing device 031 can be connected with one or more display screens 040; the specific connection manner can be selected according to actual requirements and compatibility of the devices. As an example, it can also include a DP or HDMI connection, and can also include a USB (Universal Serial Bus) or network connection, etc. The control processing device can be used to control and manage the display screen connected thereto, as an example, the control processing device 031 can be used for data transmission and decoding, such as the control processing device 031 receiving signals from an external source (such as a screen, a computer, a mobile terminal or a media player, etc.) and decoding them into a format suitable for display by the display screen; it can also be used for display control, such as overall control and scheduling of the display screen, including brightness adjustment, color correction, grayscale control, etc.; it can also be used for partition management of the display screen, which can divide the display screen into multiple independent areas, each of which can display different content.

[0042] Optionally, each master control 011 can be connected with one or more shooting devices 051; the specific connection manner can be selected according to actual requirements and compatibility of the devices. As an example, it can include wired connections such as HDMI or SDI (Serial Digital Interface, a digital video transmission standard), and can also include wireless connections such as Wi-Fi (Wireless Fidelity) or RF (Radio Frequency). The shooting device 051 can transmit shooting data to the master control.

[0043] Optionally, the display screen 040 can be an LED screen, an LCD screen, or other similar type, and can have a curved screen or a flat screen structure. It should be understood that those skilled in the art can customize the type, number, size, resolution, etc., of the display screens in the virtual shooting system according to actual needs, and this specification does not limit such customization. It should also be understood that this specification does not limit the communication connection methods between devices.

[0044] In a virtual shooting setting, a display screen shows a virtual scene created with Unreal Engine, while a corresponding real-world scene is set up in front of the screen. Typically, a certain number of physical lighting fixtures are placed in the real-world scene. Therefore, the lighting in the virtual and real scenes needs to be aligned, meaning the lighting in the real-world scene needs to be consistent with the lighting in the virtual scene displayed on the screen to ensure that the footage captured by the shooting equipment does not have a disjointed visual effect.

[0045] Currently, the method for aligning the lighting with the virtual elements involves the lighting technician reviewing the footage captured by the camera equipment and adjusting the lighting parameters of each physical light fixture via a lighting control console. For example... Figure 1B The diagram shown is a schematic of a lighting control console according to an exemplary embodiment of this specification. This control console can be connected to a lighting controller and has adjustment buttons for adjusting the lighting parameters of each lighting fixture. The lighting technician needs to continuously adjust the actual lighting to achieve alignment between the real and virtual lighting. However, due to the large number of lighting fixtures on set, each with adjustable parameters such as color or intensity, and the mutual influence between lights, the lighting technician needs to spend a considerable amount of time adjusting the alignment, and it is generally difficult to achieve the ideal alignment effect, affecting the efficiency and quality of on-site virtual shooting.

[0046] Furthermore, due to the limitations of the shooting location, the lighting control console and the shooting equipment are not placed in the same location. For example, the lighting control console may be placed behind the display screen, while the shooting equipment is placed in front of the display screen for shooting. The lighting technician needs to first set up the shooting equipment in front of the display screen, then move to other devices (such as the main control unit) that can capture the shooting image from the shooting equipment to check the shooting image, and then move back to the lighting control console to adjust the lighting. After adjustment, the lighting technician needs to go to the main control unit to check the shooting image to determine whether the lighting is aligned with the background. This process is repeated, so it is clear that the current method of aligning the lighting with the background is inefficient and causes great inconvenience to users.

[0047] Based on this, the embodiments of this specification provide a method for aligning virtual and real lights in a virtual shooting system, which can efficiently and automatically complete the alignment of virtual and real lights.

[0048] like Figure 2A The diagram shown is a schematic representation of a virtual shooting scene according to an exemplary embodiment of this specification. Figure 2AA virtual shooting system is shown in FIG. 1, which includes a display screen, a lamp cluster, a lamp controller, a master control machine of a master control end, and a shooting device; the master control machine can communicate with the lamp controller, and the lamp controller is connected with the lamp cluster; the method is applied to an operation client deployed in a mobile device, and the operation client can communicate with the master control end.

[0049] As shown in FIG. 2, a flowchart of a light virtual-real alignment method according to an exemplary embodiment of the present specification is shown, which can include the following steps: Figure 2B

[0050] In step 202, a shooting picture taken by the shooting device is displayed in a user interface.

[0051] The shooting picture includes an actual scene picture taken by the shooting device and a virtual scene picture displayed by the display screen.

[0052] In step 204, after detecting that a user selects a range to be aligned from the shooting picture, the master control end is informed so that, in the case that initial color values between two specified ranges in the range to be aligned are not matched, the master control end controls the lamp cluster to perform light adjustment through the lamp controller, and the adjusted color values between the two specified ranges are matched.

[0053] The two specified ranges include an actual specified range belonging to the actual scene picture and a virtual specified range belonging to the virtual scene picture.

[0054] As an example, the number of lamps in the lamp cluster can be arbitrary, which can be flexibly set according to an actual virtual shooting scene, and the present embodiment does not limit this.

[0055] As an example, the number of lamp controllers can also be arbitrary, which can be flexibly set according to an actual virtual shooting scene, and the present embodiment does not limit this.

[0056] For the connection relationship between the lamp controller and the lamp cluster, optionally, the two can be in a many-to-many relationship, a one-to-many relationship, or a one-to-one relationship, that is, one or more lamp controllers can be arranged, and each lamp controller can be connected with one or more lamps. Optionally, the specific connection mode between the lamp controller and the lamp can be selected according to actual needs and compatibility of the equipment. As an example, the lamp controller can have one or more interfaces, and each interface can be connected with one or more series-connected lamps.

[0057] As an example, the lamp controller and the lamp can adopt a connection mode based on a Digital Multiplex (DMX) protocol. ​

[0058] As an example, the host computer can be any computer device, including but not limited to a personal computer or a laptop computer, etc. As an example, in a virtual shooting scene, the host computer can be arranged in the area in front of the display screen in the shooting scene, or can be arranged behind the display screen. Generally, the host computer is used to control the aforementioned on-screen computer in the virtual shooting scene.

[0059] The host computer can communicate with the lamp controller, and the communication mode can be configured according to actual needs; as an example, communication can be realized through wired connection, or wireless communication can be realized by joining the same local area network. For example, the following optional modes can be used:

[0060] ①Taking the lamp controller supporting the DMX protocol as an example, the host computer and the lamp controller can be connected through a USB-to-DMX interface device, the output end of the USB-to-DMX device is connected to the DMX input end of the lamp controller, and the host computer can send DMX signals through the USB interface. The DMX signals are transmitted to the lamp controller by the USB-to-DMX device.

[0061] ②The host computer and the lamp controller can be connected to the same local area network through wired or wireless connection, and the host computer can send signals to the lamp controller directly through the network.

[0062] Of course, other communication modes can also be used in actual applications, which are not limited in the present embodiment.

[0063] The light control method for the virtual shooting system in the embodiments of the present specification can be deployed on various mobile terminals through software or hardware modification. The mobile terminal involved in the embodiments of the present specification can be a mobile terminal capable of providing an interactive interface. For example, the mobile terminal can include but is not limited to a handheld device, a tablet computer, a palm computer, a notebook computer, a smart phone, a wearable device, etc. The mobile terminal can refer to a device with wireless connection function and / or wired connection function. The wireless connection function refers to the connection with other devices through Wi-Fi, Bluetooth, etc. The mobile terminal involved in the embodiments of the present specification can also be connected to other devices through wired connection function. The mobile terminal involved in the embodiments of the present specification can be a touch screen or a non-touch screen, which is not limited in the embodiments of the present specification.

[0064] For example, the light virtual-real alignment method of the embodiments of the present specification can be deployed in a mobile terminal through software, for example, in the form of an application (APP), when a user desires to perform light virtual-real alignment, the above-mentioned application can be opened to enter a user interface for performing light virtual-real alignment, which can display a shooting picture taken by a shooting device, and the user can select a to-be-aligned range in the shooting picture that needs to be aligned, so that the application can perform subsequent steps to enable the light of the to-be-aligned range selected by the user to achieve virtual-real alignment.

[0065] In some examples, the host machine can be deployed with an Unreal Engine, which is provided with a software development kit (SDK) to allow users to develop and insert custom plugins to extend the functions of the engine. By using the SDK of the Unreal Engine, developers can add specific functions, tools or workflows to the project to meet the needs of the project. Based on this, the host end for light virtual-real alignment of the embodiments can be a plugin program running in the Unreal Engine and developed based on the software development kit provided by the Unreal Engine.

[0066] In actual applications, the number of various devices in the virtual shooting system can be flexibly configured according to actual needs, and the embodiments do not limit this. In actual applications, other devices such as network devices such as routers can also be included in the virtual shooting system according to needs, and the embodiments do not limit this.

[0067] In some examples, the host end and the operation client can communicate, and the specific communication mode can be configured according to needs; for example, the host machine and the mobile device are connected to the same router to join the same local area network. The host end can be configured with a network connection service, and the network control service can broadcast a message to each device connected to the router, and the broadcast message carries information representing the network connection service of the host end; the operation client also has a corresponding network connection service, when it receives the broadcast message sent by the router, through the information carried in the broadcast message, it identifies that the broadcast message is the information of the network connection service of the host end, then the network connection service of the operation client and the network connection service of the host end establish network connection, thereby realizing the communication between the host end and the operation client. For example, using the CS (Client / Server, client / server) architecture, the host end can be a server and the operation client can be a client, and both sides can use a communication connection based on the Transmission Control Protocol (TCP).

[0068] As an example, the user interface in step 202 displays a shooting picture, and the operation client can obtain the shooting picture in various ways; for example, the shooting device can be connected to the host machine in a wired or wireless manner, the shooting device can transmit the collected data to the host machine, and then the host end deployed in the host machine transmits the collected data to the operation client.

[0069] In some examples, the host machine can be configured with a video capture card, and the video capture card is connected to the shooting device through a transmission line; the video capture card is configured to receive the Raw data collected by the shooting device through the transmission line.

[0070] The shooting picture obtained by the shooting device can include:

[0071] The operation client receives the encoded data obtained by encoding the Raw data transmitted by the host end, and obtains the shooting picture obtained by the shooting device.

[0072] The transmission line can be configured as needed, for example, it can be an SDI transmission line, or an HDMI transmission line, etc., and the embodiment is not limited in this regard. The video capture card configured in the host machine corresponds to the transmission line, for example, the host machine is configured with an SDI video capture card, and the host machine is connected to the shooting device through an SDI transmission line. In this way, the shooting device can transmit the shooting video stream signal to the host machine in real time. In some examples, based on the virtual shooting scene of the embodiment, SDI can be selected, and the SDI interface can be used for long-distance transmission and can ensure stable signal quality.

[0073] The manufacturer of the shooting device provides an SDK, so that the video capture card can obtain the Raw data collected by the shooting device based on the SDK of the shooting device; the video capture card of the embodiment can support different types of shooting devices.

[0074] The amount of Raw data is large, and the host end of the embodiment can encode the Raw data to compress the Raw data. In actual applications, the encoding standard can be set according to actual needs, for example, standards such as H.265 (HEVC High Efficiency Video Coding) can be used, and the embodiment is not limited in this regard. Thus, when the host end of the embodiment transmits the encoded data to the operation client, it can be more efficient and fast.

[0075] In actual application, taking a shooting device as an example, the shooting device can be located in front of the display screen and faces the actual scene and the display screen for shooting. The actual scene can refer to an actual performance area that needs to be shot in a shooting site. Physical props are usually arranged in the actual shooting area, and the actual performance area is for actors to perform. The shooting range of the shooting device covers the display screen and the actual scene. Therefore, the shooting picture can include the actual scene picture shot by the shooting device and the virtual scene picture displayed by the display screen.

[0076] Optionally, in actual application, a plurality of shooting devices are arranged at different positions, and different shooting pictures shot by the plurality of shooting devices are also optional, which are not limited in the embodiment.

[0077] Since the operation client deployed in the mobile device is designed in the embodiment, the light virtual-real alignment function is provided for the user through the operation client. In this way, the user can hold the mobile device to conveniently check the light virtual-real situation in the shooting site. For example, the user can conveniently adjust the shooting device to start the light virtual-real alignment. Then, the user can check the shooting picture shot by the shooting device displayed in the user interface in time beside the shooting device, and check whether there is a range of light virtual-real misalignment through the shooting picture.

[0078] As an example, the operation client can provide the function of selecting the to-be-aligned range for the user through the user interface. The selection function can be implemented in various ways, which are not limited in the embodiment.

[0079] In some examples, the operation client can obtain the to-be-aligned range selected by the user by detecting the trigger position of the user in the shooting picture. For example, the user can select two specified ranges in the to-be-aligned range by clicking two different positions, double-clicking two different positions, or circling two different ranges in the shooting picture. The operation client determines the two specified ranges in the to-be-aligned range by detecting the trigger position of the user.

[0080] In the embodiment, the user can select one or more groups of to-be-aligned ranges, and the number of to-be-aligned ranges is not limited.

[0081] In another example, the method can further include:

[0082] The selection control is used for the user to select the to-be-aligned range.

[0083] In response to detecting that the user triggers the selection control, the to-be-aligned range selected by the user is determined according to the position of the selection control in the shooting picture.

[0084] For example, the operation client can display a selection control in the user interface, and the user can operate the selection control to select the range to be aligned; as an example, the selection control can be a pair of selection controls to select the actual specified range in the actual scene picture and the virtual specified range in the virtual scene picture, respectively.

[0085] As an example, the selection control can be an operation box, which can be in any shape such as a rectangular box, an oval box, a circular box, etc. The selection control can be movable and scalable, and the user can move or scale the selection control to flexibly set the size of the two specified ranges.

[0086] As an example, when the selection control is displayed in the user interface, the selection control can be displayed in a preset default position in the captured picture; the default position can be set according to actual needs, which is not limited in the present embodiment.

[0087] As an example, the user interface can first display a pair of selection controls to let the user select a first group of ranges to be aligned; after the user selects the first pair of ranges to be aligned, if necessary, a new pair of selection controls can be added to select a second group of ranges to be aligned, and the operation client can provide a function of adding selection controls. In this way, the user can select any number of pairs of ranges to be aligned.

[0088] As shown in Figure 2C , it is a schematic diagram of a user interface according to an example embodiment of the present specification, and the two pairs of selection controls shown in the figure are "V1 and R1" and "V2 and R2". It can be understood that the two pairs of selection controls correspond to two groups of ranges to be aligned. Optionally, in actual applications, when the user needs to select multiple groups of ranges to be aligned, different identifiers can be used to distinguish each group of ranges to be aligned. For example, Figure 2C , the first group of ranges to be aligned / the first pair of selection controls uses "V1" and "R1" as identifiers to represent the virtual specified range and the actual specified range, respectively; and the second group of ranges to be aligned / the second pair of selection controls uses "V2" and "R2" as identifiers to represent the virtual specified range and the actual specified range, respectively; so as to facilitate the user to check and distinguish each group of ranges to be aligned.

[0089] In a virtual shooting scene, virtual-real fusion is a key factor affecting the visual effect of virtual shooting. In a shooting site, display screens are arranged around an actual performance area, and actual objects (such as props such as tables and carpets) arranged in the actual performance area are usually matched with virtual objects in a virtual scene picture displayed on the display screens. For example, assuming that a virtual palace scene is presented in the virtual scene picture, actual columns that are the same as or similar to virtual columns of the virtual palace are usually arranged in the actual performance area; assuming that a virtual forest scene is presented in the virtual scene picture, actual plants that are the same as or similar to virtual plants in the virtual forest are usually arranged in the actual performance area. These matched virtual objects and actual objects for realizing virtual-real fusion should also be matched in terms of light effects in a shooting picture. If there is a large difference between the light effect of a virtual object in the shooting picture and the light effect of a matched actual object, virtual-real fusion will not be achieved, and the audience will feel that the virtual is virtual and the real is real, thereby producing a sense of conflict and inauthenticity.

[0090] Based on this, in order to facilitate user operation and guide the user to accurately select the range to be aligned, the display of the selection control in the user interface can include:

[0091] Respectively identify actual objects contained in the actual scene picture and virtual objects contained in the virtual scene picture;

[0092] From the identified actual objects and virtual objects, determine target actual objects and target virtual objects that are matched in type and matched in image position in the shooting picture;

[0093] Based on the image positions of the target actual objects and target virtual objects in the shooting picture, respectively, display a selection control; wherein the selection control includes: a selection control representing the selection of the target actual objects and a selection control representing the selection of the target virtual objects.

[0094] In the embodiments of the present specification, after the shooting picture is obtained, each actual object contained in the actual scene picture and each virtual object contained in the virtual scene picture can be identified. As an example, an image recognition technology can be used to achieve this, for example, an object detection (Object Detection) algorithm in the field of image recognition can be used to achieve this. The object detection algorithm can find all targets (objects) of interest in an image, and also determine their categories and positions in the image.

[0095] As an example, the target detection algorithm can be used to identify each object contained in the captured image, and then identify the actual object and the virtual object in each object contained in the captured image; for example, in the captured image, the virtual scene image belongs to the background, and the actual scene image belongs to the foreground. An image segmentation algorithm can be used to segment the virtual scene image belonging to the background and the actual scene image belonging to the foreground from the captured image, and then distinguish the actual object and the virtual object from each object contained in the captured image.

[0096] Alternatively, the rendering image (i.e. the virtual scene image) displayed by the display screen can be obtained, for example, in the virtual shooting scene, the rendering device can act as an image signal source, and the rendering image displayed by the display screen can be obtained by the host computer. The virtual object contained in the rendering image can be identified from the rendering image; the rendering image displayed by the display screen is captured by the shooting device and contained in the captured image as a virtual scene image; of course, the rendering image and the virtual scene image in the captured image can not be completely consistent; the actual object and the virtual object in the captured image can be identified based on the virtual object identified from the rendering image.

[0097] Then, the target actual object and the target virtual object that match in type and match in image position in the captured image can be determined from the actual objects and the virtual objects; wherein the type match can be the same type or similar type, for example, in the virtual forest scene, the virtual object is a tree, and the actual object is a wild grass, and the type match can be determined; in actual application, the condition of whether the type of the virtual object matches the type of the actual object can be set based on the actual scene; this embodiment is not limited in this regard.

[0098] In addition, the image positions of the virtual object and the actual object in the captured image can also be determined. Generally, in order to realize the matching of the virtual object and the actual object in the virtual-real fusion, the image positions in the captured image are usually not too far apart, and a preset matching condition representing the matching of the image positions can be set.

[0099] For example, the captured image is a rectangle, and the length of the captured image is greater than the width in the virtual shooting scene. Taking a coordinate system with the length as the horizontal axis and the width as the vertical axis as an example, the difference between the coordinate ranges of the virtual object and the actual object on the horizontal axis is usually not too far apart. Based on this, whether the image positions of the virtual object and the actual object in the captured image match can be determined based on the coordinate ranges of the virtual object and the actual object on the horizontal axis. For example, the target detection algorithm can detect a rectangular frame surrounding the virtual object and a rectangular frame surrounding the actual object to represent the positions of the virtual object and the actual object in the captured image. The position of the rectangular frame in the captured image can be represented by the coordinates of a diagonal vertex of a diagonal line of the rectangular frame, for example, taking the coordinates of the top-left corner and the bottom-right corner of the rectangular frame as the diagonal vertex. Figure 2DAs shown in the schematic diagram of the captured picture according to an example embodiment of the present specification, a rectangular frame of a virtual object located at the top of the picture and a rectangular frame of a real object located at the bottom are shown, and the diagonal vertices of the same type diagonal of the rectangular frame of the virtual object and the rectangular frame of the real object are taken, for example, the top coordinates of the top left corner are taken, that is Figure 2D Taking the midpoint O11 and the point O21 as an example, according to the difference between the horizontal coordinate value of the top left corner of the rectangular frame of the virtual object and the horizontal coordinate value of the top left corner of the rectangular frame of the real object, it can be determined whether the virtual object and the real object are too far apart in position to cause position mismatch. For example, a threshold value can be set, and if the absolute value of the difference is less than or equal to the set threshold value, it can be considered that the positions match, otherwise it can be considered that the positions do not match.

[0100] Based on this, the present embodiment can display selection controls representing the selection of the target real object and selection controls representing the selection of the target virtual object after identifying the matched target real object and target virtual object. There can be multiple pairs of matched target real objects and target virtual objects, and multiple pairs of selection controls can be displayed. Thus, when the user views the user interface, the user can see the selected range to be aligned by the selection controls in advance, and if it meets the user's expectation, the user can quickly trigger and reduce user operations.

[0101] In some other examples, in order to facilitate user operation, after the user selects an object as a specified range, the corresponding another object can also be automatically matched. For example, if it is detected that the user selects a target real object in the real scene picture, the client can automatically match a target virtual object matching the target real object. Conversely, if it is detected that the user selects a target virtual object in the virtual scene picture, the client can automatically match a target real object matching the target virtual object. Optionally, the client can prompt the user about the automatically matched target virtual object / target real object, and can also prompt the user to confirm whether to take the automatically matched target virtual object / target real object as another specified range. After detecting the user's confirmation information, the automatically matched target virtual object / target real object is taken as another specified range. Thus, the present embodiment can reduce user operation and help the user to quickly and accurately select the range to be aligned.

[0102] In some examples, in order to facilitate user operation, the user can trigger an operation such as "clicking" or "double-clicking" on the captured image, and the client can obtain the actual trigger position in the actual scene image and the virtual trigger position in the virtual scene image. Since the objects in the captured image are identified in the foregoing embodiments, the corresponding actual object can be detected according to the actual trigger position, and the corresponding virtual object can be detected according to the virtual trigger position. Then, the detected actual object and virtual object are displayed to the user, and the user can be prompted to confirm. If it is detected that the user confirms, the detected actual object and virtual object can be used as the range to be aligned, thereby reducing user operation and helping the user to quickly and accurately select the range to be aligned.

[0103] In the embodiment, after the range to be aligned selected by the user is obtained, it can be detected whether the user makes a mistake in selection. For example, before the step of notifying the master, the method can further include:

[0104] identifying whether the image positions of the actual specified range and the virtual specified range in the captured image match, to obtain a first identification result;

[0105] identifying whether the image contents of the actual specified range and the virtual specified range in the captured image match, to obtain a second identification result;

[0106] If the first identification result is not matched or the second identification result is not matched, a prompt message for prompting whether the range to be aligned is selected incorrectly is output.

[0107] As analyzed above, if the actual specified range and the virtual specified range in the range to be aligned selected by the user have a large difference in image positions in the captured image, it is likely that the user makes a mistake in selection. Whether the image positions match can be determined according to the implementation manner of whether the position of the virtual object matches the position of the actual object. For example, if the actual specified range and the virtual specified range are both rectangular boxes, the horizontal coordinate values of the same type of opposite corners in the two rectangular boxes are taken, and whether the image positions of the two specified ranges match is determined according to whether the absolute value of the difference between the two horizontal coordinate values is less than or equal to a set threshold value.

[0108] Alternatively, the actual specified range and the virtual specified range respectively in the image content in the captured picture do not match, such as a large type difference, and it is also possible that the user selects incorrectly. For example, as described above, the determination method of whether the type of the virtual object matches the type of the actual object, the type of the actual object contained in the image content of the actual specified range is identified, and the type of the virtual object contained in the image content of the virtual specified range is identified, and whether the two types match is determined according to whether the two types match. Based on this, the embodiment can output a prompt message for prompting whether the to-be-aligned range is selected incorrectly to prompt the user.

[0109] In the embodiment, after the selected to-be-aligned range is obtained, it can be determined whether the initial color values between the two specified ranges in the to-be-aligned range match. For example, the color values of each pixel contained in the actual specified range in the captured picture can be obtained, and a color statistical value can be calculated based on the color values of each pixel. One or more indicators can be determined as the color statistical value as needed, for example, the color average value. The color average value of the actual specified range can be calculated. Similarly, the color average value of the virtual specified range is calculated. As an example, the color value can be represented by RGB (RED, Green, Blue) value, and other color representation methods are also optional. In addition to the average value, the color value of the main color with the largest proportion of the number of pixels can also be used as the color statistical value, or other methods can be used to measure whether the initial color values between the two specified ranges in the to-be-aligned range match. The embodiment does not limit this.

[0110] Optionally, the embodiment can also display the color statistical values of the two specified ranges in the to-be-aligned range on the user interface, so that the user can view the specific differences between the color statistical values of the two specified ranges.

[0111] Optionally, the embodiment can also pre-set an error threshold. If the error threshold is less than the error threshold, the method can also include: in response to detecting that the initial color values between the two specified ranges match, outputting a prompt message for prompting that the virtual and actual light between the two specified ranges has been aligned. The specific error threshold can be configured as needed, and the embodiment does not limit this. Taking the color value as the RGB value as an example, the error threshold can include: a first error threshold corresponding to the R channel, a second error threshold corresponding to the G channel, and a third error threshold corresponding to the B channel. Whether the color values between the two specified ranges match can be determined by judging the difference between the color values of the RGB three channels and the size of the error threshold corresponding to the channel. If the difference between the color values of the three channels is greater than the error threshold corresponding to the channel, it is considered that the color values match. If the difference between the color values of one of the channels is greater than or equal to the error threshold corresponding to the channel, it is considered that the color values do not match.

[0112] Optionally, if the difference between the color statistics of the two specified ranges is greater than or equal to the error threshold, the initial color values between the two specified ranges do not match, and the light solid-void alignment processing can be started.

[0113] Optionally, in the above steps, the actions of "calculating the initial color values between the two specified ranges in the range to be aligned" and "determining whether the initial color values between the two specified ranges match" can be implemented in various ways. For example, both actions can be performed by the client, which notifies the master after obtaining the result that the initial color values between the two specified ranges do not match. Alternatively, the client can notify the master after obtaining the range to be aligned selected by the user, and then the master performs the two actions. Alternatively, the client performs the "calculation" action and notifies the master of the "initial color values between the two specified ranges", and the master performs the "determination" action and notifies the client of the result of "whether the initial color values between the two specified ranges match". Optionally, in the case where the master performs the "determination" action, the "determination" action can also be performed on the client side.

[0114] In this embodiment, the master can be notified by the operation client to enable the master to control the light fixture cluster to perform light adjustment through the light fixture controller in the case where the initial color values between the two specified ranges do not match, so that the adjusted color values between the two specified ranges match after the light adjustment.

[0115] As an example, after the master generates the light adjustment parameters for the light fixture cluster, the master sends the light adjustment parameters to the light fixture cluster through the light fixture controller, so that the light fixture cluster performs light adjustment based on the light adjustment parameters, and the adjusted color values between the two specified ranges match.

[0116] Optionally, the master can be configured with a light adjustment algorithm, and the master can generate the light adjustment parameters for the light fixture cluster based on the color averages of the two specified ranges in the range to be aligned. For example, the light adjustment parameters can be generated based on the current orientations, positions, and current light parameters of the light fixtures in the light fixture cluster, so that the adjusted color values between the two specified ranges in the range to be aligned match. Optionally, the user can select multiple groups of ranges to be aligned, and the light adjustment parameters generated by the light adjustment algorithm of the master can enable the adjusted color values between the two specified ranges in each group of ranges to be aligned selected by the user to match.

[0117] As an example, the light adjustment parameters include, but are not limited to, the lamp position, the lamp orientation, the brightness of the lamp, the color, the light effect of the lamp, and the like. The light adjustment parameters of the lamp cluster can include the light adjustment parameters of each target lamp in the lamp cluster that needs to be adjusted. That is, the generated light adjustment parameters of the lamp cluster can be the light adjustment parameters of one or more target lamps in the lamp cluster. The master control end can send the light adjustment parameters of the lamp cluster to the lamp controller, and the lamp controller sends the light adjustment parameters to the lamp cluster, so that the target lamps can be adjusted based on the corresponding light adjustment parameters.

[0118] As an example, the notification of the master control end can include: the image position of the actual specified range in the shooting picture and the color statistical value, the image position of the virtual specified range in the shooting picture and the color statistical value can be sent to the master control end, so that the master control end generates the light adjustment parameters of the lamp cluster based on the image position of the actual specified range in the shooting picture and the color statistical value, the image position of the virtual specified range in the shooting picture and the color statistical value, and the light adjustment parameters are provided to the light cluster for light adjustment, so that the adjusted color values between the two specified ranges are matched.

[0119] As an example, the master control end can determine the color adjustment parameters for adjusting the actual color value of the actual specified range to the virtual color value of each pixel in the virtual specified range; a three-dimensional lighting model can be established to simulate the lighting effect of the lamp cluster. An optimization algorithm (such as gradient descent or genetic algorithm) can be used to find the best light adjustment parameters to minimize the color difference between the virtual color value and the actual color value. The range of the actual area affected by each lamp can also be determined through the three-dimensional model of the scene and the lamp information. The brightness and color of the corresponding lamp can be adjusted according to the degree of influence.

[0120] Wherein, the shooting picture is two-dimensional, and the master control end needs to generate the light adjustment parameters of the lamp in the three-dimensional actual shooting scene; therefore, the embodiment designs to transmit the image position of the actual specified range in the shooting picture and the image position of the virtual specified range in the shooting picture, so that the master control end can determine the lamps affecting the actual specified range from the two-dimensional image position and color value, and generate the lamp adjustment parameters.

[0121] As an example, a two-dimensional shooting picture and a three-dimensional actual scene mapping can be established. For example, a mapping relationship between image coordinates and world coordinates can be established using camera parameters (focal length, light center position, etc.). Based on the relationship, the position of the actual specified range in the two-dimensional shooting picture can be determined to map to the three-dimensional actual scene. Then, based on the installation position and the angle of the light that can be illuminated of the light fixture, the light fixtures corresponding to the actual specified range (i.e., the light fixtures that have an impact on the light of the actual specified range) can be determined. The three-dimensional structure information of the scene can be obtained by using a three-dimensional reconstruction technology. Based on the three-dimensional model of the scene and the three-dimensional position of the light fixture, three-dimensional lighting simulation is performed based on the light parameters of the light fixture. The actual color value of the actual specified range is simulated to minimize the color difference between the virtual color value and the actual color value as an optimization objective. The light adjustment parameters are optimized to obtain the light adjustment parameters. The light adjustment parameters can be the target light parameters to be set for the light fixture. Thus, based on the light fixtures corresponding to the actual specified range calculated, the lighting impact of the light fixtures on the actual performance area can be simulated, and the light adjustment parameters of the light fixtures are generated. A person skilled in the art can select a specific way to generate the light adjustment parameters according to the needs.

[0122] As an example, the DMX light library in the UE supports users to create light fixtures and supports management and control of the DMX light fixtures through the DMX protocol, such as Art-Net (an Ethernet protocol based on the TCP / IP protocol family) or sACN (a kind of light control protocol) and the like. In the virtual shooting scene, the light fixture cluster needs to be modeled to record the position and angle information of each light fixture, and the information of the real light fixtures is recorded into the DMX light library of the UE.

[0123] Based on this, the master control end of the present embodiment can load the information of each light fixture in the light fixture cluster configured in the DMX light library of the UE, such as position information or angle information.

[0124] In addition, according to the connection mode of the light fixture cluster and the light fixture controller, the control mode of the light fixture controller can be configured in the master control end. For example, each light fixture controller can have one or more connection ports (commonly four connection ports or more), and each connection port represents a different control domain and can be distinguished by different identifiers, such as 0, 1, 2, 3, and the like.

[0125] A plurality of light fixtures can be connected in series and connected to one of the connection ports. If connected to the connection port represented by 0, the control domain needs to be specified as 0 when sending the light adjustment parameters to a certain light fixture in the string of light fixtures.

[0126] The light parameters that can be supported by each light fixture can include but are not limited to:

[0127] In the light mode, parameters for adjusting the brightness or color of the light can be included.

[0128] In the light effect mode, parameters for adjusting various real light effects such as lightning mode, strobe mode, gradient mode, etc. can be included.

[0129] Different modes determine the number of bytes of data that need to be sent to the lamp and the meaning of each byte.

[0130] The DMX address of the lamp can also be set, i.e. the starting position of the lamp in a domain. As an example, each domain is 512 bytes.

[0131] In the master control, the domain to be controlled (e.g. 0, 1, 2, 3), the starting address of the light (e.g. the byte position in the domain), and different light adjustment parameters according to different modes (e.g. assigning values to the bytes corresponding to the lamp) can be specified. DMX control signals can be sent to real lamps through a DMX signal controller.

[0132] As an example, the lamp can be equipped with a driving device, and the driving device can also be controlled to move as needed, thereby adjusting at least one of the position and angle of the lamp.

[0133] When real and virtual light alignment is needed, the master control can be started on the host computer, and the master control can load the real lamp information such as the position and angle of the real lamp configured in the UEDMX light library.

[0134] Next, the shooting device can be started to collect a shooting picture, which contains real and virtual scenes.

[0135] The master control can transmit the shooting picture collected by the shooting device to the operation client, and the operation client displays the shooting picture.

[0136] The operator can select an actual specified range R and a virtual specified range V in the shooting picture on the user interface of the operation client as a set of to-be-aligned ranges to be aligned. Of course, the user can also select multiple sets of to-be-aligned ranges.

[0137] The operation client can calculate the color average of the actual specified range R and the virtual specified range V in each set of to-be-aligned ranges respectively, and send the information of each set of to-be-aligned ranges to the master control.

[0138] The master control receives the information of each set of to-be-aligned ranges, which can have the following cases:

[0139] For each set of to-be-aligned ranges, if the color average values of the actual specified range R and the virtual specified range V, such as the difference values of R, G, B three channels, are within the predefined error, the light virtual-real alignment of the set of to-be-aligned ranges is completed, and a completion message of the set of to-be-aligned ranges can be sent to the operation client.

[0140] If the color values of two specified ranges in a set of to-be-aligned ranges do not match, the loaded lamp information can be used to calculate which real lamps are adjusted and the light adjustment parameters, such as color and intensity, that the lamps need to adjust; finally, the light adjustment parameters are sent to the lamp controller. The lamp controller sends data to the lamp.

[0141] Then, the master control end can send a message of the end of light virtual-real alignment to the operation client, and after receiving the message, the operation client can output a message prompting that the light virtual-real alignment is completed.

[0142] As can be seen from the above embodiment, the virtual shooting system includes a master control machine communicable with a shooting device and a lamp system controlled by the master control machine; the lamp system provides light for an actual scene; a client can obtain an overall picture shot by the shooting device, prompt a user to select a set of corresponding specified ranges from an actual scene picture and a virtual scene picture respectively; the set of corresponding specified ranges is used to represent a situation that the actual scene picture and the virtual scene picture are not aligned in light; the client can send the set of corresponding specified ranges selected by the user as reference data for adjusting light output of the lamp system to the master control end. The embodiment proposes a set of automatic light virtual-real alignment framework and interaction process, the user can select one or more sets of to-be-aligned ranges from the virtual scene and the actual scene, and can automatically adjust the light through an algorithm to make the color of the to-be-aligned ranges consistent, so as to achieve the effect of overall light virtual-real alignment. Compared with manual adjustment, the efficiency of light virtual-real alignment is greatly improved, and the effect of light virtual-real alignment is improved.

[0143] Due to the link design that the master control machine communicates with the lamp controller and the lamp controller is connected with the lamp cluster in the embodiment, the master control end can control the lamp cluster to adjust the light in the case that the initial color values between two specified ranges in the to-be-aligned ranges do not match, and the adjusted color values between the two specified ranges match after the light adjustment. In this way, the user can conveniently view the shooting picture by holding the mobile device, conveniently specify the to-be-aligned ranges through the operation client, and through the above link design, the master control end can automatically adjust the light for the user, significantly reducing the user operation and improving the efficiency of light virtual-real alignment.

[0144] As Figure 3As shown, the method is another light virtual-real alignment method in a virtual shooting system according to an example embodiment, the virtual shooting system includes a display screen, a lamp cluster, a lamp controller, a host computer deployed in a host terminal, and a shooting device;

[0145] The host computer can communicate with the lamp controller, and the lamp controller is connected with the lamp cluster.

[0146] The method is applied to the host terminal, and the host terminal can communicate with an operation client deployed in a mobile device; the method can include the following steps:

[0147] In step 302, a to-be-aligned range in a shooting picture sent by the operation client is received; the shooting picture includes an actual scene picture captured by the shooting device and a virtual scene picture displayed by the display screen; the to-be-aligned range includes two specified ranges: an actual specified range belonging to the actual scene picture and a virtual specified range belonging to the virtual scene picture; and initial color values between the two specified ranges are not matched.

[0148] In step 304, based on the to-be-aligned range, the lamp cluster is controlled by the lamp controller to perform light adjustment, so that the adjusted color values between the two specified ranges are matched.

[0149] As an example, light adjustment parameters of the lamp cluster are generated and sent to the lamp cluster through the lamp controller, so that after the lamp cluster performs light adjustment based on the light adjustment parameters, the adjusted color values between the two specified ranges are matched.

[0150] In some examples, the receiving of the to-be-aligned range in the shooting picture sent by the operation client includes:

[0151] The image position and color statistical value of the actual specified range in the shooting picture and the image position and color statistical value of the virtual specified range in the shooting picture sent by the operation client are received.

[0152] The controlling of the lamp cluster by the lamp controller to perform light adjustment based on the to-be-aligned range includes:

[0153] Based on the image position and color statistical value of the actual specified range in the shooting picture and the image position and color statistical value of the virtual specified range in the shooting picture, light adjustment parameters of the lamp cluster are generated, and the light adjustment parameters are provided to the lamp cluster to perform light adjustment, so that the adjusted color values between the two specified ranges are matched.

[0154] In some examples, the host computer is configured with a video capture card, and the video capture card is connected with the shooting device through a transmission line; the video capture card is configured to receive the Raw data collected by the shooting device through the transmission line;

[0155] The method further includes:

[0156] encoding the Raw data to obtain encoded data and sending the encoded data to the operation client, so that the operation client obtains a shooting picture taken by the shooting device.

[0157] In some examples, the host computer is deployed with an Unreal Engine, and the host end includes a plug-in program developed based on a software development kit provided by the Unreal Engine and running in the Unreal Engine.

[0158] Corresponding to the above-mentioned embodiments of the light virtual-real alignment method in the virtual shooting system, the present specification also provides embodiments of a light virtual-real alignment device in a virtual shooting system and a computer device to which the light virtual-real alignment device is applied.

[0159] The embodiment of the light virtual-real alignment device in the virtual shooting system of the present specification can be applied to a computer device, such as a server or a terminal device. The device embodiment can be implemented by software, or by hardware or a combination of software and hardware. Taking the software implementation as an example, as a logically meaningful device, it is formed by reading the corresponding computer program instructions in the non-volatile memory into the memory and running by the processor where it is located. From the hardware level, as shown in Figure 4 The embodiment of the light virtual-real alignment device in the virtual shooting system of the present specification can be applied to a computer device, such as a server or a terminal device. The device embodiment can be implemented by software, or by hardware or a combination of software and hardware. Taking the software implementation as an example, as a logically meaningful device, it is formed by reading the corresponding computer program instructions in the non-volatile memory into the memory and running by the processor where it is located. From the hardware level, as shown in Figure 4 In addition to the processor 410, the network interface 420, the memory 430, and the non-volatile memory 440 shown in

[0160] As shown in Figure 5 As shown in Figure 5 is a block diagram of a light virtual-real alignment device in a virtual shooting system according to an exemplary embodiment of the present specification, the virtual shooting system including a display screen, a lamp cluster, a lamp controller, a host computer deploying a host end, and a shooting device;

[0161] The host computer can communicate with the lamp controller, and the lamp controller is connected with the lamp cluster;

[0162] The method is applied to an operation client deployed in a mobile device, and the operation client can communicate with the host end; the device includes:

[0163] The display module 51 is used to: display the shooting screen captured by the shooting device in the user interface; the shooting screen includes the actual scene captured by the shooting device and the virtual scene displayed on the display screen;

[0164] The notification module 52 is used to: after detecting that the user selects a range to be aligned from the captured image, notify the main control terminal so that if the initial color values ​​between the two specified ranges in the range to be aligned do not match, the main control terminal controls the lighting cluster through the lighting controller to adjust the lighting so that the adjusted color values ​​between the two specified ranges match; wherein, the two specified ranges include: an actual specified range belonging to the actual scene image and a virtual specified range belonging to the virtual scene image.

[0165] like Figure 6 As shown, Figure 6 This is a block diagram of a lighting virtual-real alignment device in another virtual shooting system according to an exemplary embodiment of this specification. The virtual shooting system includes a display screen, a lighting cluster, a lighting controller, a main control unit with a main control terminal, and shooting equipment.

[0166] The main control unit can communicate with the lighting controller, and the lighting controller is connected to the lighting cluster;

[0167] The method is applied to the main control terminal, which is capable of communicating with the operating client deployed on the mobile device; the device includes:

[0168] The receiving module 61 is configured to: receive the area to be aligned in the captured image sent by the operation client; the captured image includes: the actual scene captured by the shooting device and the virtual scene displayed on the display screen; the area to be aligned includes the following two specified ranges: an actual specified range belonging to the actual scene and a virtual specified range belonging to the virtual scene; the initial color values ​​between the two specified ranges are not matched;

[0169] The adjustment module 62 is used to: based on the range to be aligned, control the lighting cluster through the lighting controller to adjust the lighting so that the adjusted color values ​​between the two specified ranges match.

[0170] The specific implementation process of the functions and roles of each module in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0171] Correspondingly, the embodiment of the present specification also provides a virtual shooting system, the virtual shooting system comprises a display screen, a lamp cluster, a lamp controller, a host computer deploying a host control terminal, a shooting device and an operation client deployed in a mobile device; the host computer can communicate with the lamp controller, the lamp controller is connected with the lamp cluster; the host control terminal and the operation client deployed in the mobile device can communicate;

[0172] The operation client is used for executing the steps of the light virtual-real alignment method embodiment in the virtual shooting system; and the host control terminal is used for executing the steps of the light virtual-real alignment method embodiment in the virtual shooting system.

[0173] Correspondingly, the embodiment of the present specification also provides a computer program product, comprising a computer program, the computer program is executed by a processor to realize the steps of the light virtual-real alignment method embodiment in the virtual shooting system.

[0174] Correspondingly, the embodiment of the present specification also provides a computer device, comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor executes the program to realize the steps of the light virtual-real alignment method embodiment in the virtual shooting system.

[0175] Correspondingly, the embodiment of the present specification also provides a computer readable storage medium, which stores a computer program, the computer program is executed by a processor to realize the steps of the light virtual-real alignment method embodiment in the virtual shooting system.

[0176] For the device embodiment, since it basically corresponds to the method embodiment, the related part is described in the part of the method embodiment. The device embodiment described above is only schematic, wherein the modules described as separate components can or can not be physically separated, and the components displayed as modules can or can not be physical modules, that is, they can be located in one place, or distributed on multiple network modules. According to the actual needs, part or all of the modules can be selected to achieve the purpose of the present specification. Those skilled in the art can understand and implement without creative labor.

[0177] The above embodiments can be applied to one or more computer devices, which are devices capable of automatically performing numerical calculation and / or information processing according to pre-set or stored instructions. The hardware of the computer devices includes, but is not limited to, microprocessors, application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, and the like.

[0178] The computer devices can be any electronic product capable of human-computer interaction with a user, such as personal computers, tablet computers, smart phones, personal digital assistants (PDAs), game consoles, interactive Internet protocol televisions (IPTVs), smart wearable devices, and the like.

[0179] The computer devices can also include network devices and / or user devices. The network devices include, but are not limited to, single network servers, server groups composed of multiple network servers, or clouds composed of a large number of hosts or network servers based on cloud computing.

[0180] The network in which the computer devices are located includes, but is not limited to, the Internet, a wide area network, a metropolitan area network, a local area network, a virtual private network (VPN), and the like.

[0181] The above describes specific embodiments of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that in the embodiments and still achieve the desired result. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In some embodiments, multi-task processing and parallel processing are possible or advantageous.

[0182] The division of steps in the above methods is only for the purpose of clear description. When implemented, a step can be combined or some steps can be split and decomposed into multiple steps, as long as the same logical relationship is included, which is within the protection scope of the present patent. Adding insignificant modifications or introducing insignificant designs in the algorithm or process, but not changing the core design of the algorithm and process, are within the protection scope of the present application.

[0183] While this specification contains many specifics, these should not be construed as limitations on the scope of any invention, but rather as descriptions of particular implementations of particular embodiments. Certain features that are, for clarity, described above and below in the context of separate embodiments, can also be provided in combination in a single embodiment. Conversely, various features that are, for brevity, described above and below in the context of a single embodiment, can also be provided separately or in any suitable subcombination. In addition, while features can be described above and below as being implemented in one manner, other implementations are also possible. For example, features can be implemented using hardware, software, or both. Further, the disclosure of these features in the context of particular embodiments should not be taken to mean that the features are necessarily limited to those embodiments. Accordingly, no limitation is placed on the scope of any of the claims, other than those implied by the specific language of the claims.

[0184] described in connection with the described embodiments or examples. The illustrative examples set forth herein are by way of example and not limitation. The description of the embodiments or examples does not preclude additional, different, or fewer embodiments or examples from being contemplated. Moreover, the description of the embodiments or examples does not preclude additional, different, or fewer embodiments or examples from being contemplated. Furthermore, the description of the embodiments or examples does not preclude additional, different, or fewer embodiments or examples from being contemplated. Moreover, the description of the embodiments or examples does not preclude additional, different, or fewer embodiments or examples from being contemplated.

[0185] Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the features of the disclosure as set forth herein. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.

[0186] It is to be understood that the application is not limited to the precise details of construction described above and illustrated in the accompanying drawings. The scope of the application is to be determined by the appended claims and their equivalents.

[0187] The above description is intended to be illustrative and not restrictive. Many other modifications within the scope of the application will be apparent to those of skill in the art upon reviewing the above description. The scope of the application should, therefore, be determined not with reference to the above description, but instead with reference to the appended claims, along with their full scope of equivalents.

Claims

1. A method for light virtual-real alignment in a virtual shooting system, the virtual shooting system comprising a display screen, a light fixture cluster, a light fixture controller, a master control host deployed with a master control terminal, and a shooting device; the master control host is capable of communicating with the light fixture controller, and the light fixture controller is connected with the light fixture cluster; the method is applied to an operation client deployed in a mobile device, and the operation client is capable of communicating with the master control terminal; the method comprises: displaying a shooting picture taken by the shooting device in a user interface; the shooting picture comprises an actual scene picture taken by the shooting device and a virtual scene picture displayed by the display screen; after detecting that a user selects a range to be aligned from the shooting picture, notifying the master control terminal, so that in a case where initial color values between two specified ranges in the range to be aligned are not matched, the master control terminal controls the light fixture cluster to perform light adjustment, and after the adjustment, the adjusted color values between the two specified ranges are matched; wherein the two specified ranges comprise an actual specified range belonging to the actual scene picture and a virtual specified range belonging to the virtual scene picture; wherein the master control terminal is configured to: establish a mapping relationship between a two-dimensional shooting picture and a three-dimensional actual scene, determine a position of the actual specified range in the three-dimensional actual scene based on the mapping relationship, and determine target light fixtures that have an influence on the light of the actual specified range in combination with installation positions and illumination angles of the light fixtures in the light fixture cluster; obtain three-dimensional structure information of the actual scene by using a three-dimensional reconstruction technology, and perform three-dimensional light simulation in combination with three-dimensional positions and light parameters of the light fixtures, to obtain actual color values of the actual specified range by simulation; for the target light fixtures, simulate the influence of the light of the target light fixtures, and solve light adjustment parameters that make a color difference between virtual color values of the virtual specified range and the simulated actual color values less than a preset threshold.

2. The method of claim 1, wherein, The method further comprises: displaying a selection control in the user interface, the selection control being configured to allow a user to select the range to be aligned; in response to detecting that the user triggers the selection control, determining the range to be aligned selected by the user according to a position of the selection control in the shooting picture. 3.The method of claim 2, wherein the displaying the selection control in the user interface comprises: identifying actual objects included in the actual scene picture and virtual objects included in the virtual scene picture, respectively; from the identified actual objects and virtual objects, determining target actual objects and target virtual objects that are of a same type and have a same image position in the shooting picture; based on the image positions of the target actual objects and target virtual objects in the shooting picture, respectively, displaying the selection control; wherein the selection control comprises a selection control representing the target actual objects and a selection control representing the target virtual objects. 4.The method of claim 1, further comprising: In response to detecting that the initial color values between the two specified ranges match, output a prompt message prompting that the virtual and real light between the two specified ranges has been aligned.

5. The method of claim 1, before the step of notifying the host, the method further comprises: identifying whether the image positions of the actual specified range and the virtual specified range in the captured picture match, obtaining a first identification result; identifying whether the image contents of the actual specified range and the virtual specified range in the captured picture match, obtaining a second identification result; if the first identification result is not matched or the second identification result is not matched, output a prompt message prompting whether the to-be-aligned range is selected incorrectly.

6. The method of claim 1, wherein, Whether the initial color values between the two specified ranges match is determined according to color values of each pixel contained in the actual specified range obtained from the captured picture and color statistics values calculated therefrom, and color values of each pixel contained in the virtual specified range obtained from the captured picture and color statistics values calculated therefrom.

7. The method of claim 1, the host is configured with a video capture card, the video capture card is connected with the shooting device through a transmission line; the video capture card is used to receive Raw data collected by the shooting device through the transmission line; the captured picture captured by the shooting device is obtained by the following way: receive the encoded data of the Raw data obtained by encoding sent by the host, to obtain the captured picture captured by the shooting device.

8. A light virtual-real alignment method in a virtual shooting system, the virtual shooting system comprising a display screen, a lamp cluster, a lamp controller, a host computer deployed with a host terminal, and a shooting device; the host computer can communicate with the lamp controller, and the lamp controller is connected with the lamp cluster; the method is applied to the host terminal, and the host terminal can communicate with an operation client deployed in a mobile device; the method comprises: receiving a to-be-aligned range in a captured picture sent by the operation client; the captured picture contains an actual scene picture captured by the shooting device and a virtual scene picture displayed by the display screen; the to-be-aligned range contains two specified ranges: an actual specified range belonging to the actual scene picture and a virtual specified range belonging to the virtual scene picture; the initial color values between the two specified ranges are not matched; based on the to-be-aligned range, controlling the lamp cluster to perform light adjustment through the lamp controller, so that the adjusted color values between the two specified ranges are matched; wherein, the light adjustment parameters of the lamp cluster are obtained by the following way: establishing a mapping relationship between a two-dimensional captured picture and a three-dimensional actual scene, based on the mapping relationship, determining the position of the actual specified range in the two-dimensional captured picture mapped to the three-dimensional actual scene, and combining the installation positions and illumination angles of each lamp in the lamp cluster, determining each target lamp that has an impact on the illumination of the actual specified range; Three-dimensional reconstruction technology is used to obtain three-dimensional structure information of an actual scene, and three-dimensional lighting simulation is performed in combination with three-dimensional positions and light parameters of lamps and lanterns to simulate actual color values in a specified range. Color difference between virtual color values in the virtual specified range and the simulated actual color values is minimized as an optimization objective, lighting adjustment parameters are solved for each target lamp and lantern to make the color difference less than a preset threshold.

9. The method of claim 8, wherein, Whether the initial color values in the two specified ranges match is determined according to color values of each pixel in the actual specified range obtained from the captured image and color statistics, and color values of each pixel in the virtual specified range obtained from the captured image and color statistics.

10. The method of claim 8, wherein the host computer is configured with a video capture card, and the video capture card is connected with the shooting device through a transmission line; the video capture card is configured to receive the Raw data collected by the shooting device through the transmission line. The method further comprises: encoding the Raw data to obtain encoded data and sending the encoded data to the operation client, so that the operation client obtains a captured image captured by the shooting device.

11. The method of claim 8, the host machine having Unreal Engine deployed thereon, the host end comprising: A plug-in program developed based on a software development kit provided by the Unreal Engine and running in the Unreal Engine.

12. A virtual shooting system, comprising a display screen, a lamp cluster, a lamp controller, a host computer deploying a host terminal, a shooting device and an operation client deployed in a mobile device; the host computer is capable of communicating with the lamp controller, the lamp controller is connected with the lamp cluster; the host terminal and the operation client deployed in the mobile device are capable of communicating with each other. The operation client is configured to execute steps of the method of any one of claims 1 to 7; and the host terminal is configured to execute steps of the method of any one of claims 8 to 11.

13. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein, The processor executes the computer program to implement steps of the method of any one of claims 1 to 11.

14. A computer program product comprising a computer program, which, when executed by a processor, implements steps of the method of any one of claims 1 to 11.

15. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements steps of the method of any one of claims 1 to 11.

Images