Image processing method and system for video production

Abstract

The present application relates to an image processing method and system for augmented reality in video production, wherein the method includes: obtaining the collected position information and video information; synchronizing each frame of the video information with the position information to obtain a real scene; synthesizing the real scene with the pre-built virtual scene to obtain the final synthesized video data. By adopting the above processing process, a Solo Go augmented reality (AR) image production tool is realized. The video image processing in the production process is performed by the realized graphic production tool, so as to achieve the purpose of augmented reality live broadcast, and the realized production tool does not need to use expensive hardware equipment to realize it, which greatly reduces the cost investment.

Description

Technical Field

[0001] The present disclosure relates to the technical field of live video production, and in particular to an image processing method and system for video production. Background Art

[0002] At present, in the field of video production technology, in order to improve the realism of the produced videos, high-definition live broadcast equipment is generally used to ensure that the final video is highly matched with the real scene. However, the equipment required for the current high-definition augmented reality live broadcast is expensive and inconvenient to carry. Although the use of ordinary mobile augmented reality software solves the inconvenience of using expensive hardware equipment, it has the problem of being unable to customize models and animations, resulting in the final video not matching the real scene with high accuracy. Summary of the invention

[0003] In view of this, the present disclosure proposes an image processing method for video production, which effectively improves the reality enhancement effect during the image processing process, so that the produced video highly restores the real scene.

[0004] According to one aspect of the present disclosure, there is provided an image processing method for augmented reality in video production, comprising:

[0005] Obtain the collected location information and video information;

[0006] Synchronize each frame of the video information with the position information to obtain a real scene;

[0007] The real scene is synthesized with the pre-built virtual scene to obtain final synthesized video data.

[0008] In a possible implementation, synchronizing each frame of the image in the video information with the position information includes performing a corresponding operation on each position data in the position information and each frame of the image.

[0009] In a possible implementation, the real scene includes each frame of image in the video information and position data corresponding to each frame of image.

[0010] In a possible implementation manner, the location information is acquired through a mobile terminal, and the video information is acquired through an image acquisition device.

[0011] In a possible implementation, the synthesis of the real scene and the pre-built virtual scene includes:

[0012] Acquire each frame image in the virtual scene;

[0013] Each frame image is associated with each position information in the real scene.

[0014] In a possible implementation, the synthesis of the real scene and the pre-built virtual scene includes:

[0015] Marking virtual objects that need to be displayed in the virtual scene;

[0016] The real scene is used as a background board, and unmarked virtual objects in the virtual scene are deleted.

[0017] According to another aspect of the present application, there is also provided an image processing system for augmented reality in video production, comprising an information acquisition module, a synchronization module and a synthesis module;

[0018] The information acquisition module is configured to acquire the collected location information and video information;

[0019] The synchronization module is configured to synchronize each frame of the video information with the position information to obtain a real scene;

[0020] The synthesis module is configured to synthesize the real scene with the pre-built virtual scene to obtain final synthesized video data.

[0021] In a possible implementation, it also includes a location information acquisition module and a video information acquisition module;

[0022] The location information collection module is configured to collect the location information;

[0023] The video information acquisition module is configured to acquire the video information.

[0024] According to another aspect of the present application, there is also provided an image processing device for augmented reality in video production, comprising:

[0025] processor;

[0026] a memory for storing processor-executable instructions;

[0027] Wherein, the processor is configured to implement any of the above-mentioned methods when executing the executable instructions.

[0028] According to another aspect of the present application, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, any of the above-mentioned methods is implemented.

[0029] The image processing method of the embodiment of the present application realizes a Solo Go augmented reality (AR) image production tool by adopting the above-mentioned processing process. The video image processing in the production process is performed by the realized graphic production tool, thereby achieving the purpose of augmented reality live broadcast. Moreover, the realized production tool does not require the use of expensive hardware equipment, which greatly reduces the cost investment.

[0030] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0032] Figure 1 A flowchart showing an image processing method according to an embodiment of the present application;

[0033] Figure 2 A schematic diagram showing a flow chart of an image processing method according to another embodiment of the present application;

[0034] Figure 3 A structural block diagram of an image processing system according to an embodiment of the present application is shown;

[0035] Figure 4 A structural block diagram of an image processing device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0036] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0037] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0038] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the main purpose of the present disclosure.

[0039] First of all, it should be noted that the image processing method of the embodiment of the present application is mainly used for image processing of the video production process in the live broadcast field. The method of the embodiment of the present application is used to enhance the realism of the produced video, making the video data integrated with the virtual scene more realistic.

[0040] Figure 1 FIG. 1 is a flowchart of an image processing method for augmented reality in video production according to an embodiment of the present disclosure. Figure 1 As shown, the method includes: step S1100, obtaining the collected location information and video information. Here, it should be pointed out that the obtained location information refers to the geographical location of the collected video data, and the location information may include multiple geographical locations, each geographical location corresponds to the specific location of different frame images in the video data, and each geographical location can be represented by longitude and latitude coordinates. The video information refers to the collected video data.

[0041] Step S1200, synchronizing each frame of the video information with the position information to obtain a real scene. That is, in the method of the embodiment of the present application, by synchronizing the acquired position information with the collected video information, it is ensured that the final real scene can have accurate geographical location information, thereby facilitating the subsequent synthesis of the virtual scene and the real scene.

[0042] Step S1300, synthesize the synchronized real scene with the pre-built virtual scene to obtain the final synthesized video data. Here, it should be noted that the virtual scene is pre-built. In a possible implementation, the virtual scene can build the required virtual objects in the editing scene as needed, and design the required animation to meet the needs of real production.

[0043] Therefore, the image processing method of the embodiment of the present application realizes a SoloGo augmented reality (AR) image production tool by adopting the above-mentioned processing process. The video image processing in the production process is performed by the realized graphic production tool, thereby achieving the purpose of augmented reality live broadcast. Moreover, the realized production tool does not require the use of expensive hardware equipment, which greatly reduces the cost investment.

[0044] It should be noted that, since it is necessary to obtain location information and video information, in a possible implementation, the location information can be directly collected through a mobile terminal device, such as a mobile phone, a tablet computer, etc. The video information can also be directly collected using an existing image collection device, such as a camera.

[0045] Furthermore, in the method of the embodiment of the present application, when synthesizing the synchronized live scene with the pre-built virtual scene, the Unreal Engine can be used directly to mix the incoming real scene with the virtual objects in the virtual scene. At the same time, the powerful rendering capability of the Unreal Engine can be used to render the mixed video data to further enhance the picture.

[0046] That is to say, in the image processing method of the present application, by using a relatively cheap, easy-to-buy, and easy-to-operate mobile terminal device (such as an iOS mobile device), or by adding an additional SLR camera as a collection terminal to collect accurate location information and high-definition video data, and then transmitting the collected location information and video data to Unreal Engine through a video transmission protocol. Then, the incoming location information and video data are synchronized through Unreal Engine, and then the synchronized real scene and virtual scene are synthesized to meet the real production needs.

[0047] Among them, those skilled in the art can understand that when synchronizing the collected video data with the position information, specifically each frame image in the video data is matched with each position data in the position information, so that each frame image is marked with accurate position data.

[0048] In a possible implementation, when the position information of each frame of the video data is matched, it can be done based on the factor of time information, that is, the image data marked at the same time are matched with the position data.

[0049] For details, see Figure 2 In the method of the embodiment of the present application, the time information of each frame position information and each frame video data collected can be injected through the Network Time Protocol (NTP). In one possible implementation, when injecting time into each frame position information and the image in each frame video data, it can be implemented by directly adding the timestamp corresponding to the currently determined time as an attribute assignment. Then, the position information and video data input of the injected time information are synchronized to obtain the synchronized real scene. Among them, when synchronizing the position information and video data of the injected time information, it can be implemented in the following way.

[0050] In this application, since the video data is collected by an additional SLR camera, in order to use the video information collected by the SLR, the location information collected by the mobile device needs to be synchronized with the video information collected by the SLR device. During synchronization, first, the two ends (i.e., the video information collection end and the location information collection end) are connected to the same NTP server, and the time difference between the video information collection end and the location information collection end and the NTP server can be calculated respectively (i.e., the time difference Δs between the video information collection end and the NTP, and the time difference Δw between the location information collection end and the NTP server).

[0051] After calculating the time difference between the two ends and the NTP server, the information sending time of the two end servers can be corrected according to the time difference between the two ends and the NTP server, so as to ensure that the two ends can send accurate NTP time code when sending each frame of information.

[0052] After receiving the video information and location information sent by both ends and the corresponding NTP time code, the PC side determines the incremental frame number of the video information and location information corresponding to the NTP time code received at the current moment, and compares the incremental frame numbers determined by both ends.

[0053] Here, it should be pointed out that the increasing frame number refers to the number corresponding to each frame of information. For example, for a video information acquisition end, a set of video data is sent at time a, and the video data contains multiple frames of images, each frame of image has a different number in sequence, wherein each number corresponds to the sequence of each frame of image in the video data.

[0054] At the same time, when determining the incremental frame number of the video information and position information corresponding to the NTP time code, an algorithm provided in the Unreal Engine can be used to implement it.

[0055] For example, when determining the incremental frame number of the video information and position information corresponding to the NTP time code received at the current moment, the video information acquisition end is end A, and the position information acquisition end is end B. End A sends a set of video data to the PC end at time point a0, and end B also sends a set of position data to the PC end at time point a0. Among them, since the transmission protocols and transmission methods adopted by ends A and B may be different, the time when the PC end receives the video data and the position data may be different. That is, the time when the PC end receives the video data and the position data may be different, and the video data and the position data need to be synchronized at this time. During synchronization, the incremental frame number (that is, the number of the last frame image during this period) and the incremental frame number in the position data (that is, the number of the last position information during this period) received by the PC end during the period from the NTP time code to the current moment are obtained, and then the incremental frame numbers corresponding to the two ends are compared.

[0056] If at a certain moment, the number of incremental frames on one end is greater than that on the other end, it means that the data (image or position) transmitted by the end with the larger number of incremental frames is fresher and needs to be cached and then wait for the number of incremental frames on the other end to increase to the same level. Finally, the matching dual-end signals are submitted to achieve dual-end signal synchronization.

[0057] Here, it should also be pointed out that the real scene obtained after synchronization includes each frame of image in the video information and the position data corresponding to each frame of image.

[0058] In a possible implementation, when synchronizing the collected position information and video information, in the process of using the NTP server to achieve synchronization, it is also possible to directly send the time code that has not been corrected by NTP to the PC end, and then perform NTP correction after the frame data is sent to the PC end, instead of performing NTP time correction at the signal sending end. Here, it should be pointed out that the method of correcting the NTP time at the PC end is the same or similar to the method of correcting at the signal sending end described above, and will not be repeated here.

[0059] Furthermore, after the PC side obtains the synchronized real scene (i.e., position and video information), AR synthesis can be performed. That is, the synchronized real scene is synthesized with the pre-built virtual scene. Among them, when synthesizing the real scene and the virtual scene, first, each frame of virtual image in the virtual scene is obtained, and then the virtual objects in each frame of virtual image are associated with each position information in the real scene. As a result, the virtual objects that need to be displayed in each frame of virtual image in the virtual scene can be consistent with each frame of image in the real scene. This is because the synchronized real scene is usually a video composed of multiple frames of images. During the playback of the video, the image is displayed frame by frame. In order to keep each frame of image and the virtual object in the corresponding virtual scene in synchronization and the display position can also be synchronized to ensure that the synthesized picture is more realistic, it is necessary to synchronize the virtual objects in each frame of virtual image in the virtual scene with the transmitted tracking information and performance to achieve the AR effect.

[0060] In a possible implementation, associating the position of a virtual object in each frame of a virtual image in a virtual scene with the position of each frame of an image in a real scene may be achieved in the following manner.

[0061] First, determine the origin in the real scene. The origin in the real scene can be set manually and can be flexibly selected according to the actual situation. That is, a certain pixel point in the current frame image in the real scene can be determined as the coordinate origin according to the actual situation. Then, the relative position of the real camera (i.e., the video information acquisition end) and the origin is matched to the relative position of the virtual camera and the origin of the virtual scene in the current frame virtual image of the virtual scene. In this way, the relative position of the virtual object to be displayed in the current frame virtual image of the virtual scene in the real scene is determined, and then the position information of the virtual objects in the real scene and the virtual scene are associated.

[0062] The determination of the coordinate origin of the current frame virtual image of the virtual scene is the same or similar to the determination of the origin of the current frame image of the real scene, and will not be repeated here.

[0063] Furthermore, when synthesizing the real scene with the virtual scene, when aligning the position of the virtual object to be displayed in each frame image of the virtual scene with each frame image in the real scene, it is necessary to first determine the virtual object to be displayed. That is, first determine the mark of the virtual object to be displayed. The mark of the virtual object to be displayed can be achieved in the following manner.

[0064] That is, when the real scene and the pre-built virtual scene are synthesized, it also includes: an operation of displaying the virtual objects that need to be displayed in the virtual scene in the real scene.

[0065] That is, when synthesizing a virtual scene with a real scene, it is necessary to display the virtual objects in the virtual scene to be displayed, and use the actual video screen as the background board, and then subtract the virtual screen that does not need to be displayed, and the subtracted part will display the actual video screen as the background board.

[0066] The specific implementation method is: the Unreal Engine software can be used for the above synthesis. In the synthesis material, the CustomDepth (custom depth) of the entire screen (that is, the final output synthesis screen) is used to subtract the SceneDepth (scene depth). The CustomDepth (custom depth) value of the entire screen is infinite. If there is a virtual object in the screen, the SceneDepth (scene depth) of the part of the virtual screen where the virtual object is located is not infinite, and the value of CustomDepth (custom depth) minus SceneDepth (scene depth) of this part of the screen is still not infinite. The part of the virtual screen where there is no virtual object has SceneDepth (scene depth) that is also infinite, and the value of CustomDepth (custom depth) minus SceneDepth (scene depth) of this part of the screen is 0. Therefore, the part of the screen where the value of CustomDepth (custom depth) minus SceneDepth (scene depth) is 0 is displayed as the video screen, and the part greater than 1 is displayed as the virtual screen, so as to achieve the AR effect. .

[0067] That is to say, in the method of the present application, by using the Unreal Engine software, the above two parameters (i.e., CustomDepth and SceneDepth) are set for the virtual objects in each frame of the virtual image in the virtual scene, thereby realizing the configuration of whether the virtual objects are realistic. The operation is simple and convenient, and easy to implement.

[0068] Here, those skilled in the art can understand that when synthesizing the real scene with the virtual scene, it means synthesizing each frame image in the real scene with each frame image in the virtual scene. That is to say, the current frame image is obtained from the real scene, and the current virtual image is obtained from the virtual scene. Then, the current frame image in the obtained real scene is used as the background board, and the virtual screen that does not need to be displayed in the current frame virtual image is deleted, so that only the virtual objects that need to be displayed in the current frame virtual image can be displayed in the current frame image as the background board.

[0069] It should be pointed out that when synthesizing each frame image in the real scene with the virtual scene, each frame image in the real scene can be acquired frame by frame in sequence, or can be flexibly selected according to actual conditions.

[0070] It should be noted that although Figure 2 The image processing method for augmented reality of the present application is introduced as an example, but those skilled in the art will appreciate that the present disclosure should not be limited thereto. In fact, the user can flexibly set the implementation method of each step according to personal preferences and / or actual application scenarios, as long as it can achieve lightweight, high-definition and greatly free design space for augmented reality image processing.

[0071] Correspondingly, based on any of the above-mentioned image processing methods for augmented reality in video production, the present application also provides an image processing system for augmented reality in video production. Since the principle of the image processing system for augmented reality provided by the present application is the same or similar to the principle of the image processing method of the present application, the repeated parts will not be repeated.

[0072] See also Figure 3 The image processing system 100 for augmented reality provided by the present application includes: an information acquisition module 110, a synchronization module 120 and a synthesis module 130. Among them, the information acquisition module 110 is configured to obtain the collected position information and video information. The synchronization module 120 is configured to synchronize each frame of the video information with the position information to obtain a real scene. The synthesis module 130 is configured to synthesize the real scene with the pre-built virtual scene to obtain the final synthesized video data.

[0073] In a possible implementation, it further includes a position information acquisition module and a video information acquisition module (not shown in the figure). The position information acquisition module is configured to acquire position information; and the video information acquisition module is configured to acquire video information.

[0074] Furthermore, according to another aspect of the present disclosure, an image processing device 200 for augmented reality is also provided. Figure 4 The image processing device 200 for augmented reality according to the embodiment of the present disclosure includes a processor 210 and a memory 220 for storing executable instructions of the processor 210. The processor 210 is configured to implement any of the above-mentioned image processing methods for augmented reality when executing the executable instructions.

[0075] Here, it should be noted that the number of processors 210 may be one or more. At the same time, in the image processing device 200 for augmented reality in the embodiment of the present disclosure, an input device 230 and an output device 240 may also be included. Among them, the processor 210, the memory 220, the input device 230 and the output device 240 may be connected through a bus or in other ways, which are not specifically limited here.

[0076] The memory 220 is a computer-readable storage medium that can be used to store software programs, computer executable programs, and various modules, such as the program or module corresponding to the image processing method for augmented reality in the embodiment of the present disclosure. The processor 210 executes various functional applications and data processing of the image processing device 200 for augmented reality by running the software programs or modules stored in the memory 220.

[0077] The input device 230 may be used to receive input numbers or signals. The signals may be key signals related to user settings and function control of the device / terminal / server. The output device 240 may include display devices such as display screens.

[0078] According to another aspect of the present disclosure, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by the processor 210, any of the above-mentioned image processing methods for augmented reality is implemented.

[0079] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The choice of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. An image processing method for augmented reality in video production, characterized in that: include: Obtain the collected location information and video information; Synchronize each frame of the video information with the position information to obtain a real scene; Synthesize the real scene with the pre-built virtual scene to obtain final synthesized video data; When synchronizing each frame image in the video information with the position information, it includes performing corresponding operations on each position data in the position information and each frame image; The real scene includes each frame of the video information and the position data corresponding to each frame of the image; that is, when the collected video data and the position information are synchronized, each frame of the video data is specifically matched with each position data in the position information, so that each frame of the image is marked with accurate position data; When the position information of each frame of the video data is matched, the image data marked at the same time are matched with the position data; Specifically, the time information of each frame position information and each frame video data collected is injected through the Network Time Protocol; when injecting time into each frame position information and each frame video data image, the timestamp corresponding to the current determined time is directly added as an attribute assignment to achieve it; then, the position information and video data input of the injected time information are synchronized to obtain the synchronized real scene; When synchronizing the location information and video data injected with time information, first, the two ends, namely the video information acquisition end and the location information acquisition end, are connected to the same NTP server, and the time difference between the video information acquisition end and the location information acquisition end and the NTP server can be calculated respectively, namely, the time difference Δs between the video information acquisition end and the NTP, and the time difference Δw between the location information acquisition end and the NTP server; After calculating the time difference between the two ends and the NTP server, the information sending time of the two end servers is corrected according to the time difference between the two ends and the NTP server, so as to ensure that the two ends can send accurate NTP time code when sending each frame of information; After receiving the video information and location information sent by both ends and the corresponding NTP time code, the PC side determines the incremental frame number of the video information and location information corresponding to the NTP time code received at the current moment, and compares the incremental frame numbers determined by both ends; The increasing frame number refers to the number corresponding to each frame of information, and each number corresponds to the sequence of each frame of image in the video data; The location information is collected directly through a mobile terminal device; the video information is also collected directly through an image acquisition device, and the location information and the video information are collected using different device terminals respectively; When the real scene is synthesized with the pre-built virtual scene, it includes: Acquire each frame image in the virtual scene; Associating each frame of image with each position information in the real scene; When the real scene is synthesized with the pre-built virtual scene, it includes: Marking virtual objects that need to be displayed in the virtual scene; Using the real scene as a background, and deleting unmarked virtual objects in the virtual scene; When synthesizing the synchronized on-site scene with the pre-built virtual scene, the virtual engine directly mixes the input real scene with the virtual objects in the virtual scene.

2. An image processing system for augmented reality in video production, characterized in that: It includes information acquisition module, synchronization module and synthesis module; The information acquisition module is configured to acquire the collected location information and video information; The synchronization module is configured to synchronize each frame of the video information with the position information to obtain a real scene; When synchronizing each frame image in the video information with the position information, it includes performing corresponding operations on each position data in the position information and each frame image; When the position information of each frame of the video data is matched, the image data marked at the same time are matched with the position data; Specifically, the time information of each frame position information and each frame video data collected is injected through the Network Time Protocol; when injecting time into each frame position information and each frame video data image, the timestamp corresponding to the current determined time is directly added as an attribute assignment to achieve it; then, the position information and video data input of the injected time information are synchronized to obtain the synchronized real scene; When synchronizing the location information and video data injected with time information, first, the two ends, namely the video information acquisition end and the location information acquisition end, are connected to the same NTP server, and the time difference between the video information acquisition end and the location information acquisition end and the NTP server can be calculated respectively, namely, the time difference Δs between the video information acquisition end and the NTP, and the time difference Δw between the location information acquisition end and the NTP server; After calculating the time difference between the two ends and the NTP server, the information sending time of the two end servers is corrected according to the time difference between the two ends and the NTP server, so as to ensure that the two ends can send accurate NTP time code when sending each frame of information; After receiving the video information and location information sent by both ends and the corresponding NTP time code, the PC side determines the incremental frame number of the video information and location information corresponding to the NTP time code received at the current moment, and compares the incremental frame numbers determined by both ends; The increasing frame number refers to the number corresponding to each frame of information, and each number corresponds to the sequence of each frame of image in the video data; The synthesis module is configured to synthesize the real scene with the pre-built virtual scene to obtain final synthesized video data; The real scene includes each frame of the video information and the position data corresponding to each frame of the image; that is, when the collected video data and the position information are synchronized, each frame of the video data is specifically matched with each position data in the position information, so that each frame of the image is marked with accurate position data; Among them, it also includes a location information acquisition module and a video information acquisition module; The location information collection module is configured to collect the location information; The video information acquisition module is configured to acquire the video information; The location information acquisition module and the video information acquisition module are implemented using different device terminals respectively; When the real scene is synthesized with the pre-built virtual scene, it includes: Acquire each frame image in the virtual scene; Associating each frame of image with each position information in the real scene; When the real scene is synthesized with the pre-built virtual scene, it includes: Marking virtual objects that need to be displayed in the virtual scene; Using the real scene as a background, and deleting unmarked virtual objects in the virtual scene; When synthesizing the synchronized on-site scene with the pre-built virtual scene, the virtual engine directly mixes the input real scene with the virtual objects in the virtual scene.

3. An image processing device for augmented reality in video production, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to implement the method of claim 1 when executing the executable instructions.

4. A non-volatile computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method of claim 1 is implemented.

Images