Video generation method, device, computer equipment and storage medium

Abstract

The embodiments of the present application disclose a video generation method, apparatus, computer equipment and storage medium; the embodiments of the present application can capture a video of a target object in a real scene, wherein the video includes at least one video frame; object detection processing is performed on the video frames in the video to obtain a target object image corresponding to each video frame; positioning processing is performed on the video according to the three-dimensional space information of a preset virtual scene video to obtain the captured posture information of the video, wherein the preset virtual scene video is generated based on a preset scene video, the preset scene video is obtained by capturing a video of a real scene, and the three-dimensional space information of the preset virtual scene video is obtained by performing three-dimensional space construction processing on the preset scene video; the target object image corresponding to each video frame and the preset virtual scene video are fused according to the captured posture information to obtain a target fused video, which can reduce the cost of generating a virtual fused video.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a video generation method, apparatus, computer equipment and storage medium. Background Art

[0002] A video that combines virtual and real worlds can refer to a completely new video that is obtained by combining a virtual world with the real world. A video that combines virtual and real worlds uses immersive technologies such as augmented reality or augmented virtuality, and combines the two together to allow physical and digital objects to coexist and interact in real time. For example, a video that combines virtual and real worlds can perfectly integrate real-life portraits into virtual scenes, thus presenting a natural and lifelike video. In related technologies, a video that combines virtual and real worlds is generally obtained by building a green screen environment and a follower to combine the real world and the virtual world, which increases the cost of generating virtual fusion videos. Summary of the invention

[0003] The embodiments of the present application propose a video generation method, apparatus, computer device and storage medium, which can generate a virtual-real fusion video without the help of a green screen environment and a follower, thereby reducing the cost of generating a virtual fusion video.

[0004] The present application provides a video generation method, including:

[0005] Capturing a video of a target object in a real scene, wherein the video includes at least one video frame;

[0006] Perform object detection processing on the video frames in the video to obtain the target object image corresponding to each video frame;

[0007] According to the three-dimensional space information of the preset virtual scene video, the video is positioned to obtain the acquisition posture information of the video, wherein the preset virtual scene video is generated based on the preset scene video, the preset scene video is obtained by performing video acquisition on the real scene, and the three-dimensional space information of the preset virtual scene video is obtained by performing three-dimensional space construction processing on the preset scene video;

[0008] The target object image corresponding to each video frame and the preset virtual scene video are fused according to the collected posture information to obtain a target fused video.

[0009] Accordingly, the embodiment of the present application further provides a video generating device, including:

[0010] A collection unit, used to collect a video of the target object in a real scene, wherein the video includes at least one video frame;

[0011] An object detection unit is used to perform object detection processing on video frames in the video to obtain a target object image corresponding to each video frame;

[0012] A positioning unit, configured to perform positioning processing on a preset virtual scene video according to the three-dimensional space information of the video, and obtain acquisition posture information of the video, wherein the preset virtual scene video is generated based on a preset scene video, the preset scene video is obtained by performing video acquisition on the real scene, and the three-dimensional space information of the preset virtual scene video is obtained by performing three-dimensional space construction processing on the preset scene video;

[0013] A fusion unit is used to fuse the target object image corresponding to each video frame with the preset virtual scene video according to the collected posture information to obtain a target fused video.

[0014] In one embodiment, the positioning unit may include:

[0015] A parsing subunit, configured to parse and process the three-dimensional space information of the preset virtual scene video to obtain at least one map point;

[0016] A matching subunit, used to match the video frame of the video with the at least one map point respectively to obtain a map point matching result of each video frame;

[0017] The mapping subunit is used to map the position and posture information of the map point corresponding to each video frame to the corresponding video frame according to the map point matching result, so as to obtain the acquisition position and posture information of the video.

[0018] In one embodiment, the video generating device may include:

[0019] A video acquisition unit, configured to acquire a preset scene video of the real scene, wherein the preset scene video includes at least one video frame;

[0020] A three-dimensional space construction unit, configured to perform three-dimensional space construction processing on the video frame of the preset scene video to obtain three-dimensional space information of the real scene in the preset scene video;

[0021] A generating unit is used to generate the preset virtual scene video and the three-dimensional space information of the preset virtual scene video according to the three-dimensional space information of the real scene in the preset scene video.

[0022] In one embodiment, the three-dimensional space construction unit may include:

[0023] An initialization subunit, configured to perform a three-dimensional space initialization process on the preset scene video to obtain initial three-dimensional space information of the real scene in the preset scene video;

[0024] an error calculation unit, used to perform error calculation on the initial three-dimensional space information to obtain three-dimensional space error information;

[0025] A key frame detection unit, configured to perform key frame detection on the video frame of the preset scene video according to the three-dimensional space error information to obtain a key frame;

[0026] An updating unit is used to update the initial three-dimensional space information according to the key frame to obtain the three-dimensional space information of the real scene in the preset scene video.

[0027] In one embodiment, the initialization subunit may include:

[0028] A video frame determination module, used to determine an initial video frame in the video frames of the preset scene video;

[0029] A feature point detection module, used to perform feature point detection on the initial video frame to obtain feature point detection information;

[0030] A positioning mark detection module, used to perform positioning mark detection on the initial video frame to obtain positioning mark detection information;

[0031] A generation module is used to generate initial three-dimensional space information of the real scene according to the feature point detection information and the positioning mark detection information.

[0032] In one embodiment, the generating module may include:

[0033] A posture calculation submodule, used to perform posture calculation processing on the feature point detection information and the positioning mark detection information respectively to obtain feature point posture information and positioning mark posture information;

[0034] A triangulation submodule is used to perform triangulation processing on the feature point pose information and the positioning mark pose information respectively to obtain the feature point three-dimensional space information and the positioning mark three-dimensional space information;

[0035] A determination module is used to determine the initial three-dimensional space information from the feature point three-dimensional space information and the positioning mark three-dimensional space information.

[0036] In one embodiment, the key frame detection unit may include:

[0037] A positioning mark detection subunit, configured to perform positioning mark detection on a video frame of the preset scene video when the three-dimensional space error information does not meet a preset error threshold;

[0038] A first determining subunit is configured to determine the video frame as an identification video frame when a positioning mark is detected in the video frame of the preset scene video;

[0039] A redundancy determination subunit, used for performing redundancy determination processing on the identified video frame;

[0040] The second determining subunit is used to determine the identification video frame as the key frame when the identification video frame passes the redundancy determination process.

[0041] In one embodiment, the updating unit may include:

[0042] An acquisition subunit, configured to acquire a key frame set of the preset scene video, wherein the key frame set includes at least one historical key frame;

[0043] A third determining subunit, configured to determine a matching key frame that matches the key frame in the historical key frames;

[0044] A calculation subunit, used for calculating the posture information corresponding to the key frame according to the matching key frame;

[0045] An updating subunit is used to update the initial three-dimensional space information according to the posture information corresponding to the key frame to obtain the three-dimensional space information of the real scene in the preset scene video.

[0046] In one embodiment, the object detection unit may include:

[0047] A feature encoding subunit, used for performing feature encoding processing on the video frame to obtain a feature image of the video frame;

[0048] A region decoding subunit, configured to perform region decoding processing on the feature image of the video frame to obtain a target object region image of the video frame;

[0049] The region refinement subunit is used to perform region refinement processing on the target object region image according to the feature image of the video frame to obtain the target object image.

[0050] In one embodiment, the region refinement subunit may include:

[0051] A first fusion module, used for fusing the target object area image and the feature image of the video frame to obtain a fused feature image;

[0052] An information acquisition module, used to acquire feature coding information of the video frame;

[0053] A second fusion module is used to fuse the fused feature image and the feature coding information to obtain fused coding information;

[0054] A decoding module is used to decode the fused encoded information to obtain the target object image.

[0055] In one embodiment, the fusion unit may include:

[0056] A layer overlay subunit, used for performing layer overlay processing on the target object image of the video frame and the video frame of the preset virtual scene video to obtain at least one overlay layer;

[0057] A setting subunit, used for setting and processing preset video display parameters according to the collected posture information to obtain the set video display parameters;

[0058] The rendering subunit is used to render the at least one overlay layer based on the set video display parameters to obtain the target fused video.

[0059] In one embodiment, the layer overlay subunit may include:

[0060] A layer generation module, used for generating a first layer and a second layer by using a preset rendering engine;

[0061] A first rendering module, configured to render the target object features and the first layer using the preset rendering engine to obtain a first rendered layer;

[0062] A second rendering module, used for superimposing the preset virtual scene video and the second layer using the preset rendering engine to obtain a second rendered image;

[0063] The superposition module is used to superimpose the first rendered layer and the second rendered layer to obtain at least one superimposed layer.

[0064] The embodiment of the present application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the method provided in various optional modes of the above-mentioned aspect.

[0065] Correspondingly, an embodiment of the present application further provides a storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, any video generation method provided in the embodiments of the present application is implemented.

[0066] The embodiments of the present application can capture a video of a target object in a real scene, wherein the video includes at least one video frame; perform object detection processing on the video frames in the video to obtain a target object image corresponding to each video frame; perform positioning processing on the video according to the three-dimensional space information of a preset virtual scene video to obtain captured posture information of the video, wherein the preset virtual scene video is generated based on a preset scene video, the preset scene video is obtained by performing video capture on the real scene, and the three-dimensional space information of the preset virtual scene video is obtained by performing three-dimensional space construction processing on the preset scene video; perform fusion processing on the target object image corresponding to each video frame and the preset virtual scene video according to the captured posture information to obtain a target fused video; output the target fused video, so as to generate a virtual-real fused video without the aid of a green screen environment and a follower, thereby reducing the cost of generating a virtual fused video. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0068] Figure 1 is a scene schematic diagram of the video generation method provided in an embodiment of the present application;

[0069] Figure 2 It is a flowchart of a video generation method provided in an embodiment of the present application;

[0070] Figure 3 is another scene schematic diagram of the video generation method provided in an embodiment of the present application;

[0071] Figure 4 is another scene schematic diagram of the video generation method provided in an embodiment of the present application;

[0072] Figure 5 is another scene schematic diagram of the video generation method provided in an embodiment of the present application;

[0073] Figure 6 is another scene schematic diagram of the video generation method provided in an embodiment of the present application;

[0074] Figure 7 is another scene schematic diagram of the video generation method provided in an embodiment of the present application;

[0075] Figure 8 is another scene schematic diagram of the video generation method provided in an embodiment of the present application;

[0076] Fig. 9 is another scene schematic diagram of the video generation method provided in an embodiment of the present application;

[0077] Fig.10 is another scene schematic diagram of the video generation method provided in an embodiment of the present application;

[0078] Fig.11 It is another flowchart of the video generation method provided in the embodiment of the present application;

[0079] Fig.12 It is another flowchart of the video generation method provided in the embodiment of the present application;

[0080] Fig.13 It is another flowchart of the video generation method provided in the embodiment of the present application;

[0081] Fig.14 is a structural schematic diagram of a video generating device provided in an embodiment of the present application;

[0082] Fig.15 It is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0083] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. However, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0084] The embodiment of the present application proposes a video generation method, which can be executed by a computer device. The computer device may include at least one of a terminal and a server. That is, the video generation method proposed in the embodiment of the present application can be executed by a terminal, a server, or a terminal and a server that can communicate with each other.

[0085] Among them, the terminals may include but are not limited to smart phones, tablet computers, laptops, personal computers (PCs), smart home appliances, wearable electronic devices, VR / AR devices, vehicle-mounted terminals, intelligent voice interaction devices, etc.

[0086] The server can be an intercommunication server or background server between multiple heterogeneous systems, an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, as well as big data and artificial intelligence platforms, etc.

[0087] It should be noted that the embodiments of the present application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, etc.

[0088] In one embodiment, if Figure 1 The terminal 10 can obtain the three-dimensional space information of the preset virtual scene video generated by the server 11. Then, the terminal 10 can collect a video of the target object in the real scene, wherein the video includes at least one video frame; perform object detection processing on the video frames in the video to obtain the target object image corresponding to each video frame; perform positioning processing on the video according to the three-dimensional space information of the preset virtual scene video to obtain the acquisition posture information of the video, wherein the preset virtual scene video is generated based on the preset scene video, the preset scene video is obtained by performing video acquisition on the real scene, and the three-dimensional space information of the preset virtual scene video is obtained by performing three-dimensional space construction processing on the preset scene video; perform fusion processing on the target object image corresponding to each video frame and the preset virtual scene video according to the acquisition posture information to obtain the target fused video; and output the target fused video.

[0089] The following are detailed descriptions respectively. It should be noted that the description order of the following embodiments is not intended to limit the preferred order of the embodiments.

[0090] The embodiments of the present application will be described from the perspective of a computer device, which may be a server or a terminal.

[0091] like Figure 2 A video generation method is provided, and the specific process includes:

[0092] 101. Capture a video of a target object in a real scene, where the video includes at least one video frame.

[0093] In one embodiment, the present application proposes a method for generating a video that combines virtuality and reality. By using the method proposed in the present application, a target object in the real world and a preset virtual scene video can be combined. Therefore, a video of the target object in a real scene can be collected.

[0094] The target object may include an object that actually exists in the display. For example, the target object may include a person, an animal, a plant, or various objects, etc. For example, the target object may be a single person. For another example, the target object may be multiple people. For another example, the target object may be multiple animals, etc.

[0095] The real scene is a scene that actually exists in the real world. For example, the real scene may be a real open place or a closed place, etc. For example, the real scene may be a real football field in the real world. For another example, the real scene may be a real room in the real world, etc.

[0096] In one embodiment, there are multiple methods for capturing videos of the target object in a real scene.

[0097] For example, the computer device may be a terminal such as a mobile phone, a television, or a VR / AR device, wherein the mobile phone or television may include a video acquisition device such as a camera. When a user wants to generate a target fusion video of virtual-real fusion, the user may capture the video by activating the video acquisition device of the terminal. For example, when the terminal is a mobile phone, the user may capture the video of the target object in a real scene by activating the camera of the mobile phone.

[0098] For another example, the computer device may be a server. The server may send a video capture instruction to the terminal, and the terminal may capture a video of the target object in a real scene.

[0099] In one embodiment, in order to fuse the target object in the real world with the preset virtual scene video, the video of the target object in the real scene can be framed to obtain at least one video frame of the video. Then, object detection processing can be performed on the video frames in the video to obtain the target object image corresponding to each video frame, so that the target object in the real world and the preset virtual scene video can be fused.

[0100] There are many methods for performing frame processing on the video of the target object in the real scene. For example, the video of the target object in the real scene can be processed by video processing software to obtain video frames of the video. For another example, the video processing function in the computer vision software library such as OpenCV can be used to perform frame processing on the video of the target object in the real scene to obtain video frames of the video.

[0101] 102. Perform object detection processing on video frames in the video to obtain a target object image corresponding to each video frame.

[0102] In one embodiment, the video is acquired by capturing the target object in a real scene, and therefore, each video frame of the video may include two parts of content, namely, the real scene as background content, and the target object as foreground content.

[0103] In order to fuse the target object in the real world with the preset virtual scene video, object detection processing can be performed on the video frames in the video to obtain the target object image corresponding to each video frame.

[0104] The target object image may include an image that distinguishes the target object area and the non-target object area in the video frame. Figure 3 As shown, Figure 3 001 in the figure may be a schematic diagram of a video frame of a video. Figure 3 002 in the figure may be a target object image corresponding to the video frame after object detection processing is performed on the video frame.

[0105] In one embodiment, there are multiple methods for performing object detection processing on video frames in a video to obtain target object images corresponding to the video frames.

[0106] In one embodiment, a target object region image for a target object may be generated. Then, the target object region image is subjected to region refinement processing to obtain a target object image. Specifically, the step of "performing object detection processing on video frames in a video to obtain a target object image corresponding to each video frame" may include:

[0107] Perform feature encoding processing on the video frame to obtain a feature image of the video frame;

[0108] Performing regional decoding processing on the feature image of the video frame to obtain a target object region image of the video frame;

[0109] Performing region refinement processing on the target object region image to obtain the target object image.

[0110] The feature image of the video frame can be used to illustrate the features of the video frame. For example, the feature image can be used to illustrate the characteristics of the color, edge, geometry and texture of the video frame, etc.

[0111] The target object region image may be used to illustrate the region where the target object is located in the video frame.

[0112] For example, the target object area image can be a grayscale image, that is, the pixel values ​​of the pixels in the target object area image are all 255 or 0. Among them, when the pixel value of the pixel in the target object area image is 255, the color presented by the pixel is white. And when the pixel value of the pixel in the target object area image is 0, the color presented by the pixel is black. For example, for the target object area image, the pixel value of the target object area in the target object area image can be 0, and the pixel value of the non-target object area in the target object area image can be 255, so that the target object area image can distinguish the target object area from the non-target object area.

[0113] For example, Figure 3 As shown, Figure 3 001 in can be a video frame, the foreground content of the video frame can be the target object, and the background content can be a real scene. Figure 3 As shown, Figure 3 003 in the figure may be a target object region image, through which the target object region and the non-target object region can be distinguished.

[0114] Among them, compared with the target area image, the accuracy of the target object area image will be lower. For example, in the target object area image, the outline of the target object area will be relatively rough and cannot completely match the outline of the target object in the video frame. Compared with the target object area image, the outline of the target object area in the target object image will be much more detailed and fine.

[0115] In one embodiment, feature encoding processing may be performed on the video frame to obtain a feature image of the video frame. Specifically, the step of "feature encoding processing may be performed on the video frame to obtain a feature image of the video frame" may include:

[0116] Perform convolution processing on the video frame to obtain a convolution image of the video frame;

[0117] Performing pooling processing on the convolution image of the video frame to obtain a pooled convolution image;

[0118] The pooled convolution image is activated to obtain the feature image of the video frame.

[0119] In one embodiment, a convolution function may be used to convolve a video frame to obtain a convolution image of the video frame. The video frame may be convoluted multiple times using multiple different convolution functions to obtain a convolution image of the video frame, wherein different convolution functions may have different function parameters. For example, a first convolution function may be used to convolve a video frame to obtain a first convolution image, and then a second convolution function may be used to convolve the first convolution image to obtain a second convolution image. Then, a third convolution function may be used to convolve the second convolution image, and so on, to obtain a final convolution image of the video frame.

[0120] In one embodiment, the convolution image of the video frame may be pooled to obtain a pooled convolution image. For example, the convolution image of the video frame may be average pooled to obtain a pooled convolution image. For another example, the video frame may be max pooled to obtain a pooled convolution image.

[0121] In one embodiment, the pooled convolution image may be activated to obtain a feature image of the video frame. For example, the pooled convolution image may be activated using an activation function to obtain a feature image of the video frame. For example, the pooled convolution image may be activated using a Sigmoid function, a ReLU function, or a Tanh function to obtain a feature image of the video frame.

[0122] In one embodiment, after obtaining the feature image of the video frame, the feature image of the video frame may be subjected to region decoding processing to obtain the target object region of the video frame. The region decoding processing of the feature image of the video frame may refer to determining where the region of the target object in the video frame is based on the feature image of the video frame.

[0123] In one embodiment, the feature image of the video frame may be convolved to obtain convolution information of the feature image. Then, the convolution information of the feature image may be pooled to obtain pooled information of the feature image. Then, the pooled information of the feature image may be activated to obtain a target object area image of the video frame. For example, the feature image may be convolved using a plurality of different convolution functions to obtain convolution information of the feature image. Then, for example, the convolution image of the video frame may be averaged pooled to obtain pooled information. Next, for example, the pooled information may be activated using a Sigmoid function to obtain a target object area image of the video frame.

[0124] In one embodiment, since the accuracy of the target object region image is relatively low, the target object region image may be subjected to region refinement processing to obtain the target object image. Specifically, the step of "performing region refinement processing on the target object region image to obtain the target object image" may include:

[0125] The target object area image and the feature image of the video frame are fused to obtain a fused feature image;

[0126] Obtain feature encoding information of video frames;

[0127] The fused feature image and the feature coding information are fused to obtain fused coding information;

[0128] The fused encoded information is decoded to obtain the target object image.

[0129] In one embodiment, the target object area image and the feature image of the video frame may be fused to obtain a fused feature image. For example, the pixel values ​​of the target object area image and the pixel values ​​of the feature image of the video frame may be added to obtain a fused feature image. By fusing the target object area image and the feature image of the video frame, the fused feature image includes the features of the target object area. In order to further enhance the features of the target object area, feature coding information of the video frame may be obtained, wherein the feature coding information may be information generated when feature coding is performed on the video frame. For example, when feature coding is performed on the video frame, feature coding may be performed on the video frame multiple times to obtain a final feature image, and feature coding information may be obtained each time the feature coding is performed.

[0130] In one embodiment, the fused feature image and the feature coding information may be fused to obtain the fused coding information. For example, the pixel values ​​of the fused feature image and the feature coding information may be weighted and added to obtain the fused coding information. By fusing the fused features and the feature coding information, the amount of information in the fused coding information may be increased, thereby improving the accuracy of identifying the target object image. Then, the fused coding information may be decoded to obtain the target object image. For example, the fused coding information may be subjected to convolution operation, pooling operation and activation processing to obtain the target object image.

[0131] In one embodiment, an artificial intelligence model may be used to perform object detection processing on video frames in a video to obtain target object images corresponding to the video frames.

[0132] Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0133] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0134] For example, any one of the artificial intelligence models such as Convolutional Neural Networks (CNN), De-Convolutional Networks (DN), Deep Neural Networks (DNN), Deep Convolutional Inverse Graphics Networks (DCIGN), Region-based Convolutional Networks (RCNN), and Faster Region-based Convolutional Networks (Faster RCNN) can be used to perform object detection processing on video frames in the video to obtain target object images corresponding to the video frames.

[0135] For example, a segmentation neural network may be pre-trained and put online in a computer device. When object detection is performed on video frames in a video, the trained segmentation neural network may be used to perform object detection on the video frames in the video.

[0136] For example, Figure 4 and Figure 5As shown, the segmentation neural network may include a shared feature encoder, a segmentation decoder and a cutout decoder. First, the video frame may be processed by the shared feature encoder and the segmentation decoder to obtain a target object region image. The target object region image may indicate the region where the target object is located in the video frame.

[0137] In one embodiment, the shared feature encoder may be composed of a convolution layer, a pooling layer, and an activation layer. The convolution layer may perform feature extraction on a video frame to obtain a feature image of the video frame. In the process of extracting features from a video frame by a convolution frame, the size of the feature image will decrease as the convolution layer deepens, and convolution images of different scales represent semantic information at different levels. Large-scale feature images contain rich low-level information, such as color, edge, geometry, and texture information. Small-scale feature images, that is, deep feature maps, reflect higher-level semantic features, such as the distinction between foreground and background.

[0138] In one embodiment, the segmentation decoder can also be composed of a convolution layer, a pooling layer, and an activation layer. During the decoding process, the feature image obtained by the shared feature encoder will be enlarged layer by layer, and the feature images of each layer will be connected to the feature map from the shared feature encoder through skipconnection. In the implementation of this application, the output layer of the segmentation decoder will restore the feature image to 1 / 2 scale of the video frame. The reason for this is that it is not necessary to restore the rough segmentation to the original scale. The small image scale can reduce the network reasoning time and improve the efficiency of segmentation. The segmentation decoder finally outputs a 0 / 1 target object area image, and the position of 1 represents the area of ​​the target object in the target object area image.

[0139] In one embodiment, although the target object area image can distinguish the target object area from the non-target object area, the regional differentiation of the target object area image will be relatively rough. For example, in the target object area image, the contour division accuracy between the target object area and the non-target object area will not be high. For example, in the target object area image, the contour of the target object area will be relatively rough and cannot completely match the contour of the target object in the video frame. Therefore, the target object area image can also be processed using a cutout decoder to obtain a target object image. Among them, the target object image can distinguish the target object area and the non-target object area in the video frame, and the regional differentiation accuracy of the target object image will be higher than that of the target object area image. For example, compared with the target object area image, the contour of the target object area in the target object image will be much more detailed and fine.

[0140] In one embodiment, the cutout decoder may also be composed of a convolutional layer, a pooling layer, and an activation layer. Unlike the segmentation decoder, the cutout decoder needs to work after the segmentation decoder is completed, so the cutout decoder needs to take the output result of the segmentation decoder as input. Specifically, the target object area image output by the segmentation decoder will be fused with the feature image of the shared feature encoder through the skipconnection structure to obtain a fused image. The fused image is then fused with the decoded feature image of the same scale as the segmentation decoder. During the fusion process, the target object area image can provide strong semantic information, guiding the cutout decoder to focus on the interpretation of edge position information, thereby obtaining the target object image.

[0141] 103. Position the video according to the three-dimensional space information of the preset virtual scene video to obtain the video acquisition posture information, wherein the preset virtual scene video is generated based on the preset scene video, the preset scene video is obtained by performing video acquisition on the real scene, and the three-dimensional space information of the preset virtual scene video is obtained by performing three-dimensional space construction processing on the preset scene video.

[0142] In one embodiment, the embodiment of the present application proposes a method for generating a video that integrates virtual and real worlds. Through the method proposed in the present application, a target object in the real world and a preset virtual scene video can be integrated.

[0143] The preset virtual scene video may refer to a video whose video content is a virtual scene. The virtual scene may include a scene that does not exist in the real world and is generated by digital technology. For example, the virtual scene may be a game scene generated by digital technology. For another example, the virtual scene may be an indoor scene or an outdoor scene generated by digital technology, etc.

[0144] In one embodiment, in order to fuse the preset virtual scene video and the target object image with a high degree of matching, that is, when the target object image is fused into the preset virtual scene video, the size and spatial sense of the target object and the virtual scene in the preset virtual scene video are matched, the preset virtual scene video can be generated by the preset scene video, wherein the preset scene video is obtained by capturing the real scene video. That is, the preset scene video and the target object video are both obtained by capturing the same real scene.

[0145] In one embodiment, the three-dimensional space information of the preset virtual scene video is obtained by performing three-dimensional space construction processing on the preset scene video. For example, the preset scene video can be processed by three-dimensional space construction using technologies such as Simultaneous Localization and Mapping (SLAM) to obtain three-dimensional space information. However, traditional SLAM technology cannot be applied to scenes of virtual-real fusion videos, because traditional SLAM technology cannot accurately restore the scene scale of the real scene, resulting in low authenticity of the generated virtual-real fusion video, and low fusion of the target object and the virtual scene. Moreover, the traditional SLAM method cannot be applied to repeated similar scenes. For example, interferences such as white walls, moving objects or shadows will reduce the accuracy of the construction of three-dimensional space information. To this end, the embodiment of the present application can collect the preset scene video with the help of a positioning mark, so that the accuracy of the three-dimensional space information of the preset scene video can be improved by the positioning mark.

[0146] The positioning mark can be used to assist in constructing the three-dimensional space information of the preset scene video. For example, the positioning mark can be a barcode or a QR code, etc.

[0147] For example, when the positioning mark is a QR code, 3-4 QR codes can be pasted in the real scene. The QR code can be printed on ordinary A4 paper. The paper should be pasted on a flat surface, such as the ground or wall, to keep the paper flat. Then, a terminal (mobile phone or camera, etc.) can be used to shoot a video of the real scene without people to obtain a preset scene video. When shooting the real scene, the video can be kept around the environment as much as possible, and the QR code can be visible in the video without obstruction.

[0148] QR codes are widely used information carrier images. By including QR codes in the preset scene video, when constructing the three-dimensional spatial information of the preset scene video, multiple three-dimensional feature points with strong contrast and different from the surroundings will be generated in the video screen through the QR code. These three-dimensional feature points will appear on and near the QR code in the real scene. Therefore, QR codes can be posted in some places where it is difficult to construct three-dimensional spatial information, so that the feature points of these areas can be strengthened through the QR code, thereby improving the accuracy of the three-dimensional spatial information construction. For example, Figure 6 As shown, Figure 6 005 in the figure may be a QR code posted in a real scene, and the position information 006 of the camera may be mapped through the QR code.

[0149] In one embodiment, the three-dimensional spatial information can be used to describe the spatial structure of a real scene. For example, the three-dimensional spatial information can be a collection of multiple pieces of information. For example, the three-dimensional spatial information can be a collection of key frames, map points, positioning identifiers, topological relationships between key frames, and a key frame library. For example, the three-dimensional spatial information can be represented by the following method:

[0150] map = {K,P,M,G,D}

[0151] Among them, K = {keyframe}, which can represent a keyframe set. P = {mappoint}, which can be a map point set. M = {marker}, which can be a set of positioning markers. G can be the topological relationship between keyframes. D can be a keyframe feature library.

[0152] Among them, the key frame can refer to the most representative video frame in the video frame, and the key frame can be the basis for constructing three-dimensional spatial information of the video. The information redundancy between similar frames is very high. The key frame is the most representative frame among the local similar frames, which can reduce the information redundancy. For example, the key frame can be a video frame with a positioning mark in a preset scene video.

[0153] Among them, the topological relationship between key frames can be used to illustrate the strength of the mutual dependence between key frames. For example, if the similarity between key frames is high, it can be said that the mutual dependence between key frames is strong. On the contrary, if the similarity between key frames is low, it can be said that the mutual dependence between key frames is low.

[0154] In one embodiment, the map point is composed of multiple feature points. Feature points are two-dimensional points, which are points obtained by extracting features from video frames. The feature points can describe the location of objects in the real scene. For example, Figure 7 As shown, Figure 7 Points similar to point 004 in can be feature points. Figure 7 It can be seen that the feature points mark all kinds of objects in the video frame. Map points are three-dimensional points used to describe three-dimensional objects in real scenes. Feature points in different video frames may correspond to the same map points in three-dimensional space, while most feature points may not have corresponding map points in three-dimensional space.

[0155] In one embodiment, the key frame feature library may include image features of key frames.

[0156] In one embodiment, the video may be positioned based on the three-dimensional space information of the preset virtual scene video to obtain the captured posture information of the video.

[0157] The collected position information may include the position information and posture information of the device that collects the video. The collected posture information may be used to convert the coordinates of the real world in the video into the coordinates of the video collection device. For example, assuming that the video collection device is a camera, the position and posture of the camera that shoots the video can be estimated by positioning the video.

[0158] For example, assuming that any video frame in the preset scene video can be expressed as:

[0159] frame={t,T,δ}

[0160] Among them, t can represent the frame number. T can represent the acquisition pose information, and the acquisition pose information can be a matrix. For example, the acquisition pose information can be a camera pose matrix. The global reference coordinate system of the video frame can be transformed into the camera coordinate system through T. δ can be a camera intrinsic parameter, which can generally be obtained through camera calibration parameters. The acquisition pose information of the preset scene video can correspond to the acquisition position information of the preset virtual scene video.

[0161] Before positioning the video according to the three-dimensional space information of the preset virtual scene video, the preset scene video can be collected and three-dimensional space construction processing can be performed on the preset scene video to obtain the three-dimensional space information. Specifically, before the step of "positioning the video according to the three-dimensional space information of the preset virtual scene video to obtain the video collection posture information", the method proposed in the embodiment of the present application also includes:

[0162] Capturing a preset scene video of a real scene, wherein the preset scene video includes at least one video frame;

[0163] Performing three-dimensional space construction processing on the video frames of the preset scene video to obtain three-dimensional space information of the real scene in the preset scene video;

[0164] According to the three-dimensional space information of the real scene in the preset scene video, a preset virtual scene video and the three-dimensional space information of the preset virtual scene video are generated.

[0165] In one embodiment, a preset scene video of a real scene may be captured, wherein the preset scene video includes at least one video frame. Figure 8 As shown, positioning marks such as QR codes can be attached to real scenes, and then videos of the real scenes with the positioning marks can be collected.

[0166] In one embodiment, a three-dimensional space construction process may be performed on the video frames of the preset scene video to obtain the three-dimensional space information of the real scene in the preset scene video. The three-dimensional space construction process may refer to calculating the parameters in the three-dimensional space information. For example, the key frames, map points, positioning marks, topological relationships between key frames, and key frame feature libraries in the three-dimensional space information may be calculated through the video frames of the preset scene video.

[0167] Specifically, the step of “performing three-dimensional space construction processing on the video frames of the preset scene video to obtain three-dimensional space information of the real scene in the preset scene video” may include:

[0168] Performing three-dimensional space initialization processing on the preset scene video to obtain initial three-dimensional space information of the real scene in the preset scene video;

[0169] Perform error calculation on the initial three-dimensional space information to obtain three-dimensional space error information;

[0170] Performing key frame detection on video frames of a preset scene video according to the three-dimensional space error information to obtain key frames;

[0171] The initial three-dimensional space information is updated according to the key frames to obtain the three-dimensional space information of the real scene in the preset scene video.

[0172] In one embodiment, a preset scene video may be subjected to a three-dimensional space initialization process to obtain initial three-dimensional space information of a real scene in the preset scene video. The purpose of initializing the preset scene video is to provide an initial value for obtaining accurate three-dimensional space information. After obtaining the initial three-dimensional space information, the initial three-dimensional space information may be continuously optimized to obtain the final three-dimensional space information. Specifically, the step of "performing a three-dimensional space initialization process on the preset scene video to obtain initial three-dimensional space information of the real scene in the preset scene video" may include:

[0173] Determining an initial video frame among video frames of a preset scene video;

[0174] Perform feature point detection on the initial video frame to obtain feature point detection information;

[0175] Performing positioning mark detection on the initial video frame to obtain positioning mark detection information;

[0176] The initial three-dimensional space information of the real scene is generated according to the feature point detection information and the positioning mark detection information.

[0177] In one embodiment, not all video frames in the preset scene video can be used to generate the initial three-dimensional space video. For example, for some video frames with low feature point content, if the initial three-dimensional space information is generated based on these video frames, the accuracy of the generated initial three-dimensional space information will be reduced. Therefore, the initial video frames that can be used as the basis for generating the initial three-dimensional space information can be determined in the video frames.

[0178] There are multiple methods to determine the initial video frame. For example, the first video frame with a positioning mark in the preset video can be used as the initial video frame. For another example, whether it can be used as the initial video frame can be determined based on the number of feature points that can be matched in two consecutive video frames. For example, if the number of feature points that can be matched in two consecutive video frames is greater than a certain value, it can be considered that the frame (previous frame) can be used as the initial video frame. For another example, if two consecutive video frames both have positioning marks, and the number of feature points that can be matched in two consecutive video frames is greater than a certain value, it can be considered that the frame (previous frame) can be used as the initial video frame.

[0179] In one embodiment, it is assumed that the initial video frame is the first video frame with a positioning mark in a preset video as the initial video frame. After obtaining the initial video frame, feature point detection can be performed on the initial video frame to obtain feature point detection information. In addition, positioning mark detection can also be performed on the initial video frame to obtain positioning mark detection information. By performing feature point detection and positioning mark detection on the initial video frame, the initial three-dimensional space information of the real scene can be generated by combining the feature point detection information and the positioning mark detection information.

[0180] The feature point detection information may include feature points existing in the initial video frame and position information of the feature points in the initial video frame. The positioning mark detection information may include feature points generated based on the positioning mark in the initial video frame and position information of the feature points in the initial video frame.

[0181] In one embodiment, there are multiple methods for detecting feature points on the initial video frame. For example, SLAM technology can be used to detect feature points on the initial video frame. For another example, deep learning or machine learning algorithms can be used to detect feature points on the initial video frame.

[0182] In one embodiment, there may be multiple methods for detecting the location mark of the initial video frame. For example, the SLAM technology may be used to detect the location mark of the initial video frame. For another example, the deep learning or machine learning algorithm may be used to detect the location mark of the initial video frame.

[0183] In one embodiment, after obtaining the feature point detection information and the positioning mark detection information, the initial three-dimensional space information of the real scene can be generated by combining the feature point detection information and the positioning mark detection information. Specifically, the step of "generating the initial three-dimensional space information of the real scene according to the feature point detection information and the positioning mark detection information" may include:

[0184] Performing posture calculation processing on the feature point detection information and the positioning mark detection information respectively to obtain the feature point posture information and the positioning mark posture information;

[0185] Perform triangulation processing on the feature point pose information and the positioning mark pose information respectively to obtain the feature point three-dimensional space information and the positioning mark three-dimensional space information;

[0186] Initial three-dimensional space information is filtered out from the three-dimensional space information of feature points and the three-dimensional space information of positioning marks.

[0187] In one embodiment, the pose calculation process may refer to the calculation of the collected pose information, that is, the position information and pose information of the device for collecting the video. The position calculation may be performed based on the feature point detection information and the positioning mark detection information, respectively, to obtain the feature point pose information and the positioning mark pose information. Among them, the feature point pose information may refer to the collected pose information generated based on the feature point detection information. And the positioning mark pose information may refer to the collected position information generated based on the positioning mark detection information. For example, both the positioning mark pose information and the feature point pose information can be a camera pose matrix.

[0188] In one embodiment, the feature point detection information may include multiple feature points, and the multiple feature points may be subjected to polar line constraints to obtain feature point pose information. The positioning mark detection information may also include multiple feature points, and the multiple feature points may be subjected to polar line constraints to obtain positioning mark pose information.

[0189] In one embodiment, the feature point pose information and the positioning mark pose information can be triangulated to obtain the feature point three-dimensional space information and the positioning mark three-dimensional space information. Triangulation can refer to observing the angle of the same feature point through different video frames to determine the distance of the feature point. For example, after the pose is known, the depth corresponding to the feature point can be calculated through triangulation to generate a point cloud; and a map is created with the first frame as the world coordinate system and data association is performed.

[0190] In one embodiment, the initial three-dimensional spatial information can be selected from the feature point three-dimensional spatial information and the positioning mark three-dimensional spatial information. For example, the information accuracy between the feature point three-dimensional spatial information and the positioning mark three-dimensional spatial information can be compared, and the one with higher information accuracy can be selected as the three-dimensional spatial information.

[0191] In one embodiment, the initial three-dimensional space information is information with low precision. In order to optimize the initial three-dimensional space information to obtain three-dimensional space information with high precision, the initial three-dimensional space information may be subjected to error calculation to obtain three-dimensional space error information. For example, the quality of the three-dimensional space information may be measured by the reprojection error. For example, the initial three-dimensional space information may be subjected to error calculation by the reprojection error to obtain the three-dimensional space error information. For example, the three-dimensional space error information may be calculated according to the following formula:

[0192] error=ψ(T,x,δ)-u

[0193] Among them, T can represent the collected pose information. x can refer to the three-dimensional coordinates of the feature points corresponding to the map points, feature points or positioning markers in the initial three-dimensional space information. u can be the two-dimensional coordinates of the image corresponding to the map points in the initial three-dimensional space information. The coordinates are the row and column coordinates corresponding to (u, v). The three-dimensional coordinates of the map point are unique, and when projected into different frames of images, they will correspond to different image coordinates. ψ() can be a projection function. According to the camera pose and internal parameters, the two-dimensional coordinates of a three-dimensional point projected onto the image are calculated, which is called the projection process. The difference between the calculated image coordinates and the true image coordinates is the reprojection error.

[0194] For example, based on the above formula, the global optimal solution can be obtained by finding the minimum reprojection error through nonlinear optimization. The nonlinear optimization here refers to using a nonlinear function to find the optimal solution of the parameters corresponding to the minimum error.

[0195] In one embodiment, after the three-dimensional space error information is calculated, the three-dimensional space error information can be compared with a preset error threshold. When the three-dimensional space error information does not meet the preset error threshold, it can be said that the accuracy of the initial three-dimensional space information does not meet the requirements, and the initial three-dimensional space information needs to be updated. When the three-dimensional space error information meets the preset error threshold, it can be said that the accuracy of the initial three-dimensional space information meets the requirements, so the initial three-dimensional space information does not need to be updated.

[0196] When the three-dimensional space error information does not meet the preset error threshold, a key frame can be determined. Then, the initial three-dimensional space information can be updated through the key frame to obtain three-dimensional space information with accuracy that meets the requirements. Specifically, the step of "key frame detection of the video frame of the preset scene video according to the three-dimensional space error information to obtain the key frame" may include:

[0197] When the three-dimensional space error information does not meet the preset error threshold, positioning mark detection is performed on the video frame of the preset scene video;

[0198] When it is detected that a video frame of the preset scene video has a positioning mark, the video frame is determined as a marked video frame;

[0199] Performing redundancy determination processing on the identified video frame;

[0200] When the identification video frame passes the redundancy determination process, the identification video frame is determined as a key frame.

[0201] Among them, the key frame may refer to the most representative video frame in the video frame, and the key frame may be the basis for constructing three-dimensional spatial information of the video. The information redundancy between similar frames is very high, and the key frame is the most representative frame among the local similar frames, which can reduce the information redundancy. For example, the key frame may be a video frame with a positioning mark in a preset scene video. Therefore, when the three-dimensional spatial error information does not meet the preset error threshold, the positioning mark detection may be performed on the video frame of the preset scene video. For example, deep learning technology or machine learning technology may be used to detect the positioning mark of the video frame of the preset scene video.

[0202] In one embodiment, when a positioning mark is detected in a video frame of a preset scene video, the video frame may be determined as a marking video frame. The marking video frame may include a video frame with a positioning mark. For example, the picture content of the marking video frame may include a positioning mark such as a QR code.

[0203] In one embodiment, since the same positioning mark may appear in multiple video frames, key frames may also be redundant. When a key frame is added, the three-dimensional space information is updated once. Redundant key frames will cause the three-dimensional space information to be updated frequently, and since these redundant key frames record the content of the same positioning mark, the accuracy of the three-dimensional space information is not improved much. In order to improve efficiency, the identification video frame can be processed for redundancy determination. When the identification video frame passes the redundancy determination process, the identification video frame can be determined as a key frame.

[0204] For example, if the distance between the identified video frame and the latest historical key frame in the key frame set is greater than the preset distance threshold, it can be said that the identified video frame has passed the redundancy judgment, and the identified video frame can be determined as a key frame. For example, the identified video frame is the 40th frame in the preset scene video, the most recent historical key frame is the 35th frame in the preset scene video, and the preset distance threshold is 4. Since the distance between the identified video frame and the latest historical key frame in the key frame set is greater than the preset distance threshold, the identified video frame can be determined as a key frame.

[0205] For another example, the redundancy of the identification video frame can be determined by the map points existing in the video frame. For example, if the number of matching map points of the identification video frame and the map points of the historical video frame is greater than a preset number threshold, the identification video frame can be determined as a key frame.

[0206] In one embodiment, in order to avoid unlimited growth of key frames, some repeated key frames need to be removed. For example, each positioning mark and feature point must be observed by at least three key frames to obtain a better triangulation result during optimization.

[0207] In one embodiment, after the key frame is detected, the initial three-dimensional space information can be updated according to the key frame to obtain the three-dimensional space information of the real scene in the preset scene video. Specifically, the step of "updating the initial three-dimensional space information according to the key frame to obtain the three-dimensional space information of the real scene in the preset scene video" can include:

[0208] Acquire a key frame set of a preset scene video, wherein the key frame set includes at least one historical key frame;

[0209] Determine a matching key frame that matches the key frame in the historical key frames;

[0210] Calculate the pose information corresponding to the key frame according to the matching key frame;

[0211] The initial three-dimensional space information is updated according to the posture information corresponding to the key frame to obtain the three-dimensional space information of the real scene in the preset scene video.

[0212] The key frame set may correspond to a key frame set K.

[0213] The matching key frame that matches the key frame may include a historical key frame that has the same map point, positioning identifier or feature point as the key frame. For example, the matching key frame may be a historical key frame that has the same map point as the key frame. For another example, the matching key frame may be a historical key frame that has the same positioning identifier as the key frame. For example, a historical key frame in the key frame set that has the same positioning identifier or map point as the key frame may be determined as a matching key frame.

[0214] In one embodiment, the position information corresponding to the key frame can be calculated based on the matching key frame. For example, the matching map points between the historical key frames can be found. Once the matching map points between the historical key frames appear in the current key frame, these matching relationships are used together with the QR code corner points to calculate the initial position of the current key frame. Then, the adjacent key frames of the current key frame can be found through reprojection, and the maps of the adjacent key frames can be found to find more matching relationships. Then, the camera pose is estimated based on the minimum reprojection error.

[0215] In one embodiment, the initial three-dimensional space information is updated according to the position and posture information corresponding to the key frame to obtain the three-dimensional space information of the real scene in the preset scene video. For example, the SLAM algorithm can be used to calculate the new three-dimensional space information according to the position and posture information corresponding to the key frame.

[0216] In one embodiment, the above steps can continuously detect new key frames, and continuously update the three-dimensional space information according to the new key frames. For example, when a new key frame is added, the map information is updated. When a closed loop is identified, the entire map is optimized, or incorrect map points are removed. The optimization objects include key frame poses, map point coordinates, and QR code poses.

[0217] In one embodiment, the last frame of the preset scene video can be relocated. When the preset scene video is processed to the last frame, if the follow-up positioning fails, relocation is required. The positioning mark can be used for positioning first. If the current frame has a QR code and can be accurately positioned, it can be directly relocated. When the precise posture of the current frame is unknown, the map point of the current frame is matched with the selected similar key frame, and the optimal position transformation matrix is ​​found by the RANSAC PnP method until the number of internal points is sufficient, and the relocation is considered correct.

[0218] In one embodiment, after obtaining the three-dimensional spatial information of the real scene in the preset scene video, the preset virtual scene video and the three-dimensional spatial information of the preset virtual scene video can be generated according to the three-dimensional spatial information of the real scene in the preset scene video. For example, the three-dimensional spatial information of the real scene in the preset scene video and the virtual scene can be fitted to obtain the preset virtual scene video. The three-dimensional spatial information of the preset scene video can be the three-dimensional spatial information of the preset virtual scene video. Among them, the virtual scene can be pre-set or selected by the user. For example, several virtual scenes can be provided for the user to choose. Then, the preset virtual scene video can be generated according to the virtual scene selected by the user.

[0219] In one embodiment, the video may be positioned according to the three-dimensional spatial information of the preset virtual scene video to obtain the captured pose information of the video. Specifically, the step of "positioning the video according to the three-dimensional spatial information of the preset virtual scene video to obtain the captured pose information of the video" may include:

[0220] Analyze and process the three-dimensional space information of the preset virtual scene video to obtain at least one map point;

[0221] Matching the video frames of the video with at least one map point respectively to obtain a map point matching result of each video frame;

[0222] According to the map point matching results, the pose information of the map point corresponding to each video frame is mapped to the video frame to obtain the captured pose information of the video.

[0223] In one embodiment, the three-dimensional spatial information of the preset virtual scene video can be parsed to obtain at least one map point of the preset virtual scene video. Since the video of the target object and the preset scene video are both obtained by collecting the same real scene, the video frame of the video and at least one map point can be matched to obtain the map point matching result of each video frame. That is, each map point can be corresponded to the corresponding video frame of the video. For example, feature point detection can be performed on the video frame of the video to obtain feature point detection information of the video frame. Then, posture calculation processing can be performed based on the feature point detection point information of the video frame to obtain feature point posture information. Then, triangulation can be performed on the feature point posture information to obtain the three-dimensional spatial information of the video. Then, the three-dimensional spatial information of the video can be matched to obtain the map point matching result of each video frame. Next, based on the map point matching result, the posture information of the map point corresponding to each video frame can be mapped to the video frame to obtain the acquisition posture information of the video.

[0224] The present application implements the three-dimensional space construction processing of the preset scene video using positioning markers, which can enhance the feature points in the video, improve the accuracy of generating three-dimensional space information, and overcome the problem that the traditional SLAM method is not suitable for virtual-reality fusion applications.

[0225] 104. According to the collected posture information, the target object image corresponding to each video frame is fused with the preset virtual scene video to obtain a target fused video.

[0226] In one embodiment, after the acquired posture information is obtained, the target object image corresponding to each video frame and the preset virtual scene video may be fused according to the acquired posture information to obtain a target fused video.

[0227] Specifically, the step of "fusing the target object image corresponding to each video frame with the preset virtual scene video according to the collected posture information to obtain the target fused video" includes:

[0228] Perform layer overlay processing on the target object image of the video frame and the video frame of the preset virtual scene video to obtain at least one overlay layer;

[0229] The preset video display parameters are set and processed according to the collected posture information to obtain the set video display parameters;

[0230] The at least one overlay layer is rendered based on the set video display parameters to obtain a target fused video.

[0231] Layers can include films containing text, graphics, etc. Layers are stacked together in order to form the final effect of the page.

[0232] In one embodiment, the target object image of the video frame and the video frame of the preset virtual scene video may be subjected to layer overlay processing to obtain at least one overlay image. Specifically, the step of "overlaying the target object image of the video frame and the video frame of the preset virtual scene video to obtain at least one overlay layer" may include:

[0233] Generate a first layer and a second layer by using a preset rendering engine;

[0234] Rendering the target object image and the first layer using a preset rendering engine to obtain a first rendered layer;

[0235] Rendering the preset virtual scene video and the second layer using a preset rendering engine to obtain a second rendered layer;

[0236] The first rendered layer and the second rendered layer are superimposed to obtain at least one superimposed layer.

[0237] In one embodiment, the first layer and the second layer may be generated by a preset rendering engine. Then, the preset rendering engine may obtain target object features and a preset virtual scene video. The preset rendering engine may convert the three-dimensional spatial information of the preset virtual scene video into two-dimensional image data, that is, project it onto a two-dimensional plane according to the three-dimensional spatial information, and the two-dimensional plane may be equivalent to the second layer, thereby obtaining a second rendered image. In addition, the preset rendering engine may render the target object image and the first layer to obtain a first rendered image. For example, the preset rendering engine may project the data of the target object image onto the first layer to obtain a first rendered layer. Then, the first rendered layer and the second rendered layer may be overlaid to obtain at least one overlaid layer.

[0238] In one embodiment, after obtaining at least one superimposed layer, the preset video display parameters can be set according to the collected posture information to obtain the set video display parameters. Fig. 9As shown, when displaying the target fusion video, a virtual virtual-real camera can be used as the shooting perspective to display the target fusion video. Among them, the preset video display parameters can be parameters corresponding to the virtual-real camera. For example, the position of the virtual-real camera can be specified by the acquisition posture information. Therefore, the position movement parameters of the virtual-real camera can be set according to the acquisition posture information to obtain the set virtual-real camera. Then, the set virtual-real camera can be used to render at least one overlay layer to obtain the target fusion video. Since the position movement parameters of the virtual-real camera are set according to the acquisition posture information, the movement of the virtual-real camera will be consistent with the movement of the camera when shooting the preset scene video, thereby creating a scene in which the target object in the real world travels to the virtual world. For example, when the target object is a real person, a dream experience of a real person traveling to the virtual world can be created through the target fusion video.

[0239] In one embodiment, after the target fusion video is generated, the target fusion video can be output. For example, the target fusion video can be output to a terminal so that the target fusion video can be viewed. For example, a user can use a terminal to shoot a video of a real person in a real scene, and then the terminal can generate a target fusion video according to the method proposed in the embodiment of the present application, and output the target fusion video so that the user can view the target fusion video. For example, Fig.10 As shown in FIG. , it can be a schematic diagram of the target fusion video. Fig.10 It can be seen that the target fusion video can integrate real people and virtual scenes to create a scene where real people travel to the virtual world.

[0240] In one embodiment, the present application can also be applied to film and television production. Compared with the related technology that uses green screens and physical followers to generate virtual and real fusion videos, the present application only needs to use a camera to generate virtual and real fusion videos, which greatly reduces the cost of video production.

[0241] In an embodiment of the present application, a video of a target object in a real scene can be collected, wherein the video includes at least one video frame; object detection processing is performed on the video frames in the video to obtain the target object image corresponding to each video frame; positioning processing is performed on the video according to the three-dimensional space information of the preset virtual scene video to obtain the acquisition posture information of the video, wherein the preset virtual scene video is generated based on the preset scene video, the preset scene video is obtained by performing video acquisition on the real scene, and the three-dimensional space information of the preset virtual scene video is obtained by performing three-dimensional space construction processing on the preset scene video; the target object image corresponding to each video frame and the preset virtual scene video are fused according to the acquisition posture information to obtain the target fused video; and the target fused video is output. Through the method proposed in the embodiment of the present application, there is no need to build a green screen environment, and the target object image of the video frame can be obtained by object detection processing, which gets rid of the limitation of the environment. Moreover, the embodiment of the present application does not need to rely on expensive physical followers, and the target fused video can be shot using ordinary terminals (such as mobile phones or cameras, etc.), which greatly reduces the cost.

[0242] According to the method described in the above embodiment, the following is further described in detail with examples.

[0243] The embodiment of the present application will take the video generation method executed by the server and the terminal community as an example to introduce the method of the embodiment of the present application.

[0244] For example, Fig.11 As shown, the process proposed in the embodiment of the present application may include an offline program and an online program. The offline program may include: the server collects a preset scene video of a real scene, wherein the preset scene video includes at least one video frame; the server performs a three-dimensional space construction process on the video frame of the preset scene video to obtain the three-dimensional space information of the real scene in the preset scene video; the server generates a preset virtual scene video and the three-dimensional space information of the preset virtual scene video according to the three-dimensional space information of the real scene in the preset scene video. The online program may include: the terminal collects a video of the target object in the real scene, wherein the video includes at least one video frame; the terminal performs object detection processing on the video frame in the video to obtain the target object image corresponding to each video frame; the terminal performs positioning processing on the video according to the three-dimensional space information of the preset virtual scene video to obtain the acquisition posture information of the video; the terminal fuses the target object image corresponding to each video frame with the preset virtual scene video according to the acquisition posture information to obtain the target fused video; the terminal outputs the target fused video.

[0245] For example, more specifically, Fig.12As shown, it is assumed that the offline program may include pasting 3-4 QR codes in the real scene; shooting a pure background video with only the real scene but no people; reconstructing the sparse 3D map of the scene using the SLAM method proposed in the embodiment of the present application, and outputting the 3D map to the online program. In the online program, you can shoot a video with people; execute the AI ​​cutout algorithm to obtain the target object image; execute real-time SLAM camera positioning to obtain the collected posture information; and output the virtual-real fusion picture of the real person and the virtual background.

[0246] In one embodiment, if Fig.11 As shown, a video generation method, the specific process is as follows:

[0247] 201. The server obtains a preset scene video of a real scene, wherein the preset scene video includes at least one video frame.

[0248] For example, a positioning mark may be affixed to a real scene, and then the server may obtain a video of the real scene with the positioning mark through a camera or a webcam.

[0249] 202. The server performs three-dimensional space construction processing on the video frames of the preset scene video to obtain three-dimensional space information of the real scene in the preset scene video.

[0250] For example, assuming that any video frame in the preset scene video can be expressed as:

[0251] frame={t,T,δ}

[0252] Wherein, t may represent the frame number. T may represent the acquired pose information, which may be a matrix. For example, the acquired pose information may be a camera pose matrix. The global reference coordinate system of the video frame may be transformed into the camera coordinate system through T. δ may be a camera intrinsic parameter, which may generally be obtained through camera calibration parameters.

[0253] For example, three-dimensional spatial information can be represented as follows:

[0254] map = {K, P, M, G, D}

[0255] Among them, K = {keyframe}, which can represent a keyframe set. P = {map point}, which can be a map point set. M = {marker}, which can be a set of positioning markers. G can be the topological relationship between keyframes. D can be a keyframe feature library.

[0256] For example, the quality of the three-dimensional space information can be measured by the reprojection error. For example, the initial three-dimensional space information can be further reprojected to calculate the error to obtain the three-dimensional space error information. For example, the three-dimensional space error information can be calculated according to the following formula:

[0257] error=ψ(T,x,δ)-u

[0258] Among them, T can represent the collected posture information. x can refer to the three-dimensional coordinates of the feature points corresponding to the map points, feature points or positioning markers in the initial three-dimensional space information. u can be the two-dimensional image coordinates corresponding to the map points in the initial three-dimensional space information. The coordinates are the row and column coordinates corresponding to (u, v). The three-dimensional coordinates of the map point are unique, and when projected into different frames of images, they will correspond to different image coordinates. ψ() can be a projection function. According to the camera posture and internal parameters, the two-dimensional coordinates of a three-dimensional point projected onto the image are calculated, which is called the projection process. The difference between the calculated image coordinates and the true image coordinates is the reprojection error.

[0259] For example, based on the above formula, the global optimal solution can be obtained by finding the minimum reprojection error through nonlinear optimization. The nonlinear optimization here refers to using a nonlinear function to find the optimal solution of the parameters corresponding to the minimum error.

[0260] For example, Fig.13 As shown, it can be determined whether there is a new video frame to be processed in the preset scene video. If so, the map can be initialized. For example, when a new video frame comes in, feature point and QR code detection will be performed at the same time. If both initializations are successful, the result of QR code initialization will be used first.

[0261] Then, if Fig.13 As shown, a key frame can be found, and frame-to-frame following processing can be performed according to the key frame. For example, the key frame can be a video frame with a positioning mark in a preset scene video.

[0262] In one embodiment, when a positioning mark is detected in a video frame of a preset scene video, the video frame may be determined as a marking video frame. The marking video frame may include a video frame with a positioning mark. For example, the picture content of the marking video frame may include a positioning mark such as a QR code.

[0263] In one embodiment, since the same positioning mark may appear in multiple video frames, key frames may also be redundant. When a key frame is added, the three-dimensional space information is updated once. Redundant key frames will cause the three-dimensional space information to be updated frequently, and since these redundant key frames record the content of the same positioning mark, the accuracy of the three-dimensional space information is not improved much. In order to improve efficiency, the identification video frame can be processed for redundancy determination. When the identification video frame passes the redundancy determination process, the identification video frame can be determined as a key frame.

[0264] For example, if the distance between the identified video frame and the latest historical key frame in the key frame set is greater than the preset distance threshold, it can be said that the identified video frame has passed the redundancy judgment, and the identified video frame can be determined as a key frame. For example, the identified video frame is the 40th frame in the preset scene video, the most recent historical key frame is the 35th frame in the preset scene video, and the preset distance threshold is 4. Since the distance between the identified video frame and the latest historical key frame in the key frame set is greater than the preset distance threshold, the identified video frame can be determined as a key frame.

[0265] For another example, the redundancy of the identification video frame can be determined by the map points existing in the video frame. For example, if the number of matching map points of the identification video frame and the map points of the historical video frame is greater than a preset number threshold, the identification video frame can be determined as a key frame.

[0266] In one embodiment, if Fig.13 As shown in the figure, in order to avoid unlimited growth of key frames, some repeated key frames need to be removed. For example, each positioning mark and feature point must be observed by at least 3 key frames to obtain a good triangulation result during optimization.

[0267] In one embodiment, the purpose of frame-to-frame following is to estimate the camera pose of the current frame based on the previous map points and key frame information. The steps can be divided into the following: matching map points between historical key frames can be found. Once the matching map points between the historical key frames appear in the current key frame, these matching relationships are used together with the QR code corner points to calculate the initial position of the current key frame. Then, the adjacent key frames of the current key frame can be found through reprojection, and the maps of the adjacent key frames can be found to find more matching relationships. Then, the camera pose is estimated based on the minimum reprojection error.

[0268] In one embodiment, if Fig.13 As shown, through the above steps, new keyframes can be continuously detected, and the three-dimensional space information can be continuously updated according to the new keyframes. For example, when a new keyframe is added, the map information is updated. When a closed loop is identified, the entire map is optimized, or incorrect map points are removed. The optimization objects include keyframe poses, map point coordinates, and QR code poses.

[0269] In one embodiment, if Fig.13As shown in the figure, the last frame of the preset scene video can be relocated. When the preset scene video is processed to the last frame, if the follow-up positioning fails, relocation is required. The positioning mark can be used for positioning first. If the current frame has a QR code and can be accurately positioned, it can be relocated directly. When the current frame has an unknown exact pose, the map point of the current frame is matched with the selected similar key frame, and the optimal position transformation matrix is ​​found by the RANSAC PnP method until the number of internal points is sufficient, then the relocation is considered correct.

[0270] 203. The server generates a preset virtual scene video and the three-dimensional space information of the preset virtual scene video according to the three-dimensional space information of the real scene in the preset scene video.

[0271] 204. The terminal collects a video of the target object in a real scene, where the video includes at least one video frame.

[0272] 205. The terminal performs object detection processing on the video frames in the video to obtain a target object image corresponding to each video frame.

[0273] For example, a segmentation neural network may be pre-trained and put online in a computer device. When object detection is performed on video frames in a video, the trained segmentation neural network may be used to perform object detection on the video frames in the video.

[0274] For example, the segmentation neural network may include a shared feature encoder, a segmentation decoder, and a cutout decoder. First, the video frame may be processed by the shared feature encoder and the segmentation decoder to obtain a target object region image. The target object region image may indicate the region where the target object is located in the video frame.

[0275] In one embodiment, the shared feature encoder may be composed of a convolution layer, a pooling layer, and an activation layer. The convolution layer may perform feature extraction on a video frame to obtain a feature image of the video frame. In the process of extracting features from a video frame by a convolution frame, the size of the feature image will decrease as the convolution layer deepens, and convolution images of different scales represent semantic information at different levels. Large-scale feature images contain rich low-level information, such as color, edge, geometry, and texture information. Small-scale feature images, that is, deep feature maps, reflect higher-level semantic features, such as the distinction between foreground and background.

[0276] In one embodiment, the segmentation decoder can also be composed of a convolution layer, a pooling layer, and an activation layer. During the decoding process, the feature image obtained by the shared feature encoder will be enlarged layer by layer, and the feature images of each layer will be connected to the feature map from the shared feature encoder through skipconnection. In the implementation of this application, the output layer of the segmentation decoder will restore the feature image to 1 / 2 scale of the video frame. The reason for this is that it is not necessary to restore the rough segmentation to the original scale. The small image scale can reduce the network reasoning time and improve the efficiency of segmentation. The segmentation decoder finally outputs a 0 / 1 target object area image, and the position of 1 represents the area of ​​the target object in the target object area image.

[0277] In one embodiment, although the target object area image can distinguish the target object area from the non-target object area, the regional differentiation of the target object area image will be relatively rough. For example, in the target object area image, the contour division accuracy between the target object area and the non-target object area will not be high. For example, in the target object area image, the contour of the target object area will be relatively rough and cannot completely match the contour of the target object in the video frame. Therefore, the target object area image can also be processed using a cutout decoder to obtain a target object image. Among them, the target object image can distinguish the target object area and the non-target object area in the video frame, and the regional differentiation accuracy of the target object image will be higher than that of the target object area image. For example, compared with the target object area image, the contour of the target object area in the target object image will be much more detailed and fine.

[0278] In one embodiment, the cutout decoder may also be composed of a convolutional layer, a pooling layer, and an activation layer. Unlike the segmentation decoder, the cutout decoder needs to work after the segmentation decoder is completed, so the cutout decoder needs to take the output result of the segmentation decoder as input. Specifically, the target object area image output by the segmentation decoder will be fused with the feature image of the shared feature encoder through the skipconnection structure to obtain a fused image. The fused image is then fused with the decoded feature image of the same scale as the segmentation decoder. During the fusion process, the target object area image can provide strong semantic information, guiding the cutout decoder to focus on the interpretation of edge position information, thereby obtaining the target object image.

[0279] 206. The terminal performs positioning processing on the video according to the three-dimensional space information of the preset virtual scene video to obtain the acquisition posture information of the video.

[0280] According to the three-dimensional space information generated in step 201 to step 203, the camera image can be directly matched with the map point, so the online program no longer relies on the QR code. In this step, the frame following step in step 202 can be executed, and the key frame addition and map update are no longer executed. The output result is the camera pose, which can be directly imported into the rendering engine to specify the position of the virtual camera.

[0281] 207. The terminal fuses the target object image corresponding to each video frame with the preset virtual scene video according to the collected posture information to obtain a target fused video.

[0282] With the collected pose information and the target object image, the virtual-real fusion result can be realized in the 3D rendering engine. First, select a virtual scene in the engine, that is, the map. Then specify two layers in the engine, one layer renders the image captured by the camera, that is, the portrait. The other layer renders the target object image. Since the opacity of the portrait is 1 and the opacity of other areas is 0, the fusion of the real person and the virtual scene can be achieved by superimposing the two layers. Finally, the position of the virtual-real camera is specified by the pose followed by SLAM. When everything is ready, the movement of the virtual camera will be consistent with the movement of the handheld camera, thus creating a dream experience of real people "traveling" to the virtual world.

[0283] 208. The terminal outputs the target fused video.

[0284] In an embodiment of the present application, a server obtains a preset scene video of a real scene, wherein the preset scene video includes at least one video frame; the server performs three-dimensional space construction processing on the video frame of the preset scene video to obtain three-dimensional space information of the real scene in the preset scene video; the server generates a preset virtual scene video and three-dimensional space information of the preset virtual scene video based on the three-dimensional space information of the real scene in the preset scene video; the terminal collects a video of the target object in the real scene, wherein the video includes at least one video frame; the terminal performs object detection processing on the video frames in the video to obtain a target object image corresponding to each video frame; the terminal performs positioning processing on the video according to the three-dimensional space information of the preset virtual scene video to obtain the collected posture information of the video; the terminal fuses the target object image corresponding to each video frame with the preset virtual scene video according to the collected posture information to obtain a target fused video; the terminal outputs the target fused video, which can realize the generation of virtual-real fused video without the help of a green screen environment and a follower, thereby reducing the cost of generating a virtual fused video.

[0285] In order to better implement the video generation method provided in the embodiment of the present application, a video generation device is also provided in one embodiment. The meanings of the terms are the same as those in the above-mentioned video generation method, and the specific implementation details can refer to the description in the method embodiment.

[0286] In one embodiment, a video generation device is provided, such as Fig.14 As shown, the video generation device includes: a collection unit 301, an object detection unit 302, a positioning unit 303 and a fusion unit 304, which are specifically as follows:

[0287] A collection unit 301 is used to collect a video of a target object in a real scene, wherein the video includes at least one video frame;

[0288] The object detection unit 302 is used to perform object detection processing on the video frames in the video to obtain a target object image corresponding to each video frame;

[0289] A positioning unit 303 is used to perform positioning processing on a preset virtual scene video according to the three-dimensional space information of the video to obtain acquisition posture information of the video, wherein the preset virtual scene video is generated based on a preset scene video, the preset scene video is obtained by performing video acquisition on the real scene, and the three-dimensional space information of the preset virtual scene video is obtained by performing three-dimensional space construction processing on the preset scene video;

[0290] The fusion unit 304 is used to fuse the target object image corresponding to each video frame with the preset virtual scene video according to the collected posture information to obtain a target fused video.

[0291] In one embodiment, the positioning unit 303 may include:

[0292] A parsing subunit, configured to parse and process the three-dimensional space information of the preset virtual scene video to obtain at least one map point;

[0293] A matching subunit, used to match the video frame of the video with the at least one map point respectively to obtain a map point matching result of each video frame;

[0294] A mapping subunit is used to map the position and posture information of the map point corresponding to each video frame to the video frame according to the map point matching result, so as to obtain the acquisition position and posture information of the video.

[0295] In one embodiment, the video generating device may include:

[0296] A video acquisition unit, configured to acquire a preset scene video of the real scene, wherein the preset scene video includes at least one video frame;

[0297] A three-dimensional space construction unit, configured to perform three-dimensional space construction processing on the video frame of the preset scene video to obtain three-dimensional space information of the real scene in the preset scene video;

[0298] A generating unit is used to generate the preset virtual scene video and the three-dimensional space information of the preset virtual scene video according to the three-dimensional space information of the real scene in the preset scene video.

[0299] In one embodiment, the three-dimensional space construction unit may include:

[0300] An initialization subunit, configured to perform a three-dimensional space initialization process on the preset scene video to obtain initial three-dimensional space information of the real scene in the preset scene video;

[0301] an error calculation unit, used to perform error calculation on the initial three-dimensional space information to obtain three-dimensional space error information;

[0302] A key frame detection unit, configured to perform key frame detection on the video frame of the preset scene video according to the three-dimensional space error information to obtain a key frame;

[0303] An updating unit is used to update the initial three-dimensional space information according to the key frame to obtain the three-dimensional space information of the real scene in the preset scene video.

[0304] In one embodiment, the initialization subunit may include:

[0305] A video frame determination module, used to determine an initial video frame in the video frames of the preset scene video;

[0306] A feature point detection module, used to perform feature point detection on the initial video frame to obtain feature point detection information;

[0307] A positioning mark detection module, used to perform positioning mark detection on the initial video frame to obtain positioning mark detection information;

[0308] A generation module is used to generate initial three-dimensional space information of the real scene according to the feature point detection information and the positioning mark detection information.

[0309] In one embodiment, the generating module may include:

[0310] A posture calculation submodule, used to perform posture calculation processing on the feature point detection information and the positioning mark detection information respectively to obtain feature point posture information and positioning mark posture information;

[0311] A triangulation submodule is used to perform triangulation processing on the feature point pose information and the positioning mark pose information respectively to obtain the feature point three-dimensional space information and the positioning mark three-dimensional space information;

[0312] A determination module is used to determine the initial three-dimensional space information from the feature point three-dimensional space information and the positioning mark three-dimensional space information.

[0313] In one embodiment, the key frame detection unit may include:

[0314] A positioning mark detection subunit, configured to perform positioning mark detection on a video frame of the preset scene video when the three-dimensional space error information does not meet a preset error threshold;

[0315] A first determining subunit is configured to determine the video frame as an identification video frame when a positioning mark is detected in the video frame of the preset scene video;

[0316] A redundancy determination subunit, used for performing redundancy determination processing on the identified video frame;

[0317] The second determining subunit is used to determine the identification video frame as the key frame when the identification video frame passes the redundancy determination process.

[0318] In one embodiment, the updating unit may include:

[0319] An acquisition subunit, configured to acquire a key frame set of the preset scene video, wherein the key frame set includes at least one historical key frame;

[0320] A third determining subunit, configured to determine a matching key frame that matches the key frame in the historical key frames;

[0321] A calculation subunit, used for calculating the posture information corresponding to the key frame according to the matching key frame;

[0322] An updating subunit is used to update the initial three-dimensional space information according to the posture information corresponding to the key frame to obtain the three-dimensional space information of the real scene in the preset scene video.

[0323] In one embodiment, the object detection unit 302 may include:

[0324] A feature encoding subunit, used for performing feature encoding processing on the video frame to obtain a feature image of the video frame;

[0325] A region decoding subunit, configured to perform region decoding processing on the feature image of the video frame to obtain a target object region image of the video frame;

[0326] The region refinement subunit is used to perform region refinement processing on the target object region image according to the feature image of the video frame to obtain the target object image.

[0327] In one embodiment, the region refinement subunit may include:

[0328] A first fusion module, used for fusing the target object area image and the feature image of the video frame to obtain a fused feature image;

[0329] An information acquisition module, used to acquire feature coding information of the video frame;

[0330] A second fusion module is used to fuse the fused feature image and the feature coding information to obtain fused coding information;

[0331] A decoding module is used to decode the fused encoded information to obtain the target object image.

[0332] In one embodiment, the fusion unit 304 may include:

[0333] A layer overlay subunit, used for performing layer overlay processing on the target object image of the video frame and the video frame of the preset virtual scene video to obtain at least one overlay layer;

[0334] A setting subunit, used for setting and processing preset video display parameters according to the collected posture information to obtain the set video display parameters;

[0335] The rendering subunit is used to render the at least one overlay layer based on the set video display parameters to obtain the target fused video.

[0336] In one embodiment, the layer overlay subunit may include:

[0337] A layer generation module, used for generating a first layer and a second layer by using a preset rendering engine;

[0338] A first rendering module, configured to render the target object features and the first layer using the preset rendering engine to obtain a first rendered layer;

[0339] A second rendering module, configured to cause the preset rendering engine to overlay the preset virtual scene video and the second layer to obtain a second rendered image;

[0340] The superposition module is used to superimpose the first rendered layer and the second rendered layer to obtain at least one superimposed layer.

[0341] In specific implementation, the above units can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above units can refer to the previous method embodiments, which will not be repeated here.

[0342] The above-mentioned video generation device can realize the generation of virtual-real fusion video without the aid of a green screen environment and a follower, thereby reducing the cost of generating virtual fusion video.

[0343] The present application also provides a computer device, which may be a terminal or a server. Fig.15 As shown, it shows a schematic diagram of the structure of the computer device involved in the embodiment of the present application, specifically:

[0344] The computer device may include components such as a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a power supply 403, and an input unit 404. Those skilled in the art will appreciate that Fig.15 The computer device structure shown in the figure does not constitute a limitation on the computer device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. Among them:

[0345] The processor 401 is the control center of the computer device, and uses various interfaces and lines to connect various parts of the entire computer device. By running or executing software programs and / or modules stored in the memory 402, and calling data stored in the memory 402, the processor 401 performs various functions of the computer device and processes data. Optionally, the processor 401 may include one or more processing cores; preferably, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user pages and application programs, etc., and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 401.

[0346] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 402 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0347] The computer device also includes a power supply 403 for supplying power to various components. Preferably, the power supply 403 can be logically connected to the processor 401 through a power management system, so as to manage charging, discharging, and power consumption through the power management system. The power supply 403 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.

[0348] The computer device may further include an input unit 404, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0349] Although not shown, the computer device may further include a display unit, etc., which will not be described in detail herein. Specifically, in this embodiment, the processor 401 in the computer device will load the executable files corresponding to the processes of one or more computer programs into the memory 402 according to the following computer program, and the processor 401 will run the computer program stored in the memory 402, thereby realizing various functions, as follows:

[0350] Capturing a video of a target object in a real scene, wherein the video includes at least one video frame;

[0351] Perform object detection processing on the video frames in the video to obtain the target object image corresponding to each video frame;

[0352] According to the three-dimensional space information of the preset virtual scene video, the video is positioned to obtain the acquisition posture information of the video, wherein the preset virtual scene video is generated based on the preset scene video, the preset scene video is obtained by performing video acquisition on the real scene, and the three-dimensional space information of the preset virtual scene video is obtained by performing three-dimensional space construction processing on the preset scene video;

[0353] The target object image corresponding to each video frame and the preset virtual scene video are fused according to the collected posture information to obtain a target fused video.

[0354] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0355] According to one aspect of the present application, a computer program product is provided, the computer program product comprising a computer program, the computer program being stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the method provided in various optional implementations in the above-mentioned embodiments.

[0356] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by a computer program, or by controlling related hardware through a computer program. The computer program may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0357] To this end, the embodiment of the present application further provides a storage medium, in which a computer program is stored, and the computer program can be loaded by a processor to execute the steps in any one of the video generation methods provided in the embodiment of the present application. For example, the computer program can execute the following steps:

[0358] Capturing a video of a target object in a real scene, wherein the video includes at least one video frame;

[0359] Perform object detection processing on the video frames in the video to obtain the target object image corresponding to each video frame;

[0360] According to the three-dimensional space information of the preset virtual scene video, the video is positioned to obtain the acquisition posture information of the video, wherein the preset virtual scene video is generated based on the preset scene video, the preset scene video is obtained by performing video acquisition on the real scene, and the three-dimensional space information of the preset virtual scene video is obtained by performing three-dimensional space construction processing on the preset scene video;

[0361] The target object image corresponding to each video frame and the preset virtual scene video are fused according to the collected posture information to obtain a target fused video.

[0362] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0363] Since the computer program stored in the storage medium can execute the steps in any one of the video generation methods provided in the embodiments of the present application, the beneficial effects that can be achieved by any one of the video generation methods provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0364] It can be understood that in the specific implementation of this application, related data such as user information is involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0365] The above is a detailed introduction to a video generation method, device, computer equipment and storage medium provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A video generation method, characterized in that: include: Capturing a video of a target object in a real scene, wherein the video includes at least one video frame; Perform object detection processing on the video frames in the video to obtain the target object image corresponding to each video frame; According to the three-dimensional space information of the preset virtual scene video, the video is positioned to obtain the acquisition posture information of the video, wherein the preset virtual scene video is generated based on the preset scene video, the preset scene video is obtained by performing video acquisition on the real scene, and the three-dimensional space information of the preset virtual scene video is obtained by performing three-dimensional space construction processing on the preset scene video; According to the acquired posture information, the target object image corresponding to each video frame is fused with the preset virtual scene video to obtain a target fused video; Among them, the object detection processing is performed on the video frames in the video to obtain the target object image corresponding to each video frame, including: performing feature encoding processing on the video frame to obtain the feature image of the video frame; performing region decoding processing on the feature image of the video frame to obtain the target object region image of the video frame; and performing region refinement processing on the target object region image according to the feature image of the video frame to obtain the target object image.

2. The method according to claim 1, characterized in that: The method of performing positioning processing on the video according to the three-dimensional space information of the preset virtual scene video to obtain the acquisition posture information of the video includes: Analyze and process the three-dimensional space information of the preset virtual scene video to obtain at least one map point; Matching the video frames of the video with the at least one map point respectively to obtain a map point matching result for each video frame; According to the map point matching result, the position and posture information of the map point corresponding to each video frame is mapped to the corresponding video frame to obtain the acquisition position and posture information of the video.

3. The method according to claim 1, characterized in that Before performing positioning processing on the video according to the three-dimensional space information of the preset virtual scene video to obtain the acquisition posture information of the video, the method further includes: Capturing a preset scene video of the real scene, wherein the preset scene video includes at least one video frame; Performing three-dimensional space construction processing on the video frames of the preset scene video to obtain three-dimensional space information of the real scene in the preset scene video; The preset virtual scene video and the three-dimensional space information of the preset virtual scene video are generated according to the three-dimensional space information of the real scene in the preset scene video.

4. The method according to claim 3, characterized in that The performing three-dimensional space construction processing on the video frame of the preset scene video to obtain the three-dimensional space information of the real scene in the preset scene video includes: Performing three-dimensional space initialization processing on the preset scene video to obtain initial three-dimensional space information of the real scene in the preset scene video; Performing error calculation on the initial three-dimensional space information to obtain three-dimensional space error information; Performing key frame detection on the video frames of the preset scene video according to the three-dimensional space error information to obtain key frames; The initial three-dimensional space information is updated according to the key frame to obtain the three-dimensional space information of the real scene in the preset scene video.

5. The method according to claim 4, characterized in that The performing three-dimensional space initialization processing on the preset scene video to obtain initial three-dimensional space information of the real scene in the preset scene video includes: Determining an initial video frame among the video frames of the preset scene video; Performing feature point detection on the initial video frame to obtain feature point detection information; Performing positioning mark detection on the initial video frame to obtain positioning mark detection information; Initial three-dimensional space information of the real scene is generated according to the feature point detection information and the positioning mark detection information.

6. The method according to claim 5, characterized in that The generating the initial three-dimensional space information of the real scene according to the feature point detection information and the positioning mark detection information includes: Performing posture calculation processing on the feature point detection information and the positioning mark detection information respectively to obtain feature point posture information and positioning mark posture information; Performing triangulation processing on the feature point pose information and the positioning mark pose information respectively to obtain the feature point three-dimensional space information and the positioning mark three-dimensional space information; The initial three-dimensional space information is determined from the feature point three-dimensional space information and the positioning mark three-dimensional space information.

7. The method according to claim 4, characterized in that The step of performing key frame detection on the video frames of the preset scene video according to the three-dimensional space error information to obtain key frames includes: When the three-dimensional space error information does not meet the preset error threshold, performing positioning mark detection on the video frame of the preset scene video; When it is detected that a video frame of the preset scene video has a positioning mark, the video frame is determined as a marked video frame; Performing redundancy determination processing on the identified video frame; When the identification video frame passes the redundancy determination process, the identification video frame is determined as the key frame.

8. The method according to claim 4, characterized in that The updating of the initial three-dimensional space information according to the key frame to obtain the three-dimensional space information of the real scene in the preset scene video includes: Acquire a key frame set of the preset scene video, wherein the key frame set includes at least one historical key frame; Determining a matching key frame that matches the key frame in the historical key frames; Calculate the pose information corresponding to the key frame according to the matching key frame; The initial three-dimensional space information is updated according to the posture information corresponding to the key frame to obtain the three-dimensional space information of the real scene in the preset scene video.

9. The method according to claim 1, characterized in that: The step of performing region refinement processing on the target object region image according to the feature image of the video frame to obtain the target object image comprises: Fusing the target object area image and the feature image of the video frame to obtain a fused feature image; Acquire feature encoding information of the video frame; Fusing the fused feature image and the feature coding information to obtain fused coding information; The fused encoded information is decoded to obtain the target object image.

10. The method according to claim 1, characterized in that The step of fusing the target object image corresponding to each video frame with the preset virtual scene video according to the acquired posture information to obtain a target fused video includes: Performing layer overlay processing on the target object image of the video frame and the video frame of the preset virtual scene video to obtain at least one overlay layer; Setting and processing preset video display parameters according to the collected posture information to obtain set video display parameters; The at least one overlay layer is rendered based on the set video display parameters to obtain the target fused video.

11. The method according to claim 10, characterized in that The step of performing layer overlay processing on the target object image of the video frame and the video frame of the preset virtual scene video to obtain at least one overlay layer includes: Generate a first layer and a second layer by using a preset rendering engine; Utilizing the preset rendering engine to render the target object image and the first layer to obtain a first rendered layer; Utilizing the preset rendering engine to render the preset virtual scene video and the second layer to obtain a second rendered layer; The first rendered layer and the second rendered layer are overlaid to obtain at least one overlaid layer.

12. A video generating device, characterized in that: include: A collection unit, used to collect a video of the target object in a real scene, wherein the video includes at least one video frame; The object detection unit is used to perform object detection processing on the video frames in the video to obtain the target object image corresponding to each video frame, including: performing feature encoding processing on the video frame to obtain the feature image of the video frame; performing region decoding processing on the feature image of the video frame to obtain the target object region image of the video frame; performing region refinement processing on the target object region image according to the feature image of the video frame to obtain the target object image; A positioning unit, configured to perform positioning processing on a preset virtual scene video according to the three-dimensional space information of the video, and obtain acquisition posture information of the video, wherein the preset virtual scene video is generated based on a preset scene video, the preset scene video is obtained by performing video acquisition on the real scene, and the three-dimensional space information of the preset virtual scene video is obtained by performing three-dimensional space construction processing on the preset scene video; A fusion unit is used to fuse the target object image corresponding to each video frame with the preset virtual scene video according to the collected posture information to obtain a target fused video.

13. A computer device, characterized in that: It comprises a memory and a processor; the memory stores a computer program, and the processor is used to run the computer program in the memory to perform the operations in the video generation method according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of computer programs, and the computer programs are suitable for being loaded by a processor to execute the steps in the video generation method according to any one of claims 1 to 11.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the video generation method according to any one of claims 1 to 11 are implemented.

Images