Video generation method and video generation apparatus, electronic device, storage medium
By combining image and video identification based on color and motion features, high-quality initial candidate frames are selected to generate target-generated 3D videos, solving the problem of poor video quality in existing technologies and achieving higher coherence and quality.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- PING AN TECH (SHENZHEN) CO LTD
- Filing Date
- 2023-04-12
- Publication Date
- 2026-06-02
AI Technical Summary
Existing technologies produce poor-quality 3D videos, with issues such as tearing and asynchrony, making it difficult to guarantee video continuity.
During the generation process, a preset image discriminator and video discriminator are used to identify the generated video by combining color features and motion features, thereby selecting high-quality initial candidate frames and generating the target 3D video.
It improves video continuity, reduces image tearing, and enhances the overall quality of the video.
Smart Images
Figure CN116886838B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, and in particular to a video generation method, video generation device, electronic device, and storage medium. Background Technology
[0002] In today's era, with the continuous exploration and development of deep learning and various algorithms, and driven by the diverse needs in the field of computer vision, the demand for AI-generated images and videos has gradually emerged. Among related technologies, generative adversarial networks (GANs) are used to generate 3D videos. However, due to the differences in generation frameworks and the breadth of feature selection labels, the generated 3D videos often suffer from poor quality, exhibiting drawbacks such as tearing, desynchronization, and inability to visualize. Therefore, providing a video generation method that improves video coherence has become an urgent technical problem to be solved. Summary of the Invention
[0003] The main objective of this application is to provide a video generation method, a video generation device, an electronic device, and a storage medium that can reduce image tearing and improve video continuity.
[0004] To achieve the above objectives, a first aspect of this application proposes a video generation method, the method comprising:
[0005] The preset 3D color feature vector, the preset action command feature vector, and the preset original pose feature vector are input into the preset adversarial network; wherein, the preset adversarial network includes a preset image generator, a preset image discriminator, and a preset video discriminator;
[0006] The preset image generator performs image generation processing on the three-dimensional color feature vector, the action command feature vector, and the original posture feature vector to obtain an initial candidate image.
[0007] A reference image is acquired, and the initial candidate image and the reference image are subjected to image identification processing by the preset image discriminator to obtain the image identification label of the initial candidate image; wherein, the preset image discriminator uses color features and motion features as identification indicators to perform the image identification processing;
[0008] Target image labels are selected from the image identification labels based on preset image labels, and preliminary candidate images are selected from the initial candidate images based on the target image labels;
[0009] The preliminary candidate images are merged to obtain initial candidate frames;
[0010] The initial candidate frames are processed by the preset video discriminator to obtain the frame discrimination tags of the initial candidate frames;
[0011] Target frame labels are selected from the frame identification labels based on preset frame labels, and target candidate frames are selected from the initial candidate frames based on the target frame labels;
[0012] The target candidate frames are merged to obtain the target generated 3D video.
[0013] In some embodiments, before performing image discrimination processing on the initial candidate image and the reference image using the preset image discriminator to obtain the image discrimination label of the initial candidate image, the method further includes:
[0014] The initial candidate image is updated by upsampling it through a preset convolutional layer.
[0015] In some embodiments, acquiring the reference image includes:
[0016] The preset three-dimensional reference video is subjected to image segmentation processing to obtain at least one of the reference images;
[0017] or,
[0018] The three-dimensional color feature vector and the action command feature vector are fused to obtain a color-action fusion vector; the color-action fusion vector is then decoded to obtain at least one reference image.
[0019] In some embodiments, the step of performing image discrimination processing on the initial candidate image and the reference image through the preset image discriminator to obtain the image discrimination label of the initial candidate image includes:
[0020] Feature extraction is performed on the initial candidate image to obtain a first color feature vector and a first action feature vector;
[0021] Feature extraction is performed on the reference image to obtain a second color feature vector and a second action feature vector;
[0022] The image color similarity is obtained by calculating the similarity between the first color feature vector and the second color feature vector.
[0023] The image action similarity is obtained by calculating the similarity between the first action feature vector and the second action feature vector.
[0024] The image identification label is determined based on the image color similarity and the image action similarity.
[0025] In some embodiments, the step of performing frame identification processing on the initial candidate frames using the preset video discriminator to obtain frame identification tags for the initial candidate frames includes:
[0026] The frame action similarity is calculated based on two adjacent initial candidate frames; wherein, the frame action similarity is used to characterize the degree of action coherence between the two initial candidate frames.
[0027] If the frame action similarity is greater than the preset coherent action similarity threshold, then the identification labels of the two adjacent initial candidate frames are both positive labels.
[0028] If the frame action similarity is less than or equal to a preset coherent action similarity threshold, then the identification labels of the two adjacent initial candidate frames are both negative labels.
[0029] In some embodiments, the preset image generator includes a mapping network and a first feedforward multilayer perceptron. The step of performing image generation processing on the three-dimensional color feature vector, the action command feature vector, and the original pose feature vector through the preset image generator to obtain an initial candidate image includes:
[0030] The mapping network is used to perform feature fusion processing on each of the three-dimensional color feature vectors and each of the action command feature vectors to obtain at least one initial image motion vector.
[0031] The first forward multilayer perceptron performs vector calculation on each of the initial image motion vectors to obtain the target image motion vector;
[0032] The motion vector of each target image and the original pose feature vector are fused to obtain a fused feature vector.
[0033] The fused feature vector is decoded to obtain the initial candidate image.
[0034] In some embodiments, the preset image generator includes a mapping network, a first feedforward multilayer perceptron, and a backward multilayer perceptron. The step of performing image generation processing on the three-dimensional color feature vector, the action command feature vector, and the original pose feature vector using the preset image generator to obtain an initial candidate image includes:
[0035] The mapping network is used to fuse the three-dimensional color feature vector and the motion command feature vector to obtain at least one initial image motion vector.
[0036] The first forward multilayer perceptron performs vector calculation processing on each of the initial image motion vectors to obtain the target image motion vector.
[0037] The target pose feature vector is obtained by performing vector calculation on the original pose feature vector through the backward multilayer perceptron.
[0038] The motion vector of each target image and the pose feature vector of each target are fused together to obtain a fused feature vector.
[0039] The fused feature vector is decoded to obtain the initial candidate image.
[0040] To achieve the above objectives, a second aspect of this application provides a video generation apparatus, the apparatus comprising:
[0041] The vector input module is used to input preset three-dimensional color feature vectors, preset action command feature vectors, and preset original pose feature vectors into a preset adversarial network; wherein, the preset adversarial network includes a preset image generator, a preset image discriminator, and a preset video discriminator;
[0042] The image generation module is used to perform image generation processing on the three-dimensional color feature vector, the action command feature vector, and the original posture feature vector through the preset image generator to obtain an initial candidate image;
[0043] An image identification module is used to acquire a reference image and perform image identification processing on the initial candidate image and the reference image through the preset image discriminator to obtain an image identification label for the initial candidate image; wherein, the preset image discriminator uses color features and motion features as identification indicators to perform the image identification processing;
[0044] An image filtering module is used to filter target image tags from the image identification tags according to preset image tags, and to filter preliminary candidate images from the initial candidate images according to the target image tags;
[0045] The frame generation module is used to merge the preliminary candidate images to obtain initial candidate frames;
[0046] The frame identification module is used to perform frame identification processing on the initial candidate frame through the preset video discriminator to obtain the frame identification tag of the initial candidate frame;
[0047] A frame filtering module is used to filter target frame tags from the frame identification tags according to preset frame tags, and to filter target candidate frames from the initial candidate frames according to the target frame tags;
[0048] The video generation module is used to merge the target candidate frames to obtain a target-generated 3D video.
[0049] To achieve the above objectives, a third aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect.
[0050] To achieve the above objectives, a fourth aspect of the present application provides a storage medium, which is a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect.
[0051] This application proposes a video generation method, device, electronic equipment, and storage medium. The method involves inputting a three-dimensional color feature vector, a motion command feature vector, and an original pose feature vector into a preset adversarial network (AAN) to generate an image using an image generator within the AAN, resulting in at least one initial candidate image. This way, the initial candidate image contains not only color features but also motion features. When the initial candidate image is evaluated by an image discriminator, the color and motion features within it are utilized, improving the accuracy of image identification labels. Furthermore, a video discriminator is added to further filter images based on frame identification labels, resulting in target candidate frames with higher identification accuracy. A target three-dimensional video is then generated based on these target candidate frames. In summary, the embodiments of this application improve the identification accuracy of initial candidate images and frames, thereby improving the image quality of target candidate frames, reducing image tearing in the video, and enhancing video continuity. Attached Figure Description
[0052] Figure 1 This is a flowchart of the video generation method provided in the embodiments of this application;
[0053] Figure 2 yes Figure 1 The flowchart of step S102 in the document;
[0054] Figure 3 yes Figure 1 The flowchart of step S102 in the document;
[0055] Figure 4 yes Figure 1 The flowchart of step S103 in the process;
[0056] Figure 5 yes Figure 1 The flowchart of step S103 in the process;
[0057] Figure 6 yes Figure 1 The flowchart of step S106 in the process;
[0058] Figure 7 This is a block diagram of the module structure of the video generation device provided in the embodiments of this application;
[0059] Figure 8 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation
[0060] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0061] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.
[0062] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.
[0063] First, let's analyze some of the terms used in this application:
[0064] Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.
[0065] Natural Language Processing (NLP): NLP uses computers to process, understand, and utilize human language (such as Chinese and English). It is a branch of artificial intelligence and an interdisciplinary field of computer science and linguistics, often referred to as computational linguistics. NLP includes syntactic analysis, semantic analysis, and discourse understanding. It is commonly used in machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information and image processing, information extraction and filtering, text classification and clustering, sentiment analysis, and opinion mining. It involves data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computation.
[0066] A multilayer perceptron (MLP) is a feedforward artificial neural network model that maps multiple input datasets to a single output dataset.
[0067] Backpropagation: The general principle of backpropagation is as follows: The training data is input into the input layer of the neural network, passes through the hidden layer, and finally reaches the output layer of the neural network to output the result; Since there is an error between the output result of the neural network and the actual result, the error between the estimated value and the actual value is calculated, and this error is backpropagated from the output layer to the hidden layer until it reaches the input layer; During the backpropagation process, the values of various parameters are adjusted according to the error; The above process is iterated continuously until convergence.
[0068] Encoder-decoder: This is a common model framework in deep learning. Many common applications are designed using the encoder-decoder framework. The encoder and decoder parts can be any text, speech, image, and video data, etc. Various models can be designed based on the encoder-decoder.
[0069] In today's era, with the continuous exploration and development of deep learning and various algorithms, and driven by the diverse needs in the field of computer vision, the demand for AI-generated images and videos has gradually emerged. Among related technologies, generative adversarial networks (GANs) are used to generate videos. However, due to the differences in generation frameworks and the breadth of feature selection labels, the generated videos often suffer from poor quality. For example, they may exhibit tearing, desynchronization, and an inability to visualize. Therefore, how to provide a video generation method that can improve video coherence has become an urgent technical problem to be solved.
[0070] Based on this, the main objective of this application is to propose a video generation method, a video generation apparatus, an electronic device, and a storage medium. The aim is to improve the accuracy of image identification labels by utilizing the color and motion features contained in the initial candidate images when identifying them using an image discriminator. Furthermore, a video discriminator is added to further filter images based on frame identification labels, thereby obtaining target candidate frames with higher identification accuracy. A 3D video is then generated from the target candidate frames, improving the video's coherence.
[0071] The video generation method provided in this application can be applied to a server, or it can be software running on a terminal or server. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The software can be an application that implements the video generation method, but it is not limited to the above forms.
[0072] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include server computers, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.
[0073] This application provides a video generation method, a video generation device, an electronic device, and a storage medium. The specific details are illustrated in the following embodiments. First, the video generation method in this application is described.
[0074] Figure 1 This is an optional flowchart of the video generation method provided in the embodiments of this application, which may include, but is not limited to, steps S101 to S108.
[0075] Step S101: Input the preset 3D color feature vector, the preset action command feature vector, and the preset original pose feature vector into the preset adversarial network; wherein, the preset adversarial network includes a preset image generator, a preset image discriminator, and a preset video discriminator;
[0076] Step S102: The three-dimensional color feature vector, action command feature vector and original pose feature vector are processed by a preset image generator to generate an initial candidate image.
[0077] Step S103: Obtain a reference image, and perform image discrimination processing on the initial candidate image and the reference image through a preset image discriminator to obtain the image discrimination label of the initial candidate image; wherein, the preset image discriminator uses color features and motion features as discrimination indicators to perform image discrimination processing.
[0078] Step S104: Select target image labels from image identification labels according to preset image labels, and select preliminary candidate images from initial candidate images according to target image labels;
[0079] Step S105: Merge the preliminary candidate images to obtain the initial candidate frames;
[0080] Step S106: Perform frame identification processing on the initial candidate frames using a preset video discriminator to obtain the frame identification tags of the initial candidate frames.
[0081] Step S107: Filter out target frame tags from frame identification tags according to preset frame tags, and filter out target candidate frames from initial candidate frames according to target frame tags;
[0082] Step S108: Merge the target candidate frames to obtain the target generated 3D video.
[0083] In steps S101 to S108 of this embodiment, the three-dimensional color feature vector, action command feature vector, and original pose feature vector are input into a preset adversarial network (AAN) to generate an image using the AAN's image generator, resulting in at least one initial candidate image. This way, the initial candidate image contains not only color features but also action features. When the image discriminator identifies the initial candidate image, it utilizes the color and action features contained within the initial candidate image, improving the accuracy of the image identification label. Furthermore, a video discriminator is added for identification, further filtering the image based on the frame identification label to obtain target candidate frames with higher identification accuracy. Based on the target candidate frames, a target 3D video is generated. In summary, this embodiment improves the image quality of the target candidate frame by increasing the identification accuracy of the initial candidate image and initial candidate frame, reducing video tearing and improving video continuity.
[0084] In step S101 of some embodiments, a three-dimensional color feature vector is used to represent the color features of the image. An action command feature vector is used to represent the action features of the image. An original pose feature vector is used to represent the pose features of the image. If the image contains a person, the action command is used to represent actions such as picking up, throwing, and walking. If the image contains an animal, the action command is used to represent actions such as running, jumping, and sitting.
[0085] It should be noted that the preset adversarial network is a generative adversarial network (GAN). The preset adversarial network in this embodiment includes a preset image generator, a preset image discriminator, and a preset video discriminator. The initial image generator generates initial candidate images. The preset image discriminator discriminates the initial candidate images to obtain image discrimination labels. The preset video discriminator discriminates the initial candidate frames to obtain frame discrimination labels.
[0086] Taking a 3D color image as an example, the 3D color image is processed by color (RGB) encoding to obtain a 3D color feature vector. Image transformation processing is then performed on the 3D color image to obtain a laser image; Fourier transform processing is then performed on the laser image to obtain a pose feature vector. It should be noted that the 3D color image does not contain the action command features referred to in the embodiments of this application, and therefore, it is impossible to obtain the action command feature vector from the 3D color image. Therefore, in related technologies, it is impossible to identify images based on action similarity. In the embodiments of this application, action commands are encoded to obtain action command feature vectors; subsequently, these action command feature vectors are integrated into the image so that the image can be identified through action feature similarity, thereby improving the identification accuracy.
[0087] The 3D color feature vector is stored in the 3D video color encoding set, the motion command feature vector is stored in the motion command encoding set, and the original pose feature vector is stored in the image pose encoding set. All three encoding sets can be stored in a preset database.
[0088] In one example, the original 3D video is acquired; the original 3D video is then color-coded (RGB encoding) to obtain a 3D video color encoding set. Specifically, the original 3D video is image-splitting to obtain at least two original images; each original image is color-coded to obtain an original image color feature vector; the color feature vectors of the at least two original images are merged to obtain the 3D video color encoding set. A residual layer can be used to extract the color features of the original images to obtain the original image color feature vectors.
[0089] In another example, action commands typically include lifting, running, standing, and sitting. Each action command is encoded to obtain an action command feature vector. At least two action command feature vectors are merged to obtain an action command encoding set.
[0090] In another example, a camera is used to photograph the target object, obtaining at least one pose image; the target object includes a person or other animal, etc.; the pose image is encoded to obtain a pose feature vector; at least two pose feature vectors are merged to obtain a human pose image encoding set. The pose image can be subjected to a Fourier transform to obtain the human pose feature vector. Alternatively, the pose image can first be converted into a laser image, and then a Fourier transform can be performed on the laser image to obtain the pose feature vector. It should be noted that the use of a laser image improves the efficiency of feature extraction.
[0091] In step S102 of some embodiments, an image is generated by an initial image generator to obtain at least one initial candidate image. Specifically, after the initial image generator performs image generation processing on the three-dimensional color feature vector, motion command feature vector, and original pose feature vector, at least one initial candidate image is obtained. This initial candidate image retains the original pose features but also incorporates color and motion features.
[0092] In one example, the 3D color feature vector, motion command feature vector, and original pose feature vector are concatenated to obtain the target concatenated vector; the target concatenated vector is then decoded by a preset decoder to obtain the initial candidate image.
[0093] In another example, the 3D color feature vector, motion command feature vector, and original pose feature vector are weighted and summed to obtain the target fusion vector. The target fusion vector is then decoded using a pre-defined decoder to obtain the initial candidate image. Among these, the original pose feature has the highest weight, the 3D color feature vector has a weight between that of the original pose feature and the motion command feature vector, and the motion command feature vector has the lowest weight.
[0094] In some embodiments, see Figure 2 The preset image generator includes a mapping network and a first feedforward multilayer perceptron. Step S102 specifically includes, but is not limited to, steps S201 to S204:
[0095] Step S201: Perform feature fusion processing on each three-dimensional color feature vector and each action command feature vector through a mapping network to obtain at least one initial image motion vector.
[0096] Step S202: Perform vector calculation on each initial image motion vector through the first feedforward multilayer perceptron to obtain the target image motion vector;
[0097] Step S203: Perform vector fusion on each target image motion vector and each original pose feature vector to obtain a fused feature vector;
[0098] Step S204: Decode the fused feature vector to obtain the initial candidate image.
[0099] Steps S201 to S204, as shown in the embodiments of this application, perform vector fusion processing through a mapping network and a first forward perceptron, which also improves the accuracy of feature fusion, thereby improving the generation quality of the initial candidate image.
[0100] It should be noted that the mapping network is also called mapping. The first forward multilayer perceptron, the second forward multilayer perceptron, and the backward multilayer perceptron are all conventional multilayer perceptrons. In this embodiment, the first and second forward multilayer perceptrons are multilayer perceptrons that use the forward algorithm for vector computation, while the backward multilayer perceptron is a multilayer perceptron that uses the backward algorithm for vector computation. The forward algorithm performs computation from the input layer -> hidden layer -> output layer, and the result of each layer affects the data of the next layer. The backward algorithm can be called backpropagation computation or reverse propagation computation. The backward algorithm assumes a weight for the output layer and works backward to infer the most suitable vector matrix until the corresponding weight is found to calculate the output matrix vector.
[0101] In some embodiments, see Figure 3 The preset image generator includes a mapping network, a first forward multilayer perceptron, and a backward multilayer perceptron. Step S102 specifically includes, but is not limited to, steps S301 to S305:
[0102] Step S301: The three-dimensional color feature vector and the action command feature vector are fused through a mapping network to obtain at least one initial image motion vector;
[0103] Step S302: Perform vector calculation processing on each initial image motion vector through the first feedforward multilayer perceptron to obtain the target image motion vector;
[0104] Step S303: The original attitude feature vector is processed by a backward multilayer perceptron to obtain the target attitude feature vector.
[0105] Step S304: Perform vector fusion processing on the motion vector of each target image and the pose feature vector of each target to obtain a fused feature vector;
[0106] Step S305: Decode the fused feature vector to obtain the initial candidate image.
[0107] Steps S301 to S305, as shown in the embodiments of this application, perform vector fusion processing through a mapping network, a first forward perceptron, and a backward multilayer perceptron, which also improves the accuracy of feature fusion and thus improves the generation quality of the initial candidate image.
[0108] In some embodiments, the preset image generator further includes a second forward perceptron. Before step S202 or step S302, the generator further includes updating the initial image motion vector. Specifically, this includes inputting the initial image motion feature vector into the second forward perceptron, and performing vector calculation on the initial image motion vector through a second forward multilayer perceptron to update the initial image motion vector. After updating the initial image motion vector, vector calculation processing is then performed on each initial image motion vector through a first forward multilayer perceptron to obtain the target image motion vector. This embodiment further improves the motion features of the target image motion vector, thereby helping to improve the generation quality of the initial candidate image.
[0109] After step S102, the image generation process has been completed, resulting in at least two initial candidate images. However, the quality of the generated initial candidate images is unknown. Therefore, step S103, image identification processing, is introduced to identify the initial candidate images and obtain image identification labels. These labels are then used to determine the generation quality of the initial candidate images.
[0110] In one embodiment, prior to step S103, the video generation method further includes: upsampling the initial candidate image through a preset convolutional layer to update the initial candidate image. This embodiment can further improve the image quality of the initial candidate image, such as image sharpness, thereby further improving the accuracy of image identification.
[0111] Specifically, a pre-defined convolutional layer is also called a convolutional kernel. In image processing, given an input image, the pixels in a small region of the input image are weighted and averaged to become each corresponding pixel in the output image. The weights are defined by a function called the convolutional kernel.
[0112] In step S103 of some embodiments, the initial candidate image is identified by a preset image discriminator to obtain an image identification label. This image identification label may include a positive label and a negative label. Specifically, a similarity calculation is performed between the initial candidate image and the reference image to obtain an image similarity score; if the image similarity score is less than a preset image similarity threshold, the identification label is a negative label; if the image similarity score is greater than or equal to the preset image similarity threshold, the identification label is a positive label. More specifically, a similarity calculation is performed between the initial candidate image and the reference image using a preset activation function to obtain the image similarity score. The activation function may be a ReLU function.
[0113] During the identification process, since the initial candidate images contain motion features in addition to color features, both color features and motion features can be used as identification indicators for image identification. Furthermore, the increased number of identification indicators improves the accuracy of image identification.
[0114] The image identification in step S103 requires identifying the initial candidate image based on the reference image; therefore, the reference image is obtained first.
[0115] In one embodiment, obtaining the reference image specifically includes:
[0116] The preset 3D reference video is subjected to image splitting processing to obtain at least one reference image.
[0117] In this embodiment, the reference image is obtained by splitting a 3D reference video, and the 3D color feature vector in step S101 is obtained by color encoding the 3D reference video so that color feature similarity can be used as a discrimination criterion. Since the reference image has no action features, the weight of action similarity needs to be reduced during image discrimination, for example, reduced to zero.
[0118] In another embodiment, see Figure 4 In step S103, obtaining the reference image specifically includes steps S401 to S402;
[0119] Step S401: Perform feature fusion processing on the three-dimensional color feature vector and the action command feature vector to obtain the color-action fusion vector;
[0120] Step S402: Decode the color motion fusion vector to obtain at least one reference image.
[0121] In steps S401 to S402 of the embodiments of this application, since the reference image is obtained by fusing three-dimensional color feature vector and action command feature vector, during image identification, image color similarity and image action similarity can be used as identification conditions to identify the initial candidate image. The weights of image color similarity and image action similarity are the same or not significantly different.
[0122] In another embodiment, see Figure 5 Step S103 specifically includes, but is not limited to, steps S501 to S505;
[0123] Step S501: Extract features from the initial candidate image to obtain a first color feature vector and a first action feature vector;
[0124] Step S502: Extract features from the reference image to obtain the second color feature vector and the second action feature vector;
[0125] Step S503: Calculate the similarity based on the first color feature vector and the second color feature vector to obtain the image color similarity.
[0126] Step S504: Calculate the similarity based on the first action feature vector and the second action feature vector to obtain the image action similarity.
[0127] Step S505: Determine image identification labels based on image color similarity and image action similarity.
[0128] Steps S501 to S505, as illustrated in this embodiment, determine image identification labels based on image color similarity and image action similarity to identify initial candidate images. Compared to related technologies, this embodiment adds image action similarity as an identification indicator, improving identification accuracy. Furthermore, the action feature vector in this embodiment is derived from and used to generate the initial candidate image by incorporating action command feature vectors. Compared to directly obtaining image action vectors in existing technologies, this improves the efficiency and accuracy of action feature extraction, further enhancing the accuracy of subsequent image identification.
[0129] In step S501 of some embodiments, a 152-layer residual network is used to extract color features from the initial candidate image to obtain a first color feature vector; a multimodal fusion network is used to extract action features from the initial candidate image to obtain a first action feature vector.
[0130] In step S502 of some embodiments, a 152-layer residual network is used to extract color features from the reference image to obtain a second color feature vector; a multimodal fusion network is used to extract action features from the reference image to obtain a second action feature vector.
[0131] In some embodiments, steps S503 and S504 are used to calculate similarity using Euclidean distance or cosine similarity. This application does not limit the specific method of similarity calculation.
[0132] In one example, step S505 specifically includes:
[0133] If the image color similarity is greater than the preset color similarity threshold, and the image action similarity is greater than the preset action similarity threshold, then the image identification label is a positive label;
[0134] If the image color similarity is greater than the preset color similarity threshold, and the image action similarity is less than or equal to the preset action similarity threshold, then the image identification label is a negative label.
[0135] If the image color similarity is less than or equal to the preset color similarity threshold, and the image action similarity is greater than the preset action similarity threshold, then the image identification label is a positive label.
[0136] If the image color similarity is less than or equal to a preset color similarity threshold, and the image action similarity is less than or equal to a preset action similarity threshold, then the image identification label is a negative label.
[0137] In another example, step S505 specifically includes:
[0138] The image fusion similarity is obtained by weighted summation based on image color similarity and image action similarity;
[0139] If the image fusion similarity is greater than the preset fusion similarity threshold, the image identification label is a positive label;
[0140] If the image fusion similarity is less than or equal to the preset fusion similarity threshold, the image identification label is a negative label.
[0141] In step S104 of some embodiments, a preset image label is used to indicate that the image quality is superior. For example, if the image identification label includes positive and negative labels, then the target image label determined according to the preset image label is a positive label. In one example, the positive label is also called a true label, and the negative label is called a false label. In another example, the true label is also called a superior label, and the negative label is called a poor label.
[0142] In step S105 of some embodiments, the initial candidate images are filtered to obtain at least two preliminary candidate images whose image identification tags are the target image tags. The initial candidate images are then merged to obtain at least two initial candidate frames. Each of the initial candidate frames includes at least two preliminary candidate images.
[0143] In some embodiments, step S105 specifically includes: merging the preliminary candidate images to obtain an initial generated 3D video; splitting the initial generated 3D video into frames to obtain at least two candidate frames; performing action recognition on each candidate image frame; and merging two candidate frames belonging to the same action to obtain an initial candidate frame. Here, belonging to the same action can mean that the action similarity between the two candidate frames is greater than a preset frame similarity threshold. Belonging to the same action can also mean that the action command recognition results of the two candidate frames are the same.
[0144] In step S106 of some embodiments, frame identification processing is added on the basis of previous image identification processing to further improve the accuracy of identification.
[0145] In another embodiment, see Figure 6 Step S106 specifically includes, but is not limited to, steps S601 to S603;
[0146] Step S601: Calculate the action similarity based on two adjacent initial candidate frames to obtain the frame action similarity; wherein, the frame action similarity is used to characterize the degree of action coherence between the two initial candidate frames.
[0147] Step S602: If the frame action similarity is greater than the preset coherent action similarity threshold, then the identification labels of the two adjacent initial candidate frames are both positive labels.
[0148] Step S603: If the frame action similarity is less than or equal to the preset coherent action similarity threshold, then the identification labels of the two adjacent initial candidate frames are both negative labels.
[0149] In the embodiments of this application, steps S601 to S603 determine frame identification labels based on the comparison result between frame action similarity and a coherent action similarity threshold. If the frame action similarity is greater than a preset coherent action similarity threshold, it indicates that the actions of two adjacent initial candidate frames have a high degree of coherence, and both are positive labels. Otherwise, they are all negative labels.
[0150] It should be noted that if image color and motion identification processing is performed on a single initial candidate frame and a reference image, the image identification condition for that initial candidate frame is a positive label. However, this does not mean that screen tearing will not occur during video generation. Therefore, after image identification processing, video identification processing is added to improve the accuracy of identification, thereby improving the quality of 3D video generation.
[0151] In some embodiments, after step S106, the video generation method further includes: updating the preset image generator, specifically including:
[0152] The parameters of the preset image generator are adjusted based on the initial candidate images and image identification labels to update the preset image generator;
[0153] And / or,
[0154] The preset image generator is updated by adjusting its parameters based on the initial candidate frames and frame identification tags.
[0155] In one example, the target candidate image is obtained by performing image generation processing on the 3D color feature vector, motion command feature vector and original pose feature vector using an updated preset image generator;
[0156] At least two target candidate images are merged to obtain a target-generated 3D video.
[0157] Specifically, by updating the preset image generator, the initial candidate images generated by the preset image generator cannot be identified by the preset image discriminator and / or the preset video discriminator, thereby improving the image generation quality and thus helping to improve the video generation quality.
[0158] In step S107 of some embodiments, a target frame label is obtained from the frame identification label according to a preset video identification label; the initial candidate frames are filtered according to the target frame label to obtain the target candidate frame.
[0159] It should be noted that the preset frame label is used to indicate that the frame quality is relatively good. For example, if the frame identification label includes positive and negative labels, then the target frame label determined based on the preset frame label is the positive label. In one example, the positive label is also called the true label, and the negative label is called the false label. In another example, the true label is also called the excellent label, and the negative label is called the poor label.
[0160] In step S108 of some embodiments, noise values are generally introduced to generate a relatively complete and accurate new image, in order to improve the robustness of the image generator. However, in the embodiments of this application, the dual identification process of image identification processing and frame identification processing improves the identification accuracy and obtains higher-quality target candidate frames, thereby improving the quality of the target-generated 3D video. For example, the target-generated 3D video has better motion continuity, reduces screen tearing and asynchrony, and makes screen transitions smoother.
[0161] It should also be noted that in this embodiment, an initial candidate image is obtained through a preset image generator, a preliminary candidate image is obtained through a preset image discriminator, a target candidate frame is obtained through a preset video discriminator, and finally, a target-generated 3D video is obtained. The target pose features of each target candidate frame may not differ much from the original pose features, but the color similarity and motion similarity of adjacent target candidate frames are higher. Since the high similarity of adjacent images can reduce screen tearing or desynchronization during video playback, this embodiment can at least avoid screen tearing in the target-generated video, thus improving the quality of video generation.
[0162] Please see Figure 7 This application also provides a video generation apparatus that can implement the above-described video generation method. Figure 7This is a block diagram of the module structure of a video generation device provided in an embodiment of this application. The device includes: a vector input module 701, an image generation module 702, an image discrimination module 703, an image filtering module 704, a frame generation module 705, a frame discrimination module 706, a frame filtering module 707, and a video generation module 708. The vector input module 701 is used to input preset three-dimensional color feature vectors, preset action command feature vectors, and preset original pose feature vectors into a preset adversarial network; wherein the preset adversarial network is a generative adversarial network, and includes a preset image generator, a preset image discriminator, and a preset video discriminator; the image generation module 702 is used to perform image generation processing on the three-dimensional color feature vectors, action command feature vectors, and original pose feature vectors through the preset image generator to obtain initial candidate images; the image discrimination module 703 is used to acquire a reference image, and perform image discrimination processing on the initial candidate image and the reference image through the preset image discriminator to obtain image discrimination labels for the initial candidate images; wherein the preset image discriminator uses color features... The image identification process uses feature and action features as identification indicators; the image filtering module 704 is used to filter target image labels from image identification labels according to preset image labels, and to filter preliminary candidate images from initial candidate images according to target image labels; the frame generation module 705 is used to merge preliminary candidate images to obtain initial candidate frames; the frame identification module 706 is used to perform frame identification processing on the initial candidate frames through a preset video discriminator to obtain frame identification labels for the initial candidate frames; the frame filtering module 707 is used to filter target frame labels from frame identification labels according to preset frame labels, and to filter target candidate frames from initial candidate frames according to target frame labels; the video generation module 708 is used to merge target candidate frames to obtain target generated 3D video.
[0163] It should be noted that the specific implementation of this video generation device is basically the same as the specific implementation of the video generation method described above, and will not be repeated here.
[0164] This application also provides an electronic device, which includes: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for communication between the processor and the memory. When the program is executed by the processor, it implements the aforementioned video generation method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.
[0165] Please see Figure 8 , Figure 8 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:
[0166] The processor 801 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.
[0167] The memory 802 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 802 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and called and executed by the processor 801 using the video generation method of the embodiments of this application.
[0168] The 803 input / output interface is used to implement information input and output.
[0169] The communication interface 804 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).
[0170] Bus 805 transmits information between various components of the device (e.g., processor 801, memory 802, input / output interface 803, and communication interface 804);
[0171] The processor 801, memory 802, input / output interface 803, and communication interface 804 are connected to each other within the device via bus 805.
[0172] This application embodiment also provides a storage medium, which is a computer-readable storage medium for computer-readable storage. The storage medium stores one or more programs, which can be executed by one or more processors to implement the above-described video generation method.
[0173] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.
[0174] The video generation method, device, electronic device, and storage medium provided in this application embodiment input a three-dimensional color feature vector, motion command feature vector, and original pose feature vector into a preset adversarial network (AAN) to generate an image through the AAN's image generator, obtaining at least one initial candidate image. Thus, the initial candidate image contains not only color features but also motion features. When the initial candidate image is identified by an image discriminator, the color and motion features contained in the initial candidate image are utilized, improving the accuracy of image identification labels. Furthermore, a video discriminator is added for identification, further filtering the image based on frame identification labels to obtain target candidate frames with higher identification accuracy. The target candidate frame is then used to generate a target three-dimensional video. In summary, this application embodiment improves the image quality of the target candidate frame by increasing the identification accuracy of the initial candidate image and initial candidate frame, reducing video tearing, and improving the overall quality of the generated video.
[0175] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.
[0176] It will be understood by those skilled in the art that Figure 1-6 The technical solutions shown do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.
[0177] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.
[0178] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.
[0179] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0180] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.
[0181] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.
[0182] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0183] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0184] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0185] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.
Claims
1. A video generation method, characterized in that, The method includes: The preset 3D color feature vector, the preset action command feature vector, and the preset original pose feature vector are input into the preset adversarial network; wherein, the preset adversarial network includes a preset image generator, a preset image discriminator, and a preset video discriminator; The preset image generator performs image generation processing on the three-dimensional color feature vector, the action command feature vector, and the original posture feature vector to obtain an initial candidate image. A reference image is acquired, and the initial candidate image and the reference image are subjected to image identification processing by the preset image discriminator to obtain the image identification label of the initial candidate image; wherein, the preset image discriminator uses color features and motion features as identification indicators to perform the image identification processing; Target image labels are selected from the image identification labels based on preset image labels, and preliminary candidate images are selected from the initial candidate images based on the target image labels; The preliminary candidate images are merged to obtain initial candidate frames; The initial candidate frames are processed by the preset video discriminator to obtain the frame discrimination tags of the initial candidate frames; Target frame labels are selected from the frame identification labels based on preset frame labels, and target candidate frames are selected from the initial candidate frames based on the target frame labels; The target candidate frames are merged to obtain the target generated 3D video.
2. The method according to claim 1, characterized in that, Before performing image discrimination processing on the initial candidate image and the reference image using the preset image discriminator to obtain the image discrimination label of the initial candidate image, the method further includes: The initial candidate image is updated by upsampling it through a preset convolutional layer.
3. The method according to claim 1, characterized in that, The acquisition of the reference image includes: The preset three-dimensional reference video is subjected to image segmentation processing to obtain at least one of the reference images; or, The three-dimensional color feature vector and the action command feature vector are fused to obtain a color-action fusion vector; the color-action fusion vector is then decoded to obtain at least one reference image.
4. The method according to claim 1, characterized in that, The step of performing image discrimination processing on the initial candidate image and the reference image through the preset image discriminator to obtain the image discrimination label of the initial candidate image includes: Feature extraction is performed on the initial candidate image to obtain a first color feature vector and a first action feature vector; Feature extraction is performed on the reference image to obtain a second color feature vector and a second action feature vector; The image color similarity is obtained by calculating the similarity between the first color feature vector and the second color feature vector. The image action similarity is obtained by calculating the similarity between the first action feature vector and the second action feature vector. The image identification label is determined based on the image color similarity and the image action similarity.
5. The method according to claim 1, characterized in that, The step of performing frame identification processing on the initial candidate frames using the preset video discriminator to obtain frame identification tags for the initial candidate frames includes: The frame action similarity is calculated based on two adjacent initial candidate frames; wherein, the frame action similarity is used to characterize the degree of action coherence between the two initial candidate frames. If the frame action similarity is greater than the preset coherent action similarity threshold, then the identification labels of the two adjacent initial candidate frames are both positive labels. If the frame action similarity is less than or equal to a preset coherent action similarity threshold, then the identification labels of the two adjacent initial candidate frames are both negative labels.
6. The method according to any one of claims 1 to 5, characterized in that, The preset image generator includes a mapping network and a first feedforward multilayer perceptron. The process of generating an initial candidate image by using the preset image generator to process the 3D color feature vector, the action command feature vector, and the original pose feature vector includes: The mapping network is used to perform feature fusion processing on each of the three-dimensional color feature vectors and each of the action command feature vectors to obtain at least one initial image motion vector. The first forward multilayer perceptron performs vector calculation on each of the initial image motion vectors to obtain the target image motion vector; The motion vector of each target image and the original pose feature vector are fused to obtain a fused feature vector. The fused feature vector is decoded to obtain the initial candidate image.
7. The method according to any one of claims 1 to 5, characterized in that, The preset image generator includes a mapping network, a first forward multilayer perceptron, and a backward multilayer perceptron. The process of generating an initial candidate image by using the preset image generator to process the 3D color feature vector, the action command feature vector, and the original pose feature vector includes: The mapping network is used to fuse the three-dimensional color feature vector and the motion command feature vector to obtain at least one initial image motion vector. The first forward multilayer perceptron performs vector calculation processing on each of the initial image motion vectors to obtain the target image motion vector. The target pose feature vector is obtained by performing vector calculation on the original pose feature vector through the backward multilayer perceptron. The motion vector of each target image and the pose feature vector of each target are fused together to obtain a fused feature vector. The fused feature vector is decoded to obtain the initial candidate image.
8. A video generation apparatus, characterized in that, The device includes: The vector input module is used to input preset three-dimensional color feature vectors, preset action command feature vectors, and preset original pose feature vectors into a preset adversarial network; wherein, the preset adversarial network includes a preset image generator, a preset image discriminator, and a preset video discriminator; The image generation module is used to perform image generation processing on the three-dimensional color feature vector, the action command feature vector, and the original posture feature vector through the preset image generator to obtain an initial candidate image; An image identification module is used to acquire a reference image and perform image identification processing on the initial candidate image and the reference image through the preset image discriminator to obtain an image identification label for the initial candidate image; wherein, the preset image discriminator uses color features and motion features as identification indicators to perform the image identification processing; An image filtering module is used to filter target image tags from the image identification tags according to preset image tags, and to filter preliminary candidate images from the initial candidate images according to the target image tags; The frame generation module is used to merge the preliminary candidate images to obtain initial candidate frames; The frame identification module is used to perform frame identification processing on the initial candidate frame through the preset video discriminator to obtain the frame identification tag of the initial candidate frame; A frame filtering module is used to filter target frame tags from the frame identification tags according to preset frame tags, and to filter target candidate frames from the initial candidate frames according to the target frame tags; The video generation module is used to merge the target candidate frames to obtain a target-generated 3D video.
9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the video generation method according to any one of claims 1 to 7.
10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the video generation method according to any one of claims 1 to 7.