3D special effect enhancement method and device, equipment and storage medium

Abstract

The invention provides a 3D special effect enhancement method and device, equipment and a storage medium. The method comprises: according to guidance information, determining a plurality of images to be enhanced and position information of a target object in each image to be enhanced from video data, the video data comprising a plurality of continuous frame images; determining three-dimensional motion information of a target object according to the position information of the target object in the plurality of to-be-enhanced images; performing image coding processing on the plurality of to-be-enhanced images to obtain feature parameters of each to-be-enhanced image; and based on the three-dimensional motion information of the target object, the feature parameters of the plurality of to-be-enhanced images and the guide information, generating 3D special effect enhanced images corresponding to the plurality of to-be-enhanced images. According to the method provided by the invention, automatic 3D special effect enhancement processing is realized.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a 3D special effects enhancement method, device, equipment and storage medium. Background Art

[0002] 3D special effects, the full name of which is Three-Dimensional Effects, is a technology widely used in the production of multimedia content such as movies, television, games, advertising, animation, etc. It uses computer graphics and three-dimensional modeling technology to create images or animation effects with a sense of depth, stereoscopic feeling and realism on a two-dimensional screen, which can greatly enhance the audience's visual experience.

[0003] Most of the existing research on 2D to 3D conversion focuses on monocular to binocular image conversion, usually directly using another view as a label for pixel-level image reconstruction.

[0004] Moreover, existing 3D special effects enhancement methods have limited usage scenarios in actual environments and require the use of binocular cameras in virtual environments or manual design and adjustment. Summary of the invention

[0005] The present application provides a 3D special effect enhancement method, device, equipment and storage medium to solve the problem in the prior art that 3D special effect enhancement processing cannot be automatically performed on non-3D videos / images.

[0006] On the one hand, the present application provides a 3D special effects enhancement method, comprising:

[0007] Determine, from the video data, a plurality of images to be enhanced and position information of a target object in each of the images to be enhanced according to the guidance information, wherein the video data includes a plurality of continuous frame images;

[0008] Determining three-dimensional motion information of the target object according to position information of the target object in the multiple images to be enhanced;

[0009] Performing image coding processing on the multiple images to be enhanced respectively to obtain feature parameters of each image to be enhanced;

[0010] Based on the three-dimensional motion information of the target object, the characteristic parameters of the multiple images to be enhanced and the guidance information, 3D special effect enhanced images corresponding to the multiple images to be enhanced are generated.

[0011] Optionally, the guidance information includes: guidance prompt text, and the determining of a plurality of images to be enhanced from the video data according to the guidance information, and the position information of the target object in each of the images to be enhanced, includes:

[0012] Performing semantic analysis on the guidance prompt text to obtain a corresponding target object;

[0013] Based on the target object, performing image analysis processing on the plurality of continuous frame images to obtain an analysis result, wherein the analysis result is used to indicate whether the corresponding frame image contains the target object;

[0014] According to the analysis result, a plurality of images to be enhanced that contain the target object are determined from the video data, and position information of the target object in each image to be enhanced is determined.

[0015] Optionally, determining the three-dimensional motion information of the target object according to the position information of the target object in the multiple images to be enhanced includes:

[0016] Determining a moving direction of the target object according to position information of the target object in the plurality of images to be enhanced;

[0017] Determining a motion trajectory of the target object in three-dimensional space according to a motion direction of the target object;

[0018] The motion trajectory is encoded to obtain three-dimensional motion information of the target object.

[0019] Optionally, the generating, based on the three-dimensional motion information of the target object, the characteristic parameters of the multiple images to be enhanced and the guidance information, 3D special effect enhanced images corresponding to the multiple images to be enhanced includes:

[0020] Inputting the three-dimensional motion information of the target object, the characteristic parameters of the plurality of images to be enhanced and the guiding information into a generation model;

[0021] The three-dimensional motion information of the target object, the characteristic parameters of the multiple images to be enhanced and the guide information are analyzed and processed by the generation model to obtain 3D special effect enhanced images corresponding to the multiple images to be enhanced.

[0022] Optionally, the guidance information further includes: disparity information and optical flow information of the multiple images to be enhanced, and the three-dimensional motion information of the target object, the characteristic parameters of the multiple images to be enhanced, and the guidance information are analyzed and processed by the generation model to obtain 3D special effect enhanced images corresponding to the multiple images to be enhanced, including:

[0023] Optimizing and adjusting the generation model according to the disparity information and the optical flow information;

[0024] The adjusted generation model is used to output 3D special effect enhanced images corresponding to the multiple images to be enhanced according to the three-dimensional motion information of the target object, the characteristic parameters of the multiple images to be enhanced and the guidance information.

[0025] Optionally, the image to be enhanced includes: a left view and a right view, and after generating 3D special effect enhanced images corresponding to the multiple images to be enhanced based on the three-dimensional motion information of the target object, the characteristic parameters of the multiple images to be enhanced and the guidance information, the method further includes:

[0026] Decoding the 3D special effect enhanced images corresponding to the multiple images to be enhanced to obtain decoded left view and right view;

[0027] The decoded left view and right view are synthesized to obtain 3D special effect enhanced videos corresponding to the multiple images to be enhanced.

[0028] On the other hand, the present application provides a device for enhancing 3D special effects, comprising:

[0029] A determination module, configured to determine a plurality of images to be enhanced and position information of a target object in each of the images to be enhanced from the video data according to the guidance information, wherein the video data includes a plurality of continuous frame images;

[0030] Determining three-dimensional motion information of the target object according to position information of the target object in the multiple images to be enhanced;

[0031] A processing module, used for performing image coding processing on the multiple images to be enhanced respectively to obtain characteristic parameters of each image to be enhanced;

[0032] Based on the three-dimensional motion information of the target object, the characteristic parameters of the multiple images to be enhanced and the guidance information, 3D special effect enhanced images corresponding to the multiple images to be enhanced are generated.

[0033] Optionally, the guidance information includes: guidance prompt text, and the processing module is further used to:

[0034] Performing semantic analysis on the guidance prompt text to obtain a corresponding target object;

[0035] Based on the target object, performing image analysis processing on the plurality of continuous frame images to obtain an analysis result, wherein the analysis result is used to indicate whether the corresponding frame image contains the target object;

[0036] The determination module is further used to determine, based on the analysis result, from the video data, a plurality of images to be enhanced that contain the target object, and to determine the position information of the target object in each image to be enhanced.

[0037] Optionally, the determination module is used to determine the moving direction of the target object according to the position information of the target object in the multiple images to be enhanced;

[0038] Determining a motion trajectory of the target object in three-dimensional space according to a motion direction of the target object;

[0039] The processing module is used to encode the motion trajectory to obtain the three-dimensional motion information of the target object.

[0040] Optionally, the processing module is used to input the three-dimensional motion information of the target object, the characteristic parameters of the multiple images to be enhanced and the guiding information into the generation model;

[0041] The three-dimensional motion information of the target object, the characteristic parameters of the multiple images to be enhanced and the guide information are analyzed and processed by the generation model to obtain 3D special effect enhanced images corresponding to the multiple images to be enhanced.

[0042] Optionally, the guide information further includes: disparity information and optical flow information of the multiple images to be enhanced, and the processing module is used to optimize and adjust the generation model according to the disparity information and optical flow information;

[0043] The adjusted generation model is used to output 3D special effect enhanced images corresponding to the multiple images to be enhanced according to the three-dimensional motion information of the target object, the characteristic parameters of the multiple images to be enhanced and the guidance information.

[0044] Optionally, the images to be enhanced include: a left view and a right view, and the processing module is used to decode the 3D special effect enhanced images corresponding to the multiple images to be enhanced to obtain the decoded left view and right view;

[0045] The decoded left view and right view are synthesized to obtain 3D special effect enhanced videos corresponding to the multiple images to be enhanced.

[0046] In a third aspect, the present application provides a device for enhancing 3D special effects, including:

[0047] Memory;

[0048] processor;

[0049] Wherein, the memory stores computer-executable instructions;

[0050] The processor executes the computer-executable instructions stored in the memory to implement the 3D special effect enhancement method as described in the first aspect and various possible implementation methods of the first aspect.

[0051] In a fourth aspect, the present application provides a computer storage medium having computer execution instructions stored thereon, wherein the computer execution instructions are executed by a processor to implement the 3D special effects enhancement method as described in the first aspect and various possible implementations of the first aspect.

[0052] The present application provides a 3D special effect enhancement method, device, equipment and storage medium, the method comprising: according to guidance information, determining multiple images to be enhanced and the position information of the target object in each image to be enhanced from the video data, and according to the position information of the target object in the multiple images to be enhanced, determining the three-dimensional motion information of the target object, performing image coding processing on the multiple images to be enhanced respectively, obtaining the characteristic parameters of each image to be enhanced, and generating 3D special effect enhanced images corresponding to the multiple images to be enhanced based on the three-dimensional motion information of the target object, the characteristic parameters of the multiple images to be enhanced and the guidance information. The method of the present application realizes automated 3D special effect enhancement processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0054] Figure 1 A 3D special effects enhancement method process provided in the embodiment of the present application Figure 1 ;

[0055] Figure 2 A 3D special effects enhancement method process provided in the embodiment of the present application Figure 2 ;

[0056] Figure 3 A schematic diagram of the structure of a 3D special effects enhancement device provided in this application;

[0057] Figure 4 A schematic diagram of the structure of a 3D special effects enhancement device provided in this application.

[0058] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0060] The terms "first", "second", "third", "fourth", etc. (if any) in the description and claims of the present invention and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in sequences other than those illustrated or described herein.

[0061] In the embodiments of the present application, the words "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0062] With the widespread application of 3D technology in movies, games, virtual reality, augmented reality and other fields, the demand for automated and intelligent processing of 3D special effects has become more urgent. The traditional 3D special effects production method mainly relies on manual design and adjustment, which is not only time-consuming and labor-intensive, but also inefficient.

[0063] The existing 3D special effects production methods have relatively limited scenarios in practical applications, mainly due to the limitations of technical implementation and cost investment. For example, some methods require the use of binocular cameras in a virtual environment for shooting, which not only requires professional equipment and technical support, but also limits the freedom of the shooting scene. In addition, some methods still require a lot of manual design and adjustment work, and cannot truly achieve automation and intelligence.

[0064] In view of the above problems existing in the prior art, the present application provides a 3D special effects enhancement method. The method uses a three-dimensional trajectory and an image to be enhanced as the input of a generation model, and guides the generation model to output an image / video enhanced with 3D special effects by determining the target position, prompt text information, disparity information of the image, and optical flow information. The method can realize automated 3D special effects enhancement processing.

[0065] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0066] Figure 1 This is a 3D special effects enhancement method provided by the embodiment of the present application. Figure 1 .like Figure 1 As shown, the 3D special effect enhancement method provided in this embodiment includes:

[0067] S101: Determine, from video data, multiple images to be enhanced and position information of a target object in each image to be enhanced according to guidance information.

[0068] The guide information includes target information for special effect enhancement, the video data includes a plurality of continuous frame images, and the target object is an object in the image to be enhanced with special effects.

[0069] It can be understood that according to preset guidance information, such as the feature description of the target object, multiple image frames to be processed that need to be enhanced with special effects are determined from a continuous video data stream or a stored video file, and for each selected image to be enhanced, image recognition and positioning technology is used to accurately extract the position information of the target object therein, including the bounding box coordinates of the target object, the center point position and the size of the pixel area occupied by the target object, etc., so as to perform subsequent targeted image enhancement processing or analysis.

[0070] Determining the target object to be enhanced in the video data and its position information through the guidance information is conducive to accurate special effect enhancement processing of the target object in the subsequent steps, so that the effect of the special effect enhancement meets the expectations.

[0071] S102: Determine three-dimensional motion information of the target object according to position information of the target object in the plurality of images to be enhanced.

[0072] It can be understood that based on the position information of the target object accurately obtained from the images to be enhanced in multiple consecutive frames, computer vision technology and three-dimensional reconstruction algorithms can be used to further determine the motion information of the target object in three-dimensional space, including but not limited to dynamic parameters such as displacement changes and trajectory of the target object between consecutive frames.

[0073] Through the time series analysis and spatial mapping of the target object's position information, a motion model of the target object in the three-dimensional scene is constructed, thereby achieving a quantitative description of the target object's motion state, which is conducive to more precise control of the target object.

[0074] S103: Perform image coding processing on the multiple images to be enhanced respectively to obtain feature parameters of each image to be enhanced.

[0075] It is understandable that before encoding the image to be enhanced, the original image needs to be preprocessed, including color space conversion, size adjustment, denoising and other operations.

[0076] The image coding process includes: converting the image data to another domain for representation, such as using discrete cosine transform to convert the image data to the frequency domain. Quantizing the transformed coefficients to approximate continuous values, reducing the precision of the data to reduce the amount of information. Encoding according to the quantized coefficient frequency, further compressing the data, and generating the final encoded data stream.

[0077] Among them, the color features of the image can be extracted through color space conversion and quantization, the texture features of the image can be extracted through transform coding and quantization, and the shape features of the image can be extracted through image segmentation, edge detection and other steps.

[0078] The input image is efficiently encoded through the image encoder, and key feature information is extracted, providing strong support for subsequent image analysis and processing.

[0079] S104: Generate 3D special effect enhanced images corresponding to the multiple images to be enhanced based on the three-dimensional motion information of the target object, the characteristic parameters of the multiple images to be enhanced and the guidance information.

[0080] It can be understood that the three-dimensional motion information of the target object, the characteristic parameters of the image to be enhanced and the guidance information are input into the multimodal large model, and the multimodal large model outputs the desired 3D special effect enhanced image according to the above information.

[0081] Among them, a multimodal large model refers to a machine learning model that can process information from different modalities, such as images, voice, text, etc. Multimodal large models usually use deep learning architectures such as Transformer to capture long-distance dependencies in the data and improve the performance of the model. Multimodal large models learn the potential associations between different modal data and map them to a common feature space. By integrating information from different modalities into a unified representation, subsequent processing and analysis are facilitated.

[0082] The embodiment of the present application provides a 3D special effect enhancement method, which determines multiple images to be enhanced and the position information of the target object in each image to be enhanced from the video data according to the guidance information; determines the three-dimensional motion information of the target object according to the position information of the target object in the multiple images to be enhanced; performs image coding processing on the multiple images to be enhanced respectively to obtain the characteristic parameters of each image to be enhanced; and generates 3D special effect enhancement images corresponding to the multiple images to be enhanced based on the three-dimensional motion information of the target object, the characteristic parameters of the multiple images to be enhanced and the guidance information. The method can realize automatic 3D special effect enhancement processing.

[0083] Figure 2 A 3D special effects enhancement method process provided in the embodiment of the present application Figure 2 This embodiment is in Figure 1 Based on the embodiment, a possible implementation method of the 3D special effect enhancement method is described in detail. Figure 2 As shown, the method includes:

[0084] S201: Perform semantic analysis on the guidance prompt text to obtain a corresponding target object.

[0085] It can be understood that semantic analysis refers to splitting the guidance prompt text into individual words or phrases. The target object can be extracted from the text according to predefined rules or templates. The text can also be trained and learned using machine learning algorithms to extract the target object.

[0086] By performing semantic analysis on the guidance prompt text, accurate understanding of the guidance prompt text and precise extraction of the target object can be achieved.

[0087] S202: Based on the target object, perform image analysis processing on a plurality of continuous frame images to obtain analysis results.

[0088] It is understandable that a large-scale image and text dataset is used for pre-training to obtain a model that can process images and text. The model learns how to align images and text into the same feature space to achieve cross-modal understanding and processing. The model can also be fine-tuned on specific tasks to further improve performance on specific tasks.

[0089] Specifically, in this embodiment, the CLIP algorithm is used to match images and texts, and then the image and text pairs are trained so that the model can understand the image and the description of the image at the same time.

[0090] Among them, the CLIP algorithm can be used to implement text-based image classification tasks. For example, given a text description of an image and a set of category labels, CLIP can determine which category the image belongs to. CLIP can also implement text-based image retrieval tasks. For example, by inputting a text description, CLIP can find the image in the image database that best matches the description.

[0091] S203: According to the analysis result, a plurality of images to be enhanced that contain the target object are determined from the video data, and position information of the target object in each image to be enhanced is determined.

[0092] It can be understood that the key frame containing the target object is screened out from the video data according to the image analysis result, and the key frame containing the target object is used as the image to be enhanced.

[0093] In each image to be enhanced, a mask image of the target object on the image is obtained through a segmentation algorithm, and the location information of the target object is extracted from the mask image.

[0094] The mask image is a binary image that uses two different pixel values ​​to represent a specific area in the image. For example, the area where the target object is located can be assigned a value of 1, while the background and other non-target areas can be assigned a value of 0.

[0095] Furthermore, a boundary detection algorithm is used to identify the boundary of the target object in the mask image. After the boundary is identified, a contour extraction algorithm is used to track the boundary points and construct the complete contour of the target object. Based on the extracted contour, a minimum rectangular box surrounding the target object is calculated, and the coordinate information of the rectangular box is used as the location information of the target object.

[0096] S204: Determine a moving direction of the target object according to position information of the target object in the multiple images to be enhanced.

[0097] It can be understood that the motion vector of the target object is obtained by comparing the position change of the target object in adjacent frames, and the motion direction of the target object is further determined according to the motion vector.

[0098] By determining the moving direction of the target object, it is convenient to construct the three-dimensional moving trajectory of the target object in the subsequent steps.

[0099] S205: Determine a motion trajectory of the target object in three-dimensional space according to the motion direction of the target object.

[0100] In three-dimensional space, the movement of the target object can be regarded as a series of vector changes. Interpolation algorithms can be used to estimate the position of the target object at different time points. Interpolation algorithms use known discrete position data to infer unknown position data, thereby obtaining a continuous trajectory.

[0101] Furthermore, the calculated trajectory data is visualized to more intuitively represent the motion path of the target object in the three-dimensional space.

[0102] S206: Encode the motion trajectory to obtain three-dimensional motion information of the target object.

[0103] It can be understood that a deep learning model can be used to encode the motion trajectory, and the position coordinates of the target object at different time points can be extracted from the encoded trajectory data.

[0104] By encoding the motion trajectory, the three-dimensional motion information of the target object is obtained, which is conducive to performing specific special effect enhancement processing on the target object according to the three-dimensional motion information in the subsequent steps.

[0105] S207: Perform image coding processing on the multiple images to be enhanced respectively to obtain feature parameters of each image to be enhanced.

[0106] Among them, step S207 is similar to the above step S103 and will not be repeated here.

[0107] S208: Inputting the three-dimensional motion information of the target object, the feature parameters of the multiple images to be enhanced, and the guiding information into a generation model.

[0108] Among them, the generative model is a type of model that can randomly generate observations based on certain implicit parameters. In deep learning, the generative model estimates the true distribution of the training data and samples new samples from it. Common generative models include: variational autoencoders, generative adversarial networks, diffusion models, etc.

[0109] Exemplarily, the three-dimensional motion information of the target object, the characteristic parameters of multiple images to be enhanced, and the guidance information are input into the diffusion model, and the three-dimensional motion information of the target object, the characteristic parameters of multiple images to be enhanced, and the guidance information are processed by the diffusion model.

[0110] S209: Optimizing and adjusting the generation model according to the disparity information and the optical flow information.

[0111] Among them, parallax information is used to represent the position difference of objects in the scene under different viewing angles. Through parallax information, the model can better simulate the perspective changes of objects under different viewing angles, thereby creating visual content with a sense of depth and three-dimensionality. Optical flow is a vector field that describes the direction and speed of movement of pixels in an image, reflecting the dynamic changes of objects in the scene. Introducing optical flow information into the generative model helps the model capture and reproduce the movement trajectory and speed changes of objects in the image, thereby improving the dynamic coherence and realism of the generated content.

[0112] When optimizing and adjusting the generative model, the calculated disparity information and optical flow information of the image need to be used as input features, and the model can learn the impact of disparity and optical flow information on the generated content by designing a reasonable network structure and loss function. The above features can also be used to construct supervisory signals, and through regularization terms or adversarial training, the model can be guided to generate content that is more in line with physical laws and visual perception.

[0113] S210: Outputting 3D special effect enhanced images corresponding to the multiple images to be enhanced using the adjusted generation model according to the three-dimensional motion information of the target object, the feature parameters of the multiple images to be enhanced and the guidance information.

[0114] It can be understood that when generating 3D special effects enhanced images, the generative model integrates the motion trajectory of the target object, the visual features of the image, and the guiding information of the special effects enhancement, and then generates the corresponding 3D special effects enhanced images through model reasoning, so that the generated special effects enhanced images both conform to the laws of physics and meet user needs.

[0115] S211: Decode the 3D special effect enhanced images corresponding to the multiple images to be enhanced to obtain decoded left view and right view.

[0116] The image to be enhanced includes a left view and a right view. In this embodiment, the 3D special effect enhanced images corresponding to the multiple images to be enhanced are decoded by a decoding model to obtain the decoded left view and right view.

[0117] S212: synthesizing the decoded left view and right view to obtain 3D special effect enhanced videos corresponding to the multiple images to be enhanced.

[0118] It is understandable that the decoded left view and right view need to be synthesized and processed in time, so as to ensure that each frame of the image is correctly matched, and then the color, brightness, contrast and other parameters in the left and right views are fine-tuned using image processing algorithms to ensure the visual coherence and consistency of the video. In addition, in order to enhance the stereoscopic and realistic sense of the video, the depth information in the view can be further optimized, and finally the corresponding 3D special effect enhanced video is output.

[0119] The embodiment of the present application provides a 3D special effect enhancement method, which performs semantic analysis on the guidance prompt text to obtain the corresponding target object, performs image analysis processing on multiple continuous frame images based on the target object to obtain the analysis result, and determines multiple images to be enhanced containing the target object from the video data according to the analysis result, and determines the position information of the target object in each image to be enhanced, determines the movement direction of the target object according to the position information of the target object in the multiple images to be enhanced, determines the movement trajectory of the target object in three-dimensional space according to the movement direction of the target object, encodes the movement trajectory to obtain the three-dimensional movement information of the target object, performs image encoding processing on multiple images to be enhanced to obtain the feature parameters of each image to be enhanced, inputs the three-dimensional movement information of the target object, the feature parameters of multiple images to be enhanced and the guidance information into the generation model, optimizes and adjusts the generation model according to the parallax information and the optical flow information, and uses the adjusted generation model to output the 3D special effect enhancement images corresponding to the multiple images to be enhanced according to the input information. The method can realize the accurate control of the 3D special effect enhancement effect.

[0120] Figure 3 This is a schematic diagram of the structure of a 3D special effects enhancement device provided in this application. Figure 3 As shown, the 3D special effect enhancement device 300 provided in this embodiment includes:

[0121] A determination module 301 is used to determine a plurality of images to be enhanced and position information of a target object in each of the images to be enhanced from the video data according to the guidance information, wherein the video data includes a plurality of continuous frame images;

[0122] Determining three-dimensional motion information of the target object according to position information of the target object in the multiple images to be enhanced;

[0123] The processing module 302 is used to perform image coding processing on the multiple images to be enhanced respectively to obtain feature parameters of each image to be enhanced;

[0124] Based on the three-dimensional motion information of the target object, the characteristic parameters of the multiple images to be enhanced and the guidance information, 3D special effect enhanced images corresponding to the multiple images to be enhanced are generated.

[0125] Optionally, the guidance information includes: guidance prompt text, and the processing module 302 is further used to:

[0126] Performing semantic analysis on the guidance prompt text to obtain a corresponding target object;

[0127] Based on the target object, performing image analysis processing on the plurality of continuous frame images to obtain an analysis result, wherein the analysis result is used to indicate whether the corresponding frame image contains the target object;

[0128] The determination module 301 is further configured to determine, based on the analysis result, from the video data, a plurality of images to be enhanced that contain the target object, and determine position information of the target object in each image to be enhanced.

[0129] Optionally, the determination module 301 is used to determine the moving direction of the target object according to the position information of the target object in the multiple images to be enhanced;

[0130] Determining a motion trajectory of the target object in three-dimensional space according to a motion direction of the target object;

[0131] The processing module 302 is used to encode the motion trajectory to obtain the three-dimensional motion information of the target object.

[0132] Optionally, the processing module 302 is used to input the three-dimensional motion information of the target object, the characteristic parameters of the multiple images to be enhanced and the guiding information into a generation model;

[0133] The three-dimensional motion information of the target object, the characteristic parameters of the multiple images to be enhanced and the guide information are analyzed and processed by the generation model to obtain 3D special effect enhanced images corresponding to the multiple images to be enhanced.

[0134] Optionally, the guide information further includes: disparity information and optical flow information of the multiple images to be enhanced, and the processing module 302 is used to optimize and adjust the generation model according to the disparity information and optical flow information;

[0135] The adjusted generation model is used to output 3D special effect enhanced images corresponding to the multiple images to be enhanced according to the three-dimensional motion information of the target object, the characteristic parameters of the multiple images to be enhanced and the guidance information.

[0136] Optionally, the image to be enhanced includes: a left view and a right view, and the processing module 302 is used to decode the 3D special effect enhanced images corresponding to the multiple images to be enhanced to obtain the decoded left view and right view;

[0137] The decoded left view and right view are synthesized to obtain 3D special effect enhanced videos corresponding to the multiple images to be enhanced.

[0138] The present embodiment provides a device for enhancing 3D special effects, which can execute the method provided in the above method embodiment. The implementation principle and technical effect are similar, and will not be described in detail in the present embodiment.

[0139] Figure 4 This is a schematic diagram of the structure of a 3D special effects enhancement device provided by this application. Figure 4 As shown, the 3D special effect enhancement device provided by the present application includes: a receiver 401, a transmitter 402, a processor 403 and a memory 404.

[0140] Receiver 401, used for receiving instructions and data;

[0141] A transmitter 402, used for sending instructions and data;

[0142] Memory 404, for storing computer-executable instructions;

[0143] The processor 403 is used to execute the computer-executable instructions stored in the memory 404 to implement the various steps performed by the 3D special effect enhancement method in the above embodiment. For details, please refer to the relevant description in the above 3D special effect enhancement method embodiment.

[0144] Optionally, the memory 404 may be independent or integrated with the processor 403 .

[0145] When the memory 404 is independently provided, the electronic device further includes a bus for connecting the memory 404 and the processor 403 .

[0146] The present application also provides a computer storage medium, in which computer execution instructions are stored. When a processor executes the computer execution instructions, a 3D special effect enhancement method as performed by the above-mentioned 3D special effect enhancement device is implemented.

[0147] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or transient medium). As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0148] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0149] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A 3D special effect enhancement method, characterized in that: The method comprises: Determine, from the video data, a plurality of images to be enhanced and position information of a target object in each of the images to be enhanced according to the guidance information, wherein the video data includes a plurality of continuous frame images; Determining three-dimensional motion information of the target object according to position information of the target object in the multiple images to be enhanced; Performing image coding processing on the multiple images to be enhanced respectively to obtain feature parameters of each image to be enhanced; Based on the three-dimensional motion information of the target object, the characteristic parameters of the multiple images to be enhanced and the guidance information, 3D special effect enhanced images corresponding to the multiple images to be enhanced are generated.

2. The method according to claim 1, characterized in that The guide information includes: guide prompt text, and the determining of a plurality of images to be enhanced from the video data according to the guide information, and the position information of the target object in each of the images to be enhanced, includes: Performing semantic analysis on the guidance prompt text to obtain a corresponding target object; Based on the target object, performing image analysis processing on the plurality of continuous frame images to obtain an analysis result, wherein the analysis result is used to indicate whether the corresponding frame image contains the target object; According to the analysis result, a plurality of images to be enhanced that contain the target object are determined from the video data, and position information of the target object in each image to be enhanced is determined.

3. The method according to claim 1, characterized in that The step of determining the three-dimensional motion information of the target object according to the position information of the target object in the plurality of images to be enhanced comprises: Determining a moving direction of the target object according to position information of the target object in the plurality of images to be enhanced; Determining a motion trajectory of the target object in three-dimensional space according to a motion direction of the target object; The motion trajectory is encoded to obtain three-dimensional motion information of the target object.

4. The method according to claim 1, characterized in that: The step of generating 3D special effect enhanced images corresponding to the multiple images to be enhanced based on the three-dimensional motion information of the target object, the characteristic parameters of the multiple images to be enhanced and the guide information comprises: Inputting the three-dimensional motion information of the target object, the characteristic parameters of the plurality of images to be enhanced and the guiding information into a generation model; The three-dimensional motion information of the target object, the characteristic parameters of the multiple images to be enhanced and the guide information are analyzed and processed by the generation model to obtain 3D special effect enhanced images corresponding to the multiple images to be enhanced.

5. The method according to claim 4, characterized in that The guide information also includes: disparity information and optical flow information of the multiple images to be enhanced; the three-dimensional motion information of the target object, the characteristic parameters of the multiple images to be enhanced and the guide information are analyzed and processed by the generation model to obtain 3D special effect enhanced images corresponding to the multiple images to be enhanced, including: Optimizing and adjusting the generation model according to the disparity information and the optical flow information; The adjusted generation model is used to output 3D special effect enhanced images corresponding to the multiple images to be enhanced according to the three-dimensional motion information of the target object, the characteristic parameters of the multiple images to be enhanced and the guidance information.

6. The method according to claim 1, characterized in that The image to be enhanced includes: a left view and a right view, and after generating 3D special effect enhanced images corresponding to the multiple images to be enhanced based on the three-dimensional motion information of the target object, the characteristic parameters of the multiple images to be enhanced and the guide information, the method further includes: Decoding the 3D special effect enhanced images corresponding to the multiple images to be enhanced to obtain decoded left view and right view; The decoded left view and right view are synthesized to obtain 3D special effect enhanced videos corresponding to the multiple images to be enhanced.

7. A device for enhancing 3D special effects, characterized in that: The device comprises: A determination module, configured to determine a plurality of images to be enhanced and position information of a target object in each of the images to be enhanced from the video data according to the guidance information, wherein the video data includes a plurality of continuous frame images; Determining three-dimensional motion information of the target object according to position information of the target object in the multiple images to be enhanced; A processing module, used for performing image coding processing on the multiple images to be enhanced respectively to obtain characteristic parameters of each image to be enhanced; Based on the three-dimensional motion information of the target object, the characteristic parameters of the multiple images to be enhanced and the guidance information, 3D special effect enhanced images corresponding to the multiple images to be enhanced are generated.

8. The device according to claim 7, characterized in that The guidance information includes: guidance prompt text, and the processing module is further used for: Performing semantic analysis on the guidance prompt text to obtain a corresponding target object; Based on the target object, performing image analysis processing on the plurality of continuous frame images to obtain an analysis result, wherein the analysis result is used to indicate whether the corresponding frame image contains the target object; The determination module is further used to determine, based on the analysis result, from the video data, a plurality of images to be enhanced that contain the target object, and to determine the position information of the target object in each image to be enhanced.

9. A device for enhancing 3D special effects, characterized in that: The device comprises: Memory; processor; Wherein, the memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement a 3D special effect enhancement method according to any one of claims 1 to 6.

10. A computer storage medium, characterized in that: The computer storage medium stores computer execution instructions, and when the computer execution instructions are executed by the processor, they are used to implement a 3D special effect enhancement method as described in any one of claims 1-6.

Images