Video conversion method and device

Abstract

The embodiment of the invention provides a video conversion method, and belongs to the technical field of video processing. The video conversion method comprises the following steps: acquiring a first video of a first format; inputting the first video into a pre-trained target neural network model to output a second video in a second format, wherein the target neural network model is obtained by training a plurality of sample video pairs; each sample video pair comprises a first sample video in a first format and a second sample video in a second format, and the second sample video is obtained by performing corresponding adjustment and format conversion on the first sample video according to a video style of the first sample video. According to the technical scheme of the embodiment of the invention, the first video can be converted into the second video through the target neural network model, and the optimal conversion effect of the corresponding video style is realized.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of video processing technology, and in particular, to a video conversion method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] With the improvement of display technology, more and more display devices are beginning to support high dynamic range videos with higher dynamic range and wider color gamut. Although there has been a significant improvement in the device level for high dynamic range videos, the proportion of high dynamic range videos is still not high at this stage. Therefore, the user's viewing experience can be improved by converting standard dynamic videos into high dynamic videos.

[0003] However, the video conversion methods in the prior art tend to be averaged, and the video conversion effects on different video styles are poor.

[0004] It should be noted that the above content is not necessarily prior art, nor is it intended to limit the scope of patent protection of this application. Summary of the invention

[0005] The embodiments of the present application provide a video conversion method, apparatus, computer device, computer-readable storage medium, and computer program product to solve or alleviate one or more of the technical problems raised above.

[0006] One aspect of an embodiment of the present application provides a video conversion method, the method comprising: Obtaining a first video in a first format; Inputting the first video into a pre-trained target neural network model to output a second video in a second format; Wherein, the target neural network model is trained based on multiple sample video pairs; each of the sample video pairs includes a first sample video in a first format and a second sample video in a second format, and the second sample video is obtained by adjusting and format-converting the first sample video according to the video style of the first sample video.

[0007] Optionally, the target neural network model includes: A global information network, configured to receive the downsampled first video and output global information; A mapping network, configured to receive the first video and the global information to output an intermediate video in the second format; A local enhancement network is used to receive the intermediate video and the global information to output a second video in the second format.

[0008] Optionally, the global information network includes: Several global information modules, M*M convolutional layers and global pooling layers; the global information module includes M*M convolutional and pooling layers; Wherein, M is a natural number greater than 0.

[0009] Optionally, the mapping network includes: A plurality of M*M convolutions and first fine-tuning modules, one M*M convolution corresponds to one first fine-tuning module; one first fine-tuning module includes two fully connected layers, one fully connected layer receives the global information to output a vector representing the amplitude, and the other fully connected layer receives the global information to output a vector representing the offset; the mapping network is used to: output the intermediate video according to the intermediate information obtained after convolution calculation of the vector representing the amplitude, the vector representing the offset and the first video of the first format.

[0010] Optionally, there are several N*N convolution and second fine-tuning modules, one of the N*N convolution corresponds to one of the second fine-tuning modules; one of the second fine-tuning modules includes two fully connected layers, one of which receives the global information to output a vector representing the amplitude, and the other fully connected layer receives the global information to output a vector representing the offset; the local enhancement network is used to: output the second video in the second format according to the intermediate information obtained after the vector representing the amplitude, the vector representing the offset and the intermediate video are convoluted; Wherein, N is a natural number greater than 0, N>M.

[0011] Optionally, the target neural network model is trained by the following operations: During training: Inputting a first picture frame sequence in the first sample video into the target neural network model to output a predicted picture frame sequence through the target neural network model, wherein the predicted picture frame sequence corresponds to the second format; Obtaining a first inter-frame difference between adjacent frames in the predicted picture frame sequence; Obtaining a second inter-frame difference between adjacent frames in a second picture frame sequence of the second sample video; Obtaining an absolute value of a difference between each of a plurality of first inter-frame gaps and a corresponding second inter-frame gap; According to the sum of the absolute values ​​of the multiple differences, the network parameters in the target neural network model are adjusted.

[0012] Another aspect of an embodiment of the present application provides a video conversion device, the device comprising: An acquisition module, used for acquiring a first video in a first format; An output module, used for inputting the first video into a pre-trained target neural network model to output a second video in a second format; Wherein, the target neural network model is trained based on multiple sample video pairs; each of the sample video pairs includes a first sample video in a first format and a second sample video in a second format, and the second sample video is obtained by adjusting and format-converting the first sample video according to the video style of the first sample video.

[0013] Another aspect of an embodiment of the present application provides a computer device, including: at least one processor; and a memory communicatively coupled to the at least one processor; Wherein: the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method as described above.

[0014] Another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method described above is implemented.

[0015] Another aspect of an embodiment of the present application provides a computer program product, including a computer program, which implements the above-mentioned method when executed by a processor.

[0016] The above technical solution adopted in the embodiment of the present application may have the following advantages: The target neural network model is obtained by training a first sample video in a first format and a second sample video in a second format obtained after video style adjustment and format conversion as a sample video pair, so that the first video can be converted into the second video through the target neural network model and achieve the best conversion effect of the corresponding video style. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0018] Figure 1 The following schematically shows an operating environment diagram of the video conversion method according to the first embodiment of the present application; Figure 2 A flowchart of a video conversion method according to Embodiment 1 of the present application is schematically shown; Figure 3The newly added flow chart of the video conversion method according to the first embodiment of the present application is schematically shown; Figure 4 A diagram schematically shows an application example of the video conversion method according to the first embodiment of the present application; Figure 5 A block diagram schematically shows a video conversion device according to the second embodiment of the present application; and Figure 6 The hardware architecture diagram of the computer device according to the third embodiment of the present application is schematically shown. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0020] It should be noted that the descriptions involving "first", "second", etc. in the embodiments of the present application are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0021] In the description of the present application, it should be understood that the numerical labels before the steps do not indicate the order in which the steps are executed, but are only used to facilitate the description of the present application and to distinguish each step, and therefore should not be understood as a limitation on the present application.

[0022] First, the following terms are explained: SDR: Standard Dynamic Range is a technology for displaying video based on brightness, contrast, color characteristics, and the limitations of CRT (cathode ray tube) displays.

[0023] HDR: High Dynamic Range is an upgrade of SDR and a technology that improves the quality of video display. HDR changes the way the brightness and color information of videos and images are represented in the signal, thereby supporting a larger brightness range, a wider color gamut, and higher-precision quantization.

[0024] Convolutional Neural Networks (CNN): A feedforward neural network with deep structure and convolutional calculation, which is one of the representative algorithms of deep learning. Convolutional neural network includes multiple convolutional layers, each of which has a different number of convolutional kernels, and each convolutional kernel corresponds to a convolution channel.

[0025] Secondly, in order to facilitate those skilled in the art to understand the technical solutions provided in the embodiments of the present application, the relevant technologies are described below: The applicant understands that the video conversion scheme based on fixed function mapping can only achieve global mapping due to the problem of the number of parameters itself, and cannot take into account the customized mapping of content in different areas of the picture content, and at the same time make adaptive adjustments to the content services of different scenes. Based on the learnable video conversion scheme, local and scene-based adaptive mapping can be achieved, and the upper limit of the effect of the converted video content will be higher, but if only a batch of sample video pairs are collected without considering the uniformity of their internal content and style, then the final result will be an evenly distributed model, which cannot achieve the best effect in different content classifications. In addition, when the model under the learnable scheme processes high-resolution videos, especially videos with a resolution of 4k or above, it often requires a lot of computing time due to the high pixel density, and the output video content is very prone to inter-frame flickering.

[0026] To this end, the embodiment of the present application provides a video conversion technology solution. In this technical solution, (1) the problem of model inapplicability caused by different video content styles is avoided, and the conversion is performed through models corresponding to different video styles to achieve the best conversion effect of different video styles; (2) the model is trained in multiple stages to achieve the best balance between efficiency and effect; (3) through the time-stable training method and time-consistency loss function, it can ensure that the video content output by the model of this solution is coherent and not prone to flickering. See below for details.

[0027] Finally, for ease of understanding, an exemplary operating environment is provided below.

[0028] like Figure 1 As shown, the operating environment diagram includes: like Figure 1 As shown, the operating environment diagram includes: a service platform 2, and clients (4A, 4B, ..., 4N).

[0029] The service platform 2 can be connected to the client terminals ( 4A, 4B, ..., 4N) via the network.

[0030] The service platform 2 may be a single server, a server cluster or a cloud computing service center.

[0031] The service platform 2 can provide video conversion services and the like to the client.

[0032] The video conversion service can convert a first video in a first format into a second video in a second format, for example, converting an SDR video into an HDR video.

[0033] The service platform 2 may be located in a data center such as a single location, or distributed in different geographical locations (e.g., in multiple locations). The service platform 2 may provide services via a network. The network includes various network devices such as routers, switches, multiplexers, hubs, modems, bridges, repeaters, firewalls, proxy devices, and / or the like. The network may include physical links such as coaxial cable links, twisted pair cable links, optical fiber links, combinations thereof, and the like, or wireless links such as cellular links, satellite links, Wi-Fi links, and the like.

[0034] The client (4A, 4B, ..., 4N) may be configured to access the content and services of the service platform 2. The client (4A, 4B, ..., 4N) may include an electronic device carrying or externally connected to a display panel, such as a mobile device, a tablet device, a laptop computer, a workstation, a virtual reality device, a gaming device, a digital streaming device, a vehicle terminal, a smart TV, a set-top box, etc., and may also include a virtualized computing instance. The virtualized computing instance may include a virtual machine, such as a simulation of a computer system, an operating system, a server, etc. The computing device may load a virtual machine based on a virtual image and / or other data defining specific software (e.g., an operating system, a dedicated application, a server) for simulation. As the demand for different types of processing services changes, different virtual machines may be loaded and / or terminated on one or more computing devices.

[0035] The client (4A, 4B, ..., 4N) may be associated with one or more users. A single user may also use one or more of the clients (4A, 4B, ..., 4N) to access the service platform 2. The client (4A, 4B, ..., 4N) may travel to various locations and use different networks to access the service platform 2.

[0036] The technical solutions of the present application are described below through multiple embodiments. It should be noted that these embodiments can be implemented in a variety of different forms and should not be construed as being limited to the embodiments described here.

[0037] Embodiment 1 Figure 2 The flowchart of the video conversion method according to the first embodiment of the present application is schematically shown.

[0038] like Figure 2 As shown, the video conversion method may include steps S200 to S202, wherein: Step S200: obtaining a first video in a first format.

[0039] Step S202, input the first video into a pre-trained target neural network model to output a second video in a second format; wherein the target neural network model is trained based on multiple sample video pairs; each of the sample video pairs includes a first sample video in a first format and a second sample video in a second format, and the second sample video is obtained by adjusting and format-converting the first sample video according to the video style of the first sample video.

[0040] In the video conversion method provided in this embodiment, the target neural network model is obtained by training a first sample video in a first format and a second sample video in a second format obtained after video style adjustment and format conversion as a sample video pair, so that the first video can be converted into the second video through the target neural network model and achieve the best conversion effect of the corresponding video style.

[0041] The following combination Figure 2 , each step in steps S200~S202 and other optional steps are explained in detail.

[0042] Step S200 , obtain a first video in a first format.

[0043] The first format may be an SDR format or other video formats, which is not limited here. The first video may be a video uploaded by a user or a video stored by a server, which is not limited here. In some embodiments, the first video may be an SDR video that needs to be converted to an HDR video. By acquiring the SDR video and converting it to an HDR video, the coverage of the HDR video can be increased at a very low cost, thereby improving the user's viewing experience.

[0044] Step S202 , inputting the first video into a pre-trained target neural network model to output a second video in a second format; Wherein, the target neural network model is trained based on multiple sample video pairs; each of the sample video pairs includes a first sample video in a first format and a second sample video in a second format, and the second sample video is obtained by adjusting and format-converting the first sample video according to the video style of the first sample video.

[0045] The video style can be determined according to different types of videos, such as movies, cartoons, and TV series, etc., or according to the content of the video, such as horror content and comedy content, etc., which is not limited here. In some embodiments, the first video is an SDR video, and the second video is an HDR video. The SDR video is converted into a corresponding style HDR video through a pre-trained target neural network model (trained based on multiple style videos).

[0046] During training: The first sample video may be an SDR sample video, and the second sample video may be an HDR sample video obtained by adjusting and converting the SDR sample video according to the video style of the SDR video by a colorist. The adjustment and format conversion process may be to convert the color space of the SDR sample video, map the content of the SDR sample video to a color space that complies with the HDR standard, and adjust the color and brightness of the picture content by content and region according to the artistic taste of the colorist, so as to obtain the best viewing experience for the second sample video.

[0047] In this embodiment, the target neural network model is trained by using a first sample video in a first format and a second sample video in a second format obtained after video style adjustment and format conversion as a sample video pair, so that the first video can be converted into the second video through the target neural network model and achieve the best conversion effect of the corresponding video style. In an optional embodiment, if Figure 6 , the target neural network model includes: a global information network, a mapping network and a local enhancement network.

[0048] A global information network, configured to receive the downsampled first video and output global information; A mapping network, configured to receive the first video and the global information to output an intermediate video in the second format; A local enhancement network is used to receive the intermediate video and the global information to output a second video in the second format.

[0049] Downsampling is a method of "compressing" or "simplifying" data to reduce the size of the file or reduce the complexity of processing. The first video can be converted in units of video frames. Global information based on the overall picture can be obtained through the operation of the global information network, and the global information can be composed of vectors. The global information is used to guide the mapping network and the local enhancement network to adaptively process the content input therein. The mapping network can receive the first video, and with the help of the global information, process the picture content of the first video, so that the first video is initially mapped to the color space and brightness range under the second format standard to obtain an intermediate video. Based on the intermediate video, and with the help of the global information, the local enhancement network adaptively fine-tunes the video content of the intermediate video to obtain a more refined second video in the second format.

[0050] In this embodiment, the optimal balance between efficiency and effect is achieved through multi-stage conversion of the global information network, the mapping network and the local enhancement network.

[0051] The global information network, the mapping network and the local enhancement network will be further exemplified below through more embodiments.

[0052] In an optional embodiment, the global information network includes: Several global information modules, M*M convolutional layers and global pooling layers; the global information module includes M*M convolutional and pooling layers; Wherein, M is a natural number greater than 0.

[0053] The pooling layer is a neural network layer commonly used in deep learning. Its function is to downsample the input data, thereby reducing the size of the data while retaining important features. Pooling is used in this global information network to help reduce the amount of calculation, reduce the number of parameters, and enhance the robustness of the model. In some embodiments, M can be 1, and the step size of the uniform pooling layer can be 2, which refers to the step size of the pooling operation sliding on the input data. In the global information network, firstly, through multiple calculations of several global information modules, the calculation process of the global information module is: first through 1*1 convolution calculation, and then through the uniform pooling layer. Since the step size of the uniform pooling layer is 2, the first video continuously reduces the resolution during the calculation process while improving the receptive field of the global information network. The receptive field refers to the area size of the input image that a neuron in a neural network can "see" or "perceive". Finally, the feature information is compressed into a vector of size 1*1 through the calculation of the 1*1 convolution layer and the global pooling layer to obtain global information. The feature information can be color space information and brightness information, etc., which are not limited here.

[0054] In this embodiment, through the calculation of the global information network, the global information is output to provide the mapping network and the local enhancement network for calculation. In addition, in the process of the global information network calculation, the receptive field of the global information network is improved while the resolution of the first video is continuously reduced, so that the global information network can capture more information, thereby improving the accuracy and generalization ability of the global information network.

[0055] In an optional embodiment, the mapping network includes: A plurality of M*M convolutions and first fine-tuning modules, one M*M convolution corresponds to one first fine-tuning module; one first fine-tuning module includes two fully connected layers, one fully connected layer receives the global information to output a vector representing the amplitude, and the other fully connected layer receives the global information to output a vector representing the offset; the mapping network is used to: output the intermediate video according to the intermediate information obtained after convolution calculation of the vector representing the amplitude, the vector representing the offset and the first video of the first format.

[0056] In some embodiments, M may be 1, and a 1*1 convolution calculation may be performed on the first video first, and then the first fine-tuning module may perform two connection layer calculations on the global information to obtain an amplitude and an offset vector, and the amplitude and the offset vector are applied to the result of the 1*1 convolution output to adjust the result, for example, The operation of the formula, Represents the result of 1*1 convolution output, Represents the result after adjustment by amplitude and offset vector, where scale and shift represent amplitude vector and offset vector respectively.

[0057] In this embodiment, through the calculation of the mapping network, the first video can be initially mapped to the color space and brightness range under the second format standard to obtain an intermediate video in the second format. In addition, the amplitude and offset vectors generated by the global information enable the mapping network to combine the global context information and dynamically adjust the output results, thereby more accurately achieving the mapping from the first format to the second format.

[0058] In an optional embodiment, the local enhancement network includes: A plurality of N*N convolution and second fine-tuning modules, wherein one N*N convolution corresponds to one second fine-tuning module; one second fine-tuning module comprises two fully connected layers, wherein one fully connected layer receives the global information to output a vector representing the amplitude, and the other fully connected layer receives the global information to output a vector representing the offset; the local enhancement network is used to: output the second video in the second format according to the intermediate information obtained after the convolution calculation of the vector representing the amplitude, the vector representing the offset and the intermediate video; Wherein, N is a natural number greater than 0, N>M.

[0059] N is greater than M, so that the local enhancement network has a larger receptive field to obtain pixel information around each pixel, so as to adaptively fine-tune the content of the intermediate video, for example, the brightness and color can be adaptively fine-tuned. In some embodiments, N can be 3 and M can be 1. The feature information of the intermediate video can be calculated by the local enhancement network through 3*3 convolution, and then the global information can be calculated by two connection layers through the first fine-tuning module to obtain the amplitude and offset vector, and the amplitude and offset vector are applied to the result of the 3*3 convolution output. The operation of the formula, Represents the result of 3*3 convolution output, It represents the output result after the amplitude and offset vector are adjusted. Scale and shift represent the amplitude vector and offset vector respectively.

[0060] In this embodiment, by using a larger convolution kernel in the local enhancement network than that in the mapping network, the local enhancement network has a larger receptive field to obtain pixel information around each pixel, thereby adaptively fine-tuning the content of the intermediate video to obtain a more refined second format video.

[0061] In an optional embodiment, if Figure 3 As shown, the target neural network model is trained by the following operations, and the training process may include: Step S300, inputting a first picture frame sequence in the first sample video into the target neural network model, so as to output a predicted picture frame sequence through the target neural network model, wherein the predicted picture frame sequence corresponds to the second format; Step S302: obtaining a first inter-frame difference between adjacent frames in the predicted picture frame sequence.

[0062] Step S304: obtaining a second inter-frame difference between adjacent frames in a second picture frame sequence of the second sample video.

[0063] Step S306: Obtain the absolute value of the difference between each of the plurality of first inter-frame gaps and the corresponding second inter-frame gap.

[0064] Step S308, adjusting the network parameters in the target neural network model according to the sum of the absolute values ​​of the multiple differences.

[0065] During the training process, the first picture frame sequence in the first sample video is input into the target neural network model to output a predicted picture frame sequence, and in each training round, a first picture frame is input into the target neural network model to output a predicted picture frame. By making the first frame gap and the second frame gap as close as possible, the network parameters of the target neural network model are adjusted so that the first frame gap and the second frame gap output by it are as close as possible. In some embodiments, the temporal consistency loss function can be used to make the first frame gap and the second frame gap as close as possible. The temporal consistency loss function can be ,in represents the first frame gap, Indicates the second picture frame, Indicates the second frame gap. In addition to calculating through the temporal consistency loss function, the target neural network model can also be evaluated through the L1 Loss loss function to guide the target neural network model to adjust the network parameters. L1 Loss is a commonly used loss function in machine learning and deep learning, which is used to measure the error between the model's prediction result and the true value. The core idea of ​​L1 Loss is to calculate the absolute difference between the predicted value and the true value, and then find the average. The network parameters can include the convolution kernel of the convolution layer, and can also include other parameter bits, which are not limited here.

[0066] In this embodiment, by making the first frame gap and the second frame gap as close as possible, and then adjusting the network parameters of the target neural network model so that the first frame gap and the second frame gap of its output are as close as possible, it is ensured that the second video output by the target neural network model has continuity and is not prone to flickering, thereby improving the user viewing experience.

[0067] In order to make this application easier to understand, the following Figure 4 An exemplary application is provided.

[0068] In this exemplary application, the process of training the target neural network (SDR-HDR lightweight network) based on the SDR original film is as follows: S1. According to the video style of the SDR original film, the colorist creates an HDR version of the SDR original film.

[0069] S2. Downsample the original SDR film and input it into the global information network of the target neural network to obtain global information.

[0070] S3, input the SDR original film and global information into the mapping network of the target neural network, and output the intermediate video through the mapping network.

[0071] S4: Input the intermediate video and global information into local enhancement, and predict the HDR result through the mapping network output.

[0072] S5. The predicted HDR content and the HDR version video created by the colorist are calculated through the loss function to optimize the network parameters of the target neural network model.

[0073] In this exemplary application, the problem of model inapplicability caused by the different styles of SDR videos is avoided. At the same time, the optimal balance between efficiency and effect is achieved through multi-stage conversion of global information network, mapping network and local enhancement network.

[0074] Embodiment 2 Figure 5 The block diagram of the video conversion device according to the second embodiment of the present application is schematically shown. The device can be divided into one or more program modules, one or more program modules are stored in a storage medium and executed by one or more processors to complete the embodiment of the present application. The program module referred to in the embodiment of the present application refers to a series of computer program instruction segments that can complete specific functions. The following description will specifically introduce the functions of each program module in this embodiment. Figure 5 As shown, the device 1600 may include: an acquisition module 1610 and an output module 1620, wherein: An acquisition module 1610 is used to acquire a first video in a first format; An output module 1620, configured to input the first video into a pre-trained target neural network model to output a second video in a second format; Wherein, the target neural network model is trained based on multiple sample video pairs; each of the sample video pairs includes a first sample video in a first format and a second sample video in a second format, and the second sample video is obtained by adjusting and format-converting the first sample video according to the video style of the first sample video.

[0075] As an optional embodiment, the target neural network model includes: A global information network, configured to receive the downsampled first video and output global information; A mapping network, configured to receive the first video and the global information to output an intermediate video in the second format; A local enhancement network is used to receive the intermediate video and the global information to output a second video in the second format.

[0076] As an optional embodiment, the global information network includes: Several global information modules, M*M convolutional layers and global pooling layers; the global information module includes M*M convolutional and pooling layers; Wherein, M is a natural number greater than 0.

[0077] As an optional embodiment, the mapping network includes: A plurality of M*M convolutions and first fine-tuning modules, one M*M convolution corresponds to one first fine-tuning module; one first fine-tuning module includes two fully connected layers, one fully connected layer receives the global information to output a vector representing the amplitude, and the other fully connected layer receives the global information to output a vector representing the offset; the mapping network is used to: output the intermediate video according to the intermediate information obtained after convolution calculation of the vector representing the amplitude, the vector representing the offset and the first video of the first format.

[0078] As an optional embodiment, the local enhancement network includes: A plurality of N*N convolution and second fine-tuning modules, wherein one N*N convolution corresponds to one second fine-tuning module; one second fine-tuning module comprises two fully connected layers, wherein one fully connected layer receives the global information to output a vector representing the amplitude, and the other fully connected layer receives the global information to output a vector representing the offset; the local enhancement network is used to: output the second video in the second format according to the intermediate information obtained after the convolution calculation of the vector representing the amplitude, the vector representing the offset and the intermediate video; Wherein, N is a natural number greater than 0, N>M.

[0079] As an optional embodiment, the target neural network model is trained by the following operations: During training: Inputting a first picture frame sequence in the first sample video into the target neural network model to output a predicted picture frame sequence through the target neural network model, wherein the predicted picture frame sequence corresponds to the second format; Obtaining a first inter-frame difference between adjacent frames in the predicted picture frame sequence; Obtaining a second inter-frame difference between adjacent frames in a second picture frame sequence of the second sample video; Obtaining an absolute value of a difference between each of a plurality of first inter-frame gaps and a corresponding second inter-frame gap; According to the sum of the absolute values ​​of the multiple differences, the network parameters in the target neural network model are adjusted.

[0080] Embodiment 3 Figure 6The schematic diagram of the hardware architecture of a computer device 10000 suitable for implementing the video conversion method according to the third embodiment of the present application is schematically shown. In some embodiments, the computer device 10000 may be a terminal device such as a smart phone, a wearable device, a tablet computer, a personal computer, a vehicle terminal, a game console, a virtual device, a workbench, a digital assistant, a set-top box, a robot, etc. In other embodiments, the computer device 10000 may be a rack server, a blade server, a tower server, or a cabinet server (including an independent server, or a server cluster composed of multiple servers), etc. Figure 6 As shown, the computer device 10000 includes but is not limited to: a memory 10010, a processor 10020, and a network interface 10030 that can communicate with each other through a system bus. Among them: The memory 10010 includes at least one type of computer-readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (such as SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 10010 can be an internal storage module of the computer device 10000, such as a hard disk or memory of the computer device 10000. In other embodiments, the memory 10010 can also be an external storage device of the computer device 10000, such as a plug-in hard disk equipped on the computer device 10000, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 10010 can also include both the internal storage module of the computer device 10000 and its external storage device. In this embodiment, the memory 10010 is generally used to store the operating system and various application software installed in the computer device 10000, such as program codes of the video conversion method, etc. In addition, the memory 10010 can also be used to temporarily store various data that have been output or will be output.

[0081] In some embodiments, the processor 10020 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other chips. The processor 10020 is generally used to control the overall operation of the computer device 10000, such as performing control and processing related to data interaction or communication with the computer device 10000. In this embodiment, the processor 10020 is used to run the program code stored in the memory 10010 or process data.

[0082] The network interface 10030 may include a wireless network interface or a wired network interface, and the network interface 10030 is generally used to establish a communication link between the computer device 10000 and other computer devices. For example, the network interface 10030 is used to connect the computer device 10000 to an external terminal through a network, and to establish a data transmission channel and a communication link between the computer device 10000 and the external terminal. The network may be a wireless or wired network such as an intranet, the Internet, the Global System of Mobile communication (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 5G network, Bluetooth, Wi-Fi, etc.

[0083] It should be pointed out that Figure 6 Only a computer device having components 10010 - 10030 is shown, but it should be understood that implementation of all of the components shown is not a requirement, and more or fewer components may alternatively be implemented.

[0084] In this embodiment, the video conversion method stored in the memory 10010 can also be divided into one or more program modules and executed by one or more processors (such as processor 10020) to complete the embodiment of the present application.

[0085] Embodiment 4 An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the video conversion method in the embodiment are implemented.

[0086] In this embodiment, the computer-readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the computer-readable storage medium can be an internal storage unit of a computer device, such as a hard disk or memory of the computer device. In other embodiments, the computer-readable storage medium can also be an external storage device of a computer device, such as a plug-in hard disk equipped on the computer device, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the computer-readable storage medium can also include both the internal storage unit of the computer device and its external storage device. In this embodiment, the computer-readable storage medium is generally used to store an operating system and various application software installed on the computer device, such as the program code of the video conversion method in the embodiment, etc. In addition, the computer-readable storage medium can also be used to temporarily store various types of data that have been output or are to be output.

[0087] Embodiment 5 An embodiment of the present application also provides a computer program product, including a computer program, which implements the method in the above embodiment when executed by a processor.

[0088] Obviously, those skilled in the art should understand that the modules or steps of the above-mentioned embodiments of the present application can be implemented by general-purpose computer devices, they can be concentrated on a single computer device, or distributed on a network composed of multiple computer devices, optionally, they can be implemented by executable program codes of computer devices, so that they can be stored in a storage device and executed by the computer device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0089] It should be noted that the above are only preferred embodiments of the present application, and the patent protection scope of the present application is not limited thereto. Any equivalent structure or equivalent process transformation made using the contents of the specification and drawings of the present application, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A video conversion method, characterized in that: The method comprises: Obtaining a first video in a first format; Inputting the first video into a pre-trained target neural network model to output a second video in a second format; Wherein, the target neural network model is trained based on multiple sample video pairs; each of the sample video pairs includes a first sample video in a first format and a second sample video in a second format, and the second sample video is obtained by adjusting and format-converting the first sample video according to the video style of the first sample video.

2. The method according to claim 1, characterized in that The target neural network model includes: A global information network, configured to receive the downsampled first video and output global information; A mapping network, configured to receive the first video and the global information to output an intermediate video in the second format; A local enhancement network is used to receive the intermediate video and the global information to output a second video in the second format.

3. The method according to claim 2, characterized in that The global information network includes: Several global information modules, M*M convolutional layers and global pooling layers; the global information module includes M*M convolutional and pooling layers; Wherein, M is a natural number greater than 0.

4. The method according to claim 3, characterized in that The mapping network includes: A plurality of M*M convolutions and first fine-tuning modules, one M*M convolution corresponds to one first fine-tuning module; one first fine-tuning module includes two fully connected layers, one fully connected layer receives the global information to output a vector representing the amplitude, and the other fully connected layer receives the global information to output a vector representing the offset; the mapping network is used to: output the intermediate video according to the intermediate information obtained after convolution calculation of the vector representing the amplitude, the vector representing the offset and the first video of the first format.

5. The method according to claim 4, characterized in that The local enhancement network comprises: A plurality of N*N convolution and second fine-tuning modules, wherein one N*N convolution corresponds to one second fine-tuning module; one second fine-tuning module comprises two fully connected layers, wherein one fully connected layer receives the global information to output a vector representing the amplitude, and the other fully connected layer receives the global information to output a vector representing the offset; the local enhancement network is used to: output the second video in the second format according to the intermediate information obtained after the convolution calculation of the vector representing the amplitude, the vector representing the offset and the intermediate video; Wherein, N is a natural number greater than 0, N>M.

6. The method according to any one of claims 1 to 5, characterized in that: The target neural network model is trained by the following operations: During training: Inputting a first picture frame sequence in the first sample video into the target neural network model to output a predicted picture frame sequence through the target neural network model, wherein the predicted picture frame sequence corresponds to the second format; Obtaining a first inter-frame difference between adjacent frames in the predicted picture frame sequence; Obtaining a second inter-frame difference between adjacent frames in a second picture frame sequence of the second sample video; Obtaining an absolute value of a difference between each of a plurality of first inter-frame gaps and a corresponding second inter-frame gap; According to the sum of the absolute values ​​of the multiple differences, the network parameters in the target neural network model are adjusted.

7. A video conversion device, characterized in that: The device comprises: An acquisition module, used for acquiring a first video in a first format; An output module, used for inputting the first video into a pre-trained target neural network model to output a second video in a second format; Wherein, the target neural network model is trained based on multiple sample video pairs; each of the sample video pairs includes a first sample video in a first format and a second sample video in a second format, and the second sample video is obtained by adjusting and format-converting the first sample video according to the video style of the first sample video.

8. A computer device, characterized in that: include: at least one processor; and a memory communicatively coupled to the at least one processor; wherein: The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to claims 1 to 6 are implemented.

Images