Data processing method, live broadcast method and device, electronic device, and storage medium

Abstract

The present disclosure provides a data processing method, a live broadcast method and device, an electronic device, and a storage medium, which relate to the field of live broadcast technology. The data processing method comprises: obtaining an initial video; inputting the initial video into a pre-trained target super-resolution model, and outputting an intermediate video, wherein the image resolution of the intermediate video is greater than the image resolution of the initial video; performing detail recovery processing on the intermediate video to obtain a target video. The technical solution of the embodiment of the present disclosure can overcome the technical problem of poor live broadcast effect due to model limitations such as high frame rate and high complexity requirements.

Description

Background Art

[0002] With the improvement of video production level, audiences have higher and higher requirements for the quality of live broadcast images. Although the method of improving resolution based on traditional algorithms does not require high computing power of the machine model, the reconstructed image output is generally very blurry, and the clarity is even lower than the original image. For example, in high frame rate and high complexity live broadcast scenarios, due to factors such as bandwidth and encoding loss, it is easy to cause blurred and distorted images. Therefore, traditional algorithms are difficult to reuse in real-time live broadcast scenarios.

[0003] In the process of applying ultra-high-definition video technology to live broadcast technology, there is a technical problem that the live broadcast effect is poor due to model limitations such as high frame rate and high complexity requirements. No solution has been proposed yet.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] The purpose of the embodiments of the present disclosure is to provide a data processing method, a live broadcast method, a data processing device, a live broadcast device, an electronic device and a computer-readable storage medium, thereby at least to a certain extent overcoming the technical problem of poor live broadcast effect due to model limitations such as high frame rate and high complexity requirements.

[0006] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0007] According to a first aspect of an embodiment of the present disclosure, a data processing method is provided, comprising: obtaining an initial video; inputting the initial video into a pre-trained target super-resolution model to output an intermediate video, wherein the image resolution of the intermediate video is greater than the image resolution of the initial video; and performing detail recovery processing on the intermediate video to obtain a target video.

[0008] In some example embodiments of the present disclosure, based on the aforementioned scheme, a target super-resolution model is obtained based on a pre-training process, and the pre-training process includes: obtaining a training data set; performing model training on a pre-constructed initial super-resolution model through the training data set to obtain an intermediate super-resolution model, wherein the initial super-resolution model includes a first convolutional layer and a re-parameterized convolutional layer, and the first convolutional layer and / or the re-parameterized convolutional layer are connected through a long and short residual network; and the network structure of the intermediate super-resolution model is reconstructed to obtain the target super-resolution model.

[0009] In some example embodiments of the present disclosure, based on the aforementioned scheme, the network structure of the intermediate super-resolution model is reconstructed to obtain a target super-resolution model, including: determining a second convolutional layer equivalent to the reparameterized convolutional layer; replacing the reparameterized convolutional layer of the intermediate super-resolution model with the second convolutional layer to obtain the target super-resolution model.

[0010] In some example embodiments of the present disclosure, based on the aforementioned scheme, a second convolutional layer equivalent to the heavy parameter convolutional layer is determined, including: performing equivalent fusion processing on each convolutional layer in the heavy parameter convolutional layer to determine the second convolutional layer; wherein the equivalent fusion processing includes Einstein summation.

[0011] In some example embodiments of the present disclosure, based on the aforementioned scheme, the target super-resolution model includes a first convolutional layer and a second convolutional layer, an initial video is input into a pre-trained target super-resolution model, and an intermediate video is output, including: inputting the initial video into the first convolutional layer to extract first feature data of the initial video; inputting the first feature data into the second convolutional layer to extract second feature data of the initial video; upsampling the second feature data to obtain expanded second feature data; and using the expanded second feature data as the intermediate video.

[0012] In some example embodiments of the present disclosure, based on the aforementioned scheme, detail recovery processing is performed on the intermediate video to obtain a target video, including: sharpening processing is performed on the intermediate video to enhance high-frequency information in the intermediate video to obtain the target video.

[0013] According to the second aspect of the embodiment of the present disclosure, a live broadcast method is also provided, including: obtaining a target video, where the target video is determined according to the data processing method of the first aspect as described above; obtaining live broadcast data, and generating a live broadcast data stream according to the target video and the live broadcast data; and pushing the live broadcast data stream to each live broadcast client for display.

[0014] In some example embodiments of the present disclosure, based on the aforementioned scheme, a live data stream is generated according to the target video and the live data, including: encoding the image data in the live data and the image data in the target video to obtain a live video stream; synthesizing the audio data in the live data and the audio data in the target video to obtain a live audio stream; and encapsulating the live video stream and the live audio stream to obtain a live data stream.

[0015] In some example embodiments of the present disclosure, based on the aforementioned scheme, the method also includes: setting a target clarity level on the live broadcast client according to the transmission status of the target video; setting a target clarity level on the live broadcast client according to the transmission status of the target video, including: if it is monitored that the target video is in a normal transmission state within a first preset time period, adding a target clarity level on the live broadcast client; or if it is monitored that the target video is in a disconnected state within a second preset time period, deleting the target clarity level in the live broadcast client.

[0016] According to a third aspect of an embodiment of the present disclosure, a data processing device is provided, including: a first acquisition unit, used to acquire an initial video; an output unit, used to input the initial video into a pre-trained target super-resolution model, and output an intermediate video, wherein the image resolution of the intermediate video is greater than the image resolution of the initial video; and a first processing unit, used to perform detail recovery processing on the intermediate video to obtain a target video.

[0017] According to the fourth aspect of the embodiment of the present disclosure, a live broadcast device is also provided, including: a third acquisition unit, used to acquire a target video, the target video is determined according to the data processing device as described in the third aspect above; a generation unit, used to acquire live broadcast data, and generate a live broadcast data stream according to the target video and the live broadcast data; a display unit, used to push the live broadcast data stream to each live broadcast client for display.

[0018] According to a fifth aspect of an embodiment of the present disclosure, there is provided an electronic device, comprising: a processor; and a memory, wherein computer-readable instructions are stored in the memory, and when the computer-readable instructions are executed by the processor, any one of the above methods is implemented.

[0019] According to a sixth aspect of an embodiment of the present disclosure, there is provided a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, any one of the above methods is implemented.

[0020] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0021] The data processing method in the example embodiment of the present disclosure obtains an initial video; inputs the initial video into a pre-trained target super-resolution model, outputs an intermediate video, wherein the image resolution of the intermediate video is greater than that of the initial video; and performs detail recovery processing on the intermediate video to obtain a target video. On the one hand, the pre-trained target super-resolution model can overcome the distortion of the image caused by excessive bandwidth and encoding loss of the input initial video in a high frame rate and high complexity live broadcast scenario; on the other hand, through detail recovery processing, the high-frequency information of the image is enhanced to achieve a visual effect that is more in line with the viewing habits of the audience.

[0022] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0024] Figure 1 A schematic diagram showing an exemplary system architecture to which a data processing method and apparatus of the present disclosure can be applied in an exemplary embodiment;

[0025] Figure 2 A schematic diagram schematically illustrates a data processing method according to some embodiments of the present disclosure;

[0026] Figure 3 A schematic diagram of a training network architecture according to some embodiments of the present disclosure is schematically shown;

[0027] Figure 4 A schematic diagram schematically illustrates an inference network architecture according to some embodiments of the present disclosure;

[0028] Figure 5 A schematic diagram schematically illustrates a heavy parameter module architecture according to some embodiments of the present disclosure;

[0029] Figure 6 Schematically showing a schematic diagram of the equivalent of a heavy parameter module according to some embodiments of the present disclosure;

[0030] Figure 7 Schematically illustrating a schematic diagram of a re-parameterized convolution according to some embodiments of the present disclosure;

[0031] Figure 8 A schematic diagram of a live broadcast method according to some embodiments of the present disclosure;

[0032] Fig. 9 A schematic diagram schematically illustrates a data processing device according to some embodiments of the present disclosure;

[0033] Fig.10 A schematic diagram of a live broadcast device according to some embodiments of the present disclosure is schematically shown;

[0034] Fig.11 A schematic diagram of the structure of a computer system of an electronic device according to some embodiments of the present disclosure is schematically shown.

[0035] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts. DETAILED DESCRIPTION

[0036] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be more comprehensive and complete and will fully convey the concept of the example embodiments to those skilled in the art.

[0037] In addition, the described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known methods, devices, implementations or operations are not shown or described in detail to avoid blurring the various aspects of the present disclosure.

[0038] Furthermore, the drawings are only schematic illustrations and are not necessarily drawn to scale. The block diagrams shown in the drawings are only functional entities and do not necessarily correspond to physically separate entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0039] Figure 1 A schematic diagram showing an exemplary system architecture of a data processing method and device to which the embodiments of the present disclosure can be applied.

[0040] like Figure 1 As shown, the system architecture 100 may include one or more of terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc. The terminal devices 101, 102, 103 may be various electronic devices with display screens, including but not limited to desktop computers, portable computers, smart phones, tablet computers, etc. It should be understood that Figure 1The number of terminal devices, networks and servers in the example is only illustrative. According to the implementation needs, there can be any number of terminal devices, networks and servers. For example, server 105 can be a server cluster composed of multiple servers. As an example, a client application can be installed in the terminal device, and the server provides a live broadcast service to each client application. For example, a user can use a terminal to install a live broadcast client application, and obtain the live broadcast service provided by the server through the live broadcast client application. The user can also use a terminal to install a browser client application, log in to the live broadcast page provided by the server through the browser client application, and obtain the live broadcast service. Usually, two types of users are involved in the live broadcast process, one type of user is an anchor user, and the other type of user is an audience user. Based on this, the user terminal can be divided into an anchor terminal and an audience terminal. The client application can provide an anchor live broadcast function and an audience viewing live broadcast function. The anchor user can use the anchor live broadcast function provided by the client application to broadcast video live, and the audience user can use the live broadcast viewing function provided by the client application to watch video content. For example, the host end with the client application installed can turn on the video shooting module of the host end, collect audio and video data in real time through the video shooting module, and send it to the server. The server sends the received audio and video data to the audience end with the client application installed, and the audience user can use the viewing function provided by the client application to watch the live broadcast content of the host user.

[0041] The data processing method provided in the embodiment of the present disclosure can be executed by the terminal devices 101, 102, and 103, and accordingly, the data processing device can also be set in the terminal devices 101, 102, and 103. The data processing method provided in the embodiment of the present disclosure can also be jointly executed by the terminal devices 101, 102, and 103 and the server 105, and accordingly, the data processing device can be set in the terminal devices 101, 102, and 103 and the server 105. In addition, the data processing method provided in the embodiment of the present disclosure can also be executed by the server 105, and accordingly, the data processing device can be set in the server 105, which is not particularly limited in this exemplary embodiment.

[0042] For example, in this example implementation, an initial video can be obtained from a server 105 deployed on a live broadcast platform and received through terminal devices 101, 102, and 103; then, the initial video is input into a pre-trained target super-resolution model and an intermediate video is output; thereafter, the server 105 continues to perform detail recovery processing on the intermediate video to obtain a target video.

[0043] However, it is easy for those skilled in the art to understand that the above operations are only for example and are not limited to this exemplary embodiment.

[0044] In one embodiment of the present disclosure, the data processing method may be executed on a terminal device or a server. The following description will be given by taking the server executing the data processing method in the present disclosure as an example. Figure 2 A schematic diagram of a data processing method according to some embodiments of the present disclosure is schematically shown.

[0045] refer to Figure 2 As shown, the data processing method may include the following steps:

[0046] Step S210, obtaining an initial video.

[0047] Specifically, the target video obtained by the embodiment of this example can be applied to a super-resolution cloud-based streaming media processing link, and the video stream of the initial video input into the processing link can be the original video stream of the live broadcast room, or it can be other types of original video streams of the live broadcast room, wherein the resolution of the original video stream can be 720P or 1080P, which is not limited in the present disclosure.

[0048] Step S220, inputting the initial video into a pre-trained target super-resolution model, and outputting an intermediate video, wherein the image resolution of the intermediate video is greater than the image resolution of the initial video.

[0049] Specifically, the target super-resolution model can be used to convert low-resolution image frames in the initial video into high-resolution image frames, wherein the intermediate video can be a video with improved resolution after the original 1080P live broadcast room video flows through the target super-resolution model.

[0050] Among them, the target super-resolution model is obtained based on a pre-training process, and the specific pre-training process includes: obtaining a training data set; training a pre-constructed initial super-resolution model through the training data set to obtain an intermediate super-resolution model, wherein the initial super-resolution model includes a first convolutional layer and a re-parameterized convolutional layer, and the first convolutional layer and / or the re-parameterized convolutional layer are connected through a long and short residual network; the network structure of the intermediate super-resolution model is reconstructed to obtain the target super-resolution model.

[0051] Specifically, the original video stream of the live broadcast room to be displayed and enhanced is obtained, and the video stream is used as a training data set to train the pre-built initial super-resolution model.

[0052] For example, the process of training the pre-built initial super-resolution model through the training data set to obtain the intermediate super-resolution model can be as follows: Figure 3 As shown, a 1080P video stream of the original game image is input to Figure 3In the 5*5 convolution layer (corresponding to the first convolution layer in the present disclosure) of the initial super-resolution model shown, a first set of training image feature data is obtained, and the first set of image training feature data is input into the re-parameterized convolution layer to obtain a second set of training image feature data, and then the second set of training image feature data is repeatedly iterated to obtain an intermediate super-resolution model, wherein the convolution layers of the initial super-resolution model are connected by a long and short residual network and an identity mapping is performed, the purpose is to further reduce the difficulty of machine learning and enhance the correlation between neighboring pixels, and as the resolution of the input 1080P video stream is larger, the re-parameterized structure is used to give the training process of the initial super-resolution model enough receptive field to learn the corresponding model parameters, and does not increase the subsequent reasoning time of the target super-resolution model. It also further improves the generation efficiency of the target video.

[0053] Specifically, the target super-resolution model is obtained by reconstructing the network structure of the intermediate super-resolution model. The specific implementation process includes the following: determining a second convolutional layer equivalent to the reparameterized convolutional layer; replacing the reparameterized convolutional layer of the intermediate super-resolution model with the second convolutional layer to obtain the target super-resolution model.

[0054] For example, Figure 3 The re-parameter module in the intermediate super-resolution model is equivalent to Figure 4 The convolution layer 3*3 in (corresponding to the second convolution layer in the present disclosure), of course, can be Figure 3 The heavy parameter module in is equivalent to the convolution layer of other levels, and the present disclosure is not limited to this. The present disclosure can effectively avoid the occurrence of blurred and distorted images due to factors such as bandwidth and encoding loss in high frame rate and high complexity live broadcast scenarios through the target super-resolution model.

[0055] Specifically, a second convolution layer equivalent to the heavy parameter convolution layer is determined, and the implementation steps include the following: performing equivalent fusion processing on each convolution layer in the heavy parameter convolution layer to determine the second convolution layer; wherein the equivalent fusion processing includes Einstein summation.

[0056] For example, Figure 5 The network structure of the heavy parameter module is shown. Figure 6 is the equivalent result of each convolutional layer in the heavy parameter module, where Figure 5 The internal schematic structure of the heavily parameterized convolution in can be shown as Figure 7 As shown in the figure, the characteristic is that the convolution layer does not add nonlinear factors, so the target super-resolution model can be obtained by using the Einstein summation formula in an equivalent fusion manner.

[0057] Specifically, the target super-resolution model includes a first convolution layer and a second convolution layer. The initial video is input into the pre-trained target super-resolution model, and an intermediate video is output, which is achieved by the following steps: inputting the initial video into the first convolution layer to extract the first feature data of the initial video; inputting the first feature data into the second convolution layer to extract the second feature data of the initial video; upsampling the second feature data to obtain expanded second feature data; and using the expanded second feature data as the intermediate video.

[0058] For example, Figure 4 As shown, if the initial live broadcast original picture 1080P video stream is input into a 5*5 convolutional layer (corresponding to the first convolutional layer in the present disclosure), after extracting the 1080P video frame features of the live broadcast original picture (that is, the first feature data in the present disclosure), the 1080P video frame features of the live broadcast original picture are continuously input into multiple 3*3 convolutional layers for refined feature extraction to obtain the extracted data (that is, the second feature data in the present disclosure), and the extracted data is expanded by upsampling to obtain the expanded data, and the expanded data (that is, the intermediate video in the present disclosure) is expanded by upsampling to enhance the data features in the live broadcast original picture.

[0059] Step S230, performing detail restoration processing on the intermediate video to obtain the target video.

[0060] Specifically, detail recovery can be to enhance the brightness of the image, enhance the display, make the edge contour clearer, etc., so as to obtain a target video with clearer picture quality and resolution, wherein the target video can be high-definition 2K quality or ultra-high-definition 4K quality, which is not limited in the present disclosure.

[0061] In the embodiment of this example, the intermediate video is processed for detail recovery to obtain the target video, and the following steps can be adopted: the intermediate video is sharpened to enhance the high-frequency information in the intermediate video to obtain the target video. That is, the present disclosure uses high-contrast filtering to achieve the sharpening effect on the expanded data, and obtains the target video with ultra-high-definition resolution by enhancing the high-frequency information in the target video screen, wherein the Gaussian blurred image ( Figure 3 The process of (result in) is performed before the target super-resolution model, which can not only reduce the inference time, but also increase the range of the Gaussian kernel, thereby enhancing the sharpening effect. Through this optimization operation, the following can be achieved: Figure 4 The inference model network architecture shown can achieve a sharpening effect with a convolution kernel size of 17 within 1.5 ms, while also improving the visual effect of the audience's viewing habits.

[0062] In summary, in the example embodiments of the present disclosure, an initial video is obtained; the initial video is input into a pre-trained target super-resolution model, and an intermediate video is output, wherein the image resolution of the intermediate video is greater than the image resolution of the initial video; and the intermediate video is processed for detail recovery to obtain a target video. On the one hand, the pre-trained target super-resolution model can overcome the distortion of the image caused by excessive bandwidth and encoding loss of the input initial video in high frame rate and high complexity live broadcast scenarios; on the other hand, through detail recovery processing, the high-frequency information of the image is enhanced to achieve a visual effect that is more in line with the viewing habits of the audience.

[0063] Figure 8 A schematic diagram of a live broadcast method according to some embodiments of the present disclosure is schematically shown.

[0064] refer to Figure 8 As shown, the live broadcast method may include the following steps:

[0065] Step S810, obtaining a target video, the target video is obtained according to the above Figure 2 The data processing method of the embodiment is determined.

[0066] Step S820, obtaining live broadcast data, and generating a live broadcast data stream according to the target video and the live broadcast data.

[0067] Step S830: Push the live broadcast data stream to each live broadcast client for display.

[0068] Among them, the live broadcast data may include the audio and video data of the live broadcast room, as well as the host's voice and camera images, etc.

[0069] Specifically, a live data stream is generated according to the target video and the live data, including: encoding the image data in the live data and the image data in the target video to obtain a live video stream; synthesizing the audio data in the live data and the audio data in the target video to obtain a live audio stream; encapsulating the live video stream and the live audio stream to obtain a live data stream.

[0070] For example, the game video (target video) with enhanced image quality is encoded with the live broadcast data to obtain the video stream of the live broadcast room, and the original game audio in the video stream of the live broadcast room is synthesized with the audio in the live broadcast data. The specific synthesis operation can be mixing and synthesizing the video stream of the live broadcast room, or muting the video stream of the live broadcast room to obtain a live audio stream, and encapsulating the live audio stream with the encoded live video stream to obtain a low-latency, high-frame-rate, high-quality ultra-high-definition game live broadcast data stream.

[0071] For example, a target clarity level is set in the client, and the clarity level is monitored through a heartbeat mechanism, wherein the target clarity level can be a 2K level or a 4K level, which is not limited in the present disclosure. Specifically, the method also includes: setting a target clarity level in the live broadcast client according to the transmission status of the target video; setting a target clarity level in the live broadcast client according to the transmission status of the target video, including: if it is monitored that the target video is in a normal transmission state within a first preset time period, adding a target clarity level in the live broadcast client; if it is monitored that the target video is in a disconnected state within a second preset time period, deleting the target clarity level in the live broadcast client.

[0072] For example, when the ultra-high-definition live data stream is pushed to the target clarity level after encapsulation is completed, a heartbeat mechanism is added to the target clarity levels of all live broadcast rooms. The specific implementation process of the heartbeat mechanism is as follows: monitor the transmission status of the target video. If the transmission status of the target video is normal within a certain period of time, add the target clarity level to the target client to ensure that the audience can watch the live video normally. If the transmission status of the target video is disconnected within a certain period of time, automatically delete the target clarity level on the target client to avoid the situation where the audience cannot watch the video after clicking the target clarity level. The present disclosure adds a heartbeat mechanism to the target clarity level of the client to avoid the situation where the user continues to click on the target clarity level when the transmission link is the video stream corresponding to the ultra-high-definition target video, resulting in a poor live broadcast experience for the audience.

[0073] It should be noted that, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0074] Furthermore, in this exemplary embodiment, a data processing apparatus is provided. Fig. 9 As shown, the data processing device 900 includes: a first acquisition unit 910 , an output unit 920 , and a first processing unit 930 .

[0075] Specifically, the first acquisition unit 910 is used to acquire an initial video;

[0076] An output unit 920, configured to input the initial video into a pre-trained target super-resolution model and output an intermediate video, wherein the image resolution of the intermediate video is greater than the image resolution of the initial video;

[0077] The first processing unit 930 is used to perform detail restoration processing on the intermediate video to obtain a target video.

[0078] In summary, in the example embodiment of the present disclosure, the initial video is acquired by the first acquisition unit 910; the output unit 920 inputs the initial video into the pre-trained target super-resolution model and outputs an intermediate video, wherein the image resolution of the intermediate video is greater than the image resolution of the initial video; the first processing unit 930 performs detail recovery processing on the intermediate video to obtain the target video. On the one hand, the pre-trained target super-resolution model can overcome the distortion of the image caused by excessive bandwidth and encoding loss of the input initial video in high frame rate and high complexity live broadcast scenarios; on the other hand, through detail recovery processing, the high-frequency information of the image is enhanced to achieve a visual effect that is more in line with the viewing habits of the audience.

[0079] In some example embodiments of the present disclosure, based on the aforementioned scheme, the device also includes: a second acquisition unit, used to acquire a training data set; a training unit, used to perform model training on a pre-constructed initial super-resolution model through the training data set to obtain an intermediate super-resolution model, wherein the initial super-resolution model includes a first convolutional layer and a re-parameterized convolutional layer, and the first convolutional layer and / or the re-parameterized convolutional layer are connected through a long and short residual network; a second processing unit, used to reconstruct the network structure of the intermediate super-resolution model to obtain a target super-resolution model.

[0080] In some example embodiments of the present disclosure, based on the aforementioned scheme, the second processing unit includes: a determination module, used to determine a second convolutional layer equivalent to the re-parameterized convolutional layer; a replacement module, used to replace the re-parameterized convolutional layer of the intermediate super-resolution model with the second convolutional layer to obtain a target super-resolution model.

[0081] In some example embodiments of the present disclosure, based on the aforementioned scheme, a determination module includes: a processing submodule, used to perform equivalent fusion processing on each convolution layer in the heavy parameter convolution layer to determine a second convolution layer; wherein the equivalent fusion processing includes Einstein summation.

[0082] In some example embodiments of the present disclosure, based on the aforementioned scheme, the target super-resolution model includes a first convolutional layer and a second convolutional layer, and an output unit, including: a first extraction module, used to input the initial video into the first convolutional layer, and extract first feature data of the initial video; a second extraction module, used to input the first feature data into the second convolutional layer, and extract second feature data of the initial video; a sampling module, used to upsample the second feature data to obtain expanded second feature data; and a determination module, used to use the expanded second feature data as an intermediate video.

[0083] In some example embodiments of the present disclosure, based on the aforementioned solution, the first processing unit includes: a sharpening module, which is used to perform sharpening processing on the intermediate video to enhance the high-frequency information in the intermediate video to obtain the target video.

[0084] In another exemplary embodiment, a live broadcast device is also provided, referring to Fig.10 As shown, the live broadcast device 1000 includes: a third acquisition unit 1010, a generation unit 1020, and a display unit 1030.

[0085] Specifically, the third acquisition unit 1010 is used to acquire a target video. The target video is based on the above Fig. 9 generated by the data processing device of the illustrated embodiment;

[0086] A generating unit 1020, configured to obtain live broadcast data and generate a live broadcast data stream according to the target video and the live broadcast data;

[0087] The display unit 1030 is used to push the live broadcast data stream to each live broadcast client for display.

[0088] In some example embodiments of the present disclosure, based on the aforementioned scheme, a generation unit includes: an encoding module, used to encode image data in the live broadcast data and image data in the target video to obtain a live video stream; a synthesis module, used to synthesize audio data in the live broadcast data and audio data in the target video to obtain a live audio stream; and an encapsulation module, used to encapsulate the live video stream and the live audio stream to obtain a live data stream.

[0089] In some example embodiments of the present disclosure, based on the aforementioned scheme, the device also includes: a setting unit, which is used to set the target clarity level in the live broadcast client according to the transmission status of the target video; the setting unit includes: an adding module, which is used to add the target clarity level in the live broadcast client if it is monitored that the target video is in a normal transmission state within a first preset time period; or a deleting module, which is used to delete the target clarity level in the live broadcast client if it is monitored that the target video is in a disconnected state within a second preset time period.

[0090] The specific details of each module of the above-mentioned data processing device have been described in detail in the corresponding data processing method, so they will not be repeated here.

[0091] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0092] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0093] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0094] In an exemplary embodiment of the present disclosure, a computer storage medium capable of implementing the above method is also provided. A program product capable of implementing the above method of this specification is stored thereon. In some possible embodiments, various aspects of the present disclosure may also be implemented in the form of a program product, which includes a program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps described in the above "Exemplary Method" section of this specification according to various exemplary embodiments of the present disclosure, such as the following steps: obtaining an initial video; inputting the initial video into a pre-trained target super-resolution model, and outputting an intermediate video, wherein the image resolution of the intermediate video is greater than the image resolution of the initial video; performing detail recovery processing on the intermediate video to obtain a target video.

[0095] In an optional embodiment, the target super-resolution model is obtained based on a pre-training process, and the pre-training process includes: obtaining a training data set; performing model training on a pre-constructed initial super-resolution model through the training data set to obtain an intermediate super-resolution model, wherein the initial super-resolution model includes a first convolutional layer and a re-parameterized convolutional layer, and the first convolutional layer and / or the re-parameterized convolutional layer are connected through a long and short residual network; and the network structure of the intermediate super-resolution model is reconstructed to obtain the target super-resolution model.

[0096] In an optional implementation, the network structure of the intermediate super-resolution model is reconstructed to obtain a target super-resolution model, including: determining a second convolutional layer equivalent to the reparameterized convolutional layer; replacing the reparameterized convolutional layer of the intermediate super-resolution model with the second convolutional layer to obtain the target super-resolution model.

[0097] In an optional implementation, determining a second convolutional layer equivalent to the heavy parameter convolutional layer includes: performing equivalent fusion processing on each convolutional layer in the heavy parameter convolutional layer to determine the second convolutional layer; wherein the equivalent fusion processing includes Einstein summation.

[0098] In an optional embodiment, the target super-resolution model includes a first convolutional layer and a second convolutional layer, and an initial video is input into a pre-trained target super-resolution model to output an intermediate video, including: inputting the initial video into the first convolutional layer to extract first feature data of the initial video; inputting the first feature data into the second convolutional layer to extract second feature data of the initial video; upsampling the second feature data to obtain expanded second feature data; and using the expanded second feature data as the intermediate video.

[0099] In an optional implementation, detail recovery processing is performed on the intermediate video to obtain the target video, including: sharpening processing is performed on the intermediate video to enhance high-frequency information in the intermediate video to obtain the target video.

[0100] In an optional embodiment, the embodiment of the present disclosure may also include a program product for implementing the above method, which may adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto, and in this document, a readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, apparatus, or device.

[0101] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0102] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0103] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0104] Program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0105] In addition, in an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided.

[0106] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0107] Refer to the following Fig.11 1100 according to this embodiment of the present disclosure is described. Fig.11 The electronic device 1100 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0108] like Fig.11 As shown, the electronic device 1100 is in the form of a general computing device. The components of the electronic device 1100 may include, but are not limited to: the at least one first processing unit 1110, the at least one storage unit 1120, a bus 1130 connecting different system components (including the storage unit 1120 and the first processing unit 1110), and a display unit 1140.

[0109] The storage unit stores a program code, and the program code can be executed by the first processing unit 1110, so that the first processing unit 1110 performs the steps described in the above "Exemplary Method" section of this specification according to various exemplary embodiments of the present disclosure. For example, the first processing unit 1110 can perform the following steps: obtaining an initial video; inputting the initial video into a pre-trained target super-resolution model, and outputting an intermediate video, wherein the image resolution of the intermediate video is greater than the image resolution of the initial video; performing detail recovery processing on the intermediate video to obtain a target video.

[0110] In an optional embodiment, the target super-resolution model is obtained based on a pre-training process, and the pre-training process includes: obtaining a training data set; performing model training on a pre-constructed initial super-resolution model through the training data set to obtain an intermediate super-resolution model, wherein the initial super-resolution model includes a first convolutional layer and a re-parameterized convolutional layer, and the first convolutional layer and / or the re-parameterized convolutional layer are connected through a long and short residual network; and the network structure of the intermediate super-resolution model is reconstructed to obtain the target super-resolution model.

[0111] In an optional implementation, the network structure of the intermediate super-resolution model is reconstructed to obtain a target super-resolution model, including: determining a second convolutional layer equivalent to the reparameterized convolutional layer; replacing the reparameterized convolutional layer of the intermediate super-resolution model with the second convolutional layer to obtain the target super-resolution model.

[0112] In an optional implementation, determining a second convolutional layer equivalent to the heavy parameter convolutional layer includes: performing equivalent fusion processing on each convolutional layer in the heavy parameter convolutional layer to determine the second convolutional layer; wherein the equivalent fusion processing includes Einstein summation.

[0113] In an optional embodiment, the target super-resolution model includes a first convolutional layer and a second convolutional layer, and an initial video is input into a pre-trained target super-resolution model to output an intermediate video, including: inputting the initial video into the first convolutional layer to extract first feature data of the initial video; inputting the first feature data into the second convolutional layer to extract second feature data of the initial video; upsampling the second feature data to obtain expanded second feature data; and using the expanded second feature data as the intermediate video.

[0114] In an optional implementation, detail recovery processing is performed on the intermediate video to obtain the target video, including: sharpening processing is performed on the intermediate video to enhance high-frequency information in the intermediate video to obtain the target video.

[0115] The storage unit 1120 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 11201 and / or a cache storage unit 11202 , and may further include a read-only storage unit (ROM) 11203 .

[0116] The storage unit 1120 may also include a program / utility 11204 having a set (at least one) of program modules 11205, such program modules 11205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0117] The bus 1130 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a first processing unit, or a local bus using any of a variety of bus architectures.

[0118] The electronic device 1100 may also communicate with one or more external devices 1200 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 1100, and / or communicate with any device that enables the electronic device 1100 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 1150. Furthermore, the electronic device 1100 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 1160. As shown, the network adapter 1160 communicates with other modules of the electronic device 1100 via a bus 1130. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 1100, including but not limited to: microcode, device drivers, redundant first processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0119] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0120] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0121] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

Claims

1. A data processing method, characterized in that: include: Get the initial video; Inputting the initial video into a pre-trained target super-resolution model, and outputting an intermediate video, wherein the image resolution of the intermediate video is greater than the image resolution of the initial video; Performing detail restoration processing on the intermediate video to obtain a target video; The target super-resolution model is obtained based on a pre-training process, and the pre-training process includes: Get the training dataset; Performing model training on a pre-constructed initial super-resolution model using the training data set to obtain an intermediate super-resolution model, wherein the initial super-resolution model includes a first convolutional layer and a re-parameterized convolutional layer, and the first convolutional layer and / or the re-parameterized convolutional layer are connected via a long-short residual network; Determine a second convolutional layer equivalent to the re-parameterized convolutional layer; The heavy parameter convolution layer of the intermediate super-resolution model is replaced by the second convolution layer to obtain a target super-resolution model.

2. The method according to claim 1, characterized in that The determining of a second convolutional layer equivalent to the re-parameterized convolutional layer comprises: Performing equivalent fusion processing on each convolution layer in the heavy parameter convolution layer to determine the second convolution layer; Wherein, the equivalent fusion processing includes Einstein summation.

3. The method according to claim 2, characterized in that The target super-resolution model includes the first convolution layer and the second convolution layer, inputs the initial video into the pre-trained target super-resolution model, and outputs an intermediate video, including: Inputting the initial video into the first convolutional layer to extract first feature data of the initial video; Inputting the first feature data into the second convolutional layer to extract second feature data of the initial video; Performing upsampling processing on the second feature data to obtain expanded second feature data; The expanded second feature data is used as the intermediate video.

4. The method according to claim 1, characterized in that Performing detail restoration processing on the intermediate video to obtain a target video includes: The intermediate video is sharpened to enhance high-frequency information in the intermediate video to obtain the target video.

5. A live broadcast method, characterized in that: include: Acquire a target video, wherein the target video is generated according to the method according to any one of claims 1 to 4; Acquire live broadcast data, and generate a live broadcast data stream according to the target video and the live broadcast data; The live broadcast data stream is pushed to each live broadcast client for display.

6. The method according to claim 5, characterized in that Generating a live data stream according to the target video and the live data includes: Encoding the image data in the live data and the image data in the target video to obtain a live video stream; The audio data in the live broadcast data is synthesized with the audio data in the target video to obtain a live broadcast audio stream; The live video stream and the live audio stream are encapsulated to obtain a live data stream.

7. The method according to claim 5, characterized in that The method further comprises: According to the transmission status of the target video, a target definition level is set on the live broadcast client; According to the transmission status of the target video, setting a target definition level on the live broadcast client includes: If it is monitored that the target video is in a normal transmission state within the first preset time period, then a target definition level is added to the live broadcast client; or If it is monitored that the target video is in a disconnected state within the second preset time period, the target definition level in the live broadcast client is deleted.

8. A data processing device, characterized in that: include: A first acquisition unit, used to acquire an initial video; An output unit, used to input the initial video into a pre-trained target super-resolution model, and output an intermediate video, wherein the image resolution of the intermediate video is greater than the image resolution of the initial video; A first processing unit, configured to perform detail restoration processing on the intermediate video to obtain a target video; Wherein, the target super-resolution model is obtained based on a pre-training process, and the device further comprises: A second acquisition unit, used to acquire a training data set; A training unit, configured to perform model training on a pre-constructed initial super-resolution model using the training data set to obtain an intermediate super-resolution model, wherein the initial super-resolution model includes a first convolutional layer and a re-parameterized convolutional layer, and the first convolutional layer and / or the re-parameterized convolutional layer are connected via a long-short residual network; A determination module, used to determine a second convolutional layer equivalent to the re-parameterized convolutional layer; A replacement module is used to replace the heavy parameter convolution layer of the intermediate super-resolution model with the second convolution layer to obtain a target super-resolution model.

9. A live broadcast device, characterized in that: include: A third acquisition unit, configured to acquire a target video, wherein the target video is generated according to the method according to any one of claims 1 to 4; A generating unit, configured to obtain live broadcast data and generate a live broadcast data stream according to the target video and the live broadcast data; The display unit is used to push the live broadcast data stream to each live broadcast client for display.

10. An electronic device, comprising: processor; as well as A memory having computer-readable instructions stored thereon, wherein the computer-readable instructions, when executed by the processor, implement the method according to any one of claims 1 to 7.

11. A computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the method according to any one of claims 1 to 7 when executed by a processor.

Images