Live data stream interaction method, device, server and readable storage medium

Abstract

The embodiments of the present application provide a live data stream interaction method, device, server and readable storage medium. The live broadcast providing terminal performs feature incomplete processing on the live video stream to be interacted with and then sends the incomplete video stream to the server during the live broadcast interaction process, which can effectively save the data transmission amount and improve the data transmission speed. The server then redraws the complete content of the incomplete video stream, compresses the redrawn video stream and sends the compressed video stream to the live broadcast viewing terminal. The performance resource consumption of the live broadcast viewing terminal can be reduced by predicting and redrawing the integrity of the live content. Moreover, only sending the compressed video stream to the live broadcast viewing terminal also greatly saves the data transmission amount. The live broadcast viewing terminal can then convert the compressed video stream to display the converted video stream, thereby greatly reducing the transmission data amount while ensuring the viewing experience, reducing bandwidth costs and live broadcast delays.

Description

Technical Field

[0001] The present application relates to the field of mobile Internet live broadcast technology, and more specifically, to a live broadcast data stream interaction method, device, server and readable storage medium. Background Art

[0002] During the live broadcast process of mobile Internet terminals, the live broadcast platform's video live broadcast delay performance may affect user interaction or affect the user's opportunity to obtain key information, thereby having a significant impact on user experience. Based on this, how to effectively improve the live broadcast platform's video live broadcast delay performance while ensuring the viewing experience is a technical problem that needs to be solved urgently in this field. Summary of the invention

[0003] In view of this, the purpose of the present application is to provide a live data stream interaction method, device, server and readable storage medium, which can greatly reduce the amount of transmitted data while ensuring the viewing experience, and reduce bandwidth costs and live broadcast delays.

[0004] According to a first aspect of the present application, a live data stream interaction method is provided, which is applied to a live data stream interaction system, wherein the live data stream interaction system includes a server and a live broadcast providing terminal and a live broadcast viewing terminal that are communicatively connected to the server. The method includes:

[0005] The live broadcast providing terminal performs feature defect processing on the live broadcast video stream to be interacted with, obtains a defective video stream, and sends the defective video stream to the server, wherein the defective video stream has content loss of at least one defective dimension compared with the live broadcast video stream;

[0006] The server redraws the complete content of the incomplete video stream, obtains a redrawn video stream corresponding to the incomplete video stream, compresses the redrawn video stream and sends the compressed video stream to the live viewing terminal;

[0007] The live viewing terminal converts the compressed video stream to display the converted video stream.

[0008] In a possible implementation of the first aspect, the live broadcast providing terminal performs feature defect processing on the live broadcast video stream to be interacted with, and in the step of obtaining the defective video stream, the feature defect processing includes one or more combinations of the following methods:

[0009] Cutting the size of the live video stream to be interacted with;

[0010] Reducing the color of the live video stream to be interacted with;

[0011] Some video frames in the live video stream to be interacted with are deleted.

[0012] In a possible implementation manner of the first aspect, the step of, by the server, redrawing the complete content of the defective video stream to obtain a redrawn video stream corresponding to the defective video stream includes:

[0013] Inputting the defective video stream into a pre-trained complete content redrawing model, and predicting defective content information in the defective video stream based on the complete content redrawing model;

[0014] The content of the defective video stream is redrawn based on the predicted defective content information to obtain a redrawn video stream corresponding to the defective video stream.

[0015] In a possible implementation of the first aspect, the method further includes:

[0016] Acquire original training image data for each target live broadcastable activity, wherein the target live broadcastable activity has regularized image features;

[0017] Performing feature defect processing on the original training image data to obtain defective training image data and annotated defective part data corresponding to the defective training image data, wherein the defective training image data and the original training image data have content loss in at least one defective dimension;

[0018] The initial content redrawing model is trained according to the incomplete training image data and the annotated incomplete part data corresponding to the incomplete training image data to obtain a complete content redrawing model.

[0019] In a possible implementation of the first aspect, the step of training the initial content redrawing model according to the mutilated training image data and the annotated mutilated part data corresponding to the mutilated training image data to obtain the complete content redrawing model includes:

[0020] Inputting the defective training image data into an initial content redrawing model, and predicting defective content information in the defective training image data based on the initial content redrawing model;

[0021] Based on the predicted incomplete content information and the corresponding annotated incomplete part data, the prediction error parameters for the predicted incomplete content information are determined, and the model parameter information of the initial content redrawing model is adjusted based on the prediction error parameters, and then iterative training is continued until the prediction error parameters converge, and the trained complete content redrawing model is output.

[0022] In a possible implementation manner of the first aspect, the step of converting the compressed video stream by the live viewing terminal to display the converted video stream includes:

[0023] The live viewing terminal inputs the compressed video stream into a pre-trained video stream conversion model corresponding to a preset resolution, converts the resolution of the compressed video stream to the preset resolution based on the video stream conversion model, and then displays the converted video stream.

[0024] In a possible implementation of the first aspect, the method further includes:

[0025] Acquire training sample data, wherein the training sample data includes first video stream sample data of a first resolution and corresponding second video stream sample data of a second resolution;

[0026] Inputting the first video stream sample data into an initial video stream conversion model, and converting the first video stream sample data to the second resolution based on the initial video stream conversion model to obtain third video stream sample data;

[0027] The initial video stream conversion model is trained according to the difference between the third video stream sample data and the second video stream sample data to obtain a trained video stream conversion model.

[0028] According to a second aspect of the present application, a live data stream interaction method is provided, which is applied to a server, wherein the server is communicatively connected with a live broadcast providing terminal and a live broadcast viewing terminal, and the method comprises:

[0029] Acquire a mutilated video stream sent by the live broadcast providing terminal after performing feature mutilation processing on the live broadcast video stream to be interacted with, wherein the mutilated video stream has content loss of at least one mutilation dimension with the live broadcast video stream;

[0030] Redrawing the complete content of the defective video stream to obtain a redrawn video stream corresponding to the defective video stream;

[0031] After compressing the redrawn video stream, the compressed video stream is sent to the live viewing terminal, so that the live viewing terminal converts the compressed video stream to display the converted video stream.

[0032] According to a third aspect of the present application, a live data stream interaction device is provided, which is applied to a server, wherein the server is communicatively connected with a live broadcast providing terminal and a live broadcast viewing terminal, and the device comprises:

[0033] An acquisition module, configured to acquire a mutilated video stream sent by the live broadcast providing terminal after performing feature mutilation processing on the live broadcast video stream to be interacted with, wherein the mutilated video stream has content loss of at least one mutilation dimension compared with the live broadcast video stream;

[0034] A redrawing module, used to redraw the complete content of the defective video stream to obtain a redrawn video stream corresponding to the defective video stream;

[0035] The sending module is used to compress the redrawn video stream and then send the compressed video stream to the live viewing terminal, so that the live viewing terminal converts the compressed video stream to display the converted video stream.

[0036] According to a fourth aspect of the present application, a server is provided, the server comprising a machine-readable storage medium and a processor, the machine-readable storage medium storing machine-executable instructions, and when the processor executes the machine-executable instructions, the server implements the aforementioned live data stream interaction method.

[0037] According to a fifth aspect of the present application, a readable storage medium is provided, wherein the readable storage medium stores machine executable instructions, and the machine executable instructions, when executed, implement the aforementioned live data stream interaction method.

[0038] Based on any of the above aspects, in an embodiment of the present application, the live broadcast providing terminal performs feature incomplete processing on the interactive live video stream and then sends the incomplete video stream to the server during the live broadcast interaction process, which can effectively save data transmission volume and improve data transmission speed. The server then redraws the complete content of the incomplete video stream, compresses the redrawn video stream, and then sends the compressed video stream to the live broadcast viewing terminal. By predicting and redrawing the integrity of the live broadcast content, the performance resource consumption of the live broadcast viewing terminal can be reduced, and only sending the compressed video stream to the live broadcast viewing terminal also greatly saves data transmission volume. The live broadcast viewing terminal can then convert the compressed video stream to display the converted video stream, thereby greatly reducing the amount of transmitted data while ensuring the viewing experience, and at the same time reducing bandwidth costs and live broadcast delays. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0040] Figure 1 A schematic diagram of an application scenario of a live broadcast system provided in an embodiment of the present application is shown;

[0041] Figure 2 One of the flow diagrams of the live data stream interaction method provided in the embodiment of the present application is shown;

[0042] Figure 3 Shows Figure 2 One of the comparison diagrams of the example of the mutilation process of step S110 shown in FIG.

[0043] Figure 4 Shows Figure 2 FIG2 is a comparison diagram of an example of mutilation processing of step S110 shown in FIG.

[0044] Figure 5 Shows Figure 2 The sub-step flow chart of step S120 shown in FIG.

[0045] Figure 6 A schematic diagram showing a flow chart of a complete content redrawing model training method provided in an embodiment of the present application is shown;

[0046] Figure 7 A schematic diagram of a process flow of a video stream conversion model training method provided in an embodiment of the present application is shown;

[0047] Figure 8 The second flowchart of the live data stream interaction method provided in the embodiment of the present application is shown;

[0048] Fig. 9 A schematic diagram of functional modules of a live data stream interaction device provided in an embodiment of the present application is shown;

[0049] Fig.10 A schematic block diagram of the structure of a server for implementing the above-mentioned live data stream interaction method provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0050] To make the purpose, technical scheme and advantages of the embodiment of the present application clearer, the technical scheme in the embodiment of the present application will be clearly and completely described below in conjunction with the drawings in the embodiment of the present application. It should be understood that the drawings in the present application only serve the purpose of explanation and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in this application shows the operations implemented according to some embodiments of the embodiment of the present application. It should be understood that the operations of the flowchart can be implemented out of sequence, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart under the guidance of the content of the present application, or remove one or more operations from the flowchart.

[0051] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0052] Reference Figure 1 As shown, Figure 1 The interactive scene schematic diagram of the live broadcast system 10 provided in the embodiment of the present application is shown. For example, the live broadcast system 10 may be a service platform for, for example, Internet live broadcast. The live broadcast system 10 may include a server 100, a live broadcast providing terminal 200, and a live broadcast viewing terminal 300. The server 100 is respectively connected to the live broadcast providing terminal 200 and the live broadcast viewing terminal 300 for communication, and is used to provide live broadcast services for the live broadcast providing terminal 200 and the live broadcast viewing terminal 300, and provide related product live broadcast services during the live broadcast process, such as game live broadcast services, etc., for the live broadcast providing terminal 200 and the live broadcast viewing terminal 300.

[0053] In some implementation scenarios, the live broadcast providing terminal 200 and the live broadcast viewing terminal 300 can be used interchangeably. For example, the anchor of the live broadcast providing terminal 200 can use the live broadcast providing terminal 200 to provide live broadcast video services to the audience, or watch the live broadcast videos provided by other anchors as an audience. For another example, the audience of the live broadcast viewing terminal 300 can also use the live broadcast viewing terminal 300 to watch the live broadcast video provided by the anchor they are concerned about, or provide live broadcast video services to other audiences as an anchor.

[0054] In this embodiment, the live broadcast providing terminal 200 and the live broadcast viewing terminal 300 may be, but are not limited to, a smart phone, a personal digital assistant, a tablet computer, a personal computer, a laptop computer, a virtual reality terminal device, an augmented reality terminal device, etc. In the specific implementation process, there may be zero, one or more live broadcast providing terminals 200 and live broadcast viewing terminals 300 connected to the server 100. Figure 1 Among them, the live broadcast providing terminal 200 and the live broadcast viewing terminal 300 can be installed with a program product for providing Internet live broadcast services. For example, the program product can be an application APP, a web page, a small program, etc. related to the Internet live broadcast service used in a computer or a smart phone.

[0055] In this embodiment, the server 100 may be a single physical server or a server group consisting of multiple physical servers for performing different data processing functions. The server group may be centralized or distributed (for example, the server 100 may be a distributed system). In some possible implementations, if the server 100 uses a single physical server, different logical server components may be allocated to the physical server based on different live broadcast service functions.

[0056] Understandably, Figure 1 The live broadcast system 10 shown is only a feasible example. In other feasible embodiments, the live broadcast system 10 may also include only Figure 1 Some of the components shown may also include other components.

[0057] Combine the following Figure 1 The application scenario shown in the figure is an exemplary description of the live data stream interaction method provided in the embodiment of the present application. Figure 2 The live data stream interaction method provided in this embodiment can be implemented by Figure 1 It should be understood that in other embodiments, the order of some steps in the live data stream interaction method of this embodiment can be interchanged according to actual needs, or some steps can be omitted or deleted. The detailed steps of the live data stream interaction method executed by the live system 10 are described as follows.

[0058] In step S110, the live broadcast providing terminal 200 performs feature defect processing on the live broadcast video stream to be interacted with, obtains the defective video stream, and sends the defective video stream to the server 100.

[0059] In this embodiment, the incomplete video stream and the live video stream may have at least one incomplete dimension of content loss. When the live video stream has partial content loss, it can be used as an incomplete video stream. As an example, the content loss of the incomplete dimension can be part of the content with a lower degree of importance in the live video stream to be interacted, such as some content at the edge of the live video or irrelevant to the live focus (such as the anchor). Among them, the incomplete video stream has partial loss of video content compared to the live video stream, such as the incomplete loss of part of the image area in a certain video frame, or the incomplete loss of part of the video frame in a continuous video frame, or the color of a certain video frame is reduced, but not limited to this. For those skilled in the art, any feature incomplete processing involving content loss can be performed on the live video stream to be interacted based on the above examples to obtain a incomplete video stream with less content features than the live video stream to be interacted, thereby effectively saving the amount of data transmission between the live broadcast providing terminal 200 and the server 100 during the data interaction process and improving the data transmission speed.

[0060] In step S120 , the server 100 redraws the complete content of the incomplete video stream, obtains a redrawn video stream corresponding to the incomplete video stream, compresses the redrawn video stream, and sends the compressed video stream to the live viewing terminal 300 .

[0061] In this embodiment, the purpose of the above incomplete video stream is to improve the data transmission efficiency. However, for the subsequent live interactive process, only transmitting the incomplete video stream to the live viewing terminal 300 cannot provide the audience with complete live content. Therefore, in order to enable the subsequent live viewing terminal 300 to provide complete live content, it is also necessary to redraw the complete content of the aforementioned incomplete video stream on the server 100 side to obtain the redrawn video stream corresponding to the incomplete video stream, so that the redrawn video stream can have complete content characteristics, that is, the same or similar content characteristics as the aforementioned live video stream to be interacted. Then, in order to improve the data transmission efficiency, the server 100 can compress the redrawn video stream and send the compressed video stream to the live viewing terminal 300. It is worth noting that there is no special limitation on the compression method of the redrawn video stream. For example, it can be achieved by reducing the resolution of the redrawn video stream, reducing the final file size of the redrawn video stream, etc., but it is not limited to this.

[0062] Step S130 : the live viewing terminal 300 converts the compressed video stream to display the converted video stream.

[0063] In this embodiment, in order to ensure the viewing experience of the audience, the live viewing terminal 300 can convert the compressed video stream to display the converted video stream. For example, the resolution of the compressed video stream can be increased to a preset resolution, such as 1080P, 2K, 4K resolution, etc.

[0064] Based on the above steps, the live broadcast providing terminal 200 performs feature incomplete processing on the interactive live video stream and then sends the incomplete video stream to the server 100 during the live broadcast interaction process, which can effectively save data transmission volume and improve data transmission speed. Then the server 100 redraws the complete content of the incomplete video stream, compresses the redrawn video stream and sends the compressed video stream to the live broadcast viewing terminal 300. By guessing and redrawing the integrity of the live content, the performance resource consumption of the live broadcast viewing terminal 300 can be reduced, and only the compressed video stream is sent to the live broadcast viewing terminal 300, which also greatly saves data transmission volume. Then the live broadcast viewing terminal 300 can convert the compressed video stream to display the converted video stream, thereby greatly reducing the amount of transmission data while ensuring the viewing experience, reducing bandwidth costs and live broadcast delays.

[0065] In a possible implementation, with respect to step S110, the live broadcast providing terminal 200 performs feature defective processing on the live broadcast video stream to be interacted with, and in the step of obtaining the defective video stream, the feature defective processing includes any one or a combination of the following methods (1)-(3):

[0066] (1) Crop the size of the interactive live video stream. For example, see Figure 3 As shown, A1 is the live video stream to be interacted with, and A2 is the incomplete video stream after the live video stream A1 to be interacted with is cropped. Taking the live broadcast activity as a game as an example, in A1, a complete character model P and local objects Q1, Q2, Q3 and Q4 can be included. The character model P and local objects Q1, Q2, Q3 and Q4 can form a panoramic view, while in A2, only the character model P and local objects Q1, Q2, Q3 and Q4 in some areas are included. Therefore, A2 is smaller in size than A1, and the area A3 outside A2 can be understood as a incomplete area, which can effectively reduce the amount of data transmission.

[0067] (2) Reduce the color saturation of the live video stream to be interacted. For example, the color saturation of the live video stream to be interacted can be adjusted to different degrees. When the color saturation is reduced, the corresponding pixel value is reduced, so that the content features of the video stream after the color saturation is reduced are reduced, thereby effectively reducing the amount of data transmission.

[0068] (3) Remove some video frames from the interactive live video stream. For example, you can delete the middle video frames of a certain time period from a complete live video stream. Figure 4 As shown, taking a live broadcast of a game as an example, the character model P moves from point L1 to point L2, and only the video frame B1 at point L1 and the video frame Bn at point L2 are retained, while the video frames B2, B3, ... Bn-1 in the middle of the movement process are deleted, thereby effectively reducing the amount of data transmission.

[0069] In a possible implementation, for step S120, this embodiment can redraw the complete content of the defective video stream based on artificial intelligence learning and prediction to obtain a redrawn video stream corresponding to the defective video stream. For example, please refer to Figure 5 , step S120 can be further implemented through the following exemplary steps.

[0070] Sub-step S121, inputting the defective video stream into a pre-trained complete content re-rendering model, and predicting the defective content information in the defective video stream based on the complete content re-rendering model.

[0071] Sub-step S122, redrawing the content of the defective video stream based on the predicted defective content information to obtain a redrawn video stream corresponding to the defective video stream.

[0072] In this embodiment, the pre-trained complete content redrawing model can have the ability to predict the incomplete content information in the incomplete video stream, and then redraw the content of the incomplete video stream based on the predicted incomplete content information to obtain the redrawn video stream corresponding to the incomplete video stream. By guessing and redrawing the integrity of the live content, the performance resource consumption of the live viewing terminal 300 can be reduced.

[0073] In a possible implementation manner, the specific training steps of the above complete content redrawing model are introduced below in conjunction with an exemplary embodiment. Figure 6 , an embodiment of the present application provides a complete content redrawing model training method, which can be implemented through the following steps, which are specifically described as follows.

[0074] Step S210: obtaining original training image data for each target live broadcastable activity.

[0075] In this embodiment, the target live broadcast activity has regularized image features. For example, the target live broadcast activity can be a game live broadcast activity. For the game scene, the image features of the scene screen during its operation are often regular. For example, panoramas, local images, character models, animation effects, etc. are all pre-designed fixed maps with regular features. Therefore, they can be applied to subsequent training processes to obtain better learning effects.

[0076] Step S220, performing feature defect processing on the original training image data to obtain defective training image data and annotated defective part data corresponding to the defective training image data, wherein the defective training image data and the original training image data have content loss in at least one defective dimension.

[0077] In this embodiment, after obtaining the original training image data, the original training image data can be subjected to feature defect processing to obtain defective training image data and annotated defective part data corresponding to the defective training image data. Figure 3 For example, the incomplete training image data may refer to A2, and the corresponding annotated incomplete part data may refer to A3. Figure 4 For example, the incomplete training image data may refer to B1 and Bn, and the corresponding annotated incomplete part data may refer to B2, B3, ..., Bn-1.

[0078] Step S230 , training the initial content redrawing model according to the incomplete training image data and the annotated incomplete part data corresponding to the incomplete training image data to obtain a complete content redrawing model.

[0079] For example, in a possible implementation, step S230 may be implemented through the following exemplary sub-steps.

[0080] Sub-step S231: input the defective training image data into the initial content redrawing model, and predict the defective content information in the defective training image data based on the initial content redrawing model.

[0081] In this embodiment, the defective training image data is input into the initial content redrawing model, so that the initial content redrawing model can predict the defective content information in the defective training image data. Figure 3 For example, by inputting A2 into the initial content redrawing model, the initial content redrawing model can predict A3 corresponding to A2. Figure 4 For example, by inputting B1 and Bn into the initial content redrawing model, the initial content redrawing model can predict B2, B3, ..., Bn-1 corresponding to B1 and Bn.

[0082] Sub-step S232, based on the predicted incomplete content information and the corresponding annotated incomplete part data, determines the prediction error parameters for the predicted incomplete content information, and adjusts the model parameter information of the initial content redrawing model based on the prediction error parameters and continues iterative training until the prediction error parameters converge, and outputs the trained complete content redrawing model.

[0083] In this embodiment, the prediction accuracy of the initial content redrawing model can be determined based on the gap between the predicted incomplete content information and the corresponding annotated incomplete data. For example, the prediction accuracy of the initial content redrawing model can be measured by determining the prediction error parameter of the predicted incomplete content information. If the gap between the predicted incomplete content information and the corresponding annotated incomplete data is larger, it means that the prediction accuracy of the initial content redrawing model is lower, that is, the larger the prediction error parameter is, at this time, it is necessary to adjust the model parameter information of the initial content redrawing model in combination with the prediction error parameter, and then continue to iterate the training until the prediction error parameter converges, and then output the trained complete content redrawing model.

[0084] For example, the model parameter information of the initial content redrawing model can be adjusted in the direction of continuously reducing the prediction error parameter until the prediction error parameter is lower than the preset error parameter, or no longer continues to decrease, indicating that the training process can be terminated, and the complete content redrawing model after training is output.

[0085] It is worth noting that in other possible implementations, it is also possible to determine whether the training process can be terminated based on other conditions. For example, after the number of iterative training reaches a preset number of training times, it can be determined that the training process can be terminated and the complete content of the training is output to redraw the model.

[0086] In a possible implementation, for the aforementioned step S130, the live viewing terminal 300 may convert the compressed video stream by artificial intelligence learning and prediction to display the converted video stream. For example, the live viewing terminal 300 inputs the compressed video stream into a pre-trained video stream conversion model corresponding to a preset resolution, converts the resolution of the compressed video stream to the preset resolution based on the video stream conversion model, and then displays the converted video stream.

[0087] Among them, the pre-trained video stream conversion model corresponding to the preset resolution can have the ability to convert the resolution of the compressed video stream into the preset resolution. The principle is usually to increase the number of pixels of the compressed video stream by predicting the pixel points of the compressed video stream, thereby improving its resolution.

[0088] In a possible implementation manner, the specific training steps of the aforementioned video stream conversion model are introduced below with reference to a specific example. Figure 7 , an embodiment of the present application also provides a video stream conversion model training method, which can be implemented through the following exemplary steps, which are specifically described as follows.

[0089] Step S310, obtaining training sample data.

[0090] In this embodiment, the training sample data may include, for example, first video stream sample data of a first resolution and corresponding second video stream sample data of a second resolution, wherein the second resolution is higher than the first resolution, for example, the first resolution may be 360P resolution, and the second resolution may be 2K resolution.

[0091] Step S320: input the first video stream sample data into the initial video stream conversion model, and convert the first video stream sample data to the second resolution based on the initial video stream conversion model to obtain third video stream sample data.

[0092] Step S330: training the initial video stream conversion model according to the difference between the third video stream sample data and the second video stream sample data to obtain a trained video stream conversion model.

[0093] In this embodiment, the difference between the third video stream sample data and the second video stream sample data can be used to determine the prediction accuracy of the initial video stream conversion model. If the difference between the predicted third video stream sample data and the corresponding second video stream sample data is larger, it means that the prediction accuracy of the initial video stream conversion model is lower. At this time, it is necessary to adjust the model parameter information of the initial video stream conversion model based on the difference, and then continue to iterate the training until the difference converges, and then output the trained video stream conversion model.

[0094] For example, the model parameter information of the initial video stream conversion model can be adjusted in the direction of continuously reducing the difference until the difference is lower than a preset difference parameter, or no longer continues to decrease, indicating that the training process can be terminated, and the trained video stream conversion model is output at this time.

[0095] It is worth noting that in other possible implementations, it is also possible to determine whether the training process can be terminated based on other conditions. For example, after the number of iterative training reaches a preset number of training times, it can be determined that the training process can be terminated and the trained video stream conversion model can be output.

[0096] Based on the same inventive concept, Figure 8 A flow chart of another live data stream interaction method provided in an embodiment of the present application is shown. Different from the above-mentioned embodiment, the live data stream interaction method is composed of Figure 1 It should be noted that the steps involved in the live data stream interaction method to be described below have been described in the above embodiment. The detailed contents of each step can be referred to the above embodiment description, and will not be described in detail here. Only the steps executed by the second server 100 are briefly described below.

[0097] Step S410, obtaining a mutilated video stream sent by the live broadcast providing terminal 200 after performing feature mutilation processing on the live broadcast video stream to be interacted with, where there is content loss in at least one mutilation dimension between the mutilated video stream and the live broadcast video stream.

[0098] Step S420, redrawing the complete content of the defective video stream to obtain a redrawn video stream corresponding to the defective video stream.

[0099] Step S430 , compressing the redrawn video stream and sending the compressed video stream to the live viewing terminal 300 , so that the live viewing terminal 300 converts the compressed video stream to display the converted video stream.

[0100] In a possible implementation, step S420 may input the incomplete video stream into a pre-trained complete content redrawing model, predict the incomplete content information in the incomplete video stream based on the complete content redrawing model, redraw the content of the incomplete video stream based on the predicted incomplete content information, and obtain a redrawn video stream corresponding to the incomplete video stream.

[0101] In a possible implementation, the training steps of the complete content redrawing model may include:

[0102] Original training image data for each target live broadcastable activity is obtained, wherein the target live broadcastable activity has regularized image features.

[0103] The original training image data is subjected to feature defect processing to obtain defective training image data and annotated defective part data corresponding to the defective training image data, wherein the defective training image data and the original training image data have content loss in at least one defective dimension.

[0104] The initial content redrawing model is trained according to the incomplete training image data and the annotated incomplete part data corresponding to the incomplete training image data to obtain a complete content redrawing model.

[0105] In a possible implementation, the step of training the initial content redrawing model according to the incomplete training image data and the annotated incomplete part data corresponding to the incomplete training image data to obtain the complete content redrawing model may include: inputting the incomplete training image data into the initial content redrawing model, and predicting the incomplete content information in the incomplete training image data based on the initial content redrawing model. Then, based on the predicted incomplete content information and the corresponding annotated incomplete part data, determining the prediction error parameter for the predicted incomplete content information, and adjusting the model parameter information of the initial content redrawing model based on the prediction error parameter, and continuing the iterative training until the prediction error parameter converges, and outputting the trained complete content redrawing model.

[0106] In a possible implementation, the step of the live viewing terminal 300 converting the compressed video stream to display the converted video stream may include: the live viewing terminal 300 inputs the compressed video stream into a pre-trained video stream conversion model corresponding to a preset resolution, converts the resolution of the compressed video stream into the preset resolution based on the video stream conversion model, and then displays the converted video stream.

[0107] Based on the same inventive concept, please refer to Fig. 9 , showing a functional module diagram of the live data stream interaction device 110 provided in an embodiment of the present application. This embodiment can divide the functional modules of the live data stream interaction device 110 according to the above method embodiment. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic, which is only a logical functional division, and there may be other division methods in actual implementation. For example, in the case of dividing each functional module according to each function, Fig. 9 The live data stream interactive device 110 shown is only a schematic diagram of the device. The live data stream interactive device 110 may include an acquisition module 111, a redrawing module 112 and a sending module 113. The functions of each functional module of the live data stream interactive device 110 are described in detail below.

[0108] The acquisition module 111 is used to acquire the mutilated video stream sent by the live broadcast providing terminal 200 after performing feature mutilation processing on the live broadcast video stream to be interacted with, and the mutilated video stream and the live broadcast video stream have content loss of at least one mutilation dimension. It can be understood that the acquisition module 111 can be used to execute the above step S410, and the detailed implementation method of the acquisition module 111 can refer to the above content related to step S410.

[0109] The redrawing module 112 is used to redraw the complete content of the defective video stream to obtain a redrawn video stream corresponding to the defective video stream. It can be understood that the redrawing module 112 can be used to perform the above step S420, and the detailed implementation of the redrawing module 112 can refer to the above content related to step S420.

[0110] The sending module 113 is used to compress the redrawn video stream and send the compressed video stream to the live viewing terminal 300, so that the live viewing terminal 300 converts the compressed video stream to display the converted video stream. It can be understood that the sending module 113 can be used to perform the above step S430, and the detailed implementation of the sending module 113 can refer to the above content related to step S430.

[0111] Based on the same inventive concept, please refer to Fig.10 , showing a structural schematic block diagram of a server 100 for executing the above-mentioned live data stream interaction method provided in an embodiment of the present application, the server 100 may include a live data stream interaction device 110, a machine-readable storage medium 120 and a processor 130.

[0112] In this embodiment, the machine-readable storage medium 120 and the processor 130 are both located in the server 100 and are separately arranged. However, it should be understood that the machine-readable storage medium 120 may also be independent of the server 100 and may be accessed by the processor 130 through a bus interface. Alternatively, the machine-readable storage medium 120 may also be integrated into the processor 130, for example, it may be a cache and / or a general register.

[0113] The processor 130 is the control center of the server 100. It uses various interfaces and lines to connect various parts of the entire server 100. By running or executing software programs and / or modules stored in the machine-readable storage medium 120, and calling data stored in the machine-readable storage medium 120, it executes various functions of the server 100 and processes data, thereby monitoring the server 100 as a whole. Optionally, the processor 130 may include one or more processing cores; for example, the processor 130 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor.

[0114] Among them, the processor 130 can be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the program execution of the live data stream interaction method provided by the above method embodiment.

[0115] The machine-readable storage medium 120 may be a ROM or other types of static storage devices that can store static information and instructions, a RAM or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), a magnetic disk storage medium or other magnetic storage devices, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The machine-readable storage medium 120 may exist independently and be connected to the processor 130 via a communication bus. The machine-readable storage medium 120 may also be integrated with the processor. Among them, the machine-readable storage medium 120 is used to store machine executable instructions for executing the scheme of the present application. The processor 130 is used to execute the machine executable instructions stored in the machine readable storage medium 120 to implement the live data stream interaction method provided by the aforementioned method embodiment.

[0116] The live data stream interaction device 110 may include software function modules (such as an acquisition module 111, a redrawing module 112, and a sending module 113) stored in a machine-readable storage medium 120. When the processor 130 executes the various software function modules included in the live data stream interaction device 110, the live data stream interaction method provided by the aforementioned method embodiment may be executed.

[0117] Since the server 100 provided in the embodiment of the present application is another implementation form of the method embodiment executed by the above-mentioned server 100, and the server 100 can be used to execute the live data stream interaction method provided by the above-mentioned method embodiment, the technical effects that can be obtained can refer to the above-mentioned method embodiment and will not be repeated here.

[0118] Furthermore, an embodiment of the present application also provides a readable storage medium containing computer executable instructions, which, when executed, can be used to implement the live data stream interaction method provided in the above method embodiment.

[0119] Of course, the storage medium containing computer executable instructions provided in the embodiment of the present application, whose computer executable instructions are not limited to the above method operations, can also execute related operations in the live data stream interaction method provided in any embodiment of the present application.

[0120] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices, and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0121] Although the present application is described herein in conjunction with various embodiments, in the process of implementing the present application for which protection is claimed, those skilled in the art may understand and implement other variations of the disclosed embodiments by viewing the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other components or steps, and "one" or "an" does not exclude multiple situations. A single processor or other unit may implement several functions listed in a claim. Certain measures are recorded in mutually different dependent claims, but this does not mean that these measures cannot be combined to produce good results.

[0122] The above are only various implementations of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A live data stream interaction method, characterized in that: Applied to a live data stream interactive system, the live data stream interactive system includes a server and a live broadcast providing terminal and a live broadcast viewing terminal connected to the server in communication, the method includes: The live broadcast providing terminal performs feature defect processing on the live broadcast video stream to be interacted with, obtains a defective video stream, and sends the defective video stream to the server, wherein the defective video stream has content loss of at least one defective dimension compared with the live broadcast video stream; The server redraws the complete content of the incomplete video stream, obtains a redrawn video stream corresponding to the incomplete video stream, compresses the redrawn video stream and sends the compressed video stream to the live viewing terminal; The live viewing terminal converts the compressed video stream to display the converted video stream; The step of converting the compressed video stream by the live viewing terminal to display the converted video stream includes: The live viewing terminal inputs the compressed video stream into a pre-trained video stream conversion model corresponding to a preset resolution, converts the resolution of the compressed video stream to the preset resolution based on the video stream conversion model, and then displays the converted video stream; The method further comprises: Acquire training sample data, wherein the training sample data includes first video stream sample data of a first resolution and corresponding second video stream sample data of a second resolution; Inputting the first video stream sample data into an initial video stream conversion model, and converting the first video stream sample data to the second resolution based on the initial video stream conversion model to obtain third video stream sample data; The initial video stream conversion model is trained according to the difference between the third video stream sample data and the second video stream sample data to obtain a trained video stream conversion model.

2. The live data stream interaction method according to claim 1, characterized in that: The live broadcast providing terminal performs feature defective processing on the live broadcast video stream to be interacted with, and in the step of obtaining the defective video stream, the feature defective processing includes one or a combination of the following methods: Cutting the size of the live video stream to be interacted with; Reducing the color of the live video stream to be interacted with; Some video frames in the live video stream to be interacted with are deleted.

3. The live data stream interaction method according to claim 1, characterized in that: The server redraws the complete content of the defective video stream to obtain a redrawn video stream corresponding to the defective video stream, including: Inputting the defective video stream into a pre-trained complete content redrawing model, and predicting defective content information in the defective video stream based on the complete content redrawing model; The content of the defective video stream is redrawn based on the predicted defective content information to obtain a redrawn video stream corresponding to the defective video stream.

4. The live data stream interaction method according to any one of claims 1 to 3, characterized in that: The method further comprises: Acquire original training image data for each target live broadcastable activity, wherein the target live broadcastable activity has regularized image features; Performing feature defect processing on the original training image data to obtain defective training image data and annotated defective part data corresponding to the defective training image data, wherein the defective training image data and the original training image data have content loss in at least one defective dimension; The initial content redrawing model is trained according to the incomplete training image data and the annotated incomplete part data corresponding to the incomplete training image data to obtain a complete content redrawing model.

5. The live data stream interaction method according to claim 4, characterized in that: The step of training the initial content redrawing model according to the incomplete training image data and the annotated incomplete part data corresponding to the incomplete training image data to obtain the complete content redrawing model includes: Inputting the defective training image data into an initial content redrawing model, and predicting defective content information in the defective training image data based on the initial content redrawing model; Based on the predicted incomplete content information and the corresponding annotated incomplete part data, the prediction error parameters for the predicted incomplete content information are determined, and the model parameter information of the initial content redrawing model is adjusted based on the prediction error parameters, and then iterative training is continued until the prediction error parameters converge, and the trained complete content redrawing model is output.

6. A live data stream interaction method, characterized in that: Applied to a server, the server is in communication connection with a live broadcast providing terminal and a live broadcast viewing terminal, the method comprises: Acquire a mutilated video stream sent by the live broadcast providing terminal after performing feature mutilation processing on the live broadcast video stream to be interacted with, wherein the mutilated video stream has content loss of at least one mutilation dimension with the live broadcast video stream; Redrawing the complete content of the defective video stream to obtain a redrawn video stream corresponding to the defective video stream; After compressing the redrawn video stream, the compressed video stream is sent to the live viewing terminal, so that the live viewing terminal converts the compressed video stream to display the converted video stream; The step of converting the compressed video stream by the live viewing terminal to display the converted video stream includes: The live viewing terminal inputs the compressed video stream into a pre-trained video stream conversion model corresponding to a preset resolution, converts the resolution of the compressed video stream to the preset resolution based on the video stream conversion model, and then displays the converted video stream; The method further comprises: Acquire training sample data, wherein the training sample data includes first video stream sample data of a first resolution and corresponding second video stream sample data of a second resolution; Inputting the first video stream sample data into an initial video stream conversion model, and converting the first video stream sample data to the second resolution based on the initial video stream conversion model to obtain third video stream sample data; The initial video stream conversion model is trained according to the difference between the third video stream sample data and the second video stream sample data to obtain a trained video stream conversion model.

7. The live data stream interaction method according to claim 6, characterized in that: The server redraws the complete content of the defective video stream to obtain a redrawn video stream corresponding to the defective video stream, including: Inputting the defective video stream into a pre-trained complete content redrawing model, and predicting defective content information in the defective video stream based on the complete content redrawing model; The content of the defective video stream is redrawn based on the predicted defective content information to obtain a redrawn video stream corresponding to the defective video stream.

8. The live data stream interaction method according to claim 6 or 7, characterized in that: The method further comprises: Acquire original training image data for each target live broadcastable activity, wherein the target live broadcastable activity has regularized image features; Performing feature defect processing on the original training image data to obtain defective training image data and annotated defective part data corresponding to the defective training image data, wherein the defective training image data and the original training image data have content loss in at least one defective dimension; The initial content redrawing model is trained according to the incomplete training image data and the annotated incomplete part data corresponding to the incomplete training image data to obtain a complete content redrawing model.

9. The live data stream interaction method according to claim 8, characterized in that: The step of training the initial content redrawing model according to the incomplete training image data and the annotated incomplete part data corresponding to the incomplete training image data to obtain the complete content redrawing model includes: Inputting the defective training image data into an initial content redrawing model, and predicting defective content information in the defective training image data based on the initial content redrawing model; Based on the predicted incomplete content information and the corresponding annotated incomplete part data, the prediction error parameters for the predicted incomplete content information are determined, and the model parameter information of the initial content redrawing model is adjusted based on the prediction error parameters, and then iterative training is continued until the prediction error parameters converge, and the trained complete content redrawing model is output.

10. A live data stream interactive device, characterized in that: Applied to a server, the server is communicatively connected with a live broadcast providing terminal and a live broadcast viewing terminal, and the device comprises: An acquisition module, configured to acquire a mutilated video stream sent by the live broadcast providing terminal after performing feature mutilation processing on the live broadcast video stream to be interacted with, wherein the mutilated video stream has content loss of at least one mutilation dimension compared with the live broadcast video stream; A redrawing module, used to redraw the complete content of the defective video stream to obtain a redrawn video stream corresponding to the defective video stream; A sending module, used for compressing the redrawn video stream and sending the compressed video stream to the live viewing terminal, so that the live viewing terminal converts the compressed video stream to display the converted video stream; The device further comprises a display module, wherein the display module is used for: Inputting the compressed video stream into a pre-trained video stream conversion model corresponding to a preset resolution through the live viewing terminal, converting the resolution of the compressed video stream to the preset resolution based on the video stream conversion model, and then displaying the converted video stream; The device also includes a training module, which is used to: Acquire training sample data, wherein the training sample data includes first video stream sample data of a first resolution and corresponding second video stream sample data of a second resolution; Inputting the first video stream sample data into an initial video stream conversion model, and converting the first video stream sample data to the second resolution based on the initial video stream conversion model to obtain third video stream sample data; The initial video stream conversion model is trained according to the difference between the third video stream sample data and the second video stream sample data to obtain a trained video stream conversion model.

11. A server, characterized in that: The server includes a machine-readable storage medium and a processor, wherein the machine-readable storage medium stores machine-executable instructions. When the processor executes the machine-executable instructions, the server implements the live data stream interaction method described in any one of claims 6-9.

12. A readable storage medium, characterized in that: The readable storage medium stores machine executable instructions, and when the machine executable instructions are executed, the live data stream interaction method described in any one of claims 6-9 is implemented.

Images