Live video processing method, device, equipment and storage medium

Abstract

The present invention relates to the field of live broadcast technology, and discloses a live video processing method, device, equipment and storage medium. The live video processing method is applied to a video live broadcast system, and the method includes: obtaining the configuration status of the anchor end and the live video stream generated by the anchor end, and pushing the live video stream to the audience end and the server end; based on the configuration status, switching the anchor end or the server end through a preset end-cloud switching strategy to encode the live video stream to obtain a coding signal; sending the coding signal to the audience end, so that the audience end can encode, process and render the live video stream based on the coding signal to obtain the final live video. The present invention can effectively improve the real-time performance of video transmission and the efficiency of coding, avoid illegal content from appearing on the audience end, ensure the compliance of the live content, and avoid the anchor from being passively punished for violations. It can improve the processing efficiency of illegal content in live videos and reduce the processing cost of live videos.

Description

Technical Field

[0001] The present invention relates to the field of live broadcast technology, and in particular to a live broadcast video processing method, device, equipment and storage medium. Background Art

[0002] With the development of the Internet and mobile communications, the live broadcast industry has risen rapidly, and the amount of live broadcast content has increased massively. However, live broadcast content is usually filled with a large amount of video and image data, which inevitably contains illegal content and sensitive information. It is not suitable for dissemination and needs to be strictly controlled and handled.

[0003] At present, in order to ensure the security of live broadcast ecological content, existing methods generally use artificial intelligence technology to review live video streams, identify suspected illegal images or text features in massive data, and then conduct final confirmation manually. Although this method reduces the cost of manual review, its efficiency is still low and lacks real-time performance. In order to ensure the smoothness of the video stream, video rendering and review are carried out at the same time, which also means that illegal content may still be seen by the audience in a short period of time. In addition, existing methods also pre-code illegal content in the server in advance to ensure that the content is reviewed before release to avoid the above-mentioned problem of illegal content leakage. However, when this method performs model recognition on the server side, it will encounter problems with high streaming bandwidth and server-side image and text recognition costs. Summary of the invention

[0004] The main purpose of the present invention is to provide a live video processing method, device, equipment and storage medium, aiming to solve the technical problem of high cost of processing illegal content in live video in existing methods.

[0005] A first aspect of the present invention provides a live video processing method, which includes: obtaining the configuration status of the anchor end and the live video stream generated by the anchor end, and pushing the live video stream to the audience end and the server end; based on the configuration status, switching the anchor end or the server end through a preset end-cloud switching strategy to encode the live video stream to obtain a coding signal; sending the coding signal to the audience end, so that the audience end can encode, process and render the live video stream based on the coding signal to obtain the final live video.

[0006] Optionally, in a first implementation method of the first aspect of the present invention, the configuration status includes a policy activation status and a processing resource status; based on the configuration status, switching the anchor end or the server end through a preset end-cloud switching strategy to encode the live video stream to obtain the coding signaling includes: in response to determining that the policy activation status is the end startup status, and the processing resource status meets the preset resource conditions, allowing the coding module of the anchor end to encode the live video stream to obtain the coding signaling; in response to determining that the policy activation status is the end stop status, and / or the processing resource status does not meet the preset resource conditions, starting the coding module of the server end to encode the live video stream to obtain the coding signaling.

[0007] Optionally, in a second implementation method of the first aspect of the present invention, the coding of the live video stream to generate the coding signal includes: performing format conversion on the live video stream to obtain a live video frame sequence arranged in time sequence; performing text area detection on the video frames in the live video frame sequence based on a preset text detection model to obtain a text area where the text in each of the video frames is located; performing text recognition on the text area based on a preset text recognition model to obtain text content; performing violation detection on the text content, and in response to determining that a violation exists in the text content, obtaining a violation text range; and generating the coding signal based on the violation text range.

[0008] Optionally, in a third implementation manner of the first aspect of the present invention, the preset text detection model is established based on the CSP-ELAN network structure obtained by adding an efficient layer aggregation network to the CSPDarknet53 structure; the preset text detection model samples text features based on a network structure consisting of 3×3 convolutional layers and 2×2 pooling layers; the preset text detection model also generates a detection anchor point box for text information in a live broadcast scenario based on a long convolution kernel; the preset text detection model is reduced in volume through int8 quantization and pruning.

[0009] Optionally, in a fourth implementation manner of the first aspect of the present invention, the preset text recognition model is established based on adding SENet to the backbone network structure of mobilenet v3, and the output features of the lowest network layer of the preset text recognition model are merged and output with the output features of the backbone network structure of mobilenet v3; the sequence modeling module of the preset text recognition model is established by an Encoder structure of a two-layer Transformer, and the high-dimensional convolution information is reduced to the channel dimension required by the sequence modeling module through pooling and dimensionality reduction, wherein the sequence modeling module is used to identify the contextual information of the text content in the text area; the preset text recognition model is also reduced in volume through pruning.

[0010] Optionally, in a fifth implementation of the first aspect of the present invention, the preset text detection model transfers the text area to the preset text recognition model through dynamic batch recognition.

[0011] Optionally, in a sixth implementation manner of the first aspect of the present invention, the preset text detection model and the preset text recognition model are encapsulated in a software development kit based on DirectML.

[0012] The second aspect of the present invention also provides a live video processing device, which includes: an acquisition module, used to obtain the configuration status of the anchor end and the live video stream generated by the anchor end, and push the live video stream to the audience end and the server end; a coding module, used to switch the anchor end or the server end to code the live video stream based on the configuration status through a preset end-cloud switching strategy to obtain a coding signal; a communication module, used to send the coding signal to the audience end, so that the audience end can code, process and render the live video stream based on the coding signal to obtain the final live video.

[0013] The third aspect of the present invention also provides a computer device, which includes: a memory and at least one processor, wherein the memory stores instructions; the at least one processor calls the instructions in the memory so that the computer device executes the live video processing method as described above.

[0014] A fourth aspect of the present invention further provides a computer-readable storage medium, on which instructions are stored, and when the instructions are executed by a processor, the live video processing method as described above is implemented.

[0015] The embodiments of the present invention provide a live video processing method, device, equipment and storage medium, which first obtain the configuration status of the anchor end and the live video stream, and push the live video stream to the audience end and the server end, and then apply the preset end-cloud switching strategy to encode the video stream according to the configuration status, so as to obtain and send the encoding signaling to the audience end, so that the audience end can encode the video stream after receiving the signaling to complete the live video output. By using the method, the present invention can flexibly and effectively switch the host end and the server end to code the live video stream, and preferentially use the host end for coding according to the configuration status, thereby reducing the cost of image and text recognition on the server end, and at the same time, the specific coding rendering is performed on the audience end, without the need to pull the user's video stream to the server end and push the coded content on the server end to the audience end again, thereby reducing the cost consumption of bandwidth resources, and is conducive to improving the real-time performance of video transmission and the efficiency of coding, ensuring that the audience sees the live video after coding processing and rendering, avoiding the appearance of illegal content on the audience end, ensuring the compliance of the live content, and avoiding the host from being passively punished for violations, effectively improving the processing efficiency of illegal content in the live video, and reducing the processing cost of the live video. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a schematic diagram of a first embodiment of a live video processing method according to an embodiment of the present invention;

[0017] Figure 2 for Figure 1 A schematic structural diagram of an embodiment of an optional video live broadcast system in the embodiment;

[0018] Figure 3 for Figure 1 A schematic flow chart of an embodiment of step 102 in the embodiment;

[0019] Figure 4 for Figure 1 A schematic flow chart of another embodiment of step 102 in the embodiment;

[0020] Figure 5 This is a functional module diagram of an embodiment of a live video processing device according to an embodiment of the present invention;

[0021] Figure 6 The figure is a functional module diagram of an embodiment of a computer device in an embodiment of the present invention. DETAILED DESCRIPTION

[0022] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0023] For ease of understanding, the specific process of the live video processing method in the embodiment of the present invention is described below. Figure 1 , Figure 1 The first embodiment of the live video processing method in the embodiment of the present invention is a flowchart of the first embodiment.

[0024] In this embodiment, the live video processing method specifically includes:

[0025] 101. Obtain the configuration status of the anchor end and the live video stream generated by the anchor end, and push the live video stream to the audience end and the server end;

[0026] 102. Based on the configuration status, the anchor end or the server end is switched to code the live video stream through a preset end-cloud switching strategy to obtain a coding signal;

[0027] 103. Send the coding signaling to the audience end, so that the audience end can code, process and render the live video stream based on the coding signaling to obtain the final live video.

[0028] In this embodiment, the host end refers to the producer and initiator of the live content, which is specifically initiated through a specific live platform software or web page in a terminal device such as a mobile phone, tablet, PC (Personal Computer), and generates audio and video data through the collection of audio and video devices such as microphones and cameras or through screen sharing technology, and pushes the stream to the server end for the audience to pull after pre-processing and encoding packaging. The audience end is the receiver and viewer of the live content, which is specifically connected to the server end through the network through the corresponding platform software or web page end, and receives and plays the live video stream sent by the host end. The server end acts as an intermediary between the host end and the audience end, and is responsible for receiving the live video stream from the host end, performing necessary processing and distribution, and ensuring that the audience end can smoothly receive and play the live content.

[0029] In this embodiment, the configuration status can be obtained through the specific monitoring mechanism set up, and the live video stream can be obtained through a camera or a recording tool. After the anchor generates the live video stream, it can be distributed to the server and the audience through the network, so that both the server and the audience can store the live video stream for subsequent coding processing and subsequent rendering based on coding signaling.

[0030] In this embodiment, after obtaining the configuration status, the situation of the anchor end can be judged according to the configuration status. When the configuration of the anchor end is not sufficient for effective coding, switch to the server end for coding, and realize flexible switching of coding between the anchor end and the server end. On the one hand, the anchor end can be used for coding to avoid resource occupation when the processing volume of the server end is too large. On the other hand, when the coding capacity of the anchor end is insufficient, it can switch to the server end for coding and obtain corresponding coding signaling. These signalings contain information such as the location, shape, size and format of the coding. After being sent to the audience end, they can instruct the audience end to code, process and render the live video stream, which can ensure real-time and effective coding of the live video stream, and can also avoid the cost increase caused by excessive occupation of server resources and the problem of reduced coding quality due to poor configuration status of the anchor end, thereby improving the coding processing efficiency of the live video.

[0031] In this embodiment, the live video stream received by the audience end will be processed and rendered in combination with the coding signaling sent from the anchor end or the server end. The player or related software on the audience end will blur, block or replace specific areas in the live video stream according to the information in the coding signaling to achieve the purpose of protecting privacy or hiding sensitive information. Among them, since coding has a higher efficiency and the coding signaling has a smaller volume than the coded video stream transmitted in the existing method, it can achieve faster coding rendering, ensuring that the audience end can receive the coded live video stream in real time and display the final live content during playback, so that the live content is processed by coding, processing and rendering, effectively avoiding the leakage of relevant sensitive information.

[0032] This embodiment first obtains the configuration status of the anchor end and the live video stream, and pushes the live video stream to the audience end and the server end, and then according to the configuration status, applies the preset end-cloud switching strategy to code the video stream, so as to obtain and send the coding signaling to the audience end, so that the audience end can code the video stream after receiving the signaling, and complete the live video output. Through this method, the present invention can flexibly and effectively switch the anchor end and the server end to code the live video stream, and preferably use the anchor end coding according to the configuration status, reducing the cost of image and text recognition on the server end, and at the same time, the specific coding rendering is performed on the audience end, without pulling the user's video stream to the server end and pushing the content after the server end coding to the audience end again, reducing the cost consumption of bandwidth resources, and is conducive to improving the real-time performance of video transmission and the efficiency of coding, ensuring that the audience sees the live video after coding processing and rendering, avoiding the appearance of illegal content on the audience end, ensuring the compliance of the live content, and avoiding the passive violation punishment of the anchor, effectively improving the processing efficiency of illegal content in the live video, and reducing the processing cost of the live video.

[0033] Optionally, the live video processing method is applied to Figure 2 In the described video live broadcast system, the video live broadcast system includes an anchor end, a service end and an audience end which are communicatively connected to each other.

[0034] In the live video broadcast system of this optional embodiment, specific data distribution can be achieved through CDN (Content Delivery Network) technology. CDN is a distributed network that can significantly reduce data transmission delays and improve content access speed and reliability by caching content to edge nodes around the world. When the viewer requests to watch a live broadcast, the server will select a CDN edge node closest to the viewer based on the viewer's location information to provide services, thereby achieving a fast and stable live viewing experience.

[0035] In this optional embodiment, the anchor end and the server end may specifically include a coding module, and the coding module is used to code the live video stream. The coding module is integrated in the live broadcast system, and the specific area in the video stream and the retrograde blur, occlusion or replacement are processed through image processing technology and algorithms to achieve the purpose of protecting privacy or hiding sensitive information. Specifically, the video stream can be captured through an interface, a preset transmission protocol or a specific capture tool, and sensitive information can be identified and selected through preset rules, image recognition algorithms or manual marking. The coding is processed by Gaussian blur, mosaic and color block replacement, and the coding-related data after coding is output to obtain the coding signal sent to the audience end. In this embodiment, it should be noted that the data output by the coding module used is the corresponding coded data rather than the data of the video stream after coding, which may specifically include information such as the location, shape, size and shape format of the coding, so as to reduce the data volume and bandwidth occupation of the corresponding coding signal and improve the subsequent processing efficiency.

[0036] In this optional embodiment, a monitoring mechanism for resource and policy monitoring is also specifically set up in the video live broadcast system. Specifically, resource and policy monitoring is set up at the anchor end to monitor the configuration status of the anchor end in real time, that is, the policy activation status used to indicate whether the corresponding preset end-cloud switching strategy is turned on, and the processing resource status used to indicate whether the anchor end has relevant application components such as a graphics card, microphone, and camera, and reflects the specific occupancy of the corresponding CPU (Central Processing Unit), GPU (Graphics Processing Unit) and memory. After obtaining the configuration status, it can be judged whether the corresponding coding work is performed on the anchor end or the server end. At the same time, a responsive end-cloud switching monitoring service is set up on the server side, and relevant anchor strategy configurations are received, as well as the configuration status that the anchor side periodically, at a fixed frequency, or responsively feeds back to the server side. Thus, both ends can determine the start and stop of the coding module according to the configuration status, and control the coding modules at both ends separately through start or stop instructions, thereby realizing the switching of end-cloud coding work, and thus realizing flexible coding processing of the live video stream according to the live video processing method described later.

[0037] Optional, see Figure 3 In one embodiment, the configuration state includes a policy activation state and a processing resource state; based on the configuration state, the anchor end or the server end is switched to code the live video stream through a preset end-cloud switching strategy to obtain a coding signal, including:

[0038] 1021. In response to determining that the policy activation state is the end startup state and the processing resource state meets the preset resource condition, allowing the coding module of the anchor end to code the live video stream to obtain a coding signaling;

[0039] 1022. In response to determining that the policy activation state is the end stop state, and / or the processing resource state does not meet the preset resource conditions, start the coding module of the server to code the live video stream to obtain the coding signaling.

[0040] In this optional embodiment, the policy enabling state refers to the switch state of whether the anchor end is allowed to perform coding operations, and the processing resource state is used to reflect whether the hardware resources of the anchor end support coding and whether the computing resources can be effectively used for coding. Among them, when the policy enabling state is the end startup state, it means that the anchor end has the authority to perform coding operations, and when the policy enabling state is the end shutdown state, it means that the anchor end does not have the authority to perform coding operations. When the processing resource state meets the preset resource conditions, it usually indicates that the anchor end has the hardware facilities required for live broadcasting, and the corresponding computing resources are sufficient for coding processing. For example, when the live broadcast end has an independent graphics card and the CPU utilization rate is lower than 80%, it means that the processing resource state meets the preset resource conditions. When the processing resource state does not meet the preset resource conditions, it usually indicates that there is a bottleneck in the hardware resources of the anchor end, and the coding operation may not be performed smoothly. For example, when the live broadcast end lacks an independent graphics card, or the CPU utilization rate is higher than 80%, it means that the processing resource state does not meet the preset resource conditions.

[0041] In this optional embodiment, when the policy enabling state is the end start state, and the processing resource state meets the preset resource conditions, it means that the host end has the authority to perform coding operations and the hardware resources are sufficient. Therefore, the coding process can be preferentially performed on the host end, allowing the coding module on the host end to code the live video stream. This can not only reduce the burden on the server end, but also improve the efficiency of coding. When the policy enabling state is the end stop state, and / or the processing resource state does not meet the preset resource conditions, it means that the coding function on the host end is not enabled, or the host end cannot perform effective coding operations. At this time, it is necessary to switch to the server end for coding processing to ensure that the coding of the live video stream continues and ensure the compliance of the live video stream.

[0042] In actual operation of this optional embodiment, the coding module on the host side and the coding module on the server side can adopt different algorithms and technologies to adapt to different hardware resources and processing requirements. For example, the coding module on the host side can adopt a lightweight algorithm to reduce the occupation of the host side hardware resources, while the coding module on the server side can adopt a more complex algorithm to improve the accuracy and efficiency of coding. At the same time, in order to further optimize the coding process, machine learning technology can also be introduced to evaluate and optimize the coding effect in real time. By collecting a large amount of coding sample data and training the machine learning model, it is possible to achieve intelligent adjustment of parameters such as coding position, shape, size and format to achieve the best coding effect.

[0043] This optional embodiment flexibly switches the end-cloud coding according to the configuration status and processing resource status of the anchor end, which can ensure the real-time performance of the live video stream, while avoiding coding delays or failures due to insufficient resources, and improving the flexibility and real-time performance of the coding process. The preset end-cloud switching strategy gives priority to using the anchor end for coding processing, which can reduce the image and text recognition cost of the server end, thereby reducing the overall coding processing cost and improving the efficiency of coding. Through real-time and effective coding processing, it is possible to effectively prevent illegal content from appearing on the audience end, ensure the compliance of the live content, and prevent the anchor from being passively punished for violations.

[0044] Optional, see Figure 4 In one embodiment, coding the live video stream to generate coding signaling includes:

[0045] 1023. Convert the format of the live video stream to obtain a live video frame sequence arranged in time sequence;

[0046] 1024. Perform text region detection on video frames in the live video frame sequence based on a preset text detection model to obtain a text region where text is located in each video frame;

[0047] 1025. Perform text recognition on the text area based on a preset text recognition model to obtain text content;

[0048] 1026. Perform violation detection on the text content, and in response to determining that there is a violation in the text content, obtain a range of the violation text;

[0049] 1027. Generate coding signaling based on the illegal text range.

[0050] In this optional embodiment, the live video stream is usually in the original RTMP (Real Time Messaging Protocol), HLS (HTTP Live Streaming, HTTP-based streaming media transmission protocol) and RMVB (Real Media Variable Bitrate) formats. Live video streams in these formats are usually not convenient for program parsing and lack uniformity. Therefore, the live video stream can be format converted first to facilitate subsequent processing. The format conversion can be specifically achieved by decoding and re-encoding, or by using tools such as FFmpeg and HandBrake to convert the original live video stream into a time-sequentially arranged video frame sequence in a coding format such as H.264, H.265 or VP9. This makes the video content easier to be parsed and processed by the program, making subsequent text detection, recognition, and violation detection operations more efficient and accurate.

[0051] In this optional embodiment, the preset text detection model is an image processing model pre-trained based on deep learning technology, which can identify text areas in video frames. For example, the preset text detection model can be established using a convolutional neural network (CNN) architecture, and by training a large amount of labeled video frame data, the feature representation of the text area is learned, so that the text detection model can process the live video frame sequence in real time, quickly locate the position and size of the text area, and thus provide accurate input data for subsequent text recognition.

[0052] In this optional embodiment, the preset text recognition model can also be an image processing model pre-trained and established based on deep learning technology, and the preset text recognition model can convert the image in the detected text area into recognizable text content. For example, the preset text recognition model can be established using a recurrent neural network (RNN) or a Transformer architecture, and by training a large amount of labeled text image data, the feature representation of the text is learned, so that the text recognition model can process the detected text area in real time and convert the text in the image into an editable text format, so that it can be used for subsequent violation detection and coding signal generation.

[0053] In this optional embodiment, violation detection of text content can be specifically implemented through hot word matching, similarity matching, semantic analysis or machine learning classifier. Among them, hot word matching needs to first obtain the illegal vocabulary in the hot word system, and then compare the text content with the vocabulary in the hot word library through text regularization and text tracking to determine whether there are illegal words. Similarity matching determines whether there are illegal content by calculating the similarity between the text content and the known illegal content. Semantic analysis uses natural language processing technology to understand the semantics of the text content and determine whether it contains illegal meanings. The machine learning classifier classifies the text content through the model obtained by training to determine whether it is illegal. In addition, in order to improve the accuracy of violation detection, a multimodal detection method can also be used, combining the advantages of multiple detection technologies to reduce the probability of false alarms and missed reports.

[0054] In this optional embodiment, after determining that there is a violation in the text content, corresponding data caching can also be performed. After determining that there is a violation in the text content, the system will store the illegal text range and related information in the cache. In this way, when the same illegal content appears again, the system can quickly retrieve the corresponding coding signaling from the cache, thereby avoiding repeated time-consuming operations such as text detection and recognition. In addition, the cache can be prioritized according to the severity and frequency of occurrence of the illegal content, ensuring that the illegal content that is most likely to appear again can be cached first.

[0055] In this optional embodiment, after determining that there is a violation in the text content, the corresponding illegal text range can be obtained by marking the coordinate position of the illegal text area, or by extracting the characteristic information of the illegal text area. The illegal text range can specifically include the position, size and shape of the illegal text in the image of the frame sequence, as well as auxiliary information such as the decoding timestamp, the text violation type, and the text semantic content. Through this information, the corresponding illegal area can be effectively determined in the image of the frame sequence and the specific situation of the violation can be understood.

[0056] In this optional embodiment, after determining the scope of the illegal text, corresponding coding data and corresponding coding signaling can be generated within the scope of the illegal text. The coding signaling can specifically include parameters such as the coordinates, shape, and color of the coding area, ensuring that the scope of the area that needs to be coded in the live video stream can be determined according to the coding signaling, so that when the audience obtains the coding signaling for coding rendering, the illegal text can be efficiently and accurately covered and blocked, ensuring that the illegal content cannot be seen by the audience, ensuring the compliance of the live content, and reducing the risk of violations by the anchor and the platform.

[0057] Optionally, in one embodiment, the preset text detection model is established based on the CSP-ELAN network structure obtained by adding an efficient layer aggregation network to the CSPDarknet53 structure.

[0058] In this optional embodiment, CSPDarknet53 is a backbone network architecture designed for real-time target detection, which is upgraded from the CSPNet (Cross Stage Partial Network) structure and includes five CSPNet structures, combining the advantages of Cross Stage Partial Connections and Darknet networks. The CSP-ELAN network structure used in the present invention further optimizes the infrastructure of CSPDarknet53 by introducing an efficient layer aggregation network (EfficientLayer Aggregation Network, ELAN). Through the efficient layer aggregation network, the shortest and longest gradient paths can be controlled, and feature information at different levels can be aggregated, so that the network can better capture the detailed features of the text area. A preset text structure model is established based on the CSP-ELAN network structure, which effectively improves the computational efficiency while maintaining the gradient path structure, can more effectively fuse features at different levels, and improve the accuracy of text detection.

[0059] Optionally, in one embodiment, the preset text detection model samples text features based on a network structure consisting of a 3×3 convolutional layer and a 2×2 pooling layer.

[0060] In this optional embodiment, unlike the second branch of CSPDarknet53 which uses 1x1+4 3x3 convolutions for feature extraction, the CSP-ELAN used in this article uses the CSPNet structure, which has better scalability and can support general computing blocks such as bottleneck, Inception and BasicBlock, compared with ELAN which only uses convolutional layer stacking. At the same time, in the process of connecting the Neck network, adding a 3x3+2x2 convolution pooling layer reduces the spatial resolution and improves the computational efficiency. Among them, the 3×3 convolution layer can capture the local features in the image, and the 2×2 pooling can further reduce the dimension of the feature map, so that the model can process images of larger sizes. Through this combination, the preset text detection model can increase the number of channels while reducing the spatial resolution, ensuring multi-scale information, and can achieve fast text area detection while maintaining high accuracy. Compared with the text detection model used by the traditional server-side coding module, the preset text detection model not only contains the gradient path idea and integrates the features of different levels, but also effectively improves the computational efficiency.

[0061] Optionally, in one embodiment, the preset text detection model also generates a detection anchor point box for text information in a live broadcast scenario based on a long strip convolution kernel.

[0062] In this optional embodiment, the long strip convolution kernel is the parameter corresponding to the convolution in the preset text detection model. Different from the common 1×1, 2×2 and 3×3 square convolution kernels, the long strip convolution kernel has a longer horizontal or vertical size, which can better adapt to the text information in the live scene. Among them, since in the live video, the text information often appears in the form of banners, bullet screens and subtitles, the shape of these text information is usually longer and narrower, so the long strip convolution kernel can be downsampled in the height direction, retaining the information in the width direction, avoiding the loss of more text information after downsampling in the width direction, or the long strip convolution kernel can also be downsampled in the width direction, while retaining the information in the height direction, so as to avoid the loss of more text information after downsampling in the height direction. Through the use of the long strip convolution kernel, the preset text detection model can more effectively capture the aspect ratio characteristics of the text information, thereby improving the accuracy of the detection. When generating the detection anchor box, the model will dynamically adjust the aspect ratio of the anchor box according to the shape and size of different text information in the live video to meet the detection requirements of various text information. In addition, the long convolution kernel can also reduce the interference of background noise and improve the model's ability to recognize text information, thereby ensuring a balance between the performance and effect of the preset text detection model with limited performance.

[0063] Optionally, in one embodiment, the preset text detection model is reduced in size through int8 quantization and pruning.

[0064] In this optional embodiment, int8 quantization is a quantization method that uniformly converts model parameters from floating point numbers or long unnumbered integers to 8-bit integers. Int8 quantization reduces the storage and computing requirements of the model by reducing the precision of the parameters. Through this quantization process, the model volume can be significantly reduced, while maintaining high precision, fast text area detection is achieved. The pruning process is a processing method that reduces the complexity and computational complexity of the model by removing unimportant parameters or connections in the model. Through pruning, the model volume can be further reduced, the reasoning speed can be improved, and the accuracy of the model is also small. The size of the above-mentioned preset text detection model can usually be reduced to less than 40M, which can effectively save storage resources and improve the utilization of computing resources.

[0065] Optionally, in one embodiment, the preset text recognition model is established based on adding SENet to the backbone network structure of mobilenet v3, and the output features of the lowest network layer of the preset text recognition model are merged and outputted with the output features of the backbone network structure of mobilenet v3.

[0066] In this optional embodiment, mobilenet v3 is a lightweight deep neural network architecture designed for mobile terminals, which significantly reduces model parameters and computation through technologies such as depthwise separable convolution and linear bottleneck. SENet (Squeeze-and-Excitation Networks) is a module that can enhance the ability to express network features. It recalibrates channel features by introducing global information, allowing the network to pay more attention to important features.

[0067] In this optional embodiment, the preset text recognition model adds a SENet module on the basis of mobilenet v3 to further enhance the model's sensitivity and recognition ability to text features. Through the introduction of the SENet module, the model can dynamically adjust the priority of each channel, so that the network pays more attention to features that are helpful for text recognition, while suppressing unimportant features, thereby improving the model's text recognition accuracy in various complex backgrounds.

[0068] In this optional embodiment, the output features of the lowest network layer of the preset text recognition model, that is, the output features of the shallow network layer of the basic knowledge of deep learning, are combined and output with the output features of the backbone network structure of Mobilenet V3. This fusion strategy can make full use of feature information at different levels, so that the model can capture the global and local features of the text at the same time. Through this fusion, the model can better understand the contextual information of the text when performing text recognition, thereby improving the accuracy and robustness of recognition.

[0069] In this optional embodiment, through the above-mentioned establishment and fusion output, a main structure of a preset text recognition model of CRNN (Convolutional Recurrent Neural Network) + CNN (Convolutional Neural Network) + CTC (Connectionist Temporal Classification) decoding is constructed, wherein CRNN is responsible for processing sequence data and extracting temporal features, CNN focuses on extracting image features, and CTC decoding is used to map these features to the final text output. This enables the model to process the temporal information and spatial information of the text at the same time, so that the deformation and distortion of the text can be better handled during the recognition process, thereby improving the accuracy of text recognition.

[0070] Optionally, in one embodiment, a sequence modeling module of a preset text recognition model is established through a two-layer Transformer Encoder structure, and the high-dimensional convolution information is reduced to the channel dimension required by the sequence modeling module through pooling and dimensionality reduction, wherein the sequence modeling module is used to identify the contextual information of the text content in the text area.

[0071] In this optional embodiment, Transformer is an advanced deep learning model based on the self-attention mechanism, and its Encoder structure captures the dependencies in the input sequence through multi-layer self-attention mechanism and position encoding. In the sequence modeling module of the preset text recognition model, the Encoder structure of Transformer may be used as part of the sequence modeling model to combine the sequence information of different positions of the text box in the input text or image, thereby maintaining or improving the accuracy of the model while reducing the number of parameters. The sequence modeling model refers to the use of a model to process and understand the text sequence in the text image, generally using a recurrent neural network or a self-attention mechanism. The purpose of sequence modeling is to capture the contextual information in the text sequence, thereby improving the accuracy of recognition.

[0072] In this optional embodiment, by using a two-layer Transformer Encoder structure, the model can model the text sequence through a self-attention mechanism. The self-attention mechanism allows the model to consider other characters in the entire text sequence while processing each character, thereby capturing long-distance dependencies. It is possible to combine sequence information at different positions of the input text. Through the use of the two-layer Transformer Encoder structure, the sequence modeling module implements a bidirectional RNN (BiLSTM, Bi-Long Short-Term Memory, bidirectional long short-term memory network) plus transformer architecture, which reduces the number of parameters while ensuring accuracy, so that the preset text recognition model can more effectively process the contextual information in the text sequence, thereby improving the understanding and recognition capabilities of the text content.

[0073] In this optional embodiment, the pooling operation can effectively reduce the spatial size of the feature map, reduce the computational complexity, and retain important feature information. The dimensionality reduction operation further compresses the feature dimension, allowing the model to process sequence data more efficiently. Through pooling and dimensionality reduction operations, high-dimensional convolutional features are converted into low-dimensional features suitable for processing by the sequence modeling model. These low-dimensional features are then input into the Encoder structure of the two-layer Transformer, allowing the model to identify contextual information of the text content in the text area. The Encoder structure of the Transformer encodes the input sequence through a self-attention mechanism to generate a feature vector that can represent the semantic information of the entire text sequence.

[0074] In this optional embodiment, since each layer of the deep learning network has a concept of channel dimension, the output of the previous layer must be consistent with the input of the next layer, and the sequence modeling module needs to receive the output after the previous CNN feature extraction, and the channel dimension needs to be consistent. Therefore, by reducing the high-dimensional convolution information to the channel dimension required by the sequence modeling module, the model can better fuse image features and text features, making text recognition more accurate. This fusion strategy not only improves the model's ability to recognize text content, but also enhances the model's robustness when processing different text styles and fonts. After experimental verification, the 80-channel sequence modeling module can achieve the accuracy of 512 channels, ensuring accuracy while effectively meeting the performance requirements of the terminal device. Similarly, optionally, the preset text recognition model can also be pruned and optimized, which can reduce the model size to less than 8M, effectively saving storage resources and improving the utilization of computing resources.

[0075] Optionally, in one embodiment, the preset text detection model transfers the text region to the preset text recognition model through dynamic batch recognition.

[0076] In this optional embodiment, after the preset text detection model detects the text area, the data of the text area needs to be transferred to the preset text recognition model for recognition. In this process, the data of the text area usually includes the data of several text position frames appearing in an image, and the traditional reasoning framework and model are all fixed batch transmission. For example, if the fixed batch is set to 32, then the data of 32 text position frames will be fixedly transmitted each time. For the model, when there are less than 32 text position frames left, it also needs to be rushed to the preset batch for processing, which is more time-consuming and reduces the efficiency of subsequent text recognition.

[0077] In this optional embodiment, dynamic batch recognition can enable dynamic batch transmission between models based on the actual number of text location boxes in the image, wherein dynamic batch recognition can be specifically achieved by dynamically adjusting the batch size to accommodate the actual number of text location boxes, optimizing batch transmission through a specific intelligent scheduling algorithm, or setting priorities for text location boxes for batch transmission and removing redundancy. This dynamic adjustment mechanism can effectively reduce unnecessary calculations and memory consumption, while avoiding the waste of resources caused by fixed batch sizes, and can more efficiently process text area data, thereby significantly improving the overall text recognition efficiency while ensuring recognition accuracy.

[0078] Optionally, in one embodiment, the preset text detection model and the preset text recognition model are encapsulated in a software development kit based on DirectML.

[0079] In this optional embodiment, since the graphics card models of different devices on the anchor end are different, and the ONNX Runtime reasoning framework reasoning method used in the traditional method has a strong dependence on the GPU and driver version of the graphics card, it cannot fully meet the needs of different hardware platforms. Therefore, the present invention uses a software development kit (SDK) based on DirectML (Direct Machine Learning) to encapsulate the preset text detection model and the preset text recognition model to achieve efficient cross-platform reasoning. Among them, DirectML allows developers to run deep learning models on DirectX 12 compatible GPUs without relying on specific drivers or hardware architectures. Through the encapsulation of DirectML, the preset text detection model and the preset text recognition model can achieve consistent performance on various hardware platforms, thereby ensuring the compatibility and stability of the model on different devices. Under the encapsulation of DirectML, the model-related operating environment, code, and data set are packaged and written into the SDK, so that the anchor or server side can complete the corresponding coding function, and the audience side can complete the corresponding coding processing and rendering function. It can make full use of the parallel computing power of the GPU to achieve efficient calculation and data processing, effectively compatible with graphics cards of different models of devices, meet the requirements of real-time reasoning, and achieve on-end coding coverage.

[0080] To perform the corresponding steps in the above method embodiment and each possible implementation method, the following provides an implementation method of a live video processing device. Figure 5 , Figure 5 This is a functional module diagram of an embodiment of a live video processing device in an embodiment of the present invention. The live video processing device includes:

[0081] The acquisition module 201 is used to acquire the configuration status of the anchor terminal and the live video stream generated by the anchor terminal, and push the live video stream to the audience terminal and the server terminal;

[0082] The coding module 202 is used to switch the anchor end or the server end to code the live video stream based on the configuration state through a preset end-cloud switching strategy to obtain a coding signal;

[0083] The communication module 203 is used to send the coding signaling to the viewer end, so that the viewer end can code, process and render the live video stream based on the coding signaling to obtain the final live video.

[0084] Optionally, in one embodiment, the live video processing device is applied to Figure 2 In the described video live broadcast system, the video live broadcast system includes an anchor end, a service end and an audience end which are communicatively connected to each other.

[0085] Optionally, in one embodiment, the configuration status includes a policy enablement status and a processing resource status, and the coding module 202 is specifically used to: in response to determining that the policy enablement status is an end start status, and the processing resource status meets preset resource conditions, allow the coding module on the anchor end to code the live video stream to obtain coding signaling; in response to determining that the policy enablement status is an end stop status, and / or the processing resource status does not meet preset resource conditions, start the coding module on the server end to code the live video stream to obtain coding signaling.

[0086] Optionally, in one embodiment, the coding module 202 is specifically used to: perform format conversion on the live video stream to obtain a live video frame sequence arranged in time sequence; perform text area detection on the video frames in the live video frame sequence based on a preset text detection model to obtain the text area where the text in each video frame is located; perform text recognition on the text area based on a preset text recognition model to obtain the text content; perform violation detection on the text content, and in response to determining that there is a violation in the text content, obtain the violation text range; generate a coding signal based on the violation text range.

[0087] Optionally, in one embodiment, the preset text detection model is established based on the CSP-ELAN network structure obtained by adding an efficient layer aggregation network to the CSPDarknet53 structure.

[0088] Optionally, in one embodiment, the preset text detection model samples text features based on a network structure consisting of a 3×3 convolutional layer and a 2×2 pooling layer.

[0089] Optionally, in one embodiment, the preset text detection model also generates a detection anchor point box for text information in a live broadcast scenario based on a long strip convolution kernel.

[0090] Optionally, in one embodiment, the preset text detection model is reduced in size through int8 quantization and pruning.

[0091] Optionally, in one embodiment, the preset text recognition model is established based on adding SENet to the backbone network structure of mobilenet v3, and the output features of the lowest network layer of the preset text recognition model are merged and outputted with the output features of the backbone network structure of mobilenet v3.

[0092] Optionally, in one embodiment, a sequence modeling module of a preset text recognition model is established through a two-layer Transformer Encoder structure, and the high-dimensional convolution information is reduced to the channel dimension required by the sequence modeling module through pooling and dimensionality reduction, wherein the sequence modeling module is used to identify the contextual information of the text content in the text area.

[0093] Optionally, in one embodiment, the preset text detection model transfers the text region to the preset text recognition model through dynamic batch recognition.

[0094] Optionally, in one embodiment, the preset text detection model and the preset text recognition model are encapsulated in a software development kit based on DirectML.

[0095] Since the embodiments of the device part correspond to the embodiments of the above-mentioned method, please refer to the above-mentioned method embodiments for the introduction of the live video processing device provided in the embodiments of the present invention. The embodiments of the present invention will not be repeated here, and it has the same beneficial effects as the above-mentioned live video processing method.

[0096] The present invention also provides a computer device, which includes a memory and a processor. The memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, the processor executes the live video processing method as described above. Figure 63 is a functional module diagram of a computer device provided by an embodiment of the present invention. The computer device 300 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 310 (for example, one or more processors) and a memory 320, and one or more storage media 330 (for example, one or more mass storage devices) storing application programs 333 or data 332. Among them, the memory 320 and the storage medium 330 can be short-term storage or permanent storage. The program stored in the storage medium 330 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the computer device 300. Furthermore, the processor 310 can be configured to communicate with the storage medium 330 to execute a series of instruction operations in the storage medium 330 on the computer device 300.

[0097] The computer device 300 may also include one or more power supplies 340, one or more wired or wireless network interfaces 350, one or more input and output interfaces 360, and / or one or more operating systems 331, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. It will be appreciated by those skilled in the art that Figure 6 The illustrated computer device structure does not constitute a limitation on the computer device, and may include more or fewer components than illustrated, or combine certain components, or arrange the components differently.

[0098] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes the live video processing method as described above.

[0099] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. If the integrated module or unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention is essentially or part of the contribution to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0100] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A live video processing method, characterized in that: The live video processing method comprises: Obtain the configuration status of the anchor end and the live video stream generated by the anchor end, and push the live video stream to the audience end and the server end; Based on the configuration state, the anchor end or the server end is switched to encode the live video stream through a preset end-cloud switching strategy to obtain an encoding signal; The coding signaling is sent to the audience end, so that the audience end can code, process and render the live video stream based on the coding signaling to obtain a final live video.

2. The live video processing method according to claim 1, characterized in that: The configuration status includes a policy enabling status and a processing resource status; The method of switching the anchor end or the server end to encode the live video stream based on the configuration state through a preset end-cloud switching strategy to obtain an encoding signaling includes: In response to determining that the policy activation state is the end startup state and the processing resource state meets the preset resource condition, allowing the coding module of the anchor end to code the live video stream to obtain the coding signaling; In response to determining that the policy activation state is an end stop state, and / or the processing resource state does not meet the preset resource condition, the coding module of the server is started to code the live video stream to obtain the coding signaling.

3. The live video processing method according to claim 1, characterized in that: The coding the live video stream to obtain the coding signaling includes: Converting the live video stream to obtain a live video frame sequence arranged in time sequence; Performing text region detection on the video frames in the live video frame sequence based on a preset text detection model to obtain a text region where the text in each of the video frames is located; Performing text recognition on the text area based on a preset text recognition model to obtain text content; Performing violation detection on the text content, and in response to determining that there is a violation in the text content, obtaining a violation text range; The coding signaling is generated based on the illegal text range.

4. The live video processing method according to claim 3, characterized in that: The preset text detection model is established based on the CSP-ELAN network structure obtained by adding an efficient layer aggregation network to the CSPDarknet53 structure; The preset text detection model samples text features based on a network structure consisting of a 3×3 convolutional layer and a 2×2 pooling layer; The preset text detection model also generates a detection anchor point frame for text information in a live broadcast scenario based on a long strip convolution kernel; The preset text detection model is reduced in size through int8 quantization and pruning.

5. The live video processing method according to claim 3, characterized in that: The preset text recognition model is established based on adding SENet to the backbone network structure of mobilenet v3, and the output features of the lowest network layer of the preset text recognition model are combined and output with the output features of the backbone network structure of mobilenet v3; The sequence modeling module of the preset text recognition model is established by a two-layer Transformer Encoder structure, and high-dimensional convolution information is reduced to the channel dimension required by the sequence modeling module through pooling and dimensionality reduction, wherein the sequence modeling module is used to identify the context information of the text content in the text area; The preset text recognition model is also reduced in size through pruning.

6. The live video processing method according to claim 3, characterized in that: The preset text detection model transmits the text area to the preset text recognition model through dynamic batch recognition.

7. The live video processing method according to any one of claims 3 to 6, characterized in that: The preset text detection model and the preset text recognition model are encapsulated in a software development kit based on DirectML.

8. A live video processing device, characterized in that: The live video processing device comprises: An acquisition module is used to acquire the configuration status of the anchor terminal and the live video stream generated by the anchor terminal, and push the live video stream to the audience terminal and the server terminal; A coding module, used for switching the anchor end or the server end to code the live video stream based on the configuration state through a preset end-cloud switching strategy to obtain a coding signal; The communication module is used to send the coding signaling to the audience end, so that the audience end can code, process and render the live video stream based on the coding signaling to obtain the final live video.

9. A computer device, characterized in that: The computer device comprises: a memory and at least one processor, wherein instructions are stored in the memory; The at least one processor calls the instructions in the memory so that the computer device executes the live video processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instructions are executed by the processor, the live video processing method as described in any one of claims 1-7 is implemented.

Images