Live video processing method and system

Abstract

The embodiment of the present application provides a live video processing method and system, which relates to the technical field of live video image processing. The server sends a live video frame sequence to a live broadcast end for screen rendering, and performs illegal character recognition on each video frame in the live video frame sequence. When illegal characters are recognized, a coding instruction is generated, and the coding instruction is sent to the live broadcast end; the live broadcast end performs coding processing on the area where illegal characters exist in the video frame during the screen rendering process based on the coding instruction. The above scheme can perform screen rendering and coding instruction generation respectively, and perform coding processing on the area where illegal characters exist based on the generated coding instruction, so that real-time coding of illegal content in the live video can be achieved without human participation, which can achieve the purpose of reducing the exposure of illegal information, protecting the security of platform content, and avoiding passive violation penalties for anchors.

Description

Technical Field

[0001] The present application relates to the technical field of live video image processing, and in particular to a live video processing method and system. Background Art

[0002] With the increasing development of Internet technology, live streaming has become an increasingly common form of Internet, and with it comes an exponential growth in image and video content. The growth of massive data has also brought a series of challenges and problems. Various illegal information is inevitably mixed in with numerous scenarios and data, among which character violations are the most common type. In order to ensure the security of live streaming ecological content, how to quickly and effectively detect character violation content from massive data is a problem that all content platform manufacturers must solve. And for live streaming platforms, in addition to the compliance requirements of the picture content, they also need to take into account a good user experience. Therefore, how to effectively prevent violations without affecting users is a new challenge. Summary of the invention

[0003] In order to solve the above technical problems, the present application provides a live video processing method and system.

[0004] In a first aspect, an embodiment of the present application provides a video processing method, which is applied to a live video processing system, wherein the live video processing system includes a server and a live broadcast terminal connected in communication, and the method includes:

[0005] The server sends the live video frame sequence to the live broadcast terminal for picture rendering;

[0006] The server performs illegal character recognition on each video frame in the live video frame sequence, generates a coding instruction when illegal characters are recognized, and sends the coding instruction to the live broadcast terminal;

[0007] The live broadcast end codes the area containing illegal characters in the video frame during the picture rendering process based on the coding instruction.

[0008] In a possible implementation, the server identifies illegal characters in each video frame in the live video frame sequence, generates a coding instruction when illegal characters are identified, and sends the coding instruction to the live broadcast end, including:

[0009] The server performs text region detection on video frames in a live video frame sequence based on a text region detection model to obtain a text region where text is located in each video frame;

[0010] Performing character recognition on the text area in each video frame based on a character recognition model to obtain text content;

[0011] Matching characters in the text content with illegal characters in an illegal word library to obtain a target position area of ​​the illegal characters in the text area;

[0012] Generate a coding instruction based on the target location area, and send the coding instruction to the live broadcast end.

[0013] In a possible implementation, the text region detection model includes a feature extraction network, a feature fusion network and an output network. The server performs text region detection on video frames in a live video frame sequence based on the text region detection model to obtain the text region where the text in each video frame is located, including:

[0014] Inputting a video frame in a live video frame sequence into the feature extraction network, performing feature extraction through a plurality of gradient flow branches in the feature extraction network, and obtaining gradient flow feature information of the video frame;

[0015] Inputting the gradient flow feature information of the video frame into a feature fusion network of a multi-scale feature pyramid structure for fusion, thereby obtaining fused feature information having both semantic information and positioning information;

[0016] The fused feature information is input into the output network, the text region where the text in the video frame is located is predicted, and the text region where the text in each video frame is located is output.

[0017] In a possible implementation, the character recognition model includes a convolutional neural network, a recursive neural network and a decoding network, and the step of performing character recognition on the text area in each video frame based on the character recognition model to obtain text content includes:

[0018] Inputting the text region in each video frame into the convolutional neural network for feature extraction to obtain a feature sequence of the text region;

[0019] Inputting the feature sequence of the text region into the recursive neural network, learning the feature vectors in the feature sequence, and outputting the probability distribution of the predicted label characters;

[0020] The probability distribution of the label characters is input into the decoding network, and the label character sequence is obtained from the circulation layer of the decoding network, and the text content is formed by the label character sequence.

[0021] In a possible implementation, the step of inputting the probability distribution of the label characters into the decoding network, obtaining the label character sequence from the recurrent layer of the decoding network, and forming the text content from the label character sequence includes:

[0022] Calculating the probability distribution of each character in the text area;

[0023] Calculating a cross entropy loss value of a character based on a probability distribution of each character in the text region and a probability distribution of the label character;

[0024] A label character sequence is obtained according to the cross entropy loss value of the character, and the text content is formed by the label character sequence.

[0025] In a possible implementation, the step of matching characters in the text content with illegal characters in an illegal word library to obtain a target position area of ​​the illegal characters in the text area includes:

[0026] Expanding the meaning of the illegal characters in the illegal word library to obtain the illegal characters after the meaning expansion;

[0027] The characters in the text content and the illegal characters after the word meaning expansion are subjected to fuzzy text discrimination, and a confidence score between the characters in the text content and the illegal characters after the word meaning expansion is calculated. When the confidence score is greater than a preset score, it is determined that the characters in the text content match the illegal characters after the word meaning expansion, and the characters matching the illegal characters after the word meaning expansion are marked in a position area of ​​the text area to obtain the target position area.

[0028] In a possible implementation, the step of generating a coding instruction based on the target location area and sending the coding instruction to the live broadcast end includes:

[0029] Generate the coding instruction based on the position size of the target position area in the video frame and the timestamp of the video frame where the target position area is located;

[0030] The generated coding instruction is sent to the live broadcast end.

[0031] In a possible implementation, the step of the server sending the live video frame sequence to the live broadcast end for picture rendering includes:

[0032] The server sends the live video frame sequence to a cloud server, and the cloud server sends the live video frame sequence to the live end for picture rendering.

[0033] In a possible implementation manner, the step of coding the area containing illegal characters in the video frame during the picture rendering process based on the coding instruction by the live broadcast end includes:

[0034] The live broadcast end repairs from the boundary of the area where the illegal character is located toward the area where the illegal character is located, and the live broadcast end repairs the pixel to be repaired using pixel information of known pixels in the adjacent area of ​​the pixel to be repaired.

[0035] In a second aspect, an embodiment of the present application further provides a live video processing system, the live video processing system comprising a server and a live broadcast terminal connected in communication;

[0036] The server is used to send the live video frame sequence to the live broadcast end for picture rendering;

[0037] The server is used to identify illegal characters in each video frame in the live video frame sequence, generate a coding instruction when illegal characters are identified, and send the coding instruction to the live broadcast end;

[0038] The live broadcast end is used to code the area where illegal characters exist in the video frame during the picture rendering process based on the coding instruction.

[0039] Based on any one of the above aspects, the live video processing method and system provided by the embodiment of the present application, the server sends the live video frame sequence to the live broadcast end for screen rendering, and identifies illegal characters for each video frame in the live video frame sequence, generates a coding instruction when illegal characters are identified, and sends the coding instruction to the live broadcast end; the live broadcast end codes the area where illegal characters exist in the video frame during the screen rendering process based on the coding instruction. The above scheme can perform screen rendering and coding instruction generation respectively, and code the area where illegal characters exist based on the generated coding instruction, so that real-time coding of illegal content in the live video can be achieved without human participation, which can achieve the purpose of reducing the exposure of illegal information, protecting the security of platform content, and avoiding passive violation penalties for anchors. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0041] Figure 1 A schematic diagram of the structure of the live video processing system provided in this embodiment;

[0042] Figure 2 A flowchart of the live video processing method provided in this embodiment;

[0043] Figure 3for Figure 2 Schematic diagram of the sub-step flow chart of step S12;

[0044] Figure 4 for Figure 3 Schematic diagram of the sub-step flow chart of step S121;

[0045] Figure 5 for Figure 3 Schematic diagram of the sub-step flow chart of step S122. DETAILED DESCRIPTION

[0046] To make the purpose, technical scheme and advantages of the embodiment of the present application clearer, the technical scheme in the embodiment of the present application will be clearly and completely described below in conjunction with the drawings in the embodiment of the present application. It should be understood that the drawings in the present application only serve the purpose of explanation and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in this application shows the operations implemented according to some embodiments of the embodiment of the present application. It should be understood that the operations of the flowchart can be implemented out of sequence, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart under the guidance of the content of the present application, or remove one or more operations from the flowchart.

[0047] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0048] At present, the industry mainly conducts compliance review of character content in the screen, and the most direct means is manual review. However, due to the large amount of Internet content, it is unrealistic to use manual review. With the development of artificial intelligence technology, the use of artificial intelligence recognition technology to review massive image and video content combined with manual review can effectively alleviate manpower requirements. For example, optical character recognition (OCR) technology is combined with manual review to detect illegal content. This method is also the method currently adopted by major platforms, but this method has a major flaw in that it is a post-review method. This review method has disadvantages in both the timely interception of illegal content and the passive protection of anchors from violations.

[0049] In order to solve the technical problems mentioned in the above background technology, the embodiment of the present application provides a live video processing method. Before introducing the specific scheme of the live video processing method, the application scenario corresponding to the scheme is first introduced. Figure 1 , Figure 1 A possible structural diagram of a live video processing system provided by the present application is shown. The live video processing system 10 may include a server 100 and a live broadcast terminal 200 which are in communication connection, wherein the live broadcast terminal 200 may include an anchor terminal 210 and an audience terminal 220 .

[0050] In the embodiment of the present application, the host terminal 210 and the audience terminal 220 may be, but are not limited to, a smart phone, a personal digital assistant, a tablet computer, a personal computer, a laptop computer, a virtual reality terminal device, an augmented reality terminal device, etc. In the specific implementation process, there may be multiple host terminals 210 and audience terminals 220 connected to the server 100. Figure 1 Only one host terminal 210 and two audience terminals 220 are shown. Among them, the host terminal 210 and the audience terminal 220 can be installed with a live broadcast service program, for example, the service program can be an application APP or a small program related to Internet live broadcast used in a computer or a smart phone.

[0051] In the embodiment of the present application, the server 100 may be a single physical server or a server group consisting of multiple physical servers for performing different data processing functions. The server group may be centralized or distributed (for example, the server 100 may be a distributed system). In some possible implementations, if the server 100 uses a single physical server, different logical server components may be allocated to the physical server based on different business functions.

[0052] Understandably, Figure 1 The structure of the live video processing system shown is only a feasible example. In other feasible embodiments, the live video processing system may also include only Figure 1 Some of the components shown may also include other components.

[0053] Combine the following Figure 1 The application scenario shown is an exemplary description of the live video processing method provided in the embodiment of the present application. Figure 2The live video processing method provided in the embodiment of the present application can be executed by the aforementioned live video processing system 10. The order of some steps in the live video processing method in the embodiment of the present application can be interchanged according to actual needs, or some steps therein can be omitted or deleted. The detailed steps of the live video processing method executed by the live video processing system 10 are introduced as follows.

[0054] In step S11, the server 100 sends the live video frame sequence to the live broadcast terminal 200 for image rendering.

[0055] In this embodiment, the server 100 can first send the live video frame sequence to a cloud server (e.g., a CDN server), and then the cloud server sends the live video frame sequence to the live broadcast terminal 200, which then renders the live video frame sequence.

[0056] In step S12, the server 100 identifies illegal characters in each video frame in the live video frame sequence, generates a coding instruction when illegal characters are identified, and sends the coding instruction to the live broadcast terminal 200.

[0057] In this embodiment, step S11 and step S12 can be performed simultaneously. For example, for a target video frame, screen rendering and illegal character recognition can be performed simultaneously. Of course, step S11 can also be performed after step S12. For example, when the illegal character recognition of the target video frame is about to be completed, the target video frame is sent to the live broadcast terminal 200 for screen rendering.

[0058] In step S13, the live broadcast end 200 codes the area containing illegal characters in the video frame during the picture rendering process based on the coding instruction.

[0059] The solution provided above can perform screen rendering and coding instruction generation respectively, and code the area where the illegal characters are located based on the generated coding instructions. In this way, real-time coding of illegal content in live videos can be achieved without human intervention, which can achieve the purpose of reducing the exposure of illegal information, protecting the security of platform content, and avoiding passive penalties for violations by anchors.

[0060] For further information, please refer to Figure 3 In this embodiment, step S12 can be implemented in the following manner.

[0061] Step S121, performing text region detection on video frames in a live video frame sequence based on a text region detection model to obtain a text region where text is located in each video frame.

[0062] Step S122, performing character recognition on the text area in each video frame based on the character recognition model to obtain text content.

[0063] Step S123, matching the characters in the text content with the illegal characters in the illegal word library to obtain the target position area of ​​the illegal characters in the text area.

[0064] Step S124, generating a coding instruction based on the target location area, and sending the coding instruction to the live broadcast end 200.

[0065] The live broadcast ecosystem determines that the live broadcast scenes are rich and diverse, especially for game live broadcast scenes. Characters may appear anywhere on the screen (game nicknames, comments, pop-up windows, etc.), and the characters on the screen are often small and easy to be close to the background color, blurred, fast-moving, blocked, and different video frames have different character clarity. These will directly affect the recognition of illegal characters and the coding accuracy of video sequence frames.

[0066] In order to solve the above problems, it is necessary to optimize the text area detection model in step S121 and the character recognition model in step S122.

[0067] In this embodiment, the text region detection model includes a feature extraction network, a feature fusion network and an output network. After the text region detection model is optimized, please refer to Figure 4 , step S121 can be implemented in the following way.

[0068] Step S1211, inputting the video frames in the live video frame sequence into the feature extraction network, performing feature extraction through multiple gradient flow branches in the feature extraction network, and obtaining gradient flow feature information of the video frames.

[0069] In this embodiment, the video frames in the live video frame sequence can be normalized in size and then input into the feature extraction network. The feature extraction network can be composed of multiple gradient flow branches, wherein a gradient flow branch includes multiple convolution modules and multiple residual modules. Exemplarily, in each gradient flow branch, a video frame with an input size of 640*640 can be processed by the Slice layer (feature extraction layer) and the Concat layer (feature processing layer) to output a feature map of size 320*320*12, and then after passing through a Conv layer (convolution layer) with a channel number of 32, a feature map of size 320*320*32 is output. At the same time, the network structure can be optimized according to the real-time coding requirements, and the number of channels in each stage can be adjusted.

[0070] In this step, richer feature information can be extracted through multiple gradient flow branches to avoid the loss of text area information in complex scenes.

[0071] Step S1212, inputting the gradient flow feature information of the video frame into a feature fusion network of a multi-scale feature pyramid structure for fusion, thereby obtaining fused feature information having both semantic information and positioning information.

[0072] In this step, a multi-scale feature pyramid structure is used for feature fusion, which can improve the diversity and robustness of features. Exemplarily, a multi-scale feature pyramid structure can be formed by a joint module of FPN and PAN. Among them, FPN is a feature pyramid from top to bottom, which passes down the strong semantic information of the high level and enhances the entire pyramid; PAN supplements FPN and passes up the positioning features of the low level. The pyramid formed in this way combines semantic information and has positioning information, where the semantic information includes information such as image texture, color, or target category.

[0073] In this embodiment, in order to adapt to text area detection, the convolution layer before upsampling in the feature pyramid can be removed to ensure that the features output in the feature extraction network can be directly upsampled.

[0074] Step S1213, inputting the fused feature information into the output network, predicting the text area where the text in the video frame is located, and outputting the text area where the text in each video frame is located.

[0075] In this embodiment, in addition to removing the convolutional layer as described above, a loss function that improves the accuracy of text region detection may be used to calculate the loss value. For example, the GIOU loss function may be used to replace the commonly used Smooth L1 loss function.

[0076] In addition, in this embodiment, according to the characteristics of the text in the anchor coding scene, an anchor box with a specific length and width of the composite text can be set. Specifically, a long convolution kernel can be used for optimization based on the data characteristics of the text aspect ratio.

[0077] The above text area detection model can accurately detect various characters in video frames and effectively meet the processing performance requirements of massive data. Compared with the mainstream target detection structure in the industry, it can effectively adapt to the character content detection of the live broadcast ecosystem and effectively support the real-time coding business.

[0078] In this embodiment, the character recognition model includes a convolutional neural network, a recursive neural network and a decoding network. Figure 5 , step S122 can be implemented in the following way.

[0079] Step S1221: input the text region in each video frame into a convolutional neural network for feature extraction to obtain a feature sequence of the text region.

[0080] In this embodiment, before the text area (the image of the text area) is input into the convolutional neural network, the size of the text area can be normalized. For example, the size of the text area can be normalized to (32, 100, 3), and the feature sequence of the text area can be obtained by performing feature extraction on the normalized text area.

[0081] Step S1222, input the feature sequence of the text area into the recursive neural network, learn the feature vectors in the feature sequence, and output the probability distribution of the predicted label characters.

[0082] In this step, a bidirectional recurrent neural network can be used to predict the feature sequence, learn each feature vector in the feature sequence, and output the probability distribution of the predicted label character.

[0083] Step S1223, input the probability distribution of the label characters into the decoding network, obtain the label character sequence from the recurrent layer of the decoding network, and the text content formed by the label character sequence.

[0084] The CTC loss function can be used in the decoding network to calculate the loss value. Compared with the softmax cross entropy loss function, it can effectively solve the problem of character recognition of indefinite length and the problem of character alignment during training, and convert the series of label character distributions obtained from the loop layer into the final label character sequence.

[0085] In order to effectively solve the complex text in the character recognition process, a large amount of text recognition data (for example, 50 million) can be generated through the character generation solution combined with the live broadcast ecological corpus to solve the data problem of model training. At the same time, in order to effectively increase the feature extraction capability of the model, the attention mechanism is introduced to assist in solving the cumulative errors that may be caused by the decoding network during the decoding process, which is used for feature alignment. The attention module is introduced between the bidirectional recursive neural networks. Due to the lack of intermediate vector propagation between the recursive neural networks, the distance between the outputs of the timestamps of each layer of the recursive neural network is calculated to obtain a 26x26 distance matrix. For the input of the second layer of the recursive neural network, the corresponding attention weights can be calculated to construct an attention model, which effectively improves the model's feature extraction capability.

[0086] In this embodiment, in order to further optimize the character recognition results, in step S1223, a weakly supervised decoding method is introduced. This method has no formal alignment between input and output, no sequence information, focuses on representation learning, and the main idea is to supervise the number of characters in the time dimension. Exemplarily, the probability distribution of each character in the text area is calculated, and the cross entropy loss value of the character is calculated based on the probability distribution of each character in the text area and the probability distribution of the label character; a label character sequence is obtained according to the cross entropy loss value of the character, and the text content is formed by the label character sequence.

[0087] In this embodiment, step S123 can be implemented in the following manner.

[0088] First, the meaning of the illegal characters in the illegal vocabulary is expanded to obtain the illegal characters after the meaning expansion.

[0089] In this embodiment, the illegal characters that need to be coded can be used to construct an illegal word library, and the scale of illegal characters can be increased by expanding the meaning of words to avoid missing illegal characters from being coded.

[0090] Next, fuzzy text discrimination is performed on the characters in the text content and the illegal characters after the word meaning expansion, and a confidence score between the characters in the text content and the illegal characters after the word meaning expansion is calculated. When the confidence score is greater than a preset score, it is determined that the characters in the text content match the illegal characters after the word meaning expansion, and the characters matching the illegal characters after the word meaning expansion are marked in the position area of ​​the text area to obtain the target position area.

[0091] In this embodiment, the accuracy and recall rate of illegal character identification can be improved by introducing fuzzy text identification, and regularization processing is used to match similar words and phonetic words to further improve the masking effect of illegal characters.

[0092] In this embodiment, step S124 can be implemented in the following manner.

[0093] The coding instruction is generated based on the position size of the target position area in the video frame and the time stamp of the video frame where the target position area is located.

[0094] The generated coding instruction is sent to the live broadcast end.

[0095] Furthermore, in step S13, the live broadcast end 200 repairs from the boundary of the area where the illegal character is located toward the area where the illegal character is located, and the live broadcast end repairs the pixel to be repaired using pixel information of known pixels in the adjacent area of ​​the pixel to be repaired. Compared with traditional coding methods such as mosaic, frosted glass and atomization, the above-mentioned repair method can make the difference between the coded area and its surrounding non-coded area smaller, which is easier for users to accept and has less impact on the viewing experience.

[0096] In the above steps, the exposure of illegal contents can be avoided by adopting the method of early and delayed coding. For example, when the time period in which illegal contents appear is from t1 to t2, the area that needs to be coded at time t1 can be coded before time t1, and the area that needs to be coded at time t2 can be coded after time t2.

[0097] This embodiment also provides a live video processing system. Please refer to Figure 1 The live video processing system 10 includes a server 100 and a live broadcast terminal 200 which are communicatively connected.

[0098] The server 100 is used to send the live video frame sequence to the live terminal 200 for picture rendering.

[0099] In this embodiment, the server 100 can first send the live video frame sequence to a cloud server (e.g., a CDN server), and then the cloud server sends the live video frame sequence to the live broadcast terminal 200, which then renders the live video frame sequence.

[0100] The server 100 is used to identify illegal characters in each video frame in the live video frame sequence, generate a coding instruction when illegal characters are identified, and send the coding instruction to the live broadcast terminal 200.

[0101] The live broadcast terminal 200 is used to code the area with illegal characters in the video frame during the picture rendering process based on the coding instruction.

[0102] In summary, the live video processing method and system provided by the embodiment of the present application is that the server sends the live video frame sequence to the live broadcast end for screen rendering, and identifies illegal characters for each video frame in the live video frame sequence, generates a coding instruction when illegal characters are identified, and sends the coding instruction to the live broadcast end; the live broadcast end performs coding processing on the area where illegal characters exist in the video frame during the screen rendering process based on the coding instruction. The above scheme can perform screen rendering and coding instruction generation respectively, and perform coding processing on the area where illegal characters exist based on the generated coding instruction, so that real-time coding of illegal content in the live video can be achieved without human participation, and the purpose of reducing the exposure of illegal information, protecting the security of platform content, and avoiding passive violation penalties for anchors can be achieved.

[0103] The embodiments described above are only some embodiments of the present application, rather than all embodiments. The components of the embodiments of the present application generally described and shown in the drawings can be arranged and designed in various configurations. Therefore, the detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of protection of the present application, but merely represents the selected embodiments of the present application. Based on this, the scope of protection of the present application shall be based on the scope of protection of the claims. In addition, based on the embodiments of the present application, all other embodiments that can be obtained by those skilled in the art without making creative work should belong to the scope of protection of the present application.

Claims

1. A live video processing method, characterized in that: Applied to a live video processing system, the live video processing system includes a server and a live broadcast terminal connected in communication, and the method includes: The server sends the live video frame sequence to the live broadcast terminal for picture rendering; The server performs text region detection on video frames in a live video frame sequence based on a text region detection model to obtain a text region where text is located in each video frame; Performing character recognition on the text area in each video frame based on a character recognition model to obtain text content; Matching characters in the text content with illegal characters in an illegal word library to obtain a target position area of ​​the illegal characters in the text area; Generate a coding instruction based on the position size of the target position area in the video frame and the timestamp of the video frame where the target position area is located; Sending the generated coding instruction to the live broadcast end; The live broadcast end codes the area containing illegal characters in the video frame during the picture rendering process based on the coding instruction.

2. The live video processing method according to claim 1, characterized in that: The text region detection model includes a feature extraction network, a feature fusion network and an output network. The server performs text region detection on video frames in a live video frame sequence based on the text region detection model to obtain the text region where the text in each video frame is located, including: Inputting a video frame in a live video frame sequence into the feature extraction network, performing feature extraction through a plurality of gradient flow branches in the feature extraction network, and obtaining gradient flow feature information of the video frame; Inputting the gradient flow feature information of the video frame into a feature fusion network of a multi-scale feature pyramid structure for fusion, thereby obtaining fused feature information having both semantic information and positioning information; The fused feature information is input into the output network, the text region where the text in the video frame is located is predicted, and the text region where the text in each video frame is located is output.

3. The live video processing method according to claim 1, characterized in that: The character recognition model includes a convolutional neural network, a recursive neural network and a decoding network. The step of performing character recognition on the text area in each video frame based on the character recognition model to obtain text content includes: Inputting the text region in each video frame into the convolutional neural network for feature extraction to obtain a feature sequence of the text region; Inputting the feature sequence of the text region into the recursive neural network, learning the feature vectors in the feature sequence, and outputting the probability distribution of the predicted label characters; The probability distribution of the label characters is input into the decoding network, and the label character sequence is obtained from the circulation layer of the decoding network, and the text content is formed by the label character sequence.

4. The live video processing method according to claim 3, characterized in that: The step of inputting the probability distribution of the label characters into the decoding network, obtaining the label character sequence from the recurrent layer of the decoding network, and forming the text content from the label character sequence comprises: Calculating the probability distribution of each character in the text area; Calculating a cross entropy loss value of a character based on a probability distribution of each character in the text region and a probability distribution of the label character; A label character sequence is obtained according to the cross entropy loss value of the character, and the text content is formed by the label character sequence.

5. The live video processing method according to claim 1, characterized in that: The step of matching the characters in the text content with the illegal characters in the illegal word library to obtain the target position area of ​​the illegal characters in the text area includes: Expanding the meaning of the illegal characters in the illegal word library to obtain the illegal characters after the meaning expansion; The characters in the text content and the illegal characters after the word meaning expansion are subjected to fuzzy text discrimination, and a confidence score between the characters in the text content and the illegal characters after the word meaning expansion is calculated. When the confidence score is greater than a preset score, it is determined that the characters in the text content match the illegal characters after the word meaning expansion, and the characters matching the illegal characters after the word meaning expansion are marked in a position area of ​​the text area to obtain the target position area.

6. The live video processing method according to any one of claims 1 to 5, characterized in that: The step of the server sending the live video frame sequence to the live broadcast end for picture rendering comprises: The server sends the live video frame sequence to a cloud server, and the cloud server sends the live video frame sequence to the live end for picture rendering.

7. The live video processing method according to any one of claims 1 to 5, characterized in that: The step of coding the area containing illegal characters in the video frame based on the coding instruction during the picture rendering process comprises: The live broadcast end repairs from the boundary of the area where the illegal character is located toward the area where the illegal character is located, and the live broadcast end repairs the pixel to be repaired using pixel information of known pixels in the adjacent area of ​​the pixel to be repaired.

8. A live video processing system, characterized in that: The live video processing system includes a server and a live broadcast terminal that are in communication connection; The server is used to send the live video frame sequence to the live broadcast end for picture rendering; The server is used to perform text region detection on video frames in a live video frame sequence based on a text region detection model to obtain a text region where text is located in each video frame; perform character recognition on the text region in each video frame based on a character recognition model to obtain text content; match characters in the text content with illegal characters in an illegal word library to obtain a target position region of the illegal characters in the text region; generate a coding instruction based on the position size of the target position region in the video frame and the timestamp of the video frame where the target position region is located; and send the generated coding instruction to the live broadcast end; The live broadcast end is used to code the area where illegal characters exist in the video frame during the picture rendering process based on the coding instruction.

Images