Automatic processing method and system for live streaming, and electronic device and storage medium

Abstract

Disclosed in the present invention are an automatic processing method and system for live streaming, and an electronic device and a storage medium. The method comprises: acquiring a presentation video stream, a visual-teaching on-site video stream and medical-consultation-station video streams; using a scene recognition model to perform scene recognition on the medical-consultation-station video streams; analyzing scene changes of the medical-consultation-station video streams in real time, and recognizing important events; on the basis of a scene recognition result and / or an important event recognition result, automatically determining medical-consultation-station video streams which require combined display; by means of a reinforcement learning method, determining a window combination and layout result of the medical-consultation-station video streams which require combined display; determining a live-streaming picture layout; and using an adaptive bitrate transmission technique to output the live-streaming picture layout to a live-streaming platform. The present invention realizes automatic combination, layout and switching of live streaming, and compared with the prior art, the present invention reduces the consumption of directing resources and time, reduces live-streaming costs, and also improves the efficiency and accuracy of medical live streaming, thereby optimizing the viewing experience of a user.

Description

Automatic processing method and system for live broadcast, electronic device and storage medium

[0001] Cross-reference statement

[0002] The present application claims priority to the Chinese patent application No. 202411230348.X, filed on September 4, 2024, and entitled "Automatic processing method and system for live broadcast, electronic device and storage medium", the whole content of which is incorporated herein by reference. TECHNICAL FIELD

[0003] The present application relates to the field of medical live broadcast, and specifically to an automatic processing method and system for live broadcast, an electronic device and a storage medium. BACKGROUND

[0004] In modern medical teaching and surgery teaching process, especially in the live broadcast teaching of digestive endoscopy surgery (such as ERCP surgery (endoscopic retrograde cholangiopancreatography)), it is usually necessary to broadcast multiple diagnosis positions in a video conference, each diagnosis position containing endoscopy, surgical field, ultrasound and other video streams. If it is a digestive endoscopy ERCP surgery, it may also contain X-ray video stream. At the same time, the teaching site has a host and a display screen for demonstration video, and the teaching site includes a teaching room and / or a consultation room, etc. The synchronous display and switching of these multiple video streams usually requires a large number of director resources and time, and a large amount of preparation and arrangement work.

[0005] In the prior art, manual director is mainly relied on to complete complex medical live broadcast tasks by manually switching different video streams and adjusting window layout. This method mainly has the following disadvantages:

[0006] Manual operation is complicated and prone to errors: the director needs to manually switch multiple video streams and adjust the window layout in real time, which is time-consuming and prone to errors;

[0007] Low efficiency: manual operation is low in efficiency and cannot quickly respond to scene changes, resulting in that important content cannot be timely displayed;

[0008] Large resource consumption: a large number of director personnel and equipment are needed to support, increasing the cost of live broadcast;

[0009] Poor viewing experience: due to the delay of manual switching, the audience cannot obtain the best viewing experience.

[0010] In view of the above-mentioned disadvantages, it is urgent to develop an automatic processing method and system for live broadcast. SUMMARY

[0011] To solve the problems of easy error operation, low efficiency, large resource consumption and switching delay in medical live broadcast, the application provides an automatic processing method and system for live broadcast, an electronic device and a storage medium.

[0012] The technical solution adopted by the application to solve the technical problems is as follows:

[0013] In a first aspect, the application provides an automatic processing method for live broadcast, comprising

[0014] Obtaining a presentation video stream, a live teaching site video stream and a medical diagnosis video stream;

[0015] Using a pre-trained scene recognition model to perform scene recognition on the medical diagnosis video stream;

[0016] Real-time analysis of the scene changes of the medical diagnosis video stream and identification of important events;

[0017] According to the results of scene recognition and / or important event identification, automatically determining the medical diagnosis video stream that needs to be combined for display;

[0018] Determining the window combination and layout result of the medical diagnosis video stream that needs to be combined for display by reinforcement learning method;

[0019] Determining the live picture layout, wherein the video stream of the live picture layout includes at least one of the presentation video stream, the live teaching site video stream and the window combination and layout result;

[0020] Using adaptive bit rate transmission technology to output the live picture layout to a live broadcast platform.

[0021] In a preferred embodiment, the method further comprises using a deep neural network-based medical live teaching multi-scene distribution strategy algorithm to optimize the window layout;

[0022] The structure of the deep neural network-based medical live teaching multi-scene distribution strategy algorithm comprises:

[0023] An input encoding module for encoding the scene type and / or important event information of the medical diagnosis video stream to obtain a dense feature vector of each medical diagnosis video stream;

[0024] A time sequence feature extraction module for capturing the dependency relationship between the time sequence information and attributes of the dense feature vector of each medical diagnosis video stream to obtain a time sequence feature representation;

[0025] A feature fusion module for calculating the correlation and importance weight between the time sequence features of each medical diagnosis video stream to obtain a weighted combined joint feature representation;

[0026] a strategy decision module configured to calculate a Q-value vector of each layout according to the joint feature;

[0027] a strategy output module configured to determine an optimal layout decision according to the Q-value vector of each layout to obtain an optimized window layout.

[0028] In a preferred embodiment, the method further comprises: pre-processing the video stream to improve the quality of the video stream and / or video frame.

[0029] In a preferred embodiment, the real-time analysis of the scene change of the medical diagnosis video stream and the identification of the important event specifically comprises: in the live broadcast process, the scene change of each medical diagnosis video stream displayed in the live broadcast is analyzed in real time, and an important event is identified, and according to the important event identification result, the window layout is adjusted.

[0030] In a preferred embodiment, the scene recognition model is a multi-scale convolutional neural network or a Transformers neural network.

[0031] In a preferred embodiment, the multi-scale convolutional neural network comprises:

[0032] an input layer configured to obtain a video frame of the medical diagnosis video stream;

[0033] a plurality of parallel convolution branches configured to operate on the video frame to generate a plurality of initial feature maps with different scale features;

[0034] a feature fusion layer configured to fuse and splice all the initial feature maps to obtain a feature vector

[0035] an output layer configured to input the feature vector into a classifier to obtain a probability distribution of each scene type.

[0036] In a preferred embodiment, the number of convolution branches is 3, one convolution branch comprises 3 layers of 3x3 first convolution layers, each first convolution layer is followed by a first ReLU activation layer and a max-pooling layer; one convolution branch comprises 2 layers of 5x5 second convolution layers, each second convolution layer is followed by a second ReLU activation and an average-pooling layer, and the other convolution branch comprises 2 layers of 7x7 third convolution layers, each third convolution layer is followed by a third ReLU activation layer and a global average-pooling layer in sequence.

[0037] In a second aspect, the present application provides an automatic processing system for live broadcast, comprising:

[0038] a video stream acquisition module configured to acquire a presentation video stream, a visual teaching site video stream, and a medical diagnosis video stream;

[0039] a scene recognition module configured to recognize the scene of the medical diagnosis video stream by using a pre-trained scene recognition model.

[0040] an analysis module configured to analyze scene changes of the medical diagnosis video stream in real time and identify important events;

[0041] a video stream determination module configured to automatically determine medical diagnosis video streams that need to be combined for display according to results of scene identification and / or important event identification;

[0042] a window combination and layout determination module configured to determine window combination and layout results of the medical diagnosis video streams that need to be combined for display by reinforcement learning;

[0043] a live picture determination module configured to determine a live picture layout, wherein video streams of the live picture layout include at least one of the presentation video stream, the view teaching site video stream, and the window combination and layout results;

[0044] a live stream output module configured to output the live picture layout to a live platform using adaptive bitrate transmission technology.

[0045] In a third aspect, the present application provides an electronic device, comprising: a memory; one or more processors; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for executing any of the automatic processing methods of live broadcast according to the first aspect.

[0046] In a fourth aspect, the present application provides a computer-readable storage medium, comprising instructions, when the instructions are run on a computer, causing the computer to execute the automatic processing method of live broadcast according to the first aspect.

[0047] The automatic processing method, system, electronic device, and storage medium of live broadcast of the present application automatically perform scene identification on video streams by a pre-trained scene identification model, analyze scene changes of the video streams in real time and identify important events, automatically determine medical diagnosis video streams that need to be combined for display according to results of scene identification and / or important event identification, determine window combination and layout results of the medical diagnosis video streams that need to be combined for display by reinforcement learning, use videoized presentation, determine a live picture layout including a presentation video stream, a view teaching site video stream, and / or window combination and layout results, and use adaptive bitrate transmission technology for transmission. The present application realizes automatic combination, layout, and switching of live broadcast, reduces consumption of director resources and time, and reduces live broadcast cost, while improving efficiency and accuracy of medical live broadcast and optimizing viewing experience of audiences. BRIEF DESCRIPTION OF DRAWINGS

[0048] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some of the embodiments of the present application, and therefore should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can also be obtained without creative labor.

[0049] Fig. 1 is a flowchart of the automatic processing method of live broadcast provided by embodiment one.

[0050] Fig. 2 is a flowchart of the automatic processing method of live broadcast provided by embodiment two.

[0051] Fig. 3 is a flowchart of the automatic processing method of live broadcast provided by embodiment three.

[0052] Fig. 4 is a framework diagram of the automatic processing system of live broadcast provided by embodiment four.

[0053] Fig. 5 is a schematic diagram of the structural electronic device provided by embodiment five. DETAILED DESCRIPTION

[0054] The present application will be further described in detail below in conjunction with the drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the present application, and not to limit the present application. In addition, it should be noted that, in order to facilitate the description, only the parts related to the present application are shown in the drawings, not all the structures.

[0055] In the medical teaching and surgery live broadcast, the manual director is mainly relied on. However, since the director needs to manually switch multiple video streams and adjust the window layout in real time, the workload is large, which leads to easy errors. Moreover, the manual operation is low in efficiency and cannot quickly respond to scene changes, which will lead to important content not being displayed in time, thereby affecting the live broadcast, and the manual switching has a delay, so that the audience (hereinafter also referred to as students) cannot obtain the best viewing experience. In addition, the manual director needs a large number of personnel and equipment support, which increases the cost of live broadcast.

[0056] Therefore, the present application provides an automatic processing method and system of live broadcast, an electronic device and a storage medium, which can automatically identify the scene of each medical diagnosis video stream, automatically determine the medical diagnosis video stream that needs to be combined and displayed and automatically layout, can realize important event recognition and automatically update the layout accordingly, take the presentation as a video stream, take the live broadcast scene as a video stream, determine the live broadcast picture layout including the presentation video stream, the live broadcast scene video stream and / or the window combination and layout result, and automatically switch and direct accordingly, so that the present application reduces the consumption of director resources and time, reduces the cost of live broadcast, improves the efficiency and accuracy of medical live broadcast, and optimizes the viewing experience of the audience.

[0057] The various embodiments of the present application can be executed independently, and can also be executed in combination with each other, and the present application does not specifically limit the embodiments. In the embodiments of the present application, "first", "second", and the like are used to describe various components, but these components should not be limited by these terms. These terms are only used to distinguish one component from another component. In the present application, "and / or" means any and all combinations of one or more relevant listed items.

[0058] Next, the automatic processing method and system of live broadcast and its technical effects are described.

[0059] Embodiment one

[0060] FIG. 1 is a flowchart of an automatic processing method of live broadcast according to an embodiment of the present application. As shown in FIG. 1, the method provided by the embodiment is suitable for the field of medical teaching sharing, and the method comprises the following steps.

[0061] Obtaining a presentation video stream;

[0062] Obtaining a teaching site video stream;

[0063] Obtaining a medical diagnosis video stream;

[0064] Using a pre-trained scene recognition model to perform scene recognition on the medical diagnosis video stream;

[0065] Real-time analyzing the scene changes of the medical diagnosis video stream and recognizing important events;

[0066] According to the results of scene recognition and / or important event recognition, automatically determining the medical diagnosis video stream that needs to be combined and displayed;

[0067] Determining the window combination and layout result of the medical diagnosis video stream that needs to be combined and displayed by reinforcement learning method;

[0068] Determining a live broadcast picture layout, and the video stream of the live broadcast picture layout comprises at least one of the following three kinds:

[0069] I. The presentation video stream;

[0070] II. The teaching site video stream;

[0071] III. The window combination and layout result;

[0072] Using adaptive bit rate transmission technology to output the live broadcast picture layout to a live broadcast platform.

[0073] It should be understood that when judging whether there is an empty box in the above method, it can be different according to different execution processes, so the above method does not represent or imply that all steps must be executed in this order, and those skilled in the art can change or change the execution order of the above steps on the basis of the present application. The following illustrates some embodiments in the above method.

[0074] In addition, it can be understood that the window combination and layout result includes the layout mode of the window layout and the video stream corresponding to the layout mode, and the live picture layout includes the layout mode of the live picture layout and the video stream corresponding to the layout mode.

[0075] Generally, the output of the live picture layout to the live platform is in a real-time updated state, that is, the latest live picture layout is output to the live platform.

[0076] Embodiment two

[0077] Referring to FIG. 2, it is a flowchart of the automatic processing method of the live broadcast provided in this embodiment. The automatic processing method of the live broadcast provided in this embodiment specifically includes the following steps:

[0078] S1, the video input module obtains medical diagnosis video streams of multiple medical diagnoses.

[0079] S2, the medical diagnosis video streams are preprocessed to improve the quality of the video streams; the medical diagnosis video streams in the following steps can all be preprocessed medical diagnosis video streams.

[0080] S3, a pre-trained scene recognition model is used to perform scene recognition on the medical diagnosis video streams. In this embodiment, the scene recognition model can be a multi-scale convolutional neural network or a Transformers neural network.

[0081] S4, automatically determine the medical diagnosis video streams that need to be combined and displayed.

[0082] S5, determine the window combination and layout result of the medical diagnosis video streams that need to be combined and displayed by reinforcement learning method; obviously, the medical diagnosis video streams that need to be combined and displayed are multiple, which can come from the same medical diagnosis or from different medical diagnoses.

[0083] S6, the video input module obtains a presentation video stream and a view teaching site video stream. In other embodiments, the view teaching site video stream can be obtained in S5.

[0084] S7, obtain the instruction of the host, which can be at least one of the voice instruction, gesture instruction of the host, and instruction of the device used by the host;

[0085] According to the instruction of the host, the live picture layout is automatically determined; for example, the instruction of the host corresponds to the picture area occupied by the demonstration video stream in the live picture layout, which can be specifically whether to live the demonstration video stream, whether to display the demonstration video stream in the smallest layout area in the live picture layout, and whether to display the demonstration video stream in the largest layout area in the live picture layout.

[0086] It can be understood that the instruction of the host can not be obtained when S7 is executed, and the live picture layout is continued to be displayed according to the previous live picture layout, or the live picture layout is fine-tuned when the window combination and layout result is obtained.

[0087] The S7 further includes a step of obtaining information of a live watching terminal as an analysis basis for automatically determining the live picture layout, for example, the information of the live watching terminal can be a device type and / or a device location, and specifically, if the learner is remotely watching the learning at a place other than the live teaching site by using the terminal (such as a mobile phone, a notebook computer, etc.), the live picture layout can include the live teaching site video stream.

[0088] Generally, the live picture layout includes the demonstration video stream and / or the window combination and layout result, and the live teaching site video stream is generally displayed alone in the case of live teaching opening, live teaching break, the host asking the audience, etc.

[0089] S8, using adaptive bit rate transmission technology, outputting the latest live picture to the live platform.

[0090] S9, automatically analyzing scene changes of each video stream in real time and identifying important events; according to the important event identification result, automatically switching the video stream to realize switching of the live platform video stream; the important event identification can be identification of whether it is an important event, and further, can include classification of the important event, and here, when the important event is identified, the process returns to S4.

[0091] In an embodiment, after the important event identification result is obtained, the medical diagnosis video stream that needs to be combined and displayed is automatically determined, and preferably, according to the important event identification result and the scene identification result, the medical diagnosis video stream that needs to be combined and displayed is automatically re-determined.

[0092] In this embodiment, as an example, the classification of the scene identification includes an endoscopic picture video, an ultrasonic picture video, an operation field picture video, and an X-ray picture video.

[0093] Here, all the medical diagnoses have their marks, and correspondingly, the medical diagnosis video stream of the medical diagnosis also has corresponding marks to determine which medical diagnosis the medical diagnosis video stream belongs to.

[0094] The method of automatically determining the medical diagnosis video stream that needs to be combined for display is not limited, and as an example but not limitation, it can be that one or more medical diagnoses are performing surgery, and the screen is moving, and all medical diagnosis video streams of the medical diagnoses that are performing surgery (or part of the medical diagnosis video streams) are determined as the medical diagnosis video stream that needs to be combined for display.

[0095] The method and logic of automatically determining the live screen layout are not limited, and for example, reinforcement learning can also be used.

[0096] In this embodiment, reinforcement learning is used to determine the combination and layout of the window and / or the live screen layout, and the live teaching site video stream is used. The live teaching site video stream can include the host video, and can also include the video with the audience. The presentation is used for string, courseware sharing, etc.

[0097] The logic of automatically determining the live screen layout is not limited, and for example, the logic includes: the priority of the presentation video stream is the highest, and when the presentation needs to be explained, the live screen layout should be changed to the presentation video.

[0098] Embodiment Three

[0099] Please refer to FIG. 3, which is a flowchart of the automatic processing method of the live broadcast provided in this embodiment, and this embodiment specifically illustrates the implementation of each step.

[0100] Step 1: Obtain the video stream, obtain multiple medical diagnosis video streams from multiple medical diagnoses through the first input unit of the video input module, and usually each medical diagnosis corresponds to multiple medical diagnosis video streams. Obtain the presentation video stream through the second input unit of the video input module. Obtain the live teaching site video stream through the third input unit of the video input module. The video input module includes a multi-channel video capture card and a high-bandwidth data interface to ensure high-quality and low-latency transmission of the video stream.

[0101] Step 2: Video stream preprocessing to improve the quality of the video stream and / or video frame: preprocess the obtained medical diagnosis video stream, including denoising, image enhancement, and format conversion. Here, denoising, image enhancement, and format conversion are performed in sequence. The denoising operation uses the Non-Local Means (NLM) algorithm to reduce video noise and improve image quality. The image enhancement operation uses the Generative Adversarial Network (GAN) technology, such as SRGAN (Super-Resolution GAN), to improve the video resolution. Format conversion converts the video frame to a fixed size suitable for input of the multi-scale convolutional neural network below, such as 224x224 pixels, and adjusts to a uniform input format.

[0102] In this step, the presentation video stream and the live teaching site video stream can also be preprocessed, such as denoising and image enhancement.

[0103] Step 3: AI scene recognition, using a pre-trained multi-scale convolutional neural network (MSCNN) as an AI scene recognition model to perform scene recognition on the pre-processed medical diagnosis video stream, i.e., to determine the content type of the video stream. The network structure of MSCNN includes multiple parallel convolution branches, each branch processing features of different scales to generate multiple initial feature maps with different scale features by operating on video frames. The input layer receives video frame images of the pre-processed medical diagnosis video stream, with a size of 224x224x3. In this embodiment, there are three parallel convolution branches. Convolution branch one processes small-scale features, including three layers of 3x3 first convolution layers with 64, 128, and 256 filters respectively, followed by a first ReLU activation layer and a max-pooling layer after each first convolution. Convolution branch two processes medium-scale features, including two layers of 5x5 second convolution layers with 64 and 128 filters respectively, followed by a second ReLU activation and an average pooling layer after each second convolution layer. Convolution branch three processes large-scale features, including two layers of 7x7 third convolution layers with 32 and 64 filters respectively, followed by a third ReLU activation layer and a global average pooling layer in sequence after each third convolution layer. The initial feature map obtained by convolution branch one has smaller scale features than the initial feature map obtained by convolution branch two, and the initial feature map obtained by convolution branch two has smaller scale features than the initial feature map obtained by convolution branch three. The feature fusion layer concatenates the outputs of the three convolution branches to form a unified feature vector, specifically: feature compression is performed through a 1x1 fourth convolution layer, followed by a fully connected layer for concatenation. The output layer inputs the fused feature vector into a Softmax classifier for scene recognition, outputting the probability distribution of each scene type.

[0104] Step 4: window combination and layout, according to the result of scene recognition in step 3, combining and laying out several medical diagnosis video streams of one or several medical diagnoses that need to be combined and displayed. The window combination and layout algorithm uses reinforcement learning (RL) technology to ensure the best viewing experience.

[0105] Here, the window combination and layout can also be: determining the medical diagnosis video stream that needs to be combined and displayed according to the result of scene recognition in step 3, and performing scene change analysis on the medical diagnosis video stream that needs to be combined and displayed to identify important events therein.

[0106] The state of the reinforcement learning includes the current window layout state, such as video stream type, window position and size; the action includes possible layout adjustment actions, such as window moving, zooming, merging and separating; the reward function is dynamically adjusted according to audience feedback and key content display effect, ensuring that important content is displayed first. Audience feedback and key content display effect can be obtained by analyzing the video stream of the teaching site.

[0107] In an embodiment, the window combination is determined by combining several medical diagnosis video streams that need to be displayed, and then the window layout is determined by reinforcement learning;

[0108] In another embodiment, there is more than one group of several medical diagnosis video streams that need to be combined for display, and a group of several medical diagnosis video streams that need to be combined for display and their corresponding window layout are determined by reinforcement learning.

[0109] Further, the layout of the window combination and layout result determined by reinforcement learning technology is optimized, and the optimized result is used as the final window combination and layout result. Preferably, a medical teaching multi-scene distribution strategy algorithm based on deep neural network is used for window layout optimization. Other optimization algorithms can also be used for window layout adjustment, such as genetic algorithm (Genetic Algorithm, GA).

[0110] The structure of the medical teaching multi-scene distribution strategy algorithm based on deep neural network includes:

[0111] An input encoding module inputs the attributes of several medical diagnosis video streams (scene recognition results and / or important event information, which is obtained according to the important event recognition result, for example, the important event recognition result is used as the important event information); it contains an embedding layer and a multi-layer perceptron (MLP) inside, which is used to encode the scene type and / or important event information of the medical diagnosis video stream; through operation, the dense feature vector of each medical diagnosis video stream is obtained;

[0112] A time sequence feature extraction module inputs the dense feature vector of each medical diagnosis video stream; it contains a long short-term memory network (LSTM) inside, which is used to capture the dependency between time sequence information and attributes; through operation, the feature representation that captures the time sequence dependency, simply referred to as time sequence feature representation, is obtained;

[0113] A feature fusion module inputs the time sequence feature representation of all medical diagnosis video streams; it contains a multi-head self-attention mechanism (Multi-Head Self-Attention) layer inside, which is used to calculate the correlation and importance weight between the time sequence features of each medical diagnosis video stream; through operation, the weighted combined joint feature representation is obtained, highlighting the features of key medical diagnosis video streams;

[0114] a strategy decision module, which inputs the joint feature representation; internally contains fully connected layers (Fully Connected Layers) for calculating the Q value of each possible layout; through operation, obtains the Q value vector of each layout;

[0115] a strategy output module, which inputs the Q value vector of each layout; internally contains a selection function (such as argmax) for determining the best layout decision; through operation, obtains the final combination and arrangement layout of the video stream window, thus obtaining the optimized window layout, i.e. the optimized window combination and layout.

[0116] Step 5: Live picture layout: according to the preset logic, input including instructions, window combination and layout results, determine the layout method of live picture.

[0117] By real-time re-determination of live picture layout, automatic switching and director of live broadcast are realized. In the live broadcast process, real-time monitoring of live picture layout is realized, and automatic switching and director are realized by using AI system.

[0118] Real-time analysis of scene changes of medical diagnosis video streams, identification of important events of medical diagnosis, such as key steps of surgery, changes of surgical field, etc. Using event-driven switching algorithm, according to the identification result of important events, automatically switch video streams to ensure that the audience sees the most relevant and important content. Automatically adjust the window layout to adapt to the scene changes; then automatically adjust the live picture layout to adapt to the application, for example, to adapt to the changes of the host's hosting process or the changes of the learning personnel's needs, etc., to provide the best viewing experience. Combined with audience (learning personnel) feedback and interaction data, dynamically optimize the director strategy to maximize the live broadcast effect.

[0119] As an example, in this embodiment, during live broadcast, only the scene changes of each medical diagnosis video stream displayed are monitored in real time and important events are identified, and according to the identification result of important events, the window layout is adjusted. It can be understood that at this time, the window combination does not need to be adjusted.

[0120] Step 6: Output live stream, real-time output the latest live picture layout to the live broadcast platform to ensure that the audience can watch high-quality medical live broadcast content in real time. The live stream output adopts adaptive bit rate (ABR) technology to dynamically adjust the video quality according to the network bandwidth to ensure smooth playback.

[0121] In other embodiments, the method further comprises the step of automatically determining certain medical diagnosis video streams of certain medical diagnosis that need to be separately displayed according to the results of scene recognition and / or important event recognition, and outputting the video streams that need to be separately displayed to the live broadcast platform using adaptive bitrate transmission technology. Further, the live broadcast picture layout is determined by the certain video streams of certain medical diagnosis that need to be separately displayed and the presentation video stream (and the demonstration site video stream).

[0122] Embodiment four

[0123] The embodiment provides an automatic processing system 100 for live broadcast, as shown in FIG. 4, the system comprises:

[0124] A video stream acquisition module 101 is configured to acquire a presentation video stream, a demonstration site video stream and medical diagnosis video streams;

[0125] A scene recognition module 104 is configured to perform scene recognition on the medical diagnosis video streams by using a pre-trained scene recognition model;

[0126] An analysis module 103 is configured to analyze the scene changes of the medical diagnosis video streams in real time and recognize important events;

[0127] A video stream determination module 105 is configured to automatically determine medical diagnosis video streams that need to be combined and displayed according to the results of scene recognition and / or important event recognition;

[0128] A window combination and layout determination module 106 is configured to determine the window combination and layout results of the medical diagnosis video streams that need to be combined and displayed by using a reinforcement learning method;

[0129] A live broadcast picture determination module 107 is configured to determine a live broadcast picture layout, wherein the video streams of the live broadcast picture layout comprise at least one of the presentation video stream, the demonstration site video stream and the window combination and layout results;

[0130] A live broadcast stream output module 108 is configured to output the live broadcast picture layout to a live broadcast platform using adaptive bitrate transmission technology.

[0131] In the embodiment, the system further comprises a video stream preprocessing module 102 configured to pre-process the video streams to improve the quality of the video streams and / or video frames.

[0132] In this embodiment, the output end of the video stream acquisition module 101 is connected with the input end of the video stream preprocessing module 102 and the input end of the live picture determination module 107, the output end of the video stream acquisition module 101 or the video stream preprocessing module 102 is connected with the input end of the video stream determination module 105, the video stream preprocessing module 102, the scene recognition module 104 and the video stream determination module 105 are sequentially arranged, the input end of the analysis module 103 corresponds to the video stream preprocessing module 102 and / or the video stream determination module 105, the input end of the video stream determination module 105 is connected with the output end of the scene recognition module 104 and / or the analysis module 103, the video stream determination module 105, the window combination and layout determination module 106, the live picture determination module 107 and the live stream output module 108 are sequentially arranged.

[0133] In this embodiment, the window combination and layout determination module 106 is further configured to optimize the window layout using a deep Q network, that is, update the window combination and layout result, the state includes a current window layout state, the action includes a possible layout adjustment action, and the reward function is dynamically adjusted according to audience feedback and key content display effect.

[0134] In this embodiment, the analysis module 103 is specifically configured to: in a live process, analyze scene changes of each medical diagnosis video stream displayed in the live and recognize important events in real time, and the window combination and layout determination module 106 is further configured to adjust the window layout according to the important event recognition result.

[0135] The automatic processing system 100 of the live can realize intelligent live director in the implementation process, and the specific implementation steps are not described here again.

[0136] Embodiment five

[0137] As shown in FIG. 5, the method according to the present application can realize an electronic device 200 of this embodiment, which comprises a memory 201, one or more processors 202, and one or more programs stored in the memory 201 and configured to be executed by the one or more processors 202, the one or more programs comprising instructions for executing the automatic processing method of the live according to any of the above embodiments.

[0138] The processor 202 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor 202 can also be any conventional processor.

[0139] The memory 201 can include various types of storage units, such as a system memory, a read-only memory (ROM), and a permanent storage device. In addition, the memory 201 can include a combination of any computer readable storage media, and the memory 201 can be a semiconductor memory chip, a magnetic disk, an optical disk.

[0140] The memory 201 stores executable codes, which, when processed by the processor 202, can cause the processor 202 to perform part or all of the above-described methods.

[0141] Embodiment six

[0142] The embodiment provides a computer readable storage medium, which includes instructions, when the instructions are run on a computer, cause the computer to perform each step of the automatic processing method of live broadcast described in any of the above embodiments.

[0143] Those skilled in the art to which the present application belongs can clearly understand that, for the convenience and brevity of description, only the division of the above functional units and modules is exemplified, and in actual application, the above functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the above-described functions. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit, and the integrated unit can be realized in the form of hardware or in the form of software functional unit. In addition, the specific names of each functional unit and module are only for convenient distinction, and do not limit the protection scope of the present application. The specific working process of the units and modules in the above system can refer to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0144] It should be noted that in the above embodiments, the description of each embodiment has its own emphasis, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0145] The effect of the automatic processing method, system and device of live broadcast is:

[0146] The present application can automatically identify the scene of each video stream, automatically determine the window which needs to be combined and display, and automatically layout, can realize important event recognition and automatically update the window combination and layout result, determine the live picture layout of the demonstration video stream, the video stream of the teaching site and / or the window combination and layout result by using the video demonstration, and automatically switch and direct according to the result, the whole process of directing can be automatically realized by the present application, different video streams do not need to be manually switched and the live picture layout does not need to be adjusted, and the error is not easy to occur; the present application is high in efficiency, can quickly respond to scene changes and display the video stream of important events in time; a large number of directing personnel and equipment support are not needed, the live broadcast cost is reduced; automatic switching is realized, manual analysis and determination of the video to be switched and manual switching operation are not needed, and the viewing experience is improved; adaptive bit rate transmission technology is used for transmission to ensure transmission stability, avoid slow transmission of too many video streams and cause live broadcast lag, and the viewing experience is improved.

[0147] The flowcharts and block diagrams in the drawings show the possible implementation architecture, function and operation of the system and method according to the embodiments of the present application. In this regard, each block in the flowchart or block diagram can represent a module, a program segment or a part of code containing one or more executable instructions for implementing the specified logic function. It should also be noted that in some alternative implementations, the functions marked in the blocks can also occur in different order from that marked in the drawings. For example, two consecutive blocks can actually be executed substantially in parallel, and they can also be executed in reverse order depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and the combination of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system performing the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0148] The above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; under the idea of the present application, the technical features in the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other changes of different aspects of the present application as described above, which are not provided in details for the sake of simplicity; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can still be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for automatic processing of live broadcasts, characterized in that, The method comprises the following steps: acquiring a presentation video stream, a live teaching site video stream, and a medical diagnosis video stream; performing scene recognition on the medical diagnosis video stream by using a pre-trained scene recognition model; analyzing the scene changes of the medical diagnosis video stream in real time and identifying important events; automatically determining medical diagnosis video streams that need to be combined for display according to the scene recognition result and / or the important event identification result; determining the window combination and layout result of the medical diagnosis video streams that need to be combined for display by using a reinforcement learning method; determining a live picture layout, wherein the video stream of the live picture layout comprises at least one of the presentation video stream, the live teaching site video stream, and the window combination and layout result; outputting the live picture layout to a live platform by using an adaptive bit rate transmission technology.

2. The method of claim 1, wherein, The method further comprises optimizing the window layout by using a deep neural network-based medical live teaching multi-scene distribution strategy algorithm. The structure of the deep neural network-based medical live teaching multi-scene distribution strategy algorithm comprises: an input encoding module configured to encode the scene type and / or important event information of the medical diagnosis video stream to obtain a dense feature vector of each medical diagnosis video stream; a time sequence feature extraction module configured to capture the dependency between the time sequence information and attributes of the dense feature vector of each medical diagnosis video stream to obtain a time sequence feature representation; a feature fusion module configured to calculate the correlation and importance weight between the time sequence features of the medical diagnosis video streams to obtain a weighted combined joint feature representation; a strategy decision module configured to calculate a Q value vector of each layout according to the joint feature; a strategy output module configured to determine an optimal layout decision by performing calculation on the Q value vector of each layout to obtain an optimized window layout.

3. The method of claim 1, wherein the live automatic processing is performed by a plurality of servers. The method further comprises pre-processing the video stream to improve the quality of the video stream and / or video frame.

4. The method of claim 1, wherein the live automatic processing is performed by a plurality of servers. The real-time analysis of the scene changes of the medical diagnosis and the identification of important events are specifically as follows: in the live process, the scene changes of each medical diagnosis video stream displayed in the live process are analyzed in real time, important events are identified, and the window layout is adjusted according to the important event identification result.

5. The method of claim 1, wherein the live automatic processing is performed by a plurality of servers. 5 The scene recognition model is a multi-scale convolutional neural network or a Transformers neural network.

6. The method of claim 5, wherein, The multi-scale convolutional neural network comprises: an input layer configured to obtain a video frame of a medical diagnosis video stream; a plurality of parallel convolution branches configured to perform calculation on the video frame to generate a plurality of initial feature maps with different scale features; a feature fusion layer configured to fuse and splice all the initial feature maps to obtain a feature vector an output layer configured to input the feature vector into a classifier to obtain a probability distribution of each scene type.

7. The method of claim 6, wherein the live automatic processing is performed by a plurality of servers. The number of the convolution branches is three, one convolution branch comprises three 3×3 first convolution layers, each first convolution layer is followed by a first ReLU activation layer and a maximum pooling layer; one convolution branch comprises two 5×5 second convolution layers, each second convolution layer is followed by a second ReLU activation and an average pooling layer, and the other convolution branch comprises two 7×7 third convolution layers, each third convolution layer is followed by a third ReLU activation layer and a global average pooling layer.

8. An automatic processing system for live broadcasts, characterized by The method comprises the following steps: a video stream acquisition module, configured to acquire a presentation video stream, a teaching site video stream, and a medical diagnosis video stream; a scene recognition module, configured to perform scene recognition on the medical diagnosis video stream by using a pre-trained scene recognition model; an analysis module, configured to analyze scene changes and identify important events in the medical diagnosis video stream in real time; a video stream determination module, configured to automatically determine medical diagnosis video streams that need to be combined for display according to a result of the scene recognition and / or a result of important event identification; a window combination and layout determination module, configured to determine a window combination and layout result of the medical diagnosis video streams that need to be combined for display by using a reinforcement learning method; a live picture determination module, configured to determine a live picture layout, wherein a video stream of the live picture layout includes at least one of the presentation video stream, the teaching site video stream, and the window combination and layout result; a live stream output module, configured to output the live picture layout to a live platform by using an adaptive bitrate transmission technology.

9. An electronic device, comprising: comprise: a memory; one or more processors; one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs comprise instructions for executing any one of the automatic processing methods of live broadcast according to claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium comprises instructions which, when executed on a computer, cause the computer to perform the automatic processing method of live broadcast according to any one of claims 1 to 8.

Images