Automatic processing method, system, electronic device and storage medium for live broadcast

Abstract

The present invention discloses an automatic processing method, system, electronic device and storage medium for live broadcast, the method comprising: obtaining a presentation video stream, a visual teaching site video stream and a medical diagnosis video stream; using a scene recognition model to perform scene recognition on the medical diagnosis video stream; analyzing the scene changes of the medical diagnosis video stream in real time and identifying important events; automatically determining the medical diagnosis video streams that need to be combined and displayed according to the results of scene recognition and / or the results of important event recognition; determining the window combination and layout results of the medical diagnosis video streams that need to be combined and displayed by a reinforcement learning method; determining the layout of the live broadcast screen; and outputting the layout of the live broadcast screen to the live broadcast platform using an adaptive bit rate transmission technology. The present invention realizes the automatic combination, layout and switching of live broadcasts, reduces the consumption of director resources and time compared to the prior art, reduces the cost of live broadcasts, and at the same time improves the efficiency and accuracy of medical live broadcasts, and optimizes the audience viewing experience.

Description

Technical Field

[0001] The present invention relates to the field of medical live broadcast technology, and in particular to an automatic processing method, system, electronic device and storage medium for live broadcast. Background Art

[0002] In modern medical teaching and surgical video teaching, especially in the live video teaching of digestive endoscopic surgery (such as ERCP surgery (endoscopic retrograde cholangiopancreatography)), it is usually necessary to broadcast multiple diagnosis positions in a video conference, each of which contains video streams such as endoscopy, surgical field, and ultrasound. If it is a digestive endoscopic ERCP surgery, it may also include an X-ray video stream. At the same time, the video teaching site has a large screen to display the host and the video of the presentation, and the video teaching site includes a classroom and / or consultation room. The simultaneous display and switching of these multiple video streams usually requires a lot of director resources and time, and requires a lot of preparation and arrangement work.

[0003] The existing technology mainly relies on manual directing to complete complex medical live broadcast tasks by manually switching different video streams and adjusting window layouts. This method has the following main disadvantages:

[0004] Manual operation is cumbersome and prone to errors: Directors need to manually switch between multiple video streams and adjust window layouts in real time, which is labor-intensive and prone to errors.

[0005] Inefficiency: Manual operations are inefficient and cannot respond quickly to scene changes, resulting in the inability to display important content in a timely manner;

[0006] High resource consumption: A large number of directors and equipment are required, which increases the cost of live broadcasting.

[0007] Poor viewing experience: Due to the delay in manual switching, viewers cannot get the best viewing experience.

[0008] In view of the above shortcomings, there is an urgent need to develop an automatic processing method and system for live broadcasting. Summary of the invention

[0009] In order to solve the problems of easy errors, low efficiency, high resource consumption and switching delay in medical live broadcast, the present invention provides an automatic processing method, system, electronic device and storage medium for live broadcast.

[0010] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0011] In a first aspect, the present invention provides a method for automatically processing live broadcast, comprising:

[0012] Obtain presentation video streams, on-site video streams of visual teaching, and medical consultation video streams;

[0013] Use the pre-trained scene recognition model to perform scene recognition on the medical diagnosis video stream;

[0014] Analyze scene changes in medical diagnosis video streams in real time and identify important events;

[0015] Automatically determine the medical diagnosis video streams that need to be combined and displayed according to the results of scene recognition and / or important event recognition;

[0016] Determine the window combination and layout results of the medical diagnosis position video streams that need to be combined and displayed by the reinforcement learning method;

[0017] Determine a live screen layout, wherein the video stream of the live screen layout includes at least one of the presentation video stream, the visual teaching site video stream, and the window combination and layout result;

[0018] Adaptive bitrate transmission technology is used to output the live screen layout to the live broadcast platform.

[0019] In a preferred embodiment, the method further comprises: optimizing the window layout using a medical visual teaching multi-scene distribution strategy algorithm based on a deep neural network;

[0020] The structure of the multi-scenario distribution strategy algorithm for medical visual teaching based on deep neural network includes:

[0021] An input encoding module is used to encode the scene type and / or important event information of the medical diagnosis video stream to obtain a dense feature vector of each medical diagnosis video stream;

[0022] A temporal feature extraction module is used to capture the time series information of the dense feature vector of each medical diagnosis video stream and the dependency between the attributes to obtain a temporal feature representation;

[0023] The feature fusion module is used to calculate the correlation and importance weight between the time series features of each medical diagnosis video stream, and obtain the weighted combined joint feature representation;

[0024] A strategy decision module, used for calculating the Q value vector of each layout according to the joint features;

[0025] The strategy output module is used to perform calculations based on the Q value vector of each layout to determine the best layout decision and obtain the optimized window layout.

[0026] In a preferred embodiment, the method further comprises: preprocessing the video stream to improve the quality of the video stream and / or video frame.

[0027] In a preferred embodiment, the real-time analysis of scene changes in medical diagnosis video streams and identification of important events is specifically: during the live broadcast process, real-time analysis of scene changes in each medical diagnosis video stream displayed in the live broadcast and identification of important events, and adjustment of the window layout according to the important event identification results.

[0028] In a preferred embodiment, the scene recognition model is a multi-scale convolutional neural network or a Transformers neural network.

[0029] In a preferred embodiment, the multi-scale convolutional neural network includes:

[0030] The input layer is used to obtain the video frames of the medical diagnosis video stream;

[0031] Several parallel convolution branches are used to operate on video frames to generate multiple initial feature maps with different scale features;

[0032] Feature fusion layer, used to fuse and splice all initial feature maps to obtain feature vectors

[0033] The output layer is used to input the feature vector into the classifier to obtain the probability distribution of each scene type.

[0034] In a preferred embodiment, the number of convolution branches is 3, one convolution branch includes 3 layers of 3×3 first convolution layers, each first convolution layer is followed by a first ReLU activation layer and a maximum pooling layer; one convolution branch includes 2 layers of 5×5 second convolution layers, each second convolution layer is followed by a second ReLU activation and an average pooling layer, and another convolution branch includes 2 layers of 7×7 third convolution layers, each third convolution layer is sequentially followed by a third ReLU activation layer and a global average pooling layer.

[0035] In a second aspect, the present invention provides an automatic processing system for live broadcast, comprising:

[0036] A video stream acquisition module is used to acquire presentation video streams, visual teaching site video streams, and medical diagnosis video streams;

[0037] A scene recognition module, used to perform scene recognition on a medical diagnosis video stream using a pre-trained scene recognition model;

[0038] An analysis module for analyzing scene changes in medical diagnosis video streams in real time and identifying important events;

[0039] A video stream determination module, used to automatically determine the medical diagnosis video streams that need to be combined and displayed according to the results of scene recognition and / or important event recognition;

[0040] A window combination and layout determination module, used to determine the window combination and layout results of the medical diagnosis position video streams that need to be combined and displayed by a reinforcement learning method;

[0041] A live screen determination module, used to determine a live screen layout, wherein the video stream of the live screen layout includes at least one of the presentation video stream, the visual teaching site video stream and the window combination and layout result;

[0042] The live streaming output module is used to output the live screen layout to the live streaming platform using adaptive bit rate transmission technology.

[0043] In a third aspect, the present invention provides an electronic device comprising: a memory; one or more processors; one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include instructions for executing any one of the automatic processing methods for live broadcast described in the first aspect.

[0044] In a fourth aspect, the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium includes instructions, and when the instructions are executed on a computer, the computer executes the automatic processing method of live broadcast as described in the first aspect.

[0045] The automatic processing method, system, electronic device and storage medium of live broadcast of the present invention automatically perform scene recognition on video streams through a pre-trained scene recognition model, analyze scene changes of video streams in real time and identify important events, automatically determine the medical diagnosis video streams that need to be combined and displayed according to the results of scene recognition and / or important event recognition, determine the window combination and layout results of the medical diagnosis video streams that need to be combined and displayed through a reinforcement learning method, adopt a video presentation, determine the live broadcast screen layout including the presentation video stream, the visual teaching site video stream and / or the window combination and layout results, and at the same time, the transmission adopts an adaptive bit rate transmission technology. The present invention realizes the automatic combination, layout and switching of live broadcasts. Compared with the prior art, it reduces the consumption of director resources and time, reduces the cost of live broadcasts, and at the same time improves the efficiency and accuracy of medical live broadcasts, and optimizes the audience's viewing experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0047] Figure 1Schematic diagram of the process of the automatic processing method of live broadcast provided in Example 1 Figure 1 .

[0048] Figure 2 Schematic diagram of the process of the automatic processing method of live broadcast provided in Example 2 Figure 2 .

[0049] Figure 3 Schematic diagram of the process of the automatic processing method of live broadcast provided in Example 3 Figure 3 .

[0050] Figure 4 This is a framework diagram of the automatic processing system for live broadcast provided in Example 4.

[0051] Figure 5 A schematic diagram of a structural electronic device provided in Example 5. DETAILED DESCRIPTION

[0052] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0053] Medical teaching and live video teaching of surgeries mainly rely on manual directing. However, since directors need to manually switch multiple video streams and adjust the window layout in real time, the workload is large and errors are easy to occur. In addition, manual operation is inefficient and cannot respond quickly to scene changes, which will result in important content not being displayed in time, thus affecting video teaching. In addition, manual switching has delays, and the audience (hereinafter referred to as students) cannot get the best viewing experience. In addition, manual directing requires a large number of personnel and equipment support, which increases the cost of live broadcasting.

[0054] To this end, the present invention provides an automatic processing method, system, electronic device and storage medium for live broadcast, which can automatically identify the scene of each medical diagnosis position video stream, automatically determine the medical diagnosis position video streams that need to be combined for display and automatically layout them, can realize important event recognition and automatically update the layout accordingly, use the presentation as the video stream, use the situation of the visual teaching site as the video stream, determine the live broadcast screen layout including the presentation video stream, the visual teaching site video stream and / or the window combination and layout results, and automatically switch and direct the broadcast accordingly, so that the present invention reduces the consumption of director resources and time, reduces the cost of live broadcast, improves the efficiency and accuracy of medical live broadcast, and optimizes the audience's viewing experience.

[0055] The following embodiments of the present invention can be performed separately, and the embodiments can also be performed in combination with each other, and the embodiments of the present invention do not specifically limit this. In the embodiments of the present invention, "first", "second", etc. are used to describe various components, but these components should not be limited by these terms. These terms are only used to distinguish one component from another component. The "and / or" mentioned in the present invention refers to any and all combinations including one or more related listed items.

[0056] The following describes the automatic processing method and system of live broadcast and its technical effects.

[0057] Embodiment 1

[0058] Figure 1 The following is a flow chart of the automatic processing method for live broadcasting in this embodiment. Figure 1 ,like Figure 1 As shown, the method provided in this embodiment is applicable to the field of medical video teaching sharing, and the method includes the following steps.

[0059] Get the presentation video stream;

[0060] Get the video stream of the teaching site;

[0061] Get the medical diagnosis video stream;

[0062] Use the pre-trained scene recognition model to perform scene recognition on the medical diagnosis video stream;

[0063] Analyze scene changes in medical diagnosis video streams in real time and identify important events;

[0064] Automatically determine the medical diagnosis video streams that need to be combined and displayed according to the results of scene recognition and / or important event recognition;

[0065] Determine the window combination and layout results of the medical diagnosis position video streams that need to be combined and displayed by the reinforcement learning method;

[0066] Determine a live screen layout, where the video stream of the live screen layout includes at least one of the following three types:

[0067] 1. Presentation video stream;

[0068] 2. Video stream of the teaching site;

[0069] 3. Window combination and layout results;

[0070] Adaptive bitrate transmission technology is used to output the live screen layout to the live broadcast platform.

[0071] It should be understood that when to judge whether there is an empty box in the above method may vary according to different execution processes, so the above method does not represent or imply that all steps must be executed in this order. Ordinary technicians in this field can transform or change the execution order of the above steps based on the present invention. Some embodiments of the above method are illustrated below.

[0072] In addition, it can be understood that the window combination and layout result includes the layout mode of the window layout and the video stream corresponding to the layout mode, and the live screen layout includes the layout mode of the live screen layout and the video stream corresponding to the layout mode.

[0073] Typically, the live broadcast screen layout is output to the live broadcast platform in a real-time updated state, that is, the latest live broadcast screen layout is output to the live broadcast platform.

[0074] Embodiment 2

[0075] See also Figure 2 , which is a schematic diagram of the process of the automatic processing method of live broadcast provided in this embodiment Figure 2 The automatic live broadcast processing method provided in this embodiment specifically includes the following steps:

[0076] S1. The video input module obtains medical diagnosis video streams of multiple medical diagnosis positions.

[0077] S2. Preprocess the medical diagnosis video stream to improve the quality of the video stream. The medical diagnosis video streams in the following steps may all be preprocessed medical diagnosis video streams.

[0078] S3. Perform scene recognition on the medical diagnosis video stream using a pre-trained scene recognition model. In this embodiment, the scene recognition model may be a multi-scale convolutional neural network or a Transformers neural network.

[0079] S4. Automatically determine the medical diagnosis video streams that need to be combined for display.

[0080] S5. Determine the window combination and layout results of the medical diagnosis position video streams that need to be combined and displayed by reinforcement learning method; obviously, there are multiple medical diagnosis position video streams that need to be combined and displayed, and these video streams can be from the same medical diagnosis position or from different medical diagnosis positions.

[0081] S6, the video input module obtains the presentation video stream and the visual teaching site video stream. In other embodiments, the visual teaching site video stream may be obtained in S5.

[0082] S7, obtaining an instruction from the host, where the instruction may be at least one of a voice instruction, a gesture instruction, and an instruction from a device used by the host;

[0083] According to the host's instructions, the live screen layout is automatically determined; for example, the host's instructions correspond to the screen occupation area of ​​the presentation video stream in the live broadcast, which may specifically be: whether to broadcast the presentation video stream, whether to display the presentation video stream in the smallest layout area in the live screen layout, and whether to display it in the largest layout area in the live screen layout;

[0084] It is understandable that there may be no host's instruction when executing S7, and the live screen layout will continue to be displayed according to the previous layout, or the live screen layout will be fine-tuned when there is a window combination and layout result.

[0085] The S7 also includes the step of obtaining information of the terminal for watching the live broadcast as an analysis basis for automatically determining the layout of the live broadcast screen. For example, the information of the terminal for watching the live broadcast may be the device type and / or device location. Specifically, if the learner is watching and learning remotely with his / her terminal (such as a mobile phone, laptop computer, etc.) outside the visual teaching site, the live broadcast screen layout may include the video stream of the visual teaching site.

[0086] Usually the live screen layout includes presentation video stream and / or window combination and layout results. When the video stream of the video teaching site is displayed separately, it usually corresponds to the opening of the video teaching, the break of the video teaching, the host asking questions to the audience, etc.

[0087] S8, using adaptive bit rate transmission technology to output the latest live images to the live broadcast platform.

[0088] S9, automatically and in real time analyze the scene changes of each video stream and identify important events; according to the important event identification results, automatically switch the video stream to realize the switching of the video stream of the live broadcast platform; important event identification can be to identify whether it is an important event, and further, it can also include classification of important events. Here, when an important event is identified, return to S4.

[0089] In one embodiment, after the important event recognition result is obtained, the medical diagnosis video streams that need to be combined and displayed are automatically determined. Preferably, the medical diagnosis video streams that need to be combined and displayed are automatically re-determined based on the important event recognition result and the scene recognition result.

[0090] In this embodiment, as an example, the classification of the scene recognition includes endoscopy image video, ultrasound image video, surgical field image video, and X-ray image video.

[0091] Here, all the medical diagnosis positions have their marks, and correspondingly, the medical diagnosis position video streams of the medical diagnosis positions also have corresponding marks, so as to determine which medical diagnosis position the medical diagnosis position video stream belongs to.

[0092] The method for automatically determining the medical diagnosis position video streams that need to be combined for display is not limited. As an example and not a limitation, it can be that one or several medical diagnosis positions are performing surgery and the picture is moving. In this case, all the medical diagnosis position video streams (or part of the medical diagnosis position video streams) of the medical diagnosis position performing surgery are determined as the medical diagnosis position video streams that need to be combined for display.

[0093] The method and logic for automatically determining the live screen layout are not limited, for example, reinforcement learning method may also be used.

[0094] In this embodiment, the video stream of the visual teaching site is used in the window combination and layout and / or the live screen layout determined by reinforcement learning. The video stream of the visual teaching site may include a video of the host station and a video with an audience. The presentation is used for connecting the scene, sharing courseware, etc.

[0095] The logic of automatically determining the live screen layout in the step is not limited, for example, the logic includes: the presentation video stream has the highest priority, and when the presentation needs to be explained, the live screen layout should become the presentation video.

[0096] Embodiment 3

[0097] See also Figure 3 , which is a schematic diagram of the process of the automatic processing method of live broadcast provided in this embodiment Figure 3 , the implementation of each step is specifically described in this embodiment.

[0098] Step 1: Acquisition of video streams: Acquisition of multiple medical diagnosis position video streams from multiple medical diagnosis positions through the first input unit of the video input module, usually each medical diagnosis position corresponds to multiple medical diagnosis position video streams. Acquisition of presentation video stream through the second input unit of the video input module. Acquisition of visual teaching site video stream through the third input unit of the video input module. The video input module includes a multi-channel video capture card and a high-bandwidth data interface to ensure high quality and low-latency transmission of video streams.

[0099] Step 2: Video stream preprocessing to improve the quality of video stream and / or video frame: The acquired medical diagnosis video stream is preprocessed, including denoising, image enhancement and format conversion. Here, denoising, image enhancement and format conversion are performed in sequence. Denoising operation uses the Non-Local Means (NLM) algorithm to reduce video noise and improve image quality. Image enhancement operation uses Generative Adversarial Network (GAN) technology, such as SRGAN (Super-Resolution GAN) to improve video resolution. Format conversion converts the video frame to a fixed size suitable for the following multi-scale convolutional neural network input, such as 224×224 pixels, and adjusts it to a unified input format.

[0100] In this step, the presentation video stream and the on-site video stream can also be pre-processed by denoising and image enhancement.

[0101] Step 3: AI scene recognition, using the pre-trained multi-scale convolutional neural network (MSCNN) as the AI ​​scene recognition model, the pre-processed medical diagnosis video stream is subjected to scene recognition, that is, the content type of the video stream is determined. The network structure of MSCNN includes multiple parallel convolution branches, each branch processes feature maps of different scales, and is used to operate the video frame to generate multiple initial feature maps with different scale features. The input layer receives the video frame image of the pre-processed medical diagnosis video stream, with a size of 224×224×3. In this embodiment, there are three parallel convolution branches. Convolution branch one processes small-scale features, including 3 layers of 3×3 first convolution layers, with 64, 128, and 256 filters respectively, and each first convolution layer is followed by the first ReLU activation layer and the maximum pooling layer. Convolution branch two processes medium-scale features, including 2 layers of 5×5 second convolution layers, with 64 and 128 filters respectively, and each second convolution layer is followed by the second ReLU activation and average pooling layer. The convolution branch three processes large-scale features, including two layers of 7×7 third convolution layers, with 32 and 64 filters respectively. Each third convolution layer is followed by a third ReLU activation layer and a global average pooling layer. The scale feature of the initial feature map obtained by the convolution branch one is smaller than the scale feature of the initial feature map obtained by the convolution branch two, and the scale feature of the initial feature map obtained by the convolution branch two is smaller than the scale feature of the initial feature map obtained by the convolution branch three. The feature fusion layer concatenates the outputs of the three convolution branches to form a unified feature vector. Specifically: feature compression is performed through a 1×1 fourth convolution layer, and then concatenated with a fully connected layer. The output layer inputs the fused feature vector into the Softmax classifier for scene recognition and outputs the probability distribution of each scene type.

[0102] Step 4: Window combination and layout: Based on the results of the scene recognition in step 3, several medical diagnosis position video streams of one or several medical diagnosis positions that need to be combined for display are combined and laid out in windows. The window combination and layout algorithm uses reinforcement learning (RL) technology to ensure the best viewing experience.

[0103] Here, the window combination and layout may also be: determining the medical diagnosis video streams that need to be combined and displayed according to the result of the scene recognition in step 3, and performing scene change analysis on the medical diagnosis video streams that need to be combined and displayed to identify important events therein.

[0104] The states of the reinforcement learning include the current window layout state, such as video stream type, window position and size; the actions include possible layout adjustment actions, such as window movement, scaling, merging and separation; the reward function is dynamically adjusted according to audience feedback and key content display effects to ensure that important content is displayed first. Audience feedback and key content display effects can be obtained by analyzing the video stream of the video teaching site.

[0105] In one embodiment, the window combination is determined by a number of medical diagnosis position video streams that need to be displayed in combination, and then the window layout is determined according to reinforcement learning;

[0106] In another embodiment, there is more than one group of medical diagnosis position video streams that need to be displayed in combination, and a group of medical diagnosis position video streams that need to be displayed in combination and their corresponding window layouts are determined by reinforcement learning method.

[0107] Furthermore, the layout of the window combination and layout results determined by the reinforcement learning technology is also optimized, and the optimized window combination and layout results are used as the final window combination and layout results. Preferably, the medical visual teaching multi-scenario distribution strategy algorithm based on deep neural network is used to optimize the window layout. Other optimization algorithms can also be used to adjust the window layout, such as genetic algorithm (GA).

[0108] The structure of the multi-scenario distribution strategy algorithm for medical visual teaching based on deep neural network includes:

[0109] An input encoding module inputs the attributes of several medical diagnosis position video streams (scene recognition results and / or important event information, and the important event information is obtained according to the important event recognition results, for example, the important event recognition results are used as the important event information); the module internally includes an embedding layer and a multi-layer perceptron (MLP) for encoding the scene type and / or important event information of the medical diagnosis position video stream; through operation, a dense feature vector of each medical diagnosis position video stream is obtained;

[0110] A temporal feature extraction module, which inputs the dense feature vector of each medical diagnosis video stream; it contains a long short-term memory network (LSTM) to capture the dependency between time series information and attributes; through calculation, a feature representation that captures the temporal dependency is obtained, referred to as the temporal feature representation;

[0111] A feature fusion module that inputs the temporal feature representation of all medical diagnosis position video streams. It contains a multi-head self-attention mechanism layer to calculate the correlation and importance weight between the temporal features of each medical diagnosis position video stream. Through calculation, a weighted combined joint feature representation is obtained to highlight the features of the key medical diagnosis position video streams.

[0112] A policy decision module, which takes in the joint feature representation as input; it contains fully connected layers to calculate the Q value of each possible layout; through calculation, the Q value vector of each layout is obtained;

[0113] A strategy output module inputs the Q value vector of each layout; it contains a selection function (such as argmax) to determine the best layout decision; through calculation, the final video stream window combination arrangement layout is obtained, and the optimized window layout is obtained, that is, the optimized window combination and layout is obtained.

[0114] Step 5: Live screen layout: According to the preset logic, input includes instructions, window combination and layout results to determine the layout of the live screen.

[0115] By re-determining the live screen layout in real time, automatic switching and directing of live broadcasts can be achieved. During the live broadcast, the live screen layout is monitored in real time, and the AI ​​system is used to achieve automatic switching and directing.

[0116] Real-time analysis of scene changes in medical diagnosis video streams to identify important events in medical diagnosis, such as key surgical steps and changes in surgical fields. Using an event-driven switching algorithm, the video stream is automatically switched based on the identification results of important events to ensure that the audience sees the most relevant and important content. Automatically adjust the window layout to adapt to scene changes; then automatically adjust the live screen layout to adapt to the application, such as changes in the host's hosting process or changes in the needs of learners, to provide the best viewing experience. Combine audience (learner) feedback and interactive data to dynamically optimize the director strategy to ensure the maximum live broadcast effect.

[0117] As an example, in this embodiment, during the live broadcast process, only the scene changes of each displayed medical examination position video stream are monitored in real time and important events are identified. According to the important event identification results, the window layout is adjusted. It can be understood that there is no need to adjust the window combination at this time.

[0118] Step 6: Output the live stream and output the latest live screen layout to the live broadcast platform in real time to ensure that the audience can watch high-quality medical live content in real time. The live stream output uses adaptive bit rate (ABR) technology to dynamically adjust the video quality according to the network bandwidth to ensure smooth playback.

[0119] In other embodiments, the method further includes: automatically determining a certain medical diagnosis position video stream that needs to be displayed separately according to the result of scene recognition and / or the result of important event recognition, and outputting the video stream that needs to be displayed separately to the live broadcast platform using adaptive bit rate transmission technology. Furthermore, the live broadcast screen layout is determined by using a certain video stream of a certain medical diagnosis position that needs to be displayed separately and a presentation video stream (and a video stream of the visual teaching site).

[0120] Embodiment 4

[0121] This embodiment provides a live broadcast automatic processing system 100, see Figure 4 , the system includes:

[0122] The video stream acquisition module 101 is used to acquire the presentation video stream, the visual teaching site video stream and the medical diagnosis video stream;

[0123] A scene recognition module 104, used to perform scene recognition on the medical diagnosis video stream using a pre-trained scene recognition model;

[0124] An analysis module 103, for analyzing scene changes of the medical diagnosis video stream in real time and identifying important events;

[0125] The video stream determination module 105 is used to automatically determine the medical diagnosis video streams that need to be combined and displayed according to the scene recognition results and / or the important event recognition results;

[0126] The window combination and layout determination module 106 is used to determine the window combination and layout results of the medical diagnosis position video streams that need to be combined and displayed by a reinforcement learning method;

[0127] A live screen determination module 107, used to determine a live screen layout, wherein the video stream of the live screen layout includes at least one of the presentation video stream, the visual teaching site video stream and the window combination and layout result;

[0128] The live streaming output module 108 is used to output the live streaming screen layout to the live streaming platform by using an adaptive bit rate transmission technology.

[0129] In this embodiment, the system further includes a video stream preprocessing module 102, which is used to preprocess the video stream to improve the quality of the video stream and / or video frame.

[0130] In this embodiment, the output end of the video stream acquisition module 101 is connected to the input end of the video stream preprocessing module 102 and the input end of the live screen determination module 107, the output end of the video stream acquisition module 101 or the video stream preprocessing module 102 is connected to the input end of the video stream determination module 105, the video stream preprocessing module 102, the scene recognition module 104 and the video stream determination module 105 are arranged in sequence, the input end of the analysis module 103 corresponds to the video stream preprocessing module 102 and / or the video stream determination module 105, the input end of the video stream determination module 105 is connected to the output end of the scene recognition module 104 and / or the analysis module 103, the video stream determination module 105, the window combination and layout determination module 106, the live screen determination module 107 and the live stream output module 108 are arranged in sequence.

[0131] In this embodiment, the window combination and layout determination module 106 is also used to optimize the window layout using a deep Q network, that is, to update the window combination and layout results. The state includes the current window layout state, the action includes possible layout adjustment actions, and the reward function is dynamically adjusted according to audience feedback and key content display effects.

[0132] In this embodiment, the analysis module 103 is specifically used to: during the live broadcast process, analyze the scene changes of each medical diagnosis position video stream displayed in real time and identify important events. The window combination and layout determination module 106 is also used to adjust the window layout according to the important event identification results.

[0133] When the automatic live broadcast processing system 100 is implemented, the automatic live broadcast processing method in any of the above embodiments can be referred to to realize intelligent live broadcast directing, and the specific implementation steps will not be repeated.

[0134] Embodiment 5

[0135] like Figure 5 According to the method of the present invention, an electronic device 200 of this embodiment can be implemented, and the electronic device 200 includes: a memory 201; one or more processors 202; one or more programs, wherein the one or more programs are stored in the memory 201 and are configured to be executed by the one or more processors 202, and the one or more programs include instructions for executing the automatic processing method of live broadcast according to any of the above embodiments.

[0136] The processor 202 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor 202 may also be any conventional processor, etc.

[0137] The memory 201 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. In addition, the memory 201 may include any combination of computer-readable storage media, such as semiconductor memory chips, magnetic disks, and optical disks.

[0138] The memory 201 stores executable codes, and when the executable codes are processed by the processor 202 , the processor 202 can execute part or all of the above-mentioned methods.

[0139] Embodiment 6

[0140] This embodiment provides a computer-readable storage medium, which includes instructions. When the instructions are executed on a computer, the computer executes each step of the automatic processing method of live broadcast described in any of the above embodiments.

[0141] The technicians in the field to which the present invention belongs can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0142] It should be noted that in the above embodiments, the description of each embodiment has its own emphasis, and for parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0143] The effects of the automatic processing method, system and device of live broadcast of the present invention are:

[0144] The present invention can automatically identify the scene of each video stream, automatically determine the windows that need to be combined and displayed and automatically layout them, can realize important event recognition and automatically update the window combination and layout results accordingly, use the video presentation to determine the live screen layout of the presentation video stream, the visual teaching site video stream and / or the window combination and layout results, and automatically switch and direct the broadcast accordingly. The present invention can automatically realize the whole process of directing, without the need to manually switch different video streams and adjust the live screen layout, and is not prone to errors; the present invention is highly efficient and can quickly respond to scene changes and display video streams of important events in a timely manner; it does not require a large number of broadcast directors and equipment support, reducing the cost of live broadcasting; it can switch automatically, without the need for manual analysis to determine which video to switch to and manual switching operations, thereby improving the viewing experience; the transmission adopts adaptive bit rate transmission technology to ensure stable transmission, avoid slow transmission of too many video streams and cause live broadcast freezes, thereby improving the viewing experience.

[0145] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system and method according to multiple embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a part of a module, a program segment or a code, and the part of the module, the program segment or the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or the flow chart, and the combination of the square boxes in the block diagram and / or the flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0146] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them. Under the concept of the present invention, the technical features in the above embodiments or different embodiments may also be combined, the steps may be implemented in any order, and there are many other changes in different aspects of the present invention as described above, which are not provided in detail for the sake of simplicity. Although the present invention has been described in detail with reference to the above embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the above embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. The automatic processing method of live broadcast is characterized by: include: Obtain presentation video streams, on-site video streams of visual teaching, and medical consultation video streams; Use the pre-trained scene recognition model to perform scene recognition on the medical diagnosis video stream; Analyze scene changes in medical diagnosis video streams in real time and identify important events; Automatically determine the medical diagnosis video streams that need to be combined and displayed according to the results of scene recognition and / or important event recognition; Determine the window combination and layout results of the medical diagnosis position video streams that need to be combined and displayed by the reinforcement learning method; Determine a live screen layout, wherein the video stream of the live screen layout includes at least one of the presentation video stream, the visual teaching site video stream, and the window combination and layout result; Adaptive bitrate transmission technology is used to output the live screen layout to the live broadcast platform.

2. The automatic processing method of live broadcast as claimed in claim 1, characterized in that: The method further includes: optimizing the window layout using a medical visual teaching multi-scene distribution strategy algorithm based on a deep neural network; The structure of the multi-scenario distribution strategy algorithm for medical visual teaching based on deep neural network includes: An input encoding module is used to encode the scene type and / or important event information of the medical diagnosis video stream to obtain a dense feature vector of each medical diagnosis video stream; A temporal feature extraction module is used to capture the time series information of the dense feature vector of each medical diagnosis video stream and the dependency between the attributes to obtain a temporal feature representation; The feature fusion module is used to calculate the correlation and importance weight between the time series features of each medical diagnosis video stream, and obtain the weighted combined joint feature representation; A strategy decision module, used for calculating the Q value vector of each layout according to the joint features; The strategy output module is used to perform calculations based on the Q value vector of each layout to determine the best layout decision and obtain the optimized window layout.

3. The automatic processing method of live broadcast as claimed in claim 1, characterized in that: The method further comprises: preprocessing the video stream to improve the quality of the video stream and / or the video frame.

4. The automatic processing method of live broadcast as claimed in claim 1, characterized in that: The real-time analysis of scene changes of medical examination positions and identification of important events specifically includes: during the live broadcast, real-time analysis of scene changes of each medical examination position video stream displayed in the live broadcast and identification of important events, and adjustment of the window layout according to the important event identification results.

5. The automatic processing method of live broadcast as claimed in claim 1, characterized in that: The scene recognition model is a multi-scale convolutional neural network or a Transformers neural network.

6. The automatic processing method of live broadcast as claimed in claim 5, characterized in that: The multi-scale convolutional neural network comprises: The input layer is used to obtain the video frames of the medical diagnosis video stream; Several parallel convolution branches are used to operate on video frames to generate multiple initial feature maps with different scale features; Feature fusion layer, used to fuse and splice all initial feature maps to obtain feature vectors The output layer is used to input the feature vector into the classifier to obtain the probability distribution of each scene type.

7. The automatic processing method of live broadcast as claimed in claim 6, characterized in that: The number of convolution branches is 3, one convolution branch includes 3 layers of 3×3 first convolution layers, each first convolution layer is followed by a first ReLU activation layer and a maximum pooling layer; one convolution branch includes 2 layers of 5×5 second convolution layers, each second convolution layer is followed by a second ReLU activation and an average pooling layer, and the other convolution branch includes 2 layers of 7×7 third convolution layers, each third convolution layer is sequentially followed by a third ReLU activation layer and a global average pooling layer.

8. The automatic processing system of live broadcast is characterized by: include: A video stream acquisition module is used to acquire presentation video streams, visual teaching site video streams, and medical diagnosis video streams; A scene recognition module, used to perform scene recognition on a medical diagnosis video stream using a pre-trained scene recognition model; An analysis module for analyzing scene changes in medical diagnosis video streams in real time and identifying important events; A video stream determination module, used to automatically determine the medical diagnosis video streams that need to be combined and displayed according to the results of scene recognition and / or important event recognition; A window combination and layout determination module, used to determine the window combination and layout results of the medical diagnosis position video streams that need to be combined and displayed by a reinforcement learning method; A live screen determination module, used to determine a live screen layout, wherein the video stream of the live screen layout includes at least one of the presentation video stream, the visual teaching site video stream and the window combination and layout result; The live streaming output module is used to output the live screen layout to the live streaming platform using adaptive bit rate transmission technology.

9. An electronic device, characterized in that: include: Memory; one or more processors; One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for executing any one of the automatic processing methods for live broadcast according to claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes instructions, which, when executed on a computer, enable the computer to execute the automatic processing method for live broadcast as described in any one of claims 1 to 7.

Images