A method for live broadcast of ward round based on a chest card recorder system and related devices

Abstract

The application provides a ward round live broadcast method based on a chest badge recorder system and related devices, including: in response to receiving a first instruction for indicating the start of ward round live broadcast, acquiring a first video stream and a first audio stream collected by a first chest badge recorder of a first user, and a second video stream and a second audio stream collected by a second chest badge recorder of a second user; obtaining first voice video stream data according to the first video stream, the first audio stream, the second video stream and the second audio stream; obtaining second voice video stream data according to the first voice video stream data and a disease data set of a third user, the second voice video stream data displaying the disease data of the third user and interaction event data of the third user and the first user; and performing live broadcast based on the second voice video stream data to improve the accuracy of ward round live broadcast.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to a method and related apparatus for live ward rounds based on a badge recorder system. Background Technology

[0002] Ward rounds refer to the medical practice where doctors regularly check the condition and vital signs of hospitalized patients, evaluate and adjust treatment plans, and simultaneously conduct teaching explanations and case discussions. This directly affects the quality of diagnosis and treatment. To meet the needs of remote consultations, teaching observations, multi-departmental collaborative diagnosis and treatment, and the recording of medical procedures, live streaming of ward rounds is increasingly being used.

[0003] When doctors conduct ward rounds live, the live streaming platform requires a live streaming cart to film the entire process. This involves complex equipment, numerous wired connections, and multiple operators who need to follow and operate the system in real time. Many details cannot be effectively conveyed through real-time video, resulting in low efficiency for ward round live streams. Improving the efficiency of ward round live streams is a problem that needs to be addressed. Summary of the Invention

[0004] This application provides a method and related apparatus for live ward rounds based on a badge recorder system, thereby improving the accuracy of live ward rounds.

[0005] In a first aspect, embodiments of this application provide a method for live-streaming ward rounds based on a badge recorder system, including: In response to receiving a first instruction to initiate a live ward round, the system acquires a first video stream and a first audio stream captured by a first badge recorder of a first user, and a second video stream and a second audio stream captured by a second badge recorder of a second user. Based on the first video stream, the first audio stream, the second video stream, and the second audio stream, the first voice and video stream data is obtained; Based on the first audio and video stream data and the third user's medical condition dataset, a second audio and video stream data is obtained. The second audio and video stream data displays the third user's medical condition data and the interaction event data between the third user and the first user. Live streaming is conducted based on the second audio and video stream data.

[0006] Secondly, embodiments of this application provide a ward round live-streaming device based on a badge recorder system, comprising: The first processing unit is configured to, in response to receiving a first instruction to initiate a live ward round, acquire a first video stream and a first audio stream captured by a first badge recorder of a first user, and a second video stream and a second audio stream captured by a second badge recorder of a second user; The second processing unit is used to obtain first voice and video stream data based on the first video stream, the first audio stream, the second video stream, and the second audio stream; The third processing unit is used to obtain second voice and video stream data based on the first voice and video stream data and the third user's medical condition dataset. The second voice and video stream data displays the third user's medical condition data and the interaction event data between the third user and the first user. The fourth processing unit is used to perform live streaming based on the second audio and video stream data.

[0007] Thirdly, embodiments of this application provide an electronic device including a processor, a memory, and one or more programs, the one or more programs being stored in the memory and configured to be executed by the processor, the programs including instructions for performing steps as described in the method of the first aspect of this application.

[0008] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program or instructions stored thereon, wherein the computer program or instructions, when executed by a processor, implement the steps of the method described in the first aspect of this application.

[0009] As can be seen in this embodiment, after receiving the ward round live broadcast start command, the system acquires multiple video and audio streams collected by the name tag recorders worn by the ward round personnel in real time. The multiple audio and video data are integrated into a unified audio and video stream data. Then, combined with the corresponding patient's condition dataset, the patient's condition data and doctor-patient interaction event information are superimposed on the live broadcast stream. Finally, the ward round live broadcast is carried out based on the fused audio and video stream, realizing multi-view audio and video acquisition, synchronous presentation of patient information, and visualization of interactive content. This effectively avoids problems such as incomplete information from a single video feed, asynchronous condition information, and missing key interactive content, thereby significantly improving the accuracy of the ward round live broadcast. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a schematic diagram of the architecture of a name tag recorder system provided in an embodiment of this application; Figure 2 This is a flowchart of a ward round live broadcast method based on a name tag recorder system provided in an embodiment of this application; Figure 3This is a schematic diagram of a ward round live broadcast scenario provided in an embodiment of this application; Figure 4 This is a schematic diagram of a process for obtaining first voice and video stream data based on the first video stream, the first audio stream, the second video stream, and the second audio stream, provided in an embodiment of this application. Figure 5 This is a schematic diagram of a process for obtaining third voice and video stream data by performing frame-level synchronization on the first video stream, the first audio stream, the second video stream, and the second audio stream, as provided in an embodiment of this application. Figure 6 This is a schematic diagram of a process for obtaining first voice and video stream data by encoding the third voice and video stream data, provided in an embodiment of this application. Figure 7 This is a schematic diagram of a process for adjusting the bitrate of the fourth voice and video stream data based on network bandwidth, a preset network bandwidth threshold, and a bitrate adjustment coefficient, provided in an embodiment of this application. Figure 8 This is a schematic diagram of a live streaming interface provided in an embodiment of this application; Figure 9 This is a schematic diagram of the structure of a ward round live broadcast device based on a name tag recorder system provided in an embodiment of this application; Figure 10 This is a schematic diagram of the structure of a computer provided in an embodiment of this application. Detailed Implementation

[0012] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0013] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0014] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0015] Ward rounds refer to the medical practice where doctors regularly check the condition and vital signs of hospitalized patients, evaluate and adjust treatment plans, and simultaneously conduct teaching explanations and case discussions. This directly affects the quality of diagnosis and treatment. To meet the needs of remote consultations, teaching observations, multi-departmental collaborative diagnosis and treatment, and the recording of medical procedures, live streaming of ward rounds is increasingly being used.

[0016] When doctors conduct ward rounds live, the live streaming platform requires a live streaming cart to film the entire process. This involves complex equipment, numerous wired connections, and multiple operators who need to follow and operate the system in real time. Many details cannot be effectively conveyed through real-time video, resulting in low efficiency for ward round live streams. Improving the efficiency of ward round live streams is a problem that needs to be addressed.

[0017] Upon receiving the command to start a live ward round, this solution acquires multiple video and audio streams from the name tag recorders worn by the personnel involved in the ward round. It integrates these multiple audio and video data into a unified audio and video stream, and then combines this with the corresponding patient's medical data set. Patient medical data and doctor-patient interaction information are overlaid into the live stream. Finally, the ward round is conducted based on the fused audio and video stream, achieving multi-view audio and video acquisition, synchronous presentation of patient information, and visualization of interactive content. This effectively avoids problems such as incomplete information from single-stream images, asynchronous patient information, and missing key interactive content, thus significantly improving the accuracy of live ward rounds.

[0018] The ward round live streaming method and related apparatus based on the badge recorder system provided in this application can be applied to, for example... Figure 1 Please refer to the name tag recorder system shown. Figure 1 , Figure 1 This is a schematic diagram of the architecture of a name tag recorder system provided in an embodiment of this application. The name tag recorder system 100 includes a client 101 and a server 102. The client 101 can communicate with the server 102 via a network. The client 101 refers to a device used by the user, such as a smartphone or computer. The client 101 also includes the name tag recorder worn by the user. In this solution, the client 101 provides an interface for the user to interact with the name tag recorder system 100. Through the client 101, the user can interact with the name tag recorder system 100.

[0019] Server 102 refers to a remote computer used for processing large amounts of computing tasks and storing data. In this scheme, server 102 is responsible for responding to a first instruction to initiate a live ward round, acquiring a first video stream and a first audio stream collected by a first user's first badge recorder, and a second video stream and a second audio stream collected by a second user's second badge recorder; obtaining first audio-visual stream data based on the first video stream, first audio stream, second video stream, and second audio stream; obtaining second audio-visual stream data based on the first audio-visual stream data and a third user's medical condition dataset, wherein the second audio-visual stream data displays the third user's medical condition data and interaction event data between the third user and the first user; and conducting a live broadcast based on the second audio-visual stream data.

[0020] Based on this, this application provides a method and related apparatus for live ward rounds based on a badge recorder system. The following is a detailed description of this application with reference to the accompanying drawings.

[0021] Please see Figure 2 , Figure 2 This is a flowchart of a ward round live broadcast method based on a name tag recorder system provided in an embodiment of this application, such as... Figure 2 As shown, the ward round live streaming method based on the name tag recorder system includes the following steps: S201, in response to receiving a first instruction to instruct the start of a ward round live broadcast, acquires a first video stream and a first audio stream collected by a first badge recorder of a first user, and a second video stream and a second audio stream collected by a second badge recorder of a second user.

[0022] The implementing entity of this plan can be... Figure 1 Server 102 of the badge recorder system 100.

[0023] The first instruction is a control command triggered by medical staff or the system to initiate the ward round live broadcast process, notifying the start of audio and video data collection and processing. The first user refers to the primary medical staff performing the ward round, usually the attending physician or chief physician, who is the core operator in the process. The second user refers to the supporting medical staff accompanying the ward round, generally resident physicians, resident physicians, nurses, or medical staff responsible for medical record keeping.

[0024] Among them, the first badge recorder and the second badge recorder refer to portable wearable data collection devices worn by the first user and the second user respectively. Both devices integrate cameras and microphones and can collect video images and audio signals in real time.

[0025] The first video stream and the first audio stream are real-time video and audio data collected and transmitted from the ward round by the first name tag recorder worn by the attending physician. The second video stream and the second audio stream are real-time video and audio data from another perspective collected and transmitted from the second name tag recorder worn by the accompanying ward round patient.

[0026] Specifically, upon receiving the first instruction to start the live ward round, the live streaming system simultaneously acquires one audio and video stream from the first name tag recorder worn by the attending physician, and another audio and video stream from the second name tag recorder worn by the accompanying ward round personnel, thus enabling real-time acquisition of audio and video data from multiple personnel and multiple perspectives during the ward round.

[0027] Please refer to Figure 3 , Figure 3 This is a schematic diagram of a ward round live broadcast scenario provided in an embodiment of this application, such as... Figure 3 As shown, the first user (attending physician) is located in the upper left corner of the screen, wearing a first name tag and recording device to capture the first video and audio streams during the ward round. The second user (accompanying medical staff) is located in the lower center of the screen, wearing a second name tag and recording device, simultaneously capturing video and audio streams from different perspectives to provide supplementary visual and conversational information. The third user (patient to be rounded) is located in the bed area on the right side of the screen, representing the patient being treated during this ward round; their medical data will be integrated with the live stream. The first and second users are both positioned near the bed, forming a dual-perspective capture layout. The attending physician's perspective primarily focuses on face-to-face interaction with the patient, while the accompanying staff's perspective covers the patient's overall condition, examination details, or medical record recording processes.

[0028] It should be emphasized that the video and audio streams collected by different name tag recorders were all authorized by third-party users.

[0029] S202, the first voice and video stream data is obtained based on the first video stream, the first audio stream, the second video stream, and the second audio stream.

[0030] When obtaining the final first audio-video stream data from multiple audio and video streams, various methods can be employed, such as multi-channel mixing and merging, primary / backup switching merging, or multi-screen splicing and merging. Specifically, multi-channel mixing and merging involves real-time synchronization and alignment of the first video stream, first audio stream, second video stream, and second audio stream. One video stream is selected as the main screen, and the other as a picture-in-picture or auxiliary screen. The two audio streams are then mixed and noise-reduced to ultimately synthesize a single first audio-video stream containing multiple perspectives and complete live audio. Primary / backup switching merging uses the first video and first audio streams of the users making the room check as the primary live stream source, and the second video and second audio streams of the accompanying users as backup supplementary sources. The system automatically selects and merges these sources based on image stability and audio clarity, integrating them to generate a stable and continuous first audio-video stream. Multi-view splicing and streaming refers to splicing or splitting the first video stream and the second video stream, while simultaneously overlaying, deduplicating, and enhancing the two audio streams. Under the premise of ensuring audio and video synchronization, the first audio and video stream data containing dual-view images and complete audio information is merged and output.

[0031] In one possible implementation, please refer to Figure 4 , Figure 4 This is a schematic diagram of a process for obtaining first voice and video stream data based on the first video stream, the first audio stream, the second video stream, and the second audio stream, as provided in an embodiment of this application. Figure 4 As shown, obtaining the first audio-video stream data based on the first video stream, the first audio stream, the second video stream, and the second audio stream includes the following steps: S401, by performing frame-level synchronization on the first video stream, the first audio stream, the second video stream, and the second audio stream, the third voice and video stream data is obtained.

[0032] The first video stream and the first audio stream refer to the video and audio data streams collected by the name tag recorder worn by the ward round doctor. The second video stream and the second audio stream refer to the video and audio data streams collected by the name tag recorder worn by the accompanying ward round staff.

[0033] Frame-level synchronization uses video frames as the smallest time unit to align the timestamps of multiple audio and video streams, ensuring that the picture and sound, as well as the main and auxiliary streams, are completely consistent in time, thus avoiding problems such as audio and video desynchronization, picture misalignment, and sound delay.

[0034] Specifically, based on global timestamps Using the first audio stream and the first video stream as the main audio and video streams, and the second audio stream and the second video stream as the auxiliary audio and video streams, frame-level synchronization alignment is performed on the main and auxiliary audio and video streams, and the synchronization error between the multiple streams is calculated and corrected. The formula for calculating synchronization error is: ; in, Main road Frame timestamp; As auxiliary road The frame's timestamp. When When the time is less than 40ms, it is determined that the frame synchronization is complete, and the timing binding of the main and auxiliary streams is completed.

[0035] Among them, the third audio and video stream data refers to the synchronous data stream formed by the initial fusion of multiple audio and video streams after frame-level time alignment, which has not yet been compressed and encoded.

[0036] Specifically, S401 identifies the timestamps of each audio and video stream, performs frame-level time alignment and synchronization processing on the first video stream, the first audio stream, the second video stream, and the second audio stream, eliminates the time delay deviation between the data streams, and forms a time-uniform third audio and video stream data.

[0037] S402, the first audio and video stream data is obtained by encoding the third audio and video stream data.

[0038] Encoding involves using video and audio coding standards (such as H.264, H.265, AAC, etc.) to compress the synchronized original audio and video data, reducing the data bit rate and size while ensuring smooth transmission and playback.

[0039] The first audio and video stream data refers to standard format audio and video streams that, after synchronization and encoding, can be directly used for network transmission, live streaming, or subsequent information overlay.

[0040] Specifically, S402 performs standardized compression encoding on the third audio and video stream data that has completed frame-level synchronization, converting it into a format suitable for transmission and live streaming, and finally obtaining the first audio and video stream data that can be directly used for subsequent processing.

[0041] As can be seen in this example, frame-level time synchronization processing is first performed on the audio and video streams collected by multiple badge recorders to ensure that the timing of the video and audio is consistent. Then, the synchronized data stream is compressed and encoded to form a standard, smooth audio and video stream suitable for transmission and live streaming. This not only solves the problem of asynchronous multi-source data, but also optimizes the data transmission efficiency, providing stable and reliable basic stream data for subsequent ward round live streaming.

[0042] In one possible implementation, please refer to Figure 5 , Figure 5This application provides a schematic diagram of a process for obtaining third voice and video stream data by performing frame-level synchronization on the first video stream, the first audio stream, the second video stream, and the second audio stream, as illustrated in the embodiments of this application. Figure 5 As shown, the third audio-video stream data is obtained by performing frame-level synchronization on the first video stream, the first audio stream, the second video stream, and the second audio stream, including the following steps: S501, synchronize the time error of the first video stream, the first audio stream, the second video stream, and the second audio stream based on the global timestamp.

[0043] The global timestamp is a globally assigned time identifier uniformly assigned by server 102. It assigns the same time base to the first video stream, first audio stream, second video stream, and second audio stream, unifying the starting point of each data stream. Time error refers to the time offset between the audio and video streams due to differences in acquisition and transmission speeds, manifesting as delays or lead-outs in the picture or sound. Time error synchronization uses the global timestamp as a reference to align the four data streams, adjusting their playback sequence and reducing time deviations between them.

[0044] Specifically, the S501 uses a unified global timestamp as a benchmark to perform timing calibration on the two video and two audio data streams collected by the badge recorders of doctors and accompanying persons, adjusting and eliminating the time offset between each data stream frame by frame to achieve preliminary time synchronization.

[0045] S502, if the time error is detected to be less than a preset time threshold, frame-level synchronization is determined to be complete, and the third voice and video stream data is obtained.

[0046] The preset time threshold refers to the maximum allowable time error value pre-set by server 102, used to determine whether synchronization meets the usable standard. Frame-level synchronization uses a single video frame as the smallest unit to achieve precise time alignment of all audio and video streams, without noticeable stuttering, misalignment, or audio-visual asynchrony. The third audio and video stream data refers to the time-consistent, playable data stream formed by the fusion of multiple audio and video streams after precise frame-level synchronization.

[0047] Specifically, the S502 monitors the time error after synchronization in real time. When the error is less than a preset threshold, it determines that frame-level synchronization is complete and merges multiple audio and video streams into a third audio and video stream with precise and consistent timing.

[0048] As can be seen in this example, by first synchronizing the time errors of multiple audio and video streams based on the global timestamp, and then judging whether the frame-level synchronization is completed by using a preset time threshold, it is possible to achieve accurate time alignment of multi-source audio and video data during ward rounds. This effectively avoids problems such as image misalignment, sound lag, and audio-visual asynchrony, ensuring that the ward round live broadcast is coherent and the content is true and accurate, and providing stable and reliable basic data for subsequent live broadcasts and information overlay.

[0049] In one possible implementation, please refer to Figure 6 , Figure 6 This is a schematic diagram of a process for obtaining first audio and video stream data by encoding the third audio and video stream data, as provided in an embodiment of this application. Figure 6 As shown, the first audio and video stream data is obtained by encoding the third audio and video stream data, including the following steps: S601, the third voice and video stream data is encoded based on a preset encoding method to obtain the fourth voice and video stream data.

[0050] Among them, the third audio and video stream data refers to the original fused audio and video data stream that has undergone precise frame-level synchronization, with time alignment and unified picture and sound.

[0051] The preset encoding method refers to the pre-configured video and audio compression encoding standards, such as H.264, H.265, and AAC, used to compress data size and unify data format. The fourth audio and video stream data consists of intermediate audio and video stream data that has completed encoding compression and format standardization, but whose bitrate has not yet been adjusted according to network conditions.

[0052] Specifically, S601 performs format conversion and data compression on the synchronized third voice and video stream data according to the set encoding standard, to obtain a fourth voice and video stream data with a standardized format suitable for network transmission.

[0053] S602, based on network bandwidth, multiple network bandwidth thresholds and bitrate adjustment coefficient, the bitrate of the fourth voice and video stream data is adjusted to obtain the first voice and video stream data.

[0054] Among these, network bandwidth refers to the actual network transmission speed of the current live streaming link, determining whether the audio and video streams can be smoothly delivered. Network bandwidth threshold refers to a preset bandwidth level standard used to judge the quality of the network environment. Bitrate adjustment factor is a compression ratio parameter dynamically set based on network bandwidth, used to increase or decrease the audio and video data transmission bitrate.

[0055] The methods for adjusting the bitrate of the fourth audio and video stream data based on network bandwidth, multiple network bandwidth thresholds, and bitrate adjustment coefficients can be as follows: Timestamp interpolation method: This method identifies the time deviation between each audio and video stream using global timestamps, performs frame interpolation to fill in the lagging data stream, and buffers and delays the leading data stream, ensuring all streams correspond to the same timestamp at the same moment, thus achieving time error synchronization. Dynamic frame rate adaptation method: This method dynamically adjusts the playback rate of video frames and the stretching or compression of audio sampling points according to the magnitude of the time error between each audio and video stream, gradually reducing the time deviation without significantly affecting the audiovisual effect, so that the multiple data streams achieve temporal consistency. Keyframe realignment method: This method uses video keyframes (such as I-frames) as reference points, detects the time difference between keyframes of each stream, discards or repeatedly inserts non-keyframes, and uniformly aligns the keyframes of each audio and video stream, quickly reducing the time error to within a preset threshold. The clock drift compensation method calculates the clock drift between the badge recorder and the back-end system in real time, applies corresponding time offset compensation to each audio and video stream, and continuously corrects the time error caused by the difference in the local clock of the device to achieve stable frame-level synchronization.

[0056] Specifically, S602 detects the current network bandwidth in real time, compares it with the preset bandwidth threshold, dynamically adapts and optimizes the transmission bitrate of the fourth voice and video stream data through the bitrate adjustment coefficient, and finally obtains the first voice and video stream data that adapts to the network state.

[0057] As can be seen in this example, by first standardizing and compressing the synchronized audio and video streams, and then adaptively adjusting the bitrate based on the actual network bandwidth, it is possible to ensure that the audio and video data format is uniform and the picture quality is clear, while avoiding stuttering, delays or screen tearing caused by network fluctuations. This makes the ward round live broadcast more stable and accurate, and improves the overall viewing experience.

[0058] In one possible implementation, please refer to Figure 7 , Figure 7 This is a schematic diagram illustrating a process for adjusting the bitrate of the fourth audio / video stream data based on network bandwidth, a preset network bandwidth threshold, and a bitrate adjustment coefficient, as provided in an embodiment of this application. Figure 7 As shown, the bitrate of the fourth audio / video stream data is adjusted based on network bandwidth, a preset network bandwidth threshold, and a bitrate adjustment coefficient, including the following steps: S701, if the network bandwidth is detected to be greater than or equal to the first network bandwidth threshold, adjust the bitrate of the fourth voice and video stream data to the first bitrate.

[0059] Among them, network bandwidth refers to the real-time transmission rate of the network used for the current live broadcast, representing the network's carrying capacity. The first network bandwidth threshold is a high-quality network bandwidth threshold preset by the system, representing an excellent network condition. The first bitrate is a high-definition image quality bitrate set by the system, used to output clear and smooth live broadcast footage of the room check when the network is good.

[0060] Specifically, the S701 detects the current network bandwidth in real time. When the network bandwidth is greater than or equal to the first network bandwidth threshold, it determines that the network environment is good and adjusts the bitrate of the fourth voice and video stream data to the high-quality first bitrate.

[0061] For example, if the first network bandwidth threshold is set to 4Mbps, and the current detected network bandwidth is 5Mbps, which meets the condition of being greater than or equal to the threshold, then the bitrate will be adjusted to the first bitrate of 4Mbps, and a high-definition ward round live broadcast will be output.

[0062] S702, if the network bandwidth is detected to be less than or equal to the second network bandwidth threshold, adjust the bitrate of the fourth voice and video stream data to the second bitrate.

[0063] The second network bandwidth threshold is a low-quality network bandwidth threshold preset by the system, representing a poor network condition. The second bitrate is a low-resolution, low-bitrate setting set by the system to ensure uninterrupted live streaming when the network is poor.

[0064] Specifically, the S702 detects the current network bandwidth in real time. When the network bandwidth is less than or equal to the second network bandwidth threshold, it determines that the network environment is poor. In order to avoid live streaming interruptions, the bitrate is adjusted to the second bitrate with smoothness as the priority.

[0065] For example, if the second network bandwidth threshold is set to 1Mbps, and the current detected network bandwidth is 0.8Mbps, which meets the condition of being less than or equal to the threshold, then the bitrate will be adjusted to the second bitrate of 1Mbps to ensure smooth live streaming without lag.

[0066] S703, if the network bandwidth is detected to be greater than the second network bandwidth threshold and less than the first network bandwidth threshold, adjust the bitrate of the fourth voice and video stream data to be the product of the network bandwidth and the bitrate adjustment coefficient.

[0067] The bitrate adjustment coefficient is a system-preset proportional coefficient used to dynamically calculate a suitable bitrate in a moderate network environment. When the network bandwidth is greater than the second network bandwidth threshold but less than the first network bandwidth threshold, it indicates that the network condition is average.

[0068] Among them, the first network bandwidth threshold is greater than the second network bandwidth threshold, and the first bit rate is greater than the second bit rate.

[0069] Specifically, when the S703 detects that the network bandwidth is between the second network bandwidth threshold and the first network bandwidth threshold, it does not use a fixed bitrate, but dynamically calculates and sets an adaptive bitrate by multiplying the current network bandwidth by the bitrate adjustment coefficient, thus balancing clarity and smoothness.

[0070] The adaptive bitrate formula is as follows: ; in, For network bandwidth, The first bitrate is 8Mbps by default. This is the second bitrate, with a default value of 1Mbps; This is the first network bandwidth threshold, with a default value of 10Mbps; This is the second network bandwidth threshold, with a default value of 1.2Mbps; This is the bitrate adjustment factor, with a default value of 0.8.

[0071] For example, with a first network bandwidth threshold of 10Mbps, a second network bandwidth threshold of 1.2Mbps, a current bandwidth of 2Mbps, and a bitrate adjustment factor of 0.8, the calculated bitrate is 2 × 0.8 = 1.6Mbps, and this bitrate is used for live streaming transmission.

[0072] As can be seen in this example, by adaptively adjusting the bitrate in three bandwidth ranges, automatically improving image quality when the network is good, automatically maintaining smoothness when the network is poor, and dynamically adapting to the intermediate network, the live ward rounds can remain stable and smooth in different network environments, significantly improving the accuracy of the live ward rounds.

[0073] S203, based on the first voice and video stream data and the third user's medical condition dataset, obtain the second voice and video stream data.

[0074] The second audio and video stream data displays the third user's medical condition data, as well as the interaction event data between the third user and the first user.

[0075] The first audio and video stream data refers to the basic audio and video stream data obtained during the ward round after being collected by multiple badge recorders, undergoing frame-level synchronization and encoding processing. The third user refers to the inpatient corresponding to this ward round, who is the subject of the ward round diagnosis and treatment.

[0076] The third-user's medical condition dataset refers to the patient's medical condition-related data obtained from the patient database, including structured data such as diagnostic information, symptoms and signs, examination results, medication use, medical history, and medical orders. Interaction event data refers to the communication content generated between the first and third users during ward rounds, such as consultation dialogues, physical examination feedback, patient complaints, and medical order explanations.

[0077] The second audio and video stream data refers to the enhanced audio and video stream that overlays patient condition data and doctor-patient interaction event information onto the basic ward round audio and video stream.

[0078] Specifically, using the processed first audio and video stream data as the underlying image and sound carrier, combined with the corresponding patient's (i.e., the third user's) medical condition dataset, the key medical condition information of the patient is analyzed and extracted in real time, and the interactive event data between the third user and the first user, such as consultation, physical examination, and explanation of medical orders, are identified and associated simultaneously. The above content is integrated and displayed in the original audio and video stream through methods such as image and text overlay and information embedding, and finally a second audio and video stream data containing ward round images, patient medical condition data, and doctor-patient interaction event data is generated.

[0079] In one possible implementation, obtaining second voice and video stream data based on the first voice and video stream data and the third user's medical condition dataset includes the following steps: By extracting features from the first audio-video stream data, video modal feature data and audio modal feature data are obtained; by extracting features from the third user's medical condition dataset, text modal feature data is obtained; an index is generated based on the video modal feature data, the audio modal feature data, and the text modal feature data; and a second audio-video stream data is obtained based on the index and the first audio-video stream data.

[0080] Among them, video modal feature extraction is based on a pre-trained human pose detection model and object detection model, which processes the synchronized video frames frame by frame to extract the patient's positional change features. (e.g., switching between lying / sitting / standing positions, limb movements), characteristics of the doctor's surgical movements. (e.g., auscultation, physical examination, wound inspection), scene interaction features Generate video modal feature sets , This is a tag for the main speaker.

[0081] The audio modality feature extraction process involves using a voiceprint recognition model to distinguish the speaker (chief physician / patient / assistant medical staff), and then using an ASR model finely tuned for the medical field to transcribe the speech stream into text content. Simultaneously extract keywords from voice commands and features from medical terminology. Generate audio modal feature sets .

[0082] Among them, text modal feature extraction is based on disease dataset. Extract text modal features such as diagnosis type, surgical type, and disease stage. Generate text modal feature sets The SessionID is based on the NTP clock service. After binding a global synchronization timestamp ti (i is the frame number) to all collected audio and video frames, it becomes a globally unique identifier SessionID generated for this ward round session. All data frames are bound to this identifier.

[0083] The extracted video, audio, and text modal features are fused and matched to establish correspondences between features. Based on the feature matching results, a multimodal index is generated for rapid location and related queries. This index enables precise binding of ward round footage, dialogue audio, and patient condition information.

[0084] After obtaining the index, the corresponding patient condition data and doctor-patient interaction event data can be embedded into the corresponding video frames and audio segments of the first audio and video stream data in the form of text overlay and information annotation, based on the generated multimodal index, and finally generate the second audio and video stream data that synchronously displays ward round audio and video, patient condition information and interaction events.

[0085] As can be seen in this example, by extracting video and audio multimodal features from the ward round audio and video streams respectively, extracting text features from the patient's condition data, and then generating an association index based on the three types of features, the patient's condition information can be accurately matched and fused with the ward round audio and video. This enables efficient association and matching of ward round footage, doctor-patient dialogue, and patient's condition data, avoiding information misalignment or omission, and further improving the accuracy of ward round live broadcasts.

[0086] In one possible implementation, generating an index based on the video modal feature data, the audio modal feature data, and the text modal feature data includes: The video modal feature data, audio modal feature data, and text modal feature data are fused based on an attention mechanism to obtain multimodal fused feature data; multiple interaction events and their occurrence times are obtained by performing action detection on the multimodal fused feature data; and an index is generated based on the multiple interaction events and their occurrence times.

[0087] In this process, an attention mechanism is used to perform temporal alignment and fusion of features from three modalities, generating multimodal fusion feature data. The fusion formula is: ; in, , , The weight coefficients for video modal feature data, audio modal feature data, and text modal feature data are respectively, satisfying the following: Based on pre-training optimization for medical ward round scenarios, default values ​​are used. , , .

[0088] Among them, it can be based on multimodal fusion feature data By using a time-series action detection model, candidate time intervals for interactive events can be identified. At the same time, generate initial category labels for the event. (such as events involving body position adjustment, physical examination, explanation of medical orders, and answering questions about the patient's condition).

[0089] The interaction events are then processed hierarchically and structurally to generate a three-level index architecture: the first-level index represents the full ward round live session corresponding to the SessionID; the second-level index represents the event category corresponding to the event classification tag; and the third-level index contains a fine-grained description of each interaction event, including start and end timestamps, keyframe IDs, and associated patient information. In addition, a standardized interaction event description can be generated for each third-level index, and a one-to-one mapping table between index items and corresponding video frames and timestamps is established. .

[0090] The generated real-time interactive event stream index and patient basic information are then synchronously overlaid onto the live stream screen through the live streaming platform and pushed to the client for display.

[0091] As can be seen in this example, by using an attention mechanism to weightedly fuse video modal feature data, audio modal feature data, and text modal feature data, key diagnostic and treatment information can be highlighted and redundant features can be suppressed, resulting in multimodal fusion features with greater representational power. Based on this, action detection can be performed to accurately identify doctor-patient interaction events and their occurrence time, and finally generate a time-aligned index. This enables a deep correlation between ward round audio and video, patient conditions, and doctor-patient interaction behaviors, greatly improving the accuracy and time consistency of the index, and making the matching of subsequent live broadcast footage with patient information more accurate and reliable.

[0092] In one possible implementation, action detection is performed on the multimodal fusion feature data to obtain multiple interaction events and the occurrence times of the multiple interaction events, including: By performing action detection on the multimodal fusion feature data, multiple candidate events are obtained; by calculating the confidence of the multiple candidate events, a confidence score for each candidate event is obtained; events with confidence scores greater than a preset score threshold are identified as interactive events; and the occurrence times of the multiple interactive events are obtained.

[0093] In the process of performing action detection on multimodal fusion feature data and obtaining multiple candidate events, it is necessary to further verify the confidence of these multiple candidate events.

[0094] Specifically, confidence scores are calculated for candidate events based on their confidence level and temporal coherence. The calculation formula is: ; in, The total number of video frames within the candidate time interval; For the first The predicted probability of the event category corresponding to the frame; retain Valid events, among which, The preset score threshold is 0.75 by default.

[0095] As can be seen, in this example, by performing action detection on multimodal fusion features to obtain candidate events, and then calculating confidence scores for each candidate event, events with scores higher than a preset threshold are selected as valid interaction events and their occurrence time is recorded. This can effectively filter out invalid behaviors such as false detection and interference, accurately extract the real doctor-patient interaction content during ward rounds, ensure the reliability and temporal accuracy of interaction event recognition, and provide solid support for the subsequent generation of high-quality indexes.

[0096] S204, Live broadcast based on the second voice and video stream data.

[0097] After obtaining the second audio and video stream data, it can be further processed, including video distortion correction and audio noise reduction, before being broadcast live.

[0098] In addition to the above, users can initiate a search command Q through the client. After receiving the search command Q, the user first processes the search command... Perform Chinese word segmentation, medical terminology normalization, and semantic feature extraction to generate retrieval feature vectors. Then, semantic features are extracted from all event index entries in the index database obtained in S203 to generate an index feature vector set. ,in This is the index item number.

[0099] Calculate the cosine similarity between the retrieval feature vector and the feature vectors of each index. The similarity calculation formula is: ; in, To retrieve the feature vector and the first The dot product of the indexed feature vectors; To retrieve the magnitude of the feature vector; For the first The magnitude of the index feature vector.

[0100] Based on similarity The index items are sorted from highest to lowest, and target index items that meet the search criteria are selected based on the mapping table. Match the start and end timestamps of the corresponding video segments to generate playback location information that allows direct navigation.

[0101] Please refer to Figure 8 , Figure 8 This is a schematic diagram of a live streaming interface provided in an embodiment of this application, such as... Figure 8 As shown, the live streaming interface is vertically laid out, displaying three separate areas: the first area, the second area, and the third area. The first area displays the third user's basic information, such as name, consultation time, and condition. The second area displays the audio and video stream captured by the first user's first badge recorder, and the third area displays the audio and video stream captured by the second user's second badge recorder. The arrangement and size of these areas can be customized by the user. The third user's basic information can also be displayed directly within the captured audio stream, rather than in a separate area.

[0102] It is evident that upon receiving the command to start the ward round live stream, the system acquires multiple video and audio streams from the name tag recorders worn by the ward round personnel in real time. These multiple audio and video data streams are then integrated into a unified audio and video stream. Combined with the corresponding patient's medical data set, patient medical data and doctor-patient interaction information are overlaid into the live stream. Finally, the ward round live stream is conducted based on the fused audio and video stream, achieving multi-view audio and video acquisition, synchronous presentation of patient information, and visualization of interactive content. This effectively avoids problems such as incomplete information from a single video stream, asynchronous patient information, and missing key interactive content, thus significantly improving the accuracy of the ward round live stream.

[0103] Please see Figure 9 , Figure 9 This is a schematic diagram of the structure of a ward round live broadcast device based on a name tag recorder system provided in an embodiment of this application, as shown below. Figure 9 As shown, the ward round live streaming device 900 based on the name tag recorder system includes: The first processing unit 901 is configured to, in response to receiving a first instruction to instruct the start of a live ward round, acquire a first video stream and a first audio stream collected by a first badge recorder of a first user, and a second video stream and a second audio stream collected by a second badge recorder of a second user; The second processing unit 902 is used to obtain first voice and video stream data based on the first video stream, the first audio stream, the second video stream, and the second audio stream; The third processing unit 903 is used to obtain second voice and video stream data based on the first voice and video stream data and the third user's medical condition dataset. The second voice and video stream data displays the third user's medical condition data and the interaction event data between the third user and the first user. The fourth processing unit 904 is used to perform live streaming based on the second voice and video stream data.

[0104] In one possible implementation, in obtaining the first audio-video stream data based on the first video stream, the first audio stream, the second video stream, and the second audio stream, the second processing unit 902 is specifically configured to: obtain the third audio-video stream data by performing frame-level synchronization on the first video stream, the first audio stream, the second video stream, and the second audio stream; and obtain the first audio-video stream data by encoding the third audio-video stream data.

[0105] In one possible implementation, in obtaining third audio-video stream data by performing frame-level synchronization on the first video stream, the first audio stream, the second video stream, and the second audio stream, the second processing unit 902 is specifically used to: perform time error synchronization on the first video stream, the first audio stream, the second video stream, and the second audio stream based on a global timestamp; detect that the time error is less than a preset time threshold, determine that frame-level synchronization is complete, and obtain third audio-video stream data.

[0106] In one possible implementation, in obtaining first audio and video stream data by encoding the third audio and video stream data, the second processing unit 902 is specifically used to: encode the third audio and video stream data based on a preset encoding method to obtain fourth audio and video stream data; and adjust the bitrate of the fourth audio and video stream data based on network bandwidth, multiple network bandwidth thresholds, and bitrate adjustment coefficients to obtain first audio and video stream data.

[0107] In one possible implementation, regarding adjusting the bitrate of the fourth audio / video stream data based on network bandwidth, a preset network bandwidth threshold, and a bitrate adjustment coefficient, the second processing unit 902 is specifically configured to: adjust the bitrate of the fourth audio / video stream data to a first bitrate if the network bandwidth is detected to be greater than or equal to a first network bandwidth threshold; adjust the bitrate of the fourth audio / video stream data to a second bitrate if the network bandwidth is detected to be less than or equal to a second network bandwidth threshold; and adjust the bitrate of the fourth audio / video stream data to the product of the network bandwidth and the bitrate adjustment coefficient if the network bandwidth is detected to be greater than the second network bandwidth threshold and less than the first network bandwidth threshold; wherein the first network bandwidth threshold is greater than the second network bandwidth threshold, and the first bitrate is greater than the second bitrate.

[0108] In one possible implementation, in obtaining the second audio-video stream data based on the first audio-video stream data and the third user's medical condition dataset, the third processing unit 903 is specifically configured to: extract features from the first audio-video stream data to obtain video modal feature data and audio modal feature data; extract features from the third user's medical condition dataset to obtain text modal feature data; generate an index based on the video modal feature data, the audio modal feature data, and the text modal feature data; and obtain the second audio-video stream data based on the index and the first audio-video stream data.

[0109] In one possible implementation, in generating an index based on the video modal feature data, the audio modal feature data, and the text modal feature data, the third processing unit 903 is specifically configured to: fuse the video modal feature data, the audio modal feature data, and the text modal feature data based on an attention mechanism to obtain multimodal fused feature data; obtain multiple interaction events and the occurrence times of the multiple interaction events by performing action detection on the multimodal fused feature data; and generate an index based on the multiple interaction events and the occurrence times of the multiple interaction events.

[0110] In one possible implementation, in obtaining multiple interaction events and their occurrence times by performing action detection on the multimodal fusion feature data, the third processing unit 903 is specifically configured to: obtain multiple candidate events by performing action detection on the multimodal fusion feature data; obtain confidence scores for the multiple candidate events by calculating confidence scores for the multiple candidate events; determine that the events among the multiple candidate events with confidence scores greater than a preset score threshold are interaction events; and acquire the multiple interaction events and their occurrence times.

[0111] It is worth noting that the specific functional implementation of the ward round live broadcast device 900 based on the badge recorder system is described above. Figure 2 The description of the ward round live broadcast method based on the badge recorder system illustrates that, for example, the first processing unit 901 is used to execute the relevant content of S201, the second processing unit 902 is used to execute the relevant content of S202, the third processing unit 903 is used to execute the relevant content of S203, and the fourth processing unit 904 is used to execute the relevant content of S204. Each unit or module in the ward round live broadcast device 900 based on the badge recorder system can be individually or entirely merged into one or more other units or modules, or some of the units or modules can be further divided into multiple functionally smaller units or modules. This achieves the same operation without affecting the technical effect of the embodiments of the present invention. The above-mentioned units or modules are divided according to logical functions. In practical applications, the function of one unit (or module) is implemented by multiple units (or modules), or the function of multiple units (or modules) is implemented by one unit (or module).

[0112] Based on the description of the above method embodiments and related device embodiments, please refer to... Figure 10 , Figure 10 This is a schematic diagram of the structure of a computer provided in an embodiment of this application. Figure 10 The computer 1000 shown includes a processor 1001, a memory 1002, a communication interface 1003, and a bus 1004. The processor 1001, memory 1002, and communication interface 1003 are interconnected via the bus 1004.

[0113] Optionally, the memory 1002 can be ROM, static storage device, dynamic storage device, or RAM.

[0114] Memory 1002 can store executable program code. When the executable program code stored in memory 1002 is executed by processor 1001, processor 1001 and communication interface 1003 are used for execution. Figure 2 The various steps of the ward round live broadcast method based on the badge recorder system in the illustrated embodiment.

[0115] The processor 1001 employs a general-purpose CPU, microprocessor, application-specific integrated circuit (ASIC), GPU, or one or more integrated circuits to execute relevant programs to perform the ward round live broadcast method based on the badge recorder system of this application's method embodiments.

[0116] The processor 1001 can also be an integrated circuit chip with signal processing capabilities. In implementation, each step of the ward round live-streaming method based on the badge recorder system of this application can be completed through the integrated logic circuits in the hardware of the processor 1001 or through software instructions. Optionally, the processor 1001 can be a general-purpose processor, DSP, ASIC, FPGA, or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The processor can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor is a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. Optional software modules are located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 1002. Processor 1001 reads the information in memory 1002 and, in conjunction with its hardware, performs the functions required by the modules included in the ward round live broadcast device 900 based on the badge recorder system of this application embodiment, or executes the ward round live broadcast method based on the badge recorder system of this application method embodiment.

[0117] The communication interface 1003 uses transceiver-related devices such as, but not limited to, transceivers.

[0118] Bus 1004 may include a pathway for transmitting information between various components of computer 1000 (e.g., memory 1002, processor 1001, communication interface 1003).

[0119] It should be noted that, although Figure 10 The computer 1000 shown only illustrates the memory, processor, and communication interface. However, those skilled in the art should understand that in specific implementations, the computer 1000 may also include other devices necessary for normal operation. Furthermore, depending on specific needs, those skilled in the art should understand that the computer 1000 may also include hardware devices for implementing other additional functions. Moreover, those skilled in the art should understand that the computer 1000 may only include the devices necessary for implementing the embodiments of this application, and may not necessarily include... Figure 10 All the devices shown.

[0120] This application provides a computer-readable storage medium storing a computer program for electronic data interchange. The computer program includes execution instructions for performing some or all of the steps of any of the ward round live broadcast methods based on a badge recorder system as described in the above embodiments of the ward round live broadcast method based on a badge recorder system. The computer includes an electronic client device.

[0121] This application provides a computer program product, which includes a computer program operable to enable the computer to perform some or all of the steps of any of the ward round live broadcast methods based on a badge recorder system described in the above method embodiments. The computer program product may be a software installation package.

[0122] It should be noted that, for the sake of simplicity, each of the aforementioned embodiments of the ward round live streaming method based on the name tag recorder system is described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps can be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to this application.

[0123] The embodiments of this application have been described in detail above. Specific examples have been used to illustrate the principle and implementation of a ward round live broadcast method and related devices based on a badge recorder system. The description of the above embodiments is only for the purpose of helping to understand the method and its core ideas. At the same time, for those skilled in the art, based on the ideas of a ward round live broadcast method and related devices based on a badge recorder system, there will be changes in the specific implementation and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

[0124] This application is described with reference to flowchart illustrations and / or block diagrams of methods, hardware products, and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0125] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. Memory may include: flash drives, read-only memory (ROM), random access memory (RAM), hard disks or optical disks, etc.

[0126] Although this application has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the accompanying drawings, disclosure, and appended claims, will understand and implement other variations of the disclosed embodiments in carrying out the claimed application. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple instances. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce a good effect.

[0127] Those skilled in the art will understand that all or part of the steps in the various methods of any of the above-described embodiments of the ward round live broadcast method based on the name tag recorder system can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage device, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0128] It is understood that any product that is controlled or configured to execute the processing method described in the flowchart of the embodiment of the ward round live broadcast method based on the name badge recorder system of this application, such as the device and computer program product of the above flowchart, falls within the scope of the related products described in this application.

[0129] Obviously, those skilled in the art can make various modifications and variations to the ward round live broadcast method and related devices based on the name badge recorder system provided in this application without departing from the spirit and scope of this application. Therefore, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application also intends to include these modifications and variations.

Claims

1. A method for live-streaming ward rounds based on a badge recorder system, characterized in that, A server for a name badge recorder system, the server communicating with at least one name badge recorder of the name badge recorder system, the method comprising: In response to receiving a first instruction to initiate a live ward round, the system acquires a first video stream and a first audio stream captured by a first badge recorder of a first user, and a second video stream and a second audio stream captured by a second badge recorder of a second user. Based on the first video stream, the first audio stream, the second video stream, and the second audio stream, the first voice and video stream data is obtained; Based on the first audio and video stream data and the third user's medical condition dataset, a second audio and video stream data is obtained. The second audio and video stream data displays the third user's medical condition data and the interaction event data between the third user and the first user. Live streaming is conducted based on the second audio and video stream data.

2. The method as described in claim 1, characterized in that, The step of obtaining the first voice-video stream data based on the first video stream, the first audio stream, the second video stream, and the second audio stream includes: The third voice and video stream data is obtained by performing frame-level synchronization on the first video stream, the first audio stream, the second video stream, and the second audio stream. The first audio and video stream data is obtained by encoding the third audio and video stream data.

3. The method as described in claim 2, characterized in that, The process of obtaining third audio-video stream data by performing frame-level synchronization on the first video stream, the first audio stream, the second video stream, and the second audio stream includes: Synchronize the time error of the first video stream, the first audio stream, the second video stream, and the second audio stream based on the global timestamp; If the time error is detected to be less than a preset time threshold, it is determined that frame-level synchronization is complete, and the third voice and video stream data is obtained.

4. The method as described in claim 2, characterized in that, The process of encoding the third audio-video stream data to obtain the first audio-video stream data includes: The third audio and video stream data is encoded based on a preset encoding method to obtain the fourth audio and video stream data. Based on network bandwidth, multiple network bandwidth thresholds, and bitrate adjustment coefficients, the bitrate of the fourth voice and video stream data is adjusted to obtain the first voice and video stream data.

5. The method as described in claim 4, characterized in that, The adjustment of the bitrate of the fourth audio / video stream data based on network bandwidth, a preset network bandwidth threshold, and a bitrate adjustment coefficient includes: If the network bandwidth is detected to be greater than or equal to the first network bandwidth threshold, the bitrate of the fourth voice and video stream data is adjusted to the first bitrate. If the network bandwidth is detected to be less than or equal to the second network bandwidth threshold, the bitrate of the fourth voice and video stream data is adjusted to the second bitrate. If the network bandwidth is detected to be greater than the second network bandwidth threshold and less than the first network bandwidth threshold, the bitrate of the fourth voice and video stream data is adjusted to be the product of the network bandwidth and the bitrate adjustment coefficient. Wherein, the first network bandwidth threshold is greater than the second network bandwidth threshold, and the first bit rate is greater than the second bit rate.

6. The method according to any one of claims 1-5, characterized in that, The step of obtaining the second voice and video stream data based on the first voice and video stream data and the third user's medical condition dataset includes: By extracting features from the first audio and video stream data, video modal feature data and audio modal feature data are obtained; By extracting features from the third user's medical condition dataset, text modal feature data is obtained; An index is generated based on the video modal feature data, the audio modal feature data, and the text modal feature data; Based on the index and the first audio / video stream data, the second audio / video stream data is obtained.

7. The method as described in claim 6, characterized in that, The process of generating an index based on the video modal feature data, the audio modal feature data, and the text modal feature data includes: The video modal feature data, the audio modal feature data, and the text modal feature data are fused based on an attention mechanism to obtain multimodal fused feature data; By performing action detection on the multimodal fusion feature data, multiple interaction events and the occurrence times of the multiple interaction events are obtained; An index is generated based on the multiple interaction events and the occurrence time of the multiple interaction events.

8. The method as described in claim 7, characterized in that, The step of performing action detection on the multimodal fused feature data to obtain multiple interaction events and the occurrence times of the multiple interaction events includes: By performing action detection on the multimodal fusion feature data, multiple candidate events are obtained; The confidence scores of the multiple candidate events are obtained by calculating the confidence scores of the multiple candidate events. Among the multiple candidate events, events with a confidence score greater than a preset score threshold are identified as interactive events; Obtain multiple interaction events and the occurrence time of the multiple interaction events.

9. An electronic device, characterized in that, The device includes: The system includes a memory, a processor, and executable program code stored in the memory and executable on the processor, wherein the processor executes the executable program code to perform the steps of the ward round live broadcast method based on a badge recorder system as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores executable program code, which includes execution instructions for performing the steps of the ward round live broadcast method based on the badge recorder system as described in any one of claims 1-8.

Images