Method and system for retrieving diagnosis data in multidisciplinary consultation scenarios

Abstract

The application discloses a kind of multidisciplinary consultation scene in diagnosis and treatment data call method and system, method includes: response to the first operation permission certificate of current speaker's medical staff sent by first intelligent chest card and instruction voice data, instruction voice data is carried out voice recognition, and instruction voice text is obtained, and the first operation permission certificate is used to show that access and operation permission of intelligent chest card system and patient diagnosis and treatment data have;According to the first operation permission certificate, the target diagnosis and treatment operation corresponding to the diagnosis and treatment audio-video clip of target patient in target diagnosis and treatment link indicated by instruction voice text is acquired;Diagnosis and treatment audio-video clip is pushed to meeting display terminal and plays.The application can realize the instant, accurate call of audio-video clip corresponding to specified diagnosis and treatment operation of target patient in hospital multidisciplinary consultation scene, and improve the efficiency and flexibility of consultation.

Description

Technical Field

[0001] This application relates to the fields of medical diagnostic technology, voice analysis technology, or electronic digital data processing technology, and in particular to a method and system for retrieving diagnostic data in a multidisciplinary consultation scenario. Background Technology

[0002] In hospital multidisciplinary team (MDT) consultations and other meeting scenarios, medical staff need to quickly retrieve patients' past medical audio and video records for disease analysis and treatment plan discussions. Traditional methods require dedicated personnel to prepare the data in advance and manually retrieve the medical records, which cannot be accessed instantly, seriously delaying the discussion process and resulting in extremely poor flexibility.

[0003] Current diagnostic data retrieval strategies require manual input of multiple dimensions such as patient ID and time range, and cannot accurately locate video clips of specific diagnostic and treatment procedures such as extubation using spoken voice commands. This results in high retrieval barriers and insufficient accuracy. Furthermore, medical staff must rely on fixed terminals such as keyboards and mice to retrieve diagnostic data during discussions, hindering seamless integration of case discussions and data review, and posing a risk of unauthorized access to medical data. Additionally, the current solution is not designed for mobile interactive devices such as smart badges, and cannot support the real-time retrieval needs of consultation scenarios with independent access control and data processing capabilities.

[0004] It is evident that existing diagnostic and treatment data retrieval solutions cannot meet the actual needs of hospitals in multidisciplinary consultation scenarios for the rapid, accurate, safe, and convenient retrieval of audio and video clips of specific diagnostic and treatment procedures for patients, which seriously affects consultation efficiency and the quality of diagnostic and treatment decisions. Summary of the Invention

[0005] In view of this, this application provides a method and system for retrieving medical data in a multidisciplinary consultation scenario. The system collects and preprocesses the voice commands of medical staff using a smart badge, then uploads them. A server then performs voice recognition, intent parsing, entity extraction, and precise audio-visual matching and segment extraction. Finally, the target medical audio-visual segment is pushed to the conference display terminal for playback. This allows for precise location of video segments of specific medical procedures through spoken commands, enabling voice-triggered instant retrieval of medical audio-visual materials in multidisciplinary hospital consultation scenarios. No prior preparation of materials is required, significantly improving the efficiency and flexibility of case discussions. Furthermore, the system enables end-to-end operation without fixed terminals based on a smart badge wearable terminal, while ensuring the security and compliance of medical data access.

[0006] In a first aspect, embodiments of this application provide a method for retrieving diagnostic and treatment data in a multidisciplinary consultation scenario, applied to a server of a smart badge system. The smart badge system includes smart badges worn by participating medical personnel, a conference display terminal, and the server, wherein the server is communicatively connected to both the conference display terminal and the smart badges. The method includes: In response to the first smart badge sending the first operation permission certificate and instruction voice data of the currently speaking medical staff, the instruction voice data is subjected to speech recognition to obtain instruction voice text. The first operation permission certificate is used to represent that the medical staff has access and operation permissions to the smart badge system and patient diagnosis and treatment data. Based on the first operation permission certificate, obtain the audio and video clips corresponding to the target diagnosis and treatment operation of the target patient in the target diagnosis and treatment process indicated by the instruction voice text; The diagnostic audio and video clips are pushed to the conference display terminal for playback.

[0007] Secondly, this application also provides an intelligent name tag system, which includes intelligent name tags worn by participating medical personnel, a conference display terminal, and a server. The server is communicatively connected to the conference display terminal and the intelligent name tag, respectively. The server is used to execute the steps in the first aspect of the embodiments of this application.

[0008] Thirdly, embodiments of this application provide an electronic device, including a processing module, a memory, a communication interface, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the processing module, and the programs include instructions for performing the steps in the first aspect of embodiments of this application.

[0009] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program for electronic data interchange, wherein the computer program causes a computer to perform some or all of the steps described in the first aspect of embodiments of this application.

[0010] Fifthly, embodiments of this application provide a computer program product, wherein the computer program product includes a non-transitory computer-readable storage medium storing a computer program operable to cause a computer to perform some or all of the steps described in the first aspect of embodiments of this application. The computer program product may be a software installation package.

[0011] As can be seen, the method and system for retrieving diagnostic and treatment data in a multidisciplinary consultation scenario provided in this application, in response to the first operation permission certificate and instruction voice data of the currently speaking medical staff sent by the first smart badge, the server performs speech recognition on the instruction voice data to obtain the instruction voice text; based on the first operation permission certificate, it retrieves the audio and video segment corresponding to the target diagnostic and treatment operation of the target patient in the target diagnostic and treatment stage indicated by the instruction voice text; and pushes the audio and video segment to the conference display terminal for playback. Thus, compared to the existing hospital information system's diagnostic and treatment audio and video retrieval strategy, which requires manual input of multiple dimensions such as patient ID and time interval, this application can accurately locate video segments of specific diagnostic and treatment operations through spoken instructions, realizing voice-triggered instant retrieval of diagnostic and treatment audio and video in multidisciplinary consultation scenarios in hospitals, without the need for prior preparation of materials, significantly improving the efficiency and flexibility of case discussions, and enabling full-process operation without fixed terminals based on the smart badge wearable terminal, while ensuring the security and compliance of medical data access. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a schematic diagram of the architecture of an intelligent name tag system provided in an embodiment of this application; Figure 2 This is a flowchart illustrating a method for retrieving diagnostic data in a multidisciplinary consultation scenario provided in an embodiment of this application; Figure 3 This is a flowchart illustrating how to determine audio and video clips related to medical treatment based on medical-related entities, as provided in an embodiment of this application. Figure 4 This is a flowchart illustrating how to determine diagnostic audio and video segments based on a set of audio and video recordings, as provided in an embodiment of this application. Figure 5 This is a schematic diagram of a smart name tag provided in an embodiment of this application; Figure 6 This is a schematic diagram of a multidisciplinary consultation scenario provided in an embodiment of this application; Figure 7 This is a schematic diagram of another multidisciplinary consultation scenario provided in an embodiment of this application; Figure 8 This is a schematic diagram of the status of a smart badge in a multidisciplinary consultation scenario provided by an embodiment of this application; Figure 9This is a block diagram of the functional units of an intelligent name tag system provided in an embodiment of this application; Figure 10 This is a structural block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0014] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0015] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0016] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article indicates that the preceding and following related objects have an "or" relationship.

[0017] In this application's embodiments, "multiple" refers to two or more. In this application's embodiments, "connection" refers to various connection methods, such as direct or indirect connections, to achieve communication between devices; this application's embodiments do not impose any limitations on this.

[0018] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0019] The following describes the relevant content, concepts, meanings, technical issues, technical solutions, and beneficial effects involved in the embodiments of this application.

[0020] Current diagnostic data retrieval strategies require manual input of multiple dimensions such as patient ID and time range, and cannot accurately locate video clips of specific diagnostic and treatment procedures such as extubation using spoken voice commands. This results in high retrieval barriers and insufficient accuracy. Furthermore, medical staff must rely on fixed terminals such as keyboards and mice to retrieve diagnostic data during discussions, hindering seamless integration of case discussions and data review, and posing a risk of unauthorized access to medical data. Additionally, the current solution is not designed for mobile interactive devices such as smart badges, and cannot support the real-time retrieval needs of consultation scenarios with independent access control and data processing capabilities.

[0021] It is evident that existing diagnostic and treatment data retrieval solutions cannot meet the actual needs of hospitals in multidisciplinary consultation scenarios for the rapid, accurate, safe, and convenient retrieval of audio and video clips of specific diagnostic and treatment procedures for patients, which seriously affects consultation efficiency and the quality of diagnostic and treatment decisions.

[0022] To address the aforementioned issues, this application provides a method and system for retrieving medical data in a multidisciplinary consultation scenario. The system collects and pre-processes the voice commands from medical staff using a smart badge, then uploads them. A server performs voice recognition, intent parsing, entity extraction, and precise audio-visual matching and segment extraction. Finally, the target medical audio-visual segment is pushed to the conference display terminal for playback. This allows for precise location of video segments of specific medical procedures through spoken commands, enabling voice-triggered instant retrieval of medical audio-visual materials in multidisciplinary hospital consultations. No prior preparation of materials is required, significantly improving the efficiency and flexibility of case discussions. Furthermore, the system enables end-to-end operation without fixed terminals based on a smart badge-wearable terminal, while ensuring the security and compliance of medical data access.

[0023] First, combined Figure 1 The intelligent name tag system in the embodiments of this application will be described. Figure 1 This is a schematic diagram of the architecture of an intelligent name tag system provided in an embodiment of this application, such as... Figure 1 As shown, the intelligent name tag system 100 includes: a server 110, an intelligent name tag 120, and a conference display terminal 130. The server 110 is communicatively connected to the intelligent name tag 120 and the conference display terminal 130, respectively.

[0024] Among them, server 110 is the core data processing and control center, relying on the underlying data support of the hospital information system, and integrates speech recognition module 111, intent recognition module 112, entity recognition module 113, patient query module 114, audio and video index module 115, and audio and video segment positioning module 116.

[0025] Specifically, the speech recognition module 111 is used to transcribe the command speech data uploaded by the smart badge 120 into a medical scenario-specific format, generating standardized command speech text; the intent recognition module 112, based on a pre-trained semantic intent recognition model, identifies the core operational intent of medical staff; the entity recognition module 113, through a medical semantic entity recognition model, accurately extracts multiple medical-related entities such as patient identifiers, treatment procedures, and treatment operations from the command text; the patient query module 114, based on the extracted entity information and operation permission credentials, matches the full-process treatment records of the target patient in the patient treatment information database, and filters out the audio and video record set corresponding to the target treatment procedure; the audio and video indexing module 115, relying on a pre-built treatment operation video index library, performs fine-grained matching on candidate audio and video records to locate the audio and video file and start and end timestamps corresponding to the target treatment operation; the audio and video segment positioning module 116, based on the timestamp, extracts the audio and video segment corresponding to the target treatment operation from the treatment audio and video database, completes secondary permission verification and privacy desensitization, and pushes the audio and video segment to the conference display terminal 130.

[0026] The smart badge 120 is a wearable interactive terminal worn by medical staff, integrating a voice acquisition module 121, a processing module 122, a display screen 123, and a speaker 124. Specifically, the smart badge 120 is typically worn on the chest of medical staff, providing a convenient, terminal-free operation entry point for multidisciplinary consultation scenarios: the voice acquisition module 121 acquires the spoken commands of medical staff through a MEMS (Micro-Electro-Mechanical System) microphone array, while simultaneously performing environmental noise suppression and voice enhancement to extract clean and effective command voice; the processing module 122 performs preprocessing on the acquired voice, such as endpoint detection and frame segmentation and windowing, and generates the operation permission certificate of the currently speaking medical staff by combining voiceprint verification, and uploads the command voice data and permission certificate to the server 110; the display screen 123 is used to provide visual feedback on the operation status to medical staff; and the speaker 124 is used for voice interaction guidance and operation result broadcasting.

[0027] Among them, the conference display terminal 130 is a terminal for displaying medical data. It is deployed in the multidisciplinary consultation conference room to receive medical audio and video clips pushed by the server 110, complete decoding, rendering and full-screen playback, and support participating medical staff to conduct case discussions and plan evaluations based on the audio and video content, so as to achieve seamless connection between medical data and conference discussions.

[0028] As can be seen, in this embodiment, the intelligent badge system 100, as a sub-functional system of the hospital information system, is built upon the mature patient diagnosis and treatment data system, access control system, and audio-visual database resources of the hospital information system. It uses the intelligent badge 120 as the front-end mobile interaction carrier, the server 110 as the core processing hub, and the conference display terminal 130 as the display terminal, forming a lightweight, convenient, and wearable data retrieval and interaction subsystem for multidisciplinary consultation scenarios. It achieves voice-triggered retrieval, precise positioning, and real-time push of diagnosis and treatment audio-visual data in consultation scenarios, solving the technical problems of cumbersome operation, insufficient accuracy, and security risks associated with traditional retrieval methods.

[0029] The following is combined with Figure 2 This application provides a further explanation of the method for retrieving diagnostic and treatment data in a multidisciplinary consultation scenario, as provided in the embodiments of this application. Please refer to [link to relevant documentation]. Figure 2 , Figure 2 This is a flowchart illustrating a method for retrieving diagnostic and treatment data in a multidisciplinary consultation scenario, as provided in an embodiment of this application. Figure 1 Server 110 in the middle, such as Figure 2 As shown, the method includes the following steps: Step S210: In response to the first operation permission certificate and instruction voice data of the currently speaking medical staff sent by the first smart badge, perform speech recognition on the instruction voice data to obtain the instruction voice text.

[0030] In this embodiment, during a multidisciplinary consultation, multiple participating medical staff can each wear a corresponding smart badge. All smart badges act as distributed audio pickup terminals, synchronously collecting audio data from the consultation room and uploading it to the server in real time after preprocessing. The server performs signal-to-noise ratio evaluation, speech enhancement processing, and sound source localization analysis on the multiple audio streams from the smart badges, selecting the audio stream with the best speech quality and clearest instructions for processing. During voice command recognition and execution, the server adopts a speech exclusivity strategy, processing only the best audio data stream, temporarily blocking interference from other audio streams, avoiding interference from parallel audio data, and preventing misrecognition of instructions due to multiple people speaking simultaneously or environmental noise.

[0031] Furthermore, smart badges can also adopt an identity binding mode, pre-linked to an individual medical staff member's employee ID, account, or biometric data, permanently associating them with the corresponding medical staff member's identity information. This ensures that during multidisciplinary consultations, only the specific voice commands and actions of the bound medical staff member can be collected, avoiding identity confusion and permission errors caused by multiple personnel using the same badge, and further enhancing the security of command tracing and data access. The smart badge can also dynamically switch voice collection modes. For example, under normal circumstances, it can collect any audio data within the scene; when the user presses the dedicated collection button, only the specific audio data of the bound medical staff member is collected. This application does not impose any restrictions on this.

[0032] In one possible embodiment, the first smart badge is used to perform the following operations: receiving raw voice data of the medical staff currently speaking in the multidisciplinary consultation scenario; performing noise removal and non-voice segment removal processing on the raw voice data to extract a complete instruction voice segment, and identifying the instruction voice segment as the instruction voice data; performing voiceprint similarity matching on the instruction voice segment according to a pre-stored authorized medical staff voiceprint database to determine the identity identifier of the currently speaking medical staff and the first operation permission credential, wherein the authorized medical staff voiceprint database pre-stores the identity identifiers and operation permission credentials of multiple authorized medical staff; and sending the instruction voice data and the first operation permission credential of the currently speaking medical staff to the server.

[0033] Specifically, the first smart badge acquires multi-channel raw voice data through a built-in MEMS microphone array, which can be represented as a vector composed of multiple time-domain audio signals: ; Where t is the time sampling point, and M is the number of microphone channels. Let T represent the raw signal collected by the m-th microphone, and let T denote the vector transpose operation, used to construct a column vector from a row vector.

[0034] To achieve noise removal from the original speech data, the First Smart Badge employs a beamforming algorithm to spatially filter the multi-channel signals, enhancing the target speech from the direction of the medical staff's voice while suppressing ambient noise and reverberation interference, resulting in a noise-reduced and enhanced speech signal, as shown in the formula below: ; Where t is the time sampling point, Let be the weighting coefficient vector of the beamformer, H represent the conjugate transpose operation of the vector, and y(t) represent the single-channel noise-reduced and enhanced speech signal obtained after beamforming spatial filtering.

[0035] Furthermore, based on this, non-speech segments are removed from the enhanced speech signal. Speech endpoint detection (VAD) is used to distinguish between valid speech segments and silent / noisy segments, thereby extracting complete and continuous instruction speech segments, which are then identified as instruction speech data.

[0036] To identify the current medical staff member and their primary access authorization, the first smart badge extracts MFCC features and their differential features from the aforementioned instruction speech segment. A fixed-dimensional speaker embedding vector is then obtained through a pre-trained voiceprint model, as shown in the formula below: ; Where e represents the real-time speaker embedding vector of the currently speaking medical staff member, used to uniquely represent the voiceprint biometrics of that medical staff member, and R represents the set of real numbers. This represents a d-dimensional real vector space, where d is the dimension of the voiceprint embedding vector.

[0037] Furthermore, the real-time speaker embedding vector *e* of the medical staff is matched with voiceprint templates pre-stored in the authorized medical staff voiceprint database. Identity verification is achieved by calculating cosine similarity. When the similarity meets a preset threshold, identity confirmation is completed, and the corresponding operation permission credential is retrieved from the voiceprint database. Finally, the first smart badge sends the instruction voice data and the corresponding first operation permission credential to the server. The calculation of cosine similarity for identity verification can be seen in the following formula: ; in, This represents the cosine similarity between the current speaker's voiceprint and the authorized person's voiceprint, used to quantify the degree of similarity between the two, with a value range of []. [1,1], the closer the value is to 1, the more consistent the voiceprint features are. This represents the pre-stored voiceprint template vector of the k-th authorized medical staff member in the authorized medical staff voiceprint database. Describes the L2 norm of vector e. Representing vectors The L2 norm.

[0038] The first access permission credential is used to represent the user's access and operation permissions to the smart badge system and patient medical data. Specifically, the authorized medical personnel voiceprint database pre-stores the identity identifiers of multiple authorized medical personnel and their corresponding operation permission information. After the smart badge system verifies and successfully matches the voiceprint identity of the currently speaking medical personnel, it directly retrieves the pre-stored operation permission information corresponding to that medical personnel as the first access permission credential. This credential contains the medical personnel's identity identifier, department affiliation, professional title, patient data access scope, and medical operation authorization type, serving as the legal basis for medical personnel to access the patient medical audio and video database and perform medical data retrieval operations.

[0039] It should be noted that this application only provides one method for obtaining operation permission credentials for medical staff. Alternatively, after voiceprint identity matching and identity authentication are completed, the smart badge can temporarily generate a time-sensitive dynamic permission credential, such as a digital permission token. This credential is generated through encryption and bound to the current smart badge device, and the server performs subsequent access and retrieval operations for medical data based on this dynamic permission credential.

[0040] Furthermore, after receiving the first operation permission certificate and command voice data, the server can continue to verify the identity of the current speaker through voiceprint recognition technology, and link the hospital information system to complete the operation permission verification based on the identified identity information. Only the corresponding patient's diagnosis and treatment data can be retrieved within the authorized scope, thereby realizing hierarchical control and secure access to medical data and preventing unauthorized personnel from exceeding their authority.

[0041] In one possible embodiment, the step of performing speech recognition on the instruction speech data to obtain instruction speech text includes: using an end-to-end automatic speech recognition (ASR) model specifically fine-tuned for multidisciplinary medical consultation scenarios to perform speech-to-text processing on the instruction speech data to obtain an original text sequence; performing standardization preprocessing on the original text sequence, sequentially performing operations such as removing interjections, stop words, and correcting homophone errors, and standardizing the medical terms in the original text sequence according to the hospital's unified medical terminology expression standards to eliminate textual ambiguity caused by colloquial expressions, thereby obtaining the standardized instruction speech text.

[0042] The end-to-end automatic speech recognition (ASR) model is trained and optimized using medical-specific corpora, which focuses on improving the recognition accuracy of medical terms related to case discussions and diagnostic procedures, as well as colloquial expressions. This effectively solves the technical problems of general speech recognition models in medical scenarios, such as bias in the recognition of medical terms and inaccurate semantic understanding.

[0043] Specifically, the training and processing of an end-to-end automatic speech recognition (ASR) model may include the following steps: First, acoustic features are extracted from the command speech data, and the log-Mel spectrum features are calculated and used as the model input sequence: ; in, The sequence represents the acoustic features of the speech, where T is the total number of speech frames. ∈ Let D be the log-Mel feature of the T-th frame. Represents the D-dimensional real number feature space; Next, the speech acoustic feature sequence The encoder of the input end-to-end automatic speech recognition (ASR) model performs context modeling and temporal coding, as shown in the following formula: Enc(X, ); Where Enc() is the encoder network mapping function, H= The high-level speech feature sequence output by the encoder. This represents a fixed-dimensional high-level feature vector output by the encoder after encoding the acoustic features of the T-th frame of the input speech. It is used to characterize the local acoustic information and contextual semantic information of the speech in that frame. Here are the encoder parameters, and T is the total number of frames in the input speech; Then, during the model training phase, CTC loss, or Connectionist Temporal Classification (CTC) loss, is used to achieve automatic alignment of speech and text. Its core objective is to maximize the probability of the observed sequence, as shown in the following formula: ; in, This is the CTC loss value. Y represents the frame-level sequence containing whitespace characters output by the model, and Y represents the labeled text sequence. This indicates that the sequence π, after removing spaces, matches the label Y. Given an input X, the probability of outputting the path π. Secondly, to further improve the accuracy of medical terminology recognition, the training process introduces a label smooth cross-entropy loss joint constraint decoder, as shown in the following formula: ; in, The cross-entropy loss is given by U, where U is the length of the text sequence and C is the total number of characters in the dictionary. This represents the true distribution after label smoothing. The predicted probability of the model outputting character c at time t; Furthermore, the overall training objective of the model is to minimize the joint loss, as shown in the following formula: ; in, ∈(0,1) is the loss balance coefficient, used to adjust the weights of the two losses; Finally, in the inference phase, the text sequence with the highest probability, i.e., the original transcribed text, is obtained through cluster search. The specific formula is shown below: ; in, X represents the optimal text sequence obtained after beam search, which is the original text sequence output by the end-to-end automatic speech recognition (ASR) model. X represents the log-Mel acoustic feature sequence input to the ASR model, i.e., the speech feature input received by the model. Y represents the candidate text character sequence, which is the various combinations of medical instruction texts that the model may output. The maximization operator represents the search for the text sequence that maximizes the posterior probability P(Y|X) among all candidate text sequences Y, which is then used as the final recognition result.

[0044] As can be seen, in this embodiment, a safe and orderly voice interaction system for medical scenarios is constructed through distributed sound pickup of multiple intelligent badges, voice enhancement processing, and voiceprint identity verification. Combined with an end-to-end voice recognition model specifically optimized for the medical field, high-quality and robust voice command parsing and standardized processing are achieved. While improving the efficiency and ease of use of voice interaction in multidisciplinary consultations, full-process access control and data security are also achieved.

[0045] Step S220: Obtain the audio and video clips corresponding to the target diagnosis and treatment operation of the target patient in the target diagnosis and treatment process indicated by the instruction voice text, based on the first operation permission certificate.

[0046] The "diagnosis and treatment stage" defines the location or business scenario where the diagnosis and treatment occur, mainly including various departments, intensive care units, wards, and operating rooms, such as cardiology wards, neurosurgery operating rooms, intensive care units, and emergency departments. This clarifies the spatial and scenario attribution of the audio and video data. The "diagnosis and treatment operation" characterizes the specific medical actions performed, generally represented by the operation name, treatment item, or examination item, such as wound dressing, thoracentesis, CT scan, vital sign monitoring, and intraoperative anesthesia. This allows for precise location of the specific medical records that medical staff need to retrieve via voice commands. Therefore, the diagnosis and treatment stage defines the scenario and department where the diagnosis and treatment occur, while the diagnosis and treatment operation corresponds to the specific medical actions performed within that scenario. The two work together to clearly define the scope and content of the target audio and video clips that medical staff need to retrieve.

[0047] In one possible embodiment, obtaining the audio-visual segment corresponding to the target medical operation of the target patient in the target medical treatment process indicated by the instruction voice text based on the first operation permission credential includes: obtaining multiple medical related entities based on the instruction voice text, a pre-trained semantic intent recognition model, and a medical semantic entity recognition model, wherein the multiple medical related entities include a patient identifier entity, a medical treatment process entity, and a medical operation entity; and obtaining the audio-visual segment corresponding to the target medical operation of the target patient in the target medical treatment process based on the multiple medical related entities, the first operation permission credential, a pre-stored patient medical information database, a medical audio-visual database, and a pre-built medical operation video index library.

[0048] The patient identification entity is used to identify the patient and includes at least one of the following: patient name, hospital number, bed number, consultation number, and patient ID. The entity representing the diagnosis and treatment process is used to determine the scenario or department in which the diagnosis and treatment occur, including at least one of the following: department, intensive care unit, ward, and operating room; The diagnostic and treatment operation entity is used to determine the diagnostic and treatment behavior, including at least one of the operation name, diagnostic and treatment items, and examination items.

[0049] In one possible embodiment, obtaining multiple medical-related entities based on the instruction voice text, a pre-trained semantic intent recognition model, and a medical semantic entity recognition model includes: performing intent recognition on the instruction voice text based on the semantic intent recognition model to determine the intent of the currently speaking medical staff member to specify the treatment process for the target patient via video retrieval; and performing entity recognition on the instruction voice text based on the medical semantic entity recognition model to extract the multiple medical-related entities.

[0050] The medical semantic entity recognition model is a sequence labeling model fine-tuned based on medical domain corpora. It automatically identifies and extracts medical-related entities such as patients, treatment processes, and procedures from standardized instruction speech text, transforming unstructured medical instructions into structured search criteria. This model takes instruction text sequences as input and, through context encoding and label reasoning, outputs entity category labels corresponding to each character or word. This allows for precise location of target patients, treatment scenarios, and specific treatment procedures, providing structured data support for subsequent matching and retrieval of audio and video clips related to medical procedures.

[0051] For example, if the instruction voice text is "retrieve Zhang San's vital signs monitoring video in the ICU", the semantic intent recognition model will determine the intent as "retrieve patient diagnosis and treatment operation video", and the medical semantic entity recognition model will further extract the medical related entities from the instruction voice text: the patient is Zhang San, the diagnosis and treatment process is ICU, and the diagnosis and treatment operation is vital signs monitoring.

[0052] Specifically, the training and processing of the medical semantic entity recognition model may include the following steps: First, during model processing, the instruction speech text is converted into character-level or word-level embedded sequences: ; Where E is the text embedding sequence and L is the text length. The vector representation of the Lth character or word is used to characterize the local semantic features and contextual information of the text at that position. Next, the text embedding sequence E is input into an encoding structure composed of multiple Transformers or recurrent networks for context modeling to obtain the global semantic features of the text. The specific formula is shown below: U=Encoder(E, ); Encoder() is the semantic encoding function. Here are the trainable parameters of the model, and U is the encoded high-level text semantic feature sequence. Then, during the model training and inference phases, a Conditional Random Field (CRF) layer is used to globally constrain the entity label sequence, ensuring that the output entity labels satisfy the medical entity annotation rules. The optimal label sequence inference formula is as follows: ; in, The optimal entity label sequence output by the model. This means selecting the label sequence Z that maximizes the overall score as the final output. Label the i-th position as Category emission score, For from the label Transferred to The transfer score is used to ensure the continuity and rationality of entity annotation; Finally, model training aims to maximize the log-likelihood probability of the labeled sequence. The loss function is used to calculate the proportion of the true sequence score among all candidate sequence scores, and the negative logarithm is taken to achieve supervised training of the medical entity recognition results. This enables the model to more accurately extract key medical entities such as patients, treatment processes, and treatment procedures. The loss function is expressed as: ; in, This represents the CRF loss value of the medical semantic entity recognition model, used to measure the difference between the entity label sequence predicted by the model and the true label sequence. The training objective is to minimize this loss. Represents a sequence of real entity labels The corresponding total score, that is, the overall matching score of the manually annotated correct medical entity sequence under the given text features U, Z represents the sequence of high-level semantic features of the text output by the encoder of the medical semantic entity recognition model, and Z represents the sequence of arbitrary candidate medical entity labels generated by the model. Z represents the total score corresponding to any candidate entity label sequence, reflecting the reasonableness of the sequence as a medical entity recognition result.

[0053] Furthermore, based on the extracted medical-related entities, core search elements are determined, and the legality of permissions is verified by combining the first operation permission certificate. Relying on the pre-built patient diagnosis and treatment information database, diagnosis and treatment audio and video database, and diagnosis and treatment operation video index library, the target diagnosis and treatment audio and video segments are matched, located, and screened in compliance. This achieves accurate and secure mapping from the semantics of voice commands to the target diagnosis and treatment data, and quickly locates the diagnosis and treatment audio and video segments that match the command requirements while ensuring the compliance of data access permissions.

[0054] As can be seen, in this embodiment, unstructured voice commands are transformed into standardized retrieval elements through semantic intent recognition and medical entity extraction. Combined with operation permission verification and collaborative retrieval from multiple medical databases, the accurate positioning and compliant acquisition of target medical audio and video segments are achieved. This not only improves the efficiency of medical command parsing and data retrieval but also effectively ensures the security and standardization of access to medical data.

[0055] Step S230: Push the diagnostic audio and video clips to the conference display terminal for playback.

[0056] Specifically, for medical audio and video clips, format adaptation processing is first performed, including real-time transcoding, to ensure that the encoding format and resolution fully match the decoding requirements and display parameters of the conference display terminals (such as MDT conference screens, dedicated consultation display devices, etc.). This avoids playback failures due to format incompatibility. Simultaneously, the streaming ratio and resolution requirements of the conference screen are adapted to generate a video stream conforming to the RTMP / RTSP streaming standard, ensuring smooth and uninterrupted audio and video playback. Subsequently, based on the hospital's local area network, the transcoded medical audio and video clip stream is pushed to the designated conference display terminal. During the push process, a "pull successful" status indicator is generated simultaneously, ensuring accurate and rapid transmission of audio and video data to the conference display device without loss or distortion.

[0057] After the push is completed, the conference display terminal receives the audio and video streams and completes decoding, automatically starts the playback program, and synchronously presents the diagnosis and treatment audio and video clips to the participating medical staff for viewing and discussion during multidisciplinary consultations, while providing real-time feedback on the playback status.

[0058] In one possible embodiment, after the diagnostic audio and video clips are pushed to the conference display terminal for playback, the first smart badge is further configured to perform the following operations: outputting a playback completion prompt message through the badge display screen, and playing the playback completion prompt message through the badge speaker; and storing the playback information for this playback operation, the playback information including the identity of the currently speaking medical staff member, playback time, playback content, and playback result.

[0059] Among these features, by retaining the identity of the medical staff currently speaking, the actual entity responsible for retrieving and playing the medical audio and video recordings can be clearly identified, enabling precise binding of the operation to the authorized personnel, avoiding abuse of authority and unauthorized operations, and providing core evidence for subsequent authority verification and accountability. Recording the playback time can completely restore the timeline of the operation, corroborating it with the consultation process and treatment timeline, facilitating subsequent consultation review and treatment process verification. Recording the playback content can clearly retain the target patient, treatment steps, and corresponding audio and video information for this retrieval, clarifying the scope of medical data retrieval and ensuring that the audio and video retrieval behavior is traceable. Recording the playback result can reflect whether the streaming and playback steps are executed normally, facilitating the investigation of playback anomalies, and simultaneously retaining the complete execution status of the process.

[0060] In this embodiment of the application, by uniformly storing all elements of playback information, a standardized operation audit log is formed. This not only meets the mandatory management requirements of the medical industry for the use of medical data and compliance traceability, but also provides complete data support for subsequent operation verification, dispute resolution, and process optimization, further strengthening the standardization and security of medical data retrieval in multidisciplinary consultation scenarios.

[0061] As can be seen, in this embodiment, the voice commands of medical staff are collected by the smart badge, pre-processed and uploaded, and then the server completes voice recognition, intent parsing, entity extraction, and accurate audio-visual matching and segment extraction. Finally, the target diagnosis and treatment audio-visual segment is pushed to the conference display terminal for playback. The video segment of a specific diagnosis and treatment operation can be accurately located through spoken commands, realizing voice-triggered instant retrieval of diagnosis and treatment audio and video in multidisciplinary consultation scenarios in hospitals. There is no need to prepare materials in advance, which greatly improves the efficiency and flexibility of case discussions. Furthermore, the smart badge wearable terminal enables full-process operation without fixed terminals, while ensuring the security and compliance of medical data access.

[0062] Please refer to details. Figure 3 , Figure 3 This application provides a flowchart for determining medical audio and video segments based on medical-related entities, as illustrated in an embodiment of the present application. Figure 3As shown, the step of obtaining the audio and video clips corresponding to the target medical operation of the target patient in the target medical process based on the multiple medical associated entities, the first operation permission certificate, the pre-stored patient diagnosis and treatment information database, the diagnosis and treatment audio and video database, and the pre-built diagnosis and treatment operation video index library includes the following steps: S301, Generate a first search instruction that includes the plurality of medical associated entities and the first operation permission credential.

[0063] The first search command is a structured command that integrates search elements and permission verification elements. It can be directly parsed and executed by the patient's medical information database, the medical audio and video database, and the medical operation video index. The command simultaneously carries two core layers of information: "what to search" and "whether it can be searched." Specifically, the extracted patient identifier, medical process, and medical operation are used as the core conditions for precise retrieval. The first operation permission certificate is then embedded into the command for integrated encapsulation.

[0064] In one possible embodiment, before generating the first search instruction, the integrity of the extracted core medical entities such as patient identifiers, treatment procedures, and treatment operations is checked. If any key entities are missing, a voice interaction is initiated through the speaker of the first smart badge to guide the currently speaking medical staff to supplement the corresponding missing information. After the entity integrity check is passed, the qualified core search conditions and the first operation permission certificate are integrated and encapsulated to form a first search instruction that can be directly parsed and executed by the database and carries both search content and permission verification information.

[0065] S302, query the patient diagnosis and treatment information database according to the first search instruction to obtain the audio and video recording set of the target patient in the target diagnosis and treatment process.

[0066] The patient diagnosis and treatment information database pre-stores multiple audio and video record sets corresponding to multiple patients in multiple diagnosis and treatment stages, and each audio and video record set includes at least one audio and video record corresponding to a diagnosis and treatment operation.

[0067] Specifically, the audio and video records in the database are structured medical data entries, not simply audio and video files. They contain metadata information such as unique patient identifiers, medical process identifiers, medical operation tags, audio and video storage paths, acquisition timestamps, acquisition device numbers, and permission association identifiers. At the same time, they form an association mapping with the actual audio and video segments in the medical audio and video database, possessing both unique location identifiers and medical business attributes.

[0068] In one possible embodiment, the step of querying the patient's medical information database according to the first search instruction to obtain the audio and video recording set of the target patient at the target medical treatment stage includes: accessing the patient's medical information database according to the first operation permission credential, wherein the patient's medical information database pre-stores multiple full-process medical record sets for multiple patients, and each full-process medical record set includes multiple audio and video records corresponding to a single patient at multiple medical treatment stages; obtaining the full-process medical record set of the target patient according to the patient identifier entity index included in the first search instruction; filtering the full-process medical record set of the target patient from the full-process medical record set of the target patient according to the medical treatment stage entity included in the first search instruction to obtain multiple audio and video records corresponding to the target medical treatment stage; and generating the audio and video recording set of the target patient at the target medical treatment stage based on the multiple audio and video records corresponding to the target medical treatment stage.

[0069] In this embodiment, the first operation permission credential is used to complete the legitimate access verification of the patient's medical information database, preventing unauthorized access from the source and ensuring the security of medical data. The database uses a single patient as the core unit and stores the entire process of medical treatment records from admission to discharge. Each record is associated with the audio and video data of the corresponding stage, realizing centralized and full-cycle management of patient medical data. Then, using the patient identifier entity as an index, the set of full-process medical records exclusive to the target patient is quickly located, irrelevant patient data is removed, and a second fine-tuning is performed through the medical stage entity to retain only the audio and video records of the target scenario. Finally, the records are integrated to form an exclusive set of audio and video records. The progressive retrieval method not only greatly improves the positioning efficiency but also accurately matches the retrieval needs, while strictly adhering to the medical data permission control specifications.

[0070] For example, after the first operation permission certificate is verified and database access is obtained, the system retrieves the complete set of Zhang San's medical records in the patient medical information database based on the patient identifier entity "Zhang San" in the first search instruction. This includes all aspects of Zhang San's medical treatment, such as emergency treatment, ICU monitoring, and surgical ward rehabilitation. Then, based on the medical treatment entity "ICU", the system selects audio and video records such as vital sign monitoring, intravenous infusion care, and bedside examinations corresponding to the ICU monitoring stage from the complete set of records. Finally, the system integrates the above audio and video records to generate a set of audio and video records of Zhang San in the ICU medical treatment stage.

[0071] S303, determine the audio and video segment corresponding to the target diagnosis and treatment operation in the target diagnosis and treatment process for the target patient based on the audio and video recording set, the diagnosis and treatment operation video index library, and the diagnosis and treatment audio and video database.

[0072] Among them, the diagnosis and treatment operation video index library is a fast retrieval index built according to the type of diagnosis and treatment operation, which is used to efficiently match the audio and video association information corresponding to the target operation. The diagnosis and treatment audio and video database is the core data source for storing actual diagnosis and treatment audio and video segments. Through collaborative matching, the audio and video segments corresponding to the target patient and the target diagnosis and treatment operation under the target diagnosis and treatment link are accurately determined from the audio and video record set, so as to achieve the final accurate locking of the retrieval results.

[0073] As can be seen, in this embodiment, by generating structured search instructions and verifying core entities, performing pre-permission verification and hierarchical progressive database queries, and combining the collaborative matching of the diagnosis and treatment operation index library and the audio and video database, the accurate, efficient and compliant positioning of target diagnosis and treatment audio and video segments is achieved. While ensuring the security of access to diagnosis and treatment data, the retrieval efficiency and matching accuracy of diagnosis and treatment data in multidisciplinary consultation scenarios are greatly improved.

[0074] Please refer to details. Figure 4 , Figure 4 This application provides a flowchart for determining diagnostic audio and video segments based on a set of audio and video recordings, as illustrated in an embodiment of the present application. Figure 4 As shown, determining the audio-visual segment corresponding to the target medical procedure for the target patient in the target medical process based on the audio-visual recording set, the medical procedure video index library, and the medical audio-visual database includes the following steps: S401, perform fine-grained matching on the audio and video record set according to the diagnostic and treatment operation video index library, filter out the target audio and video record corresponding to the target diagnostic and treatment operation in the audio and video record set, and determine the start and end timestamps of the target diagnostic and treatment operation.

[0075] The diagnostic and treatment operation video index library pre-stores metadata for multiple diagnostic and treatment audio and video recordings. The metadata includes patient identification, diagnostic and treatment steps, diagnostic and treatment operations, and start and end timestamps of the diagnostic and treatment operations.

[0076] Specifically, using patient identification, treatment process, and target treatment operation as three matching dimensions, the search elements are aligned with the metadata pre-stored in the treatment operation video index library one by one. First, index entries under the same patient and treatment process are locked, and then the treatment operation name / operation type is accurately tagged and matched, thereby filtering out the unique target audio and video record from the audio and video record set. Since the index library has pre-labeled the start and end timestamps of the corresponding treatment operation for each audio and video record, the timestamp can be directly extracted after the matching is completed, thus completing the precise positioning of the target operation segment in the time dimension.

[0077] S402, the target audio and video file is obtained from the diagnosis and treatment audio and video database according to the target audio and video record, wherein the target audio and video record includes the file identifier of the target audio and video file.

[0078] The target audio and video record contains a file identifier, which is a unique identifier used to locate the corresponding original audio and video file in the medical audio and video database. It can adopt a structured coding form that combines patient information, treatment process, collection time and device number to ensure the uniqueness and traceability of the identifier. A simple example is as follows: ZS_ICU_20260114_0015_V01 (where ZS represents patient Zhang San, ICU is the treatment process, 20260114 is the collection date, 0015 is the collection time period, and V01 is the collection device number). The system can quickly and accurately read the corresponding complete audio and video file from the medical audio and video database through this identifier.

[0079] S403, the target audio and video file is extracted according to the start and end timestamps of the target diagnosis and treatment operation to obtain the diagnosis and treatment audio and video segments corresponding to the target diagnosis and treatment operation of the target patient in the target diagnosis and treatment process.

[0080] Specifically, the complete target audio and video file is precisely extracted based on the start and end timestamps, and irrelevant audio and video content before and after the diagnosis and treatment operation is removed. Finally, exclusive diagnosis and treatment audio and video segments containing only the target patient, the target diagnosis and treatment process, and the target diagnosis and treatment operation are obtained, realizing lightweight and precise extraction of the data required for consultation.

[0081] As can be seen, in this embodiment, through multi-dimensional matching, precise time positioning, and lightweight extraction, the rapid and accurate extraction of audio and video of the target diagnosis and treatment operation is achieved, effectively eliminating irrelevant content and ensuring that the acquired audio and video segments are highly matched with the target patient, the target diagnosis and treatment process, and the target operation. This not only improves the efficiency of data retrieval but also ensures the standardization and traceability of the data, providing accurate, efficient, and compliant audio and video data support for MDT consultations.

[0082] Combination Figure 1 Please see Figure 5 and Figure 6 , Figure 5 This is a schematic diagram of a smart name tag provided in an embodiment of this application, such as... Figure 5 , Figure 6 As shown, the smart badge 120 includes a display screen 123 and a speaker 124. The display screen 123 is used to provide medical staff with real-time feedback on the execution status of instructions and playback completion prompts, realizing visual interaction. The speaker 124 is used for voice command acquisition, status broadcasting, and interactive prompts. In addition, the smart badge 120 also includes a power button 125, a recording button 126, and a video recording button 127, which provide medical staff with a manual operation interface, allowing them to flexibly control the badge's start and stop, voice command input, and audio and video recording, thus meeting the dual needs of automatic voice interaction and manual emergency operation.

[0083] Specifically, please refer to Figure 6 , Figure 7 , Figure 6 This is a schematic diagram of a multidisciplinary consultation scenario provided in an embodiment of this application. Figure 7 This is a schematic diagram of another multidisciplinary consultation scenario provided in an embodiment of this application, such as... Figure 6 , Figure 7 As shown, the application effect of the diagnostic data retrieval solution provided in this application in a real multidisciplinary team (MDT) consultation scenario is intuitively presented: During a consultation discussion, medical staff wearing smart badges 120 initiate a request to retrieve diagnostic audio and video data via voice command, "Check Zhang San's intubation status in the ICU." After completing voice acquisition, intent recognition, and entity extraction, the smart badge 120 sends the structured retrieval request and authorization credentials to the server 110. The server 110, after completing authorization verification, multi-source database retrieval, and audio and video processing, pushes the obtained diagnostic audio and video clips to the conference display terminal 130 (MDT conference screen) for playback. The screen simultaneously displays "Currently Playing - Zhang San - ICU Intubation Video" and the playback progress, allowing all participating medical staff to view and discuss simultaneously. At the same time, the server 110 provides feedback on the execution status to the smart badge 120.

[0084] It is evident that in multidisciplinary consultation scenarios, medical staff can quickly access precise audio and video clips corresponding to target patients, target treatment steps, and target treatment operations simply through natural speech without interrupting the consultation process or manually operating the conference terminal, significantly improving the efficiency and focus of consultation discussions. At the same time, the entire process relies on the access control and operation tracking of smart badges to strictly ensure compliant access and traceability of medical data, meeting the core needs of efficient, secure, and accurate access to medical data in multidisciplinary consultations.

[0085] Further, please refer to Figure 8 , Figure 8 This is a schematic diagram of the status of a smart badge in a multidisciplinary consultation scenario provided by an embodiment of this application, such as... Figure 8 As shown, the closed-loop interaction effect and status feedback mechanism of the medical data retrieval solution provided in this application in a multidisciplinary consultation scenario are demonstrated: After the backend server 110 completes the audio and video processing and pushes it to the conference screen for playback, the display screen 123 of the smart badge 120 will provide real-time feedback with a visual status prompt that "Played successfully: ICU intubation video of patient Zhang San", and at the same time, it will cooperate with the speaker 124 to broadcast the voice, ensuring that medical staff can clearly know the operation results while focusing on the consultation; the "query" button attached to the screen also provides an entry point for subsequent operations, further enhancing the convenience of human-computer interaction and information transparency, and effectively improving the consultation experience and the traceability of data retrieval.

[0086] This application embodiment can divide the electronic device into functional units according to the above method example. For example, each function can be divided into a separate functional unit, or two or more functions can be integrated into one processing unit. The integrated unit can be implemented in hardware or as a software functional unit. It should be noted that the unit division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0087] Please see Figure 9 , Figure 9 This is a functional unit block diagram of an intelligent name tag system provided in this application embodiment. The intelligent name tag system 100 includes: a response unit 901 and a processing unit 902; wherein, the response unit 901 is used to respond to the first operation permission certificate and instruction voice data of the currently speaking medical staff sent by the first intelligent name tag, perform speech recognition on the instruction voice data to obtain instruction voice text, and the first operation permission certificate is used to represent having access and operation permissions to the intelligent name tag system and patient diagnosis and treatment data; the processing unit 902 is used to obtain the diagnosis and treatment audio and video segment corresponding to the target diagnosis and treatment operation of the target patient in the target diagnosis and treatment stage indicated by the instruction voice text according to the first operation permission certificate; and push the diagnosis and treatment audio and video segment to the conference display terminal for playback.

[0088] In one possible embodiment, the processing unit 902 obtains the audio-visual segment corresponding to the target treatment operation of the target patient in the target treatment process indicated by the instruction voice text based on the first operation permission credential. Specifically, the processing unit 902 is used to: obtain multiple medical associated entities based on the instruction voice text, a pre-trained semantic intent recognition model, and a medical semantic entity recognition model, wherein the multiple medical associated entities include a patient identifier entity, a treatment process entity, and a treatment operation entity; and obtain the audio-visual segment corresponding to the target treatment operation of the target patient in the target treatment process based on the multiple medical associated entities, the first operation permission credential, a pre-stored patient treatment information database, a treatment audio-visual database, and a pre-built treatment operation video index library.

[0089] In one possible embodiment, the processing unit 902 obtains the audio and video segments corresponding to the target treatment operation of the target patient in the target treatment stage based on the plurality of medical associated entities, the first operation permission credential, a pre-stored patient treatment information database, a treatment audio and video database, and a pre-built treatment operation video index library. Specifically, the processing unit 902 is configured to: generate a first search instruction containing the plurality of medical associated entities and the first operation permission credential; query the patient treatment information database according to the first search instruction to obtain a set of audio and video records of the target patient in the target treatment stage, wherein the patient treatment information database pre-stores multiple sets of audio and video records corresponding to multiple patients in multiple treatment stages, and each audio and video record set includes at least one audio and video record corresponding to a treatment operation; and determine the audio and video segments corresponding to the target treatment operation of the target patient in the target treatment stage based on the audio and video record sets, the treatment operation video index library, and the treatment audio and video database.

[0090] In one possible embodiment, the processing unit 902 determines the medical audio-visual segment corresponding to the target medical operation in the target medical treatment process for the target patient based on the audio-visual recording set, the medical operation video index library, and the medical audio-visual database. Specifically, the processing unit 902 is configured to: perform fine-grained matching on the audio-visual recording set according to the medical operation video index library, filter out the target audio-visual record corresponding to the target medical operation in the audio-visual recording set, determine the start and end timestamps of the target medical operation, wherein the medical operation video index library pre-stores metadata for multiple medical audio-visual records, the metadata including patient identifier, medical treatment process, medical operation, and start and end timestamps of the medical operation; read the target audio-visual file from the medical audio-visual database based on the target audio-visual record, the target audio-visual record including the file identifier of the target audio-visual file; and extract the target audio-visual file based on the start and end timestamps of the target medical operation to obtain the medical audio-visual segment corresponding to the target medical operation in the target medical treatment process for the target patient.

[0091] In one possible embodiment, the processing unit 902 queries the patient treatment information database according to the first search instruction to obtain a set of audio and video records of the target patient at the target treatment stage. Specifically, the processing unit 902 is configured to: access the patient treatment information database according to the first operation permission credential, wherein the patient treatment information database pre-stores multiple sets of full-process treatment records for multiple patients, and each full-process treatment record set includes multiple audio and video records corresponding to multiple treatment stages for a single patient; obtain the full-process treatment record set of the target patient according to the patient identifier entity index included in the first search instruction; filter the full-process treatment record set of the target patient from the full-process treatment record set of the target patient according to the treatment stage entity included in the first search instruction to obtain multiple audio and video records corresponding to the target treatment stage; and generate the audio and video record set of the target patient at the target treatment stage based on the multiple audio and video records corresponding to the target treatment stage.

[0092] In one possible embodiment, multiple medical-related entities are obtained based on the instruction voice text, a pre-trained semantic intent recognition model, and a medical semantic entity recognition model. The processing unit 902 is specifically used to: perform intent recognition on the instruction voice text based on the semantic intent recognition model to determine that the intention of the currently speaking medical staff is to retrieve the operation video of the designated diagnosis and treatment procedure for the target patient; and perform entity recognition on the instruction voice text based on the medical semantic entity recognition model to extract the multiple medical-related entities.

[0093] In one possible embodiment, the first smart badge is used to perform the following operations: receiving raw voice data of the medical staff currently speaking in the multidisciplinary consultation scenario; performing noise removal and non-voice segment removal processing on the raw voice data to extract a complete instruction voice segment, and identifying the instruction voice segment as the instruction voice data; performing voiceprint similarity matching on the instruction voice segment according to a pre-stored authorized medical staff voiceprint database to determine the identity identifier of the currently speaking medical staff and the first operation permission credential, wherein the authorized medical staff voiceprint database pre-stores the identity identifiers and operation permission credentials of multiple authorized medical staff; and sending the instruction voice data and the first operation permission credential of the currently speaking medical staff to the server.

[0094] In one possible embodiment, after the diagnostic audio and video clips are pushed to the conference display terminal for playback, the first smart badge is further configured to perform the following operations: outputting a playback completion prompt message through the badge display screen, and playing the playback completion prompt message through the badge speaker; and storing the playback information for this playback operation, the playback information including the identity of the currently speaking medical staff member, playback time, playback content, and playback result.

[0095] As can be seen, in this embodiment, the voice commands of medical staff are collected by the smart badge, pre-processed and uploaded, and then the server completes voice recognition, intent parsing, entity extraction, and accurate audio-visual matching and segment extraction. Finally, the target diagnosis and treatment audio-visual segment is pushed to the conference display terminal for playback. The video segment of a specific diagnosis and treatment operation can be accurately located through spoken commands, realizing voice-triggered instant retrieval of diagnosis and treatment audio and video in multidisciplinary consultation scenarios in hospitals. There is no need to prepare materials in advance, which greatly improves the efficiency and flexibility of case discussions. Furthermore, the smart badge wearable terminal enables full-process operation without fixed terminals, while ensuring the security and compliance of medical data access.

[0096] It is understood that since the method embodiments and the device embodiments are different presentations of the same technical concept, the content of the method embodiment section in this application should be adapted to the device embodiment section in a synchronous manner, and will not be repeated here.

[0097] Figure 10 This is a structural block diagram of an electronic device provided in an embodiment of this application. For example... Figure 10 As shown, electronic device 1000 may include one or more components: a processor 1001 and a memory 1002 coupled to the processor 1001, wherein the memory 1002 may store one or more computer programs, which may be configured to implement the methods described in the examples above when executed by one or more processors 1001. Electronic device 1000 may be as follows: Figure 1 The server shown is 110.

[0098] Processor 1001 may include one or more processing cores. Processor 1001 connects to various parts within the electronic device 1000 using various interfaces and lines, and performs various functions and processes data of the electronic device 1000 by running or executing instructions, programs, code sets, or instruction sets stored in memory 1002, and by calling data stored in memory 1002. Optionally, processor 1001 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). Processor 1001 may integrate one or more of a Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. It is understood that the aforementioned modem may also not be integrated into processor 1001, but may be implemented separately through a communication chip.

[0099] The memory 1002 may include random access memory (RAM) or read-only memory (ROM). The memory 1002 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 1002 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as touch functionality, sound playback functionality, image playback functionality, etc.), and instructions for implementing the above-described method examples. The data storage area may also store data created during the use of the electronic device 1000.

[0100] It is understood that the electronic device 1000 may include more or fewer structural elements than those shown in the above block diagram, such as a power module, physical buttons, WiFi (Wireless Fidelity) module, speaker, Bluetooth module, sensor, etc., without limitation.

[0101] This application also provides a computer storage medium storing a computer program / instructions thereon, which, when executed by a processor, implements some or all of the steps of any of the methods described in the above method embodiments.

[0102] This application also provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program operable to cause a computer to perform some or all of the steps of any of the methods described in the above method embodiments.

[0103] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0104] In the several embodiments provided in this application, it should be understood that the disclosed methods, apparatuses, and systems can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for example, the division of units is merely a logical functional division, and there may be other division methods in actual implementation; for example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0105] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0106] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can be physically comprised separately, or two or more units can be integrated into one unit. The integrated unit described above can be implemented in hardware or in the form of hardware plus software functional units.

[0107] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute partial steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes: a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, volatile memory, or non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM), etc., which are various media capable of storing program code.

[0108] While the present invention has been disclosed above, it is not limited thereto. Any person skilled in the art can easily conceive of variations or substitutions without departing from the spirit and scope of the present invention, and various modifications and alterations can be made, including combinations of the different functions and implementation steps described above, as well as software and hardware implementation methods, all of which are within the protection scope of the present invention.

Claims

1. A method for retrieving diagnostic and treatment data in a multidisciplinary consultation scenario, characterized in that, A server is used in a smart badge system, the smart badge system including smart badges worn by participating medical staff, a conference display terminal, and the server, the server being communicatively connected to the conference display terminal and the smart badges respectively; the method includes: In response to the first smart badge sending the first operation permission certificate and instruction voice data of the currently speaking medical staff, the instruction voice data is subjected to speech recognition to obtain instruction voice text. The first operation permission certificate is used to represent that the medical staff has access and operation permissions to the smart badge system and patient diagnosis and treatment data. Based on the first operation permission certificate, obtain the audio and video clips corresponding to the target diagnosis and treatment operation of the target patient in the target diagnosis and treatment process indicated by the instruction voice text; The diagnostic audio and video clips are pushed to the conference display terminal for playback.

2. The method according to claim 1, characterized in that, The step of obtaining the audio and video clips corresponding to the target treatment operation of the target patient in the target treatment process indicated by the instruction voice text according to the first operation permission certificate includes: Based on the instruction voice text, the pre-trained semantic intent recognition model, and the medical semantic entity recognition model, multiple medical related entities are obtained, including patient identification entities, diagnosis and treatment process entities, and diagnosis and treatment operation entities. Based on the multiple medical-related entities, the first operation permission certificate, the pre-stored patient diagnosis and treatment information database, the diagnosis and treatment audio and video database, and the pre-built diagnosis and treatment operation video index library, obtain the diagnosis and treatment audio and video segments corresponding to the target diagnosis and treatment operation of the target patient in the target diagnosis and treatment process.

3. The method according to claim 2, characterized in that, The step of obtaining the audio and video clips corresponding to the target medical operation in the target medical process for the target patient based on the multiple medical-related entities, the first operation permission certificate, the pre-stored patient medical information database, the medical audio and video database, and the pre-built medical operation video index library includes: Generate a first search instruction that includes the plurality of medical-related entities and the first operation permission credential; The patient diagnosis and treatment information database is queried according to the first search instruction to obtain the audio and video record set of the target patient in the target diagnosis and treatment process. The patient diagnosis and treatment information database pre-stores multiple audio and video record sets corresponding to multiple patients in multiple diagnosis and treatment processes. Each audio and video record set includes at least one audio and video record corresponding to a diagnosis and treatment operation. Based on the audio and video recording set, the diagnostic and treatment operation video index library, and the diagnostic and treatment audio and video database, determine the diagnostic and treatment audio and video segments corresponding to the target diagnostic and treatment operation for the target patient in the target diagnostic and treatment process.

4. The method according to claim 3, characterized in that, The step of determining the audio-visual segment corresponding to the target medical procedure for the target patient in the target medical process based on the audio-visual recording set, the medical procedure video index library, and the medical audio-visual database includes: The audio and video recording set is matched in a fine-grained manner according to the video index library of the diagnosis and treatment operation, and the target audio and video recording corresponding to the target diagnosis and treatment operation is selected in the audio and video recording set. The start and end timestamps of the target diagnosis and treatment operation are determined. The video index library of the diagnosis and treatment operation pre-stores metadata of multiple diagnosis and treatment audio and video recordings. The metadata includes patient identifier, diagnosis and treatment process, diagnosis and treatment operation and start and end timestamps of diagnosis and treatment operation. The target audio and video file is obtained by reading the target audio and video record from the diagnostic audio and video database, wherein the target audio and video record includes the file identifier of the target audio and video file; The target audio and video file is extracted based on the start and end timestamps of the target diagnosis and treatment operation to obtain the diagnosis and treatment audio and video segments corresponding to the target diagnosis and treatment operation of the target patient in the target diagnosis and treatment process.

5. The method according to claim 3, characterized in that, The step of querying the patient's medical information database according to the first search instruction to obtain the set of audio and video records of the target patient in the target medical treatment process includes: Accessing the patient's medical information database based on the first access permission credential, the patient's medical information database pre-stores multiple sets of complete medical records for multiple patients, each complete medical record set including multiple audio and video records corresponding to multiple stages of a single patient's medical treatment; and... The complete set of medical records for the target patient is obtained based on the patient identifier entity index contained in the first search instruction. Based on the diagnosis and treatment entity contained in the first search instruction, multiple audio and video records corresponding to the target diagnosis and treatment stage are obtained from the full-process diagnosis and treatment record set of the target patient; A set of audio and video records of the target patient at the target treatment stage is generated based on multiple audio and video records corresponding to the target treatment stage.

6. The method according to claim 2, characterized in that, The process of obtaining multiple medical-related entities based on the instruction speech text, a pre-trained semantic intent recognition model, and a medical semantic entity recognition model includes: Based on the semantic intent recognition model, the intent of the instruction speech text is recognized, and the intent of the medical staff currently speaking is to retrieve the operation video of the designated treatment procedure for the target patient. The medical semantic entity recognition model is used to perform entity recognition on the instruction voice text, and the multiple medical related entities are extracted.

7. The method according to claim 1, characterized in that, The first smart badge is used to perform the following operations: Receive the raw voice data of the medical staff currently speaking in the multidisciplinary consultation scenario; The original speech data is subjected to noise removal and non-speech segment removal to extract the complete instruction speech segment, and the instruction speech segment is identified as the instruction speech data. The voiceprint similarity of the instruction voice segment is matched according to the pre-stored authorized medical staff voiceprint database to determine the identity of the medical staff currently speaking and the first operation permission certificate. The authorized medical staff voiceprint database contains multiple authorized medical staff identity and operation permission certificates. The voice data of the currently speaking medical staff member and the first operation permission certificate are sent to the server.

8. The method according to claim 7, characterized in that, After the diagnostic audio-visual segment is pushed to the conference display terminal for playback, the first smart badge is also used to perform the following operations, including: The playback completion notification is displayed on the name tag screen, and the playback completion notification is played through the name tag speaker; and... The playback information for this playback operation is stored, including the identity of the medical staff currently speaking, the playback time, the playback content, and the playback result.

9. An intelligent name tag system, characterized in that, The smart badge system includes smart badges worn by participating medical staff, a conference display terminal, and a server. The server is communicatively connected to the conference display terminal and the smart badges, respectively. The server is used to perform the steps in the method as described in any one of claims 1-8.

10. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, The computer program / instructions are executed by the processor to implement the steps of the method according to any one of claims 1-8.

Images