Intelligent blood sampling auxiliary system and auxiliary method

Abstract

The embodiment of the application discloses a kind of intelligent blood sampling auxiliary systems and auxiliary methods, including: image acquisition unit;Sound acquisition unit;Display unit;Processing unit is respectively connected with the image acquisition unit, the sound acquisition unit and the display unit, and the processing unit is configured to: based on the real-time image identification key object in blood sampling operation and operation action, and generate visual guidance information superimposed display on the display unit;Based on the environmental sound identification key sound event in blood sampling process;Visual recognition result and sound recognition result are fused, and the completion state of blood sampling step is judged;According to preset mode, selectively output prompt information.According to the application, it realizes the leap from "passive monitoring" to "active assistance", and systematically improves the accuracy of blood sampling operation, the reliability of determination and user experience in multiple scenarios.

Description

Technical Field

[0001] This application relates to the field of medical devices, and more particularly to an intelligent blood collection auxiliary system and auxiliary method. Background Technology

[0002] Disposable blood collection needles are among the most commonly used medical devices in clinical testing. Their standard operating procedure includes multiple steps such as disinfection, puncture, tube insertion, needle removal, and retrieval. With increasingly stringent medical quality requirements, machine vision-based blood collection assistance systems have emerged in recent years. These systems typically use cameras to capture images of the operating area, employ deep learning algorithms to identify key objects such as blood collection needles and tubes, and determine the sequence and completeness of the operational steps, issuing alarms when omissions or errors occur.

[0003] However, in actual clinical applications, the aforementioned vision-based blood collection assistance systems primarily remain at the "process monitoring" level. Their core function is to assess the compliance of completed operations, rather than providing real-time, visual, and proactive guidance to the operator. Specifically, these systems merely mark detected objects on the screen with rectangles or issue alarms when errors occur. They cannot provide quantifiable assistance for experience-based skills such as determining the puncture angle and the force and direction of blood collection tube insertion. Furthermore, when hand obstruction or blind spots occur during the procedure, purely visual systems are prone to missing crucial actions, leading to incorrect step determinations. Additionally, these systems employ fixed prompting strategies and cannot adaptively adjust based on the operator's proficiency or usage scenarios (such as peak outpatient hours or teaching / training).

[0004] It is evident that existing vision-based blood collection assistance systems still have shortcomings in terms of operation guidance capabilities, multimodal perception fusion, and scene-adaptive interaction, making it difficult to meet the higher clinical demands for intelligent and scenario-based assistance. Summary of the Invention

[0005] This application provides an intelligent blood collection assistance system and method, which realizes a leap from "passive monitoring" to "active assistance", systematically improving the accuracy of blood collection operations, the reliability of judgment, and the user experience in multiple scenarios.

[0006] To address the aforementioned technical problems, the embodiments of this application disclose the following technical solutions: On one hand, a battery cell structure is provided, comprising: 1. an intelligent blood collection auxiliary system, characterized in that it comprises: The image acquisition unit is used to acquire real-time images of the blood collection operation area. The sound acquisition unit is used to collect ambient sounds during the blood collection process; The display unit is used to display the real-time image; The processing unit is connected to the image acquisition unit, the sound acquisition unit, and the display unit, respectively, and the processing unit is configured as follows: Based on the real-time image recognition of key objects and operational actions in the blood collection operation, visual guidance information is generated and superimposed on the display unit; Key sound events during the blood collection process are identified based on the aforementioned environmental sound. The visual recognition results are fused with the sound recognition results to determine the completion status of the blood collection step; Selectively output prompt information according to preset patterns.

[0007] In addition to one or more of the features disclosed above, or as an alternative, the visual guidance information includes: during the puncture preparation phase, a virtual marker of a suggested puncture point superimposed on the display unit based on the location of the blood vessel as determined by the real-time image.

[0008] In addition to one or more of the features disclosed above, or alternatively, the visual guidance information includes: during the blood collection tube insertion phase, a dynamic arrow superimposed near the blood collection tube or needle holder to indicate the direction of rotation or insertion depth.

[0009] In addition to one or more of the features disclosed above, or as an alternative, the key sound events identified by the processing unit based on the environmental sound include one or more of the following: a crisp sound of a protective sleeve being pulled out, a muffled sound of a blood collection tube stopper being punctured, and the sound of a tourniquet being released.

[0010] In addition to one or more of the features disclosed above, or as an alternative, the processing unit is configured to: determine that the step is completed when both the visual recognition result and the sound recognition result indicate that the same blood collection step is completed; and enter an uncertain state and output a verification prompt when only a single modality recognizes that the step is completed.

[0011] In addition to one or more of the features disclosed above, or as an alternative, the preset modes include a teaching mode and an efficiency mode; In the teaching mode, the processing unit controls the display unit and / or speaker to output a step-by-step explanation of the entire process and to score the operation. In the high-efficiency mode, the processing unit only outputs alarm prompts for detected operational errors and does not output prompts for regular procedures.

[0012] In addition to one or more of the features disclosed above, or as an alternative, the processing unit is also configured to automatically switch the preset mode based on the current time period, nurse identity information, or user manual selection.

[0013] In addition to one or more of the features disclosed above, or as an alternative, the processing unit is also configured to: when the visual recognition and sound recognition make inconsistent determinations on the completion status of the step, mark the corresponding image fragments and sound fragments as low-confidence samples and store them in the storage unit for subsequent model optimization.

[0014] On the other hand, a smart blood collection assistance method is further disclosed, applied to the smart blood collection assistance system as described in any of the above claims, comprising the following steps: Real-time images of the blood collection area are acquired using the image acquisition unit; The ambient sound during the blood collection process is collected using a sound acquisition unit; The processing unit identifies key objects and actions in the blood collection operation based on the real-time image and generates visual guidance information that is overlaid on the display unit. The processing unit identifies key sound events during the blood collection process based on the ambient sound. The processing unit fuses the visual recognition results with the sound recognition results to determine the completion status of the blood collection step. The processing unit selectively outputs prompt information according to a preset pattern.

[0015] In addition to one or more of the features disclosed above, or as an alternative, the visual guidance information includes: during the puncture preparation phase, overlaying a virtual marker suggesting the puncture point on the display unit based on the vessel location located by the real-time image; and / or during the blood collection tube insertion phase, overlaying a dynamic arrow indicating the rotation direction or insertion depth.

[0016] One of the above technical solutions has the following advantages or beneficial effects: Visual and auditory information are acquired by the image acquisition unit and the sound acquisition unit respectively. The processing unit then fuses and determines the visual and auditory recognition results. Simultaneously, the generated visual guidance information (such as puncture point markings and dynamic arrows) is overlaid on the display unit. Based on a preset mode (teaching or high-efficiency), prompts are selectively output, constructing a system architecture that upgrades from single-visual monitoring to multi-modal fusion assistance. This technical approach allows blood collection assistance to move beyond simply judging the compliance of completed operations and proactively intervene in the process: On the one hand, visual guidance information transforms experiential operational skills (such as puncture point selection and cannulation angle) into visualized quantitative guidance, directly reducing the learning cost and operational errors for novice nurses; on the other hand, the dual visual and auditory confirmation mechanism utilizes the redundant and complementary characteristics of different modal information to effectively compensate for the blind spots of single vision in occluded scenarios, making step determination more robust; at the same time, the patterned output strategy adjusts the prompt density according to scenario differences, avoiding interference from redundant information for experienced nurses. Therefore, this solution achieves a leap from "passive monitoring" to "active assistance" without changing nurses' core operating habits, systematically improving the accuracy of blood collection operations, the reliability of judgment, and the user experience in multiple scenarios. Attached Figure Description

[0017] The technical solution and other beneficial effects of this application will become apparent from the following detailed description of specific embodiments in conjunction with the accompanying drawings.

[0018] Figure 1 This is a topology diagram of an intelligent blood collection auxiliary system according to one embodiment of this application; Figure 2 This is a flowchart of an intelligent blood collection assistance method according to one embodiment of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and beneficial effects of this application clearer, the following detailed description, in conjunction with the accompanying drawings and specific embodiments, further illustrates this application. It should be understood that the specific embodiments described in this specification are merely for explaining this application and are not intended to limit it.

[0020] In this application, unless otherwise expressly specified and limited, "above" or "below" the second feature can include direct contact between the first and second features, or contact between the first and second features through another feature between them. Furthermore, "above," "over," and "on top" of the second feature includes the first feature being directly above or diagonally above the second feature, or simply indicates that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature includes the first feature being directly above or diagonally above the second feature, or simply indicates that the first feature is at a lower horizontal level than the second feature.

[0021] In clinical blood collection practice, machine vision-based assistive systems have been gradually applied. These systems typically use cameras to identify key objects such as blood collection needles and tubes, determine the sequence and completeness of operational steps, and issue alarms when omissions or errors occur. The industry generally believes that the core value of these systems lies in "process monitoring" and "compliance assessment," that is, reducing human error through verification of operational results. Based on this understanding, improvements to existing technologies mainly focus on enhancing the accuracy of visual recognition, optimizing target detection models, and expanding the categories of recognizable objects.

[0022] However, the inventors of this application discovered a commonly overlooked cognitive bias in their long-term clinical observations and system trials: equating "blood collection assistance" with "post-operation error correction." Under this mindset, the system is positioned as a "referee" rather than a "coach"—it can only provide feedback after the operation is completed or an error occurs, but cannot offer proactive, visual guidance during the operation. This passive assistance mode has limited value for inexperienced novice nurses, because even if the system indicates an "operation error," the nurse still doesn't know "what is the correct way to do it." Furthermore, the fixed prompts of a purely visual system create redundant interference for experienced nurses during peak hours, and lack detailed explanation capabilities in teaching scenarios, essentially failing to establish a connection between "assistance" and the "scenario."

[0023] The inventors further discovered that the root of the problem lies in the fact that the industry has long narrowed the technical problem of "blood collection assistance" to "how to accurately identify operations and trigger alarms," ​​while failing to realize that the real technical problem is "how to build an intelligent assistance system that can actively guide, perceive multiple modalities, and adapt to different scenarios." In other words, existing technologies have always focused on the single dimension of "identification," while ignoring the three equally crucial functional dimensions of "guidance," "integration," and "adaptation."

[0024] At a deeper level, the inventors recognized that blood collection is essentially a series of physical events with temporal logic. These events generate not only visual signals (such as needle movement and tube insertion) but also auditory signals (such as the crisp sound of the protective sheath being pulled out and the dull thud of the rubber stopper being pierced). However, in existing technologies, almost no one incorporates sound signals into the perception scope of blood collection assistance systems. This is because the medical device field is accustomed to viewing blood collection as a "visually intensive operation," believing that it does not require sound perception; at the same time, signal processing engineers lack a deep understanding of clinical scenarios and find it difficult to establish a correspondence between sound events and operational steps.

[0025] It is precisely based on this interdisciplinary cognitive gap and reflection on existing technological paths that the inventors of this invention have taken the lead in proposing a core technical problem that has long been overlooked in the industry: how to construct a blood collection assistance system that can provide visual operation guidance, integrate visual and auditory information for step determination, and adaptively adjust prompt strategies according to the usage scenario, thereby upgrading blood collection assistance from "passive monitoring" to "active assistance." The formulation of this problem does not stem from a superficial summary of the shortcomings of existing technologies, but from a redefinition of the essence of "assistance"—assistance should not be post-event correction, but in-event guidance. Before this problem was raised, those skilled in the art generally believed that improving the accuracy of visual recognition would solve the problem, and would not have considered introducing AR guidance, multimodal fusion, and mode adaptation mechanisms. After the problem was clarified, although constructing a system using visual overlay, sound recognition, and mode switching also required creative effort, the discovery of the problem itself had already crossed the conventional cognitive boundaries of ordinary people skilled in the art.

[0026] In view of this, this application discloses an intelligent blood collection assistance system and method. By introducing visual and auditory dual-modal fusion perception, visual active guidance, and scene adaptive switching between teaching mode and high-efficiency mode, it aims to upgrade blood collection assistance from the traditional "passive monitoring" to "active assistance", thereby improving the accuracy of operation, the reliability of judgment, and the user experience in multiple scenarios.

[0027] like Figure 1 The embodiment shown provides an intelligent blood collection assistance system, which mainly includes an image acquisition unit, a sound acquisition unit, a display unit, and a processing unit.

[0028] Image acquisition units (such as industrial cameras or network cameras) are installed at the front of the nurse's workstation, covering the area where the nurse performs blood collection. They are used to acquire high-definition video streams and images of the blood collection area in real time. The acquisition resolution is no less than 1280×720, and the frame rate is no less than 25fps to ensure clear capture of key actions and object details during the blood collection process.

[0029] A sound acquisition unit (e.g., a microphone array) is positioned near the nurse's workstation to collect ambient sounds during blood collection in real time. The sound acquisition unit may include multiple microphones arranged in an array to facilitate subsequent sound source localization and noise suppression.

[0030] The display unit (e.g., a touch screen) is located in front of the nurse's workbench or embedded in the workbench to display the video footage captured by the image acquisition unit in real time, and to overlay the visual guidance information generated by the processing unit.

[0031] The processing unit is connected to the image acquisition unit, sound acquisition unit, and display unit, respectively. The processing unit can be an embedded AI computing platform (such as an NVIDIA Jetson series or a domestically produced edge computing device equipped with an NPU), or a general-purpose computer with graphics processing capabilities. The processing unit internally deploys deep learning models and signal processing algorithms for real-time analysis of images and sound.

[0032] The processing unit is configured to perform the following functions: Based on real-time image recognition of key objects (blood collection needle, blood collection tube, puncture needle tip, etc.) and operational actions (puncture, insertion, needle removal, etc.) in blood collection operations. Generate visual guidance information and overlay it onto the display unit; Key sound events during the blood collection process are identified based on environmental sound recognition. The visual recognition results are fused with the sound recognition results to comprehensively determine the completion status of the blood collection step; Selectively output prompts based on preset modes (teaching mode or high-efficiency mode).

[0033] By employing dual-modal perception through image and sound acquisition units, combined with fusion judgment and patterned output from the processing unit, a system architecture was constructed that upgrades from single-vision monitoring to multi-modal fusion assistance. This architecture not only recognizes operational actions but also provides visual guidance before the operator performs the action and adaptively adjusts prompt strategies based on scenario differences. Thus, while maintaining system versatility, it significantly improves the ability to guide novice nurses and adapt to different scenarios for experienced workers.

[0034] To address the issue of puncture failures caused by novice nurses' lack of experience in vascular localization, visual guidance information includes: during the puncture preparation phase, a virtual marker suggesting the puncture point is superimposed on the display unit based on the vascular location located by the real-time image.

[0035] Specifically, when the nurse is preparing to perform a venipuncture, the processing unit first preprocesses the real-time image input from the image acquisition unit, including image enhancement, noise reduction, and contrast adjustment. Subsequently, a pre-trained deep learning model (such as U-Net or Mask R-CNN) is used to perform semantic segmentation on the arm region to identify the direction and distribution of blood vessels.

[0036] Based on the segmentation results, the processing unit further extracts the centerline of the blood vessel and calculates the projection position of the blood vessel on the skin surface. To determine the optimal puncture point, the system can comprehensively evaluate the following factors: blood vessel diameter (selecting the thicker part), blood vessel depth (selecting the shallower part), blood vessel direction (selecting the straight segment), and avoiding venous valves and bifurcation points.

[0037] After determining the optimal puncture point, the processing unit generates a virtual marker, such as an "X" mark or a circular bullseye mark, and displays it on the display unit in an overlay manner, precisely aligned with the actual skin location in the live video feed. This marker can be set to a semi-transparent style with an indicator arrow to prompt the nurse that this location is the recommended needle insertion point.

[0038] In addition, the system can overlay a virtual route line of the blood vessel on the screen to help nurses understand the direction in which the needle tip should be advanced after puncture.

[0039] By using image recognition to locate blood vessels and overlaying virtual markers suggesting puncture points, the traditional puncture point selection, which relies on personal experience, is transformed into a visualized, quantitative guide. This technology allows novice nurses to intuitively understand the optimal needle insertion location, reducing the puncture failure rate caused by inaccurate vessel location, while also minimizing pain and discomfort for patients due to repeated punctures.

[0040] To address the challenge of nurses accurately determining the rotation direction and insertion depth when inserting blood collection tubes, visual guidance information includes dynamic arrows superimposed near the blood collection tube or needle holder during the insertion phase to indicate the rotation direction or insertion depth.

[0041] Specifically, when the nurse completes the puncture and prepares to insert the vacuum blood collection tube, the processing unit overlays a dynamic arrow on the display unit based on the image recognition results to assist in the cannulation operation.

[0042] Specifically, the system first uses a target detection module to identify the positions of the blood collection needle (tube plug puncture needle) on the needle holder and the rubber stopper of the vacuum blood collection tube. Based on their relative positions, the system calculates the angle at which the rubber stopper needs to be rotated or the direction of insertion.

[0043] The forms in which dynamic arrows can be displayed can include: Rotation direction arrow: When the blood collection tube needs to be rotated at a certain angle to align with the blood collection needle, the arrow on the screen indicates the rotation direction (clockwise or counterclockwise) in an arc or a circular manner. The arrow length or dynamic animation indicates the required rotation angle.

[0044] Insertion depth indication: During the insertion process, the system monitors the position of the blood collection tube plug relative to the blood collection needle in real time, and indicates the current insertion depth to the nurse through the change in the length of the vertical arrow or the form of a progress bar. When the preset full puncture depth is reached, the arrow turns green and is accompanied by a "in place" prompt.

[0045] The dynamic arrow updates its position and orientation in real time as the blood collection tube moves, ensuring that it always maintains a relative positional relationship with the target object in the image.

[0046] By overlaying dynamic arrows to indicate the direction of rotation or insertion depth, the error-prone procedure of inserting blood collection tubes is transformed into visual guidance. This method enables nurses to accurately control the insertion angle and depth, avoiding problems such as failed rubber stopper puncture, bent needles, or insufficient blood collection due to improper operation. It is especially valuable for novice nurses in terms of operational guidance.

[0047] To address the issue that a single visual system cannot accurately perceive the completion status of key steps when the operating area is obstructed, the processing unit identifies key sound events based on the environmental sound, including one or more of the following: the crisp sound of the protective sleeve being pulled out, the muffled sound of the blood collection tube stopper being punctured, and the sound of the pressure band being released.

[0048] Specifically, the sound analysis module in the processing unit processes the audio stream input from the sound acquisition unit in real time using the following method: First, the audio stream is preprocessed, including noise reduction, echo cancellation, and endpoint detection, to extract valid sound segments. Then, acoustic features of each sound segment are extracted, such as Mel-frequency cepstral coefficients (MFCC), short-time energy, and zero-crossing rate.

[0049] The processing unit is pre-programmed with a trained deep learning classification model (such as a convolutional neural network or a long short-term memory network) for classifying and recognizing sound segments. This model is pre-trained by collecting sound data from real blood collection environments and manually labeling it.

[0050] In this embodiment, the key sound events that the system can identify include, but are not limited to: The crisp sound of the protective sheath being pulled out: It has the characteristics of high frequency, short duration, and suddenness, corresponding to the nurse's action of removing the protective sheath from the tip of the blood collection needle; The muffled sound of the blood collection tube stopper being punctured: It has the characteristics of low to medium frequency, short duration, and damped attenuation, corresponding to the instantaneous sound when the blood collection tube stopper is punctured by the blood collection needle. The sound of a tourniquet being released: It has an elastic rebound characteristic, corresponding to the "snap" sound when a nurse releases the tourniquet.

[0051] The system sets a confidence threshold for each type of sound event. When the recognition confidence exceeds the threshold, the corresponding event signal is output.

[0052] By adding sound acquisition and recognition functions, the system can perceive auxiliary signals in addition to visual information. Sound events such as the removal of the protective sleeve, the puncture of the rubber stopper, and the loosening of the tourniquet are all indicative signals with clear temporal significance in the blood collection process. By using these signals to corroborate visual information, the perception blind spots of a single visual modality in occluded scenarios can be effectively compensated for.

[0053] In order to establish a fusion judgment mechanism of vision and sound to solve the problem of misjudgment caused by interference in single-modal recognition, the processing unit is configured to: determine that the step is completed when both the visual recognition result and the sound recognition result indicate that the same blood collection step is completed; and enter an uncertain state and output a verification prompt when only a single modality recognizes that the step is completed.

[0054] Specifically, when determining whether a blood collection step (such as "removing the protective cover", "puncture completed", "cannula completed", "releasing the tourniquet") has been completed, the processing unit adopts a visual and auditory dual-modal fusion strategy.

[0055] Taking the "removing the protective cover" step as an example: Visual determination: If the target detection module detects that the protective cover has detached from the blood collection needle in consecutive frames and the blood collection needle body is exposed, it will output a "remove protective cover" visual confirmation signal. Sound detection: If the sound analysis module detects a crisp sound when the protective case is pulled out, it will output a "protective case removed" sound confirmation signal. Fusion determination: When both visual and audio confirmation signals are present, the processing unit determines that the "remove the protective cover" step is completed; when only a single modality is confirmed, the system enters an uncertain state.

[0056] In uncertain situations, the processing unit outputs a prompt message such as "Please confirm that the protective cover has been removed" through the human-computer interaction unit, but does not perform any blocking or alarm actions to avoid misjudgment interfering with normal operation. At the same time, the system marks the image and audio clips before and after this moment as low-confidence samples and stores them in the storage unit for subsequent manual review or model optimization.

[0057] By cross-referencing visual and auditory modalities, redundant determination of the completion status of the blood collection process is achieved. This fusion strategy effectively addresses scenarios where single-modal recognition fails (such as visual failure when the needle tip is obscured by a hand, or auditory failure due to excessive environmental noise), significantly improving the robustness and accuracy of the determination. Simultaneously, a conservative strategy of "providing without blocking" for inconsistent states avoids interference with normal operations due to modal conflicts.

[0058] To address the issue that existing blood collection assistance systems have fixed prompting strategies that cannot adapt to different user skill levels and usage scenarios, the preset modes include a teaching mode and an efficient mode. In the teaching mode, the processing unit controls the display unit and / or speaker to output a step-by-step explanation of the entire process and to score the operation. In the high-efficiency mode, the processing unit only outputs alarm prompts for detected operational errors and does not output prompts for regular procedures.

[0059] Specifically, the processing unit has two built-in preset modes: 1. Teaching Mode: Suitable for training intern nurses or mentoring newly hired nurses. In this mode, the system provides detailed operation guidance and feedback: The entire process is explained step by step: voice guidance is output through the speaker, such as "Please take out the blood collection needle and check if the packaging is intact", "Now remove the protective cover and listen for a crisp sound", "Now perform the puncture, please align with the virtual marker on the screen", etc. Operational scoring: The system provides quantitative scoring based on indicators such as operational standardization, time taken, and completion of key steps, and generates a scoring report after the operation is completed for the instructors to refer to. Error correction: When an operational error is detected, the system provides a detailed explanation of the cause of the error and the correct operating procedure, rather than simply issuing an alarm.

[0060] 2. High-Efficiency Mode: Suitable for peak outpatient hours or scenarios involving skilled nurses. In this mode, the system simplifies prompts: Only output alarm prompts for detected operational errors (such as "Incorrect cap color, please check"); Do not output standard step prompts (such as "Please remove the lancet" or "Please remove the protective cover"). Visual guidance information (such as virtual puncture points) remains displayed, but voice prompts are significantly reduced to minimize interference with the operator.

[0061] Furthermore, mode switching can be triggered in the following ways: Manual selection: Nurses can manually select the current mode via a touchscreen or physical button; Automatic identification: The system automatically switches based on time period (e.g., setting to high-efficiency mode during weekdays and teaching mode during teaching periods) or nurse's identity information (identified via RFID card).

[0062] By setting up teaching and high-efficiency modes and supporting adaptive switching, the system can provide differentiated interactive experiences based on different user needs and usage scenarios. The teaching mode meets the need for detailed guidance in training scenarios, while the high-efficiency mode avoids redundant prompts that may interfere with experienced users, improving the system's applicability and user acceptance in different application scenarios. This modular design allows the same hardware system to serve the entire chain of needs, from novice training to efficient clinical practice.

[0063] To address the data processing challenges of the system under uncertain recognition conditions and to provide a data foundation for continuous model optimization, the processing unit is also used to: mark the corresponding image and sound segments as low-confidence samples and store them in the storage unit when visual recognition and sound recognition make inconsistent judgments on the completion status of a step, for use in subsequent model optimization.

[0064] Specifically, when the processing unit encounters an inconsistency during the visual and sound fusion determination process (i.e., only one of the visual and sound confirmation steps is completed), the system automatically extracts image and sound segments of preset durations (e.g., 2 seconds before and after) before and after the current moment and marks them as "low confidence samples".

[0065] Each low-confidence sample is accompanied by metadata, including: timestamp; nurse ID; step name (e.g., "remove protective cover"); visual recognition confidence; voice recognition confidence; and final judgment result (uncertain).

[0066] These low-confidence samples are stored in storage units and can be used later for: Manual review: The head nurse or quality control personnel will conduct regular spot checks to confirm whether there are any actual operational errors that have not been detected by the system (false negatives) or whether the system has misjudged (false positives). Model optimization: Incorporate low-confidence samples into the training dataset, and incrementally train or fine-tune the visual recognition model and the sound recognition model to gradually improve the recognition accuracy of the model in complex scenarios.

[0067] By identifying, labeling, and storing inconsistent samples, a closed-loop mechanism is constructed, moving from "operational uncertainty" to "data feedback" and then to "model optimization." This design not only provides the system with a data foundation for continuous self-evolution but also offers traceable review samples for medical quality monitoring, demonstrating the self-learning and improveability characteristics of intelligent medical devices.

[0068] This embodiment also provides an intelligent blood collection assistance method applicable to any of the above systems, including the following steps: Step S1: Acquire real-time images of the blood collection operation area using the image acquisition unit; Step S2: Collect ambient sounds during the blood collection process using the sound acquisition unit; Step S3: The processing unit identifies key objects and operational actions in the blood collection operation based on the real-time image and generates visual guidance information that is overlaid on the display unit; Step S4: The processing unit identifies key sound events during the blood collection process based on the ambient sound. Step S5: The processing unit fuses the visual recognition results with the sound recognition results to determine the completion status of the blood collection step; Step S6: The processing unit selectively outputs prompt information according to a preset pattern.

[0069] The above steps are not strictly sequential but are processed in parallel: images and sounds are continuously acquired, recognition and fusion are performed in real time, visual guidance information is continuously updated, and prompts are output in a timely manner according to the judgment results and mode settings.

[0070] The method flow corresponding to the system claims is provided, clarifying the complete operation path from multimodal acquisition to fusion determination and then to patterned output. This method flow is highly programmable and scalable, can be directly deployed on existing blood collection workstations without changing nurses' core operating habits, and has high clinical acceptance.

[0071] Furthermore, in order to clarify how to achieve the virtual marking of puncture points and dynamic arrow indication functions, this embodiment refines the visual guidance information generation steps in the method.

[0072] Specifically, in step S3, the specific methods for generating visual guidance information include: During the puncture preparation stage, the processing unit identifies the location of blood vessels through an image segmentation algorithm, calculates the optimal puncture point, and overlays a virtual marker of the suggested puncture point on the display unit. During the blood collection tube insertion stage, the processing unit identifies the relative position of the blood collection tube and the blood collection needle through target detection and generates a dynamic arrow to indicate the rotation direction or insertion depth.

[0073] The above guidance information is displayed in a semi-transparent overlay to ensure that it does not obstruct the operator's observation of the real scene.

[0074] By incorporating specific forms of visual guidance information—virtual puncture points and dynamic arrows—into the methodology, the implementation steps for proactive guidance are clearly defined. This method upgrades the system from "passive monitoring" to "proactive guidance," demonstrating significant teaching support value, especially in training scenarios.

[0075] To clarify how to classify and determine key sound events, this embodiment refines the sound recognition steps in the method.

[0076] Specifically, in step S4, the specific methods for identifying key sound events include: The collected ambient sound is denoised and endpoint detected to extract valid sound segments; Extract acoustic features such as Mel frequency cepstral coefficients from sound segments; The features are classified using a pre-trained deep learning classification model, and the sound event categories are output (protective case pulled out, rubber plug punctured, nasal band loosened, etc.). When the recognition confidence level exceeds the preset threshold, the corresponding event signal is output.

[0077] The specific methods and steps for sound recognition are clearly defined, enabling those skilled in the art to reproduce the function. By introducing sound modalities, the system can acquire auxiliary information beyond vision, especially in scenarios with visual occlusion, where sound signals can serve as independent evidence of the completion of key steps.

[0078] To clarify how to handle situations where visual and audio information are inconsistent, this embodiment refines the fusion determination step in the method.

[0079] Specifically, in step S5, the logic for the fusion determination is as follows: When both visual and audio recognition results indicate that the same blood collection step has been completed, the step is considered complete. When only a single modality recognizes that a step has been completed, it is determined to be an uncertain state and a prompt message such as "Please confirm the operation" is output, but no blocking or alarm is executed; If neither modality recognizes the step as completed, the monitoring status continues.

[0080] The logical rules for fusion-based decision-making are concretized into an executable sequence of steps. This method employs a dual-modal redundancy verification mechanism to ensure decision accuracy while adopting a conservative approach to handling uncertain states, thus avoiding false positive interference caused by misjudgments of a single modality.

[0081] To clarify how to adjust the output strategy according to different scenarios, this embodiment refines the patterned prompting steps in the method.

[0082] Specifically, in step S6, the specific methods for selectively outputting prompt information according to the preset mode include: If the current mode is teaching mode, the system outputs a full-process step-by-step explanation in voice, and scores each operation step in real time. After the operation is completed, a score report is generated. If the current system is in high-efficiency mode, it will only output alarm prompts for detected operational errors and will not output regular step prompts. The mode can be switched automatically by user manual selection, time period recognition, or nurse identity recognition.

[0083] The specific implementation method of the patterned prompts was clarified. This method enables the same system to adapt to different use cases—providing detailed guidance during training and maintaining simplicity and efficiency during high-frequency operations, greatly improving the system's adaptability and user experience.

[0084] The above steps are provided only to help understand the method, structure, and core ideas of this application. Those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the scope of protection of the claims.

Claims

1. An intelligent blood collection assistance system, characterized in that, include: The image acquisition unit is used to acquire real-time images of the blood collection operation area. The sound acquisition unit is used to collect ambient sounds during the blood collection process; The display unit is used to display the real-time image; The processing unit is connected to the image acquisition unit, the sound acquisition unit, and the display unit, respectively, and the processing unit is configured as follows: Based on the real-time image recognition of key objects and operational actions in the blood collection operation, visual guidance information is generated and superimposed on the display unit; Key sound events during the blood collection process are identified based on the aforementioned environmental sound. The visual recognition results are fused with the sound recognition results to determine the completion status of the blood collection step; Selectively output prompt information according to preset patterns.

2. The intelligent blood collection auxiliary system according to claim 1, characterized in that, The visual guidance information includes: during the puncture preparation phase, a virtual marker suggesting a puncture point is superimposed on the display unit based on the blood vessel location located by the real-time image.

3. The intelligent blood collection auxiliary system according to claim 1, characterized in that, The visual guidance information includes: during the blood collection tube insertion phase, a dynamic arrow superimposed near the blood collection tube or needle holder to indicate the direction of rotation or insertion depth.

4. The intelligent blood collection auxiliary system according to claim 1, characterized in that, The key sound events identified by the processing unit based on the environmental sound include one or more of the following: the crisp sound of the protective sleeve being pulled out, the muffled sound of the blood collection tube stopper being punctured, and the sound of the pressure band being released.

5. The intelligent blood collection auxiliary system according to claim 1, characterized in that, The processing unit is configured to: determine that the step is completed when both the visual recognition result and the sound recognition result indicate that the same blood collection step is completed; and enter an uncertain state and output a verification prompt when only a single modality recognizes that the step is completed.

6. The intelligent blood collection auxiliary system according to claim 1, characterized in that, The preset modes include a teaching mode and an efficiency mode; In the teaching mode, the processing unit controls the display unit and / or speaker to output a step-by-step explanation of the entire process and to score the operation. In the high-efficiency mode, the processing unit only outputs alarm prompts for detected operational errors and does not output prompts for regular procedures.

7. The intelligent blood collection auxiliary system according to claim 6, characterized in that, The processing unit is also used to automatically switch the preset mode according to the current time period, nurse identity information, or user manual selection.

8. The intelligent blood collection auxiliary system according to claim 1, characterized in that, The processing unit is also used to: when the visual recognition and sound recognition make inconsistent judgments on the completion status of the step, mark the corresponding image fragments and sound fragments as low-confidence samples and store them in the storage unit for subsequent model optimization.

9. An intelligent blood collection assistance method, applied to the intelligent blood collection assistance system as described in any one of claims 1-8, characterized in that, Includes the following steps: Real-time images of the blood collection area are acquired using the image acquisition unit; The ambient sound during the blood collection process is collected using a sound acquisition unit; The processing unit identifies key objects and actions in the blood collection operation based on the real-time image and generates visual guidance information that is overlaid on the display unit. The processing unit identifies key sound events during the blood collection process based on the ambient sound. The processing unit fuses the visual recognition results with the sound recognition results to determine the completion status of the blood collection step. The processing unit selectively outputs prompt information according to a preset pattern.

10. The intelligent blood collection assistance method according to claim 9, characterized in that, The visual guidance information includes: during the puncture preparation stage, a virtual marker suggesting the puncture point is superimposed on the display unit based on the blood vessel location located by the real-time image; and / or during the blood collection tube insertion stage, a dynamic arrow is superimposed to indicate the rotation direction or insertion depth.

Images