A puncture navigation interaction system and method based on offline voice and gesture recognition fusion

Abstract

The present application relates to the technical field of robot navigation, in particular to a puncture navigation interaction system and method based on offline voice and gesture recognition fusion, which completes voice collection and recognition locally through an offline voice recognition module, without network transmission, eliminating the risk of patient privacy leakage and the influence of network delay, and ensuring real-time response in the surgical environment; the gesture recognition module adopts a hands-off collection method, so that the surgical operator can input instructions without touching any physical device, strictly meeting the sterile operation specification; the modal fusion decision unit performs consistency verification and priority scheduling on the dual-modal recognition results, outputs a single effective control instruction, effectively avoids instruction conflicts and misoperations, and significantly improves the interaction fault tolerance; the control unit converts the instruction into a driving signal and controls the puncture navigation robot to perform the corresponding action, realizing non-contact, high-reliability and low-delay human-computer interaction, and comprehensively guaranteeing the safety, accuracy and sterility of the puncture navigation surgery.

Description

Technical Field

[0001] This invention relates to the field of robot navigation technology, and in particular to a puncture navigation interaction system and method based on offline voice and gesture recognition fusion. Background Technology

[0002] The puncture navigation robot is a core piece of equipment in minimally invasive puncture surgery, widely used in biopsies, ablation, and particle implantation of solid organs such as the lungs, liver, and prostate. During puncture navigation, the surgeon needs to interact with the navigation system in real time, sending commands such as path adjustment, target confirmation, and needle insertion control. Currently, the interaction methods of puncture navigation systems are mainly divided into two categories: contact and non-contact. Contact interaction (such as keyboards, touch screens, and foot switches) requires the operator to touch the device, which can easily cause contamination of sterile areas, and is cumbersome and affects surgical efficiency. Non-contact interaction mainly includes single-modal voice interaction or gesture interaction. Voice interaction often uses online recognition technology, which requires network data transmission, posing risks of patient privacy leakage, network latency, and recognition failure due to unstable operating room networks. Gesture interaction is mostly used for simple control and cannot cover the entire puncture navigation process. Single-modal error tolerance is low, and surgical risks can easily arise due to misrecognition.

[0003] In existing technologies, some solutions attempt to introduce multimodal interaction, but these are mostly simple superpositions of voice and gestures, lacking deep integration and priority scheduling mechanisms, resulting in a high probability of command conflicts and misoperations. Meanwhile, the voice recognition module has not been specifically optimized for the complex noise environment of the operating room (such as laminar flow fan steady-state noise, electrosurgical pulse noise, monitor alarm sounds, etc.), resulting in insufficient recognition accuracy and anti-interference capabilities, making it difficult to meet the stringent requirements of high reliability and low latency for puncture-guided surgery. Furthermore, existing gesture recognition methods are mostly based on contact sensors or require operators to wear markers, still posing risks to aseptic operation and failing to achieve precise, remote control throughout the entire process.

[0004] Therefore, there is an urgent need for a dual-modal interactive system and method that is adapted to puncture navigation surgery scenarios and takes into account asepticity, real-time performance and safety, in order to solve the problems of aseptic contamination, privacy leakage, low interaction fault tolerance, command conflict and poor real-time performance in existing technologies, thereby assisting doctors to complete complex puncture operations safely and efficiently. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide a puncture navigation interaction system and method based on the fusion of offline voice and gesture recognition.

[0006] In a first aspect, embodiments of the present invention provide a puncture navigation interaction system based on the fusion of offline voice and gesture recognition, comprising: The offline speech recognition module is used to collect and recognize the surgeon's voice commands locally and offline, and generate speech recognition results. The gesture recognition module is used to remotely collect and recognize the hand gesture signals of the surgical operator and generate gesture recognition results; The modal fusion decision unit has its input terminals connected to the output terminals of the offline speech recognition module and the gesture recognition module, respectively. The modal fusion decision unit is used to receive the speech recognition results and gesture recognition results, perform consistency verification and priority scheduling on the two, and output a single effective control command. The control unit, whose input is connected to the output of the modal fusion decision unit, is used to receive valid control commands and convert them into drive signals. The puncture navigation robot has its control end connected to the output end of the control unit. The puncture navigation robot responds to the drive signal to perform puncture navigation actions.

[0007] In conjunction with the first aspect, the offline speech recognition module includes a microphone array, an analog-to-digital converter, a dedicated signal processing chip, and a flash memory; the flash memory internally stores a pre-trained lightweight acoustic model, an instruction matching model, and a fixed set of speech instructions; the dedicated signal processing chip is used to run the lightweight acoustic model and the instruction matching model, and to complete the feature extraction and instruction mapping of the speech signal locally.

[0008] In conjunction with the first aspect, the gesture recognition module includes an optical acquisition unit and a gesture processing unit; the optical acquisition unit is an infrared depth camera used to capture the three-dimensional spatial information of gesture movements remotely; the gesture processing unit is used to extract gesture contour features, joint angle features and motion trajectory features from the three-dimensional spatial information, generate gesture feature vectors, and match the gesture feature vectors with a preset gesture command set.

[0009] In conjunction with the first aspect, the modal fusion decision unit includes: The timestamp alignment subunit is used to take the end time of the first instruction from the speech recognition result and the end time of the second instruction from the gesture recognition result as a unified time reference, and initiates consistency comparison only when the time difference between the end time of the first instruction and the end time of the second instruction is less than a preset value. The priority scheduling subunit is used to execute scheduling strategies based on the operating stage of the puncture navigation system and the type of instruction, such as voice instructions, gesture instructions, or only outputting consistent instructions. The conflict warning subunit is used to generate a pop-up flashing prompt signal and a voice broadcast warning signal when there is a command conflict.

[0010] Secondly, this application also provides a puncture navigation interactive control method based on offline voice and gesture recognition fusion, applied to the system described above; the method includes: Voice command signals are collected through an offline voice recognition module, and gesture action signals are collected remotely through a gesture recognition module. The offline speech recognition module performs front-end audio enhancement processing and endpoint detection on the speech command signal locally to generate an effective speech feature vector; the gesture recognition module preprocesses the gesture action signal and extracts gesture features to generate a gesture feature vector. The offline speech recognition module infers and identifies valid speech feature vectors and generates voice control commands; at the same time, the gesture recognition module matches and processes the gesture feature vectors to generate gesture control commands. Voice control commands and gesture control commands are input into the modality fusion decision unit for consistency verification and priority scheduling, and then the target control command is output. The target control command is sent to the control unit of the puncture navigation system, and the control unit drives the puncture navigation robot to perform the corresponding mechanical action.

[0011] In conjunction with the second aspect, the steps for performing front-end audio enhancement processing on the voice command signal include: The digital speech signal output from the analog-to-digital converter is subjected to front-end audio enhancement processing, which includes adaptive noise suppression and dereverberation processing, to obtain the enhanced speech signal.

[0012] Endpoint detection is performed on the enhanced speech signal, and background suppression processing is performed based on a preset instruction keyword library to extract effective speech feature vectors; wherein, during the background suppression process, the enhanced speech signal is determined to be a valid speech instruction only if it contains keywords from the instruction keyword library.

[0013] Valid speech commands are converted into speech feature vectors to obtain valid speech feature vectors.

[0014] In conjunction with the second aspect, the steps of preprocessing the gesture action signal and extracting gesture features through the gesture recognition module to generate a gesture feature vector include: The gesture signal is denoised to obtain the denoised gesture signal.

[0015] The denoised gesture signal is normalized to obtain the normalized gesture signal.

[0016] Based on the normalized gesture action signal, target category features are extracted to obtain gesture features for multiple target categories.

[0017] Combine all gesture features to generate a gesture feature vector.

[0018] In conjunction with the second aspect, the steps of performing consistency verification and priority scheduling on the input modality fusion decision unit for voice control commands and gesture control commands, and outputting the target control command, include: Obtain the end time of the first command of the voice control command and the end time of the second command of the gesture control command; Calculate the time difference based on the end time of the first instruction and the end time of the second instruction of the gesture control instruction; Determine whether the time difference is greater than or equal to a preset duration threshold; If so, confirm that voice control commands and gesture control commands are independent commands; Based on the priority scheduling rules of voice control commands and gesture control commands, target control commands are generated.

[0019] In conjunction with the second aspect, after determining whether the time difference is greater than or equal to the preset duration threshold, the process also includes: If not, determine whether an emergency stop command exists among the voice control commands and gesture control commands; If so, immediately output an emergency stop command and terminate the subsequent verification process; If not, determine whether the recognition results of voice control commands and gesture control commands are the same; If so, the recognition result will be used as the target control command.

[0020] If not, determine the target control command based on the current operating stage of the puncture navigation system.

[0021] In conjunction with the second aspect, the method also includes: When a command fails to be recognized or is executed abnormally, the modal fusion decision unit immediately triggers an audible and visual alarm signal to prompt the surgical operator to reissue the command. When the command is executed successfully, the control unit displays a success status icon through a visual interface and drives the offline speech recognition module to broadcast a voice prompt.

[0022] The embodiments of this invention bring the following beneficial effects: This application provides a puncture navigation interaction system and method based on offline voice and gesture recognition fusion. The system includes: an offline voice recognition module, used to locally collect and recognize the voice commands of the surgical operator offline, and generate voice recognition results; a gesture recognition module, used to remotely collect and recognize the gesture action signals of the surgical operator, and generate gesture recognition results; a modal fusion decision unit, whose input end is connected to the output end of the offline voice recognition module and the output end of the gesture recognition module respectively, the modal fusion decision unit is used to receive the voice recognition results and gesture recognition results, and perform consistency verification and priority scheduling on the two, and output a single valid control command; a control unit, whose input end is connected to the output end of the modal fusion decision unit, the control unit is used to receive valid control commands and convert them into drive signals; and a puncture navigation robot, whose control end is connected to the output end of the control unit, the puncture navigation robot responds to the drive signal to perform puncture navigation actions.

[0023] This application utilizes an offline speech recognition module to complete speech acquisition and recognition locally, eliminating the need for network transmission, thus removing the risk of patient privacy leakage and the impact of network latency, and ensuring real-time response in the surgical environment. The gesture recognition module adopts a remote acquisition method, allowing the surgical operator to input commands without touching any physical equipment, strictly meeting aseptic operation standards. The modal fusion decision unit performs consistency verification and priority scheduling on the dual-modal recognition results, outputting a single effective control command, effectively avoiding command conflicts and misoperations, and significantly improving the interaction fault tolerance rate. The control unit converts the commands into drive signals and controls the puncture navigation robot to perform corresponding actions, realizing non-contact, highly reliable, and low-latency human-computer interaction, comprehensively ensuring the safety, accuracy, and aseptic nature of puncture navigation surgery.

[0024] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention are realized and obtained in accordance with the structures particularly pointed out in the description, claims and drawings.

[0025] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0026] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0027] Figure 1 This is a schematic diagram of the structure of a puncture navigation interaction system based on offline voice and gesture recognition fusion provided in an embodiment of the present invention; Figure 2 This is a flowchart illustrating the puncture navigation interaction method based on offline voice and gesture recognition fusion provided in an embodiment of the present invention.

[0028] Figure label: 1-Offline speech recognition module, 11-Microphone array, 12-Analog-to-digital converter, 13-Dedicated signal processing chip, 14-Flash memory; 2-Gesture recognition module, 21-Optical acquisition unit, 22-Gesture processing unit; 3-Modal fusion decision-making unit, 31-Timestamp alignment sub-unit, 32-Priority scheduling sub-unit, 33-Conflict early warning sub-unit; 4-Control unit; 5-Puncture navigation robot. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] To facilitate understanding of this embodiment, the technical terms used in this application will be briefly introduced below.

[0031] Salt-and-pepper noise refers to randomly occurring, isolated bright or dark spots in an image, resembling grains of salt and pepper. In 2D images used for gesture recognition, sudden changes in ambient lighting or random sensor interference can introduce salt-and-pepper noise, which is typically eliminated using median filtering.

[0032] Median filtering: a non-linear filtering method that replaces the original value of a pixel with the median gray value of all points in the neighborhood of the pixel. It can effectively remove salt-and-pepper noise while preserving edge information relatively well.

[0033] Outliers: These are isolated points in point cloud data that are far from the main point set. In gesture depth point clouds, pseudo-depth information caused by interference or reflection from other infrared devices is considered an outlier and needs to be removed using spatial connectivity rules.

[0034] Adaptive deblurring algorithm: used to restore image edge blur caused by rapid hand movements. This algorithm dynamically estimates the blur kernel based on local gradient information and deconvolves to restore the contour.

[0035] Convex hull algorithm: Given a set of points on a two-dimensional plane, the convex hull is the smallest convex polygon that contains all points. In gesture contour analysis, the convex hull can be used to locate fingertip positions and calculate the concavity and convexity characteristics of the hand contour.

[0036] Fingertip detection: Based on depth point cloud or contour curvature analysis, the spatial coordinates of the fingertip are located, which is a prerequisite for extracting joint angle features.

[0037] L2 normalization: The sum of squares of the components of the feature vector is reduced to 1, that is, the vector is divided by its Euclidean norm. This is used to eliminate the dimensional differences between different feature dimensions, making the subsequent matching algorithm more stable.

[0038] After introducing the technical terms used in this application, the application scenarios and design concepts of the embodiments of this application will be briefly described below.

[0039] Existing puncture navigation interaction methods have drawbacks such as contact operation leading to sterile contamination, networked voice recognition posing a privacy risk, and low fault tolerance in single-modal interaction.

[0040] Based on this, this application provides a puncture navigation interaction system and method based on offline voice and gesture recognition fusion.

[0041] Example 1 This application provides a puncture navigation interaction system based on offline voice and gesture recognition fusion, combined with Figure 1 As shown, the system includes: an offline speech recognition module 1, a gesture recognition module 2, a modal fusion decision unit 3, a control unit 4, and a puncture navigation robot 5.

[0042] The offline speech recognition module 1 is used to collect and recognize the surgeon's voice commands locally and generate speech recognition results.

[0043] The gesture recognition module 2 is used to collect and recognize the gesture signals of the surgical operator remotely and generate gesture recognition results.

[0044] The input end of the modal fusion decision unit 3 is connected to the output end of the offline speech recognition module 1 and the output end of the gesture recognition module 2 respectively. The modal fusion decision unit 3 is used to receive the speech recognition results and gesture recognition results, and to perform consistency verification and priority scheduling on the two, and output a single effective control command.

[0045] The input terminal of the control unit 4 is connected to the output terminal of the modal fusion decision unit 3. The control unit 4 is used to receive valid control commands and convert them into drive signals.

[0046] The puncture navigation robot 5 has its control end connected to the output end of the control unit 4, and the puncture navigation robot 5 performs puncture navigation actions in response to the drive signal.

[0047] This application utilizes an offline speech recognition module 1 to complete speech acquisition and recognition locally, eliminating the need for network transmission, thus removing the risk of patient privacy leakage and the impact of network latency, and ensuring real-time response in the surgical environment. The gesture recognition module 2 adopts a remote acquisition method, allowing the surgical operator to input commands without touching any physical equipment, strictly meeting aseptic operation standards. The modal fusion decision unit 3 performs consistency verification and priority scheduling on the above dual-modal recognition results, outputting a single effective control command, effectively avoiding command conflicts and misoperations, and significantly improving the interaction fault tolerance rate. The control unit 4 converts the commands into drive signals and controls the puncture navigation robot to perform corresponding actions, realizing non-contact, highly reliable, and low-latency human-computer interaction, comprehensively ensuring the safety, accuracy, and aseptic nature of puncture navigation surgery.

[0048] In conjunction with the first aspect, the offline speech recognition module 1 includes a microphone array 11, an analog-to-digital converter 12, a dedicated signal processing chip 13, and flash memory 14. The flash memory 14 internally stores a pre-trained lightweight acoustic model, an instruction matching model, and a fixed set of speech instructions; the dedicated signal processing chip is used to run the lightweight acoustic model and the instruction matching model, and to complete the feature extraction and instruction mapping of the speech signal locally.

[0049] The microphone array 11 preferably uses a dual-microphone or ring microphone array, which is placed within a range of 30cm to 100cm in front of the operating table, facing the operator's usual standing or sitting position, to ensure the directional acquisition and clarity of voice commands.

[0050] In practical implementation, the microphone array 11 is connected to an audio analog-to-digital converter 12 (such as ADC102) to convert the analog voice signal into a digital signal. In this embodiment, the frequency is set to 16kHz to ensure the integrity of the voice signal. Then, a dedicated signal processing chip 13 (specifically a DSP / MCU processor, such as STM32H743, with a main frequency of 400MHz) calls flash memory 14 (such as... Figure 1 The local inference is performed using a small CNN acoustic model, a command matching model, and a fixed voice command set pre-trained in Flash104. In this embodiment, the fixed voice command set specifically includes 7 commands: "Confirm target point", "Adjust puncture angle +5 degrees", "Adjust puncture angle -5 degrees", "Insert needle", "Retract needle", "Pause", and "Emergency stop", and the total number of command types does not exceed 10.

[0051] The offline speech recognition module 1 operates entirely without network access, with all calculations performed within the dedicated signal processing chip 13. Therefore, the raw speech data never leaves the device, fundamentally eliminating the risk of patient privacy leaks. Furthermore, since there is no need to wait for a cloud response, it avoids the impact of network latency and operating room network instability. Real-world testing shows that on the STM32H743 processor, the single-frame inference latency is ≤15ms, and the total latency for recognizing the entire instruction is ≤80ms, far below the 100ms real-time threshold perceptible to human-computer interaction. Even if the operating room network is interrupted or electromagnetic interference is present, the speech recognition function can still operate stably.

[0052] By using a fixed, limited instruction set (no more than 10 classes) in flash memory 14, the number of output layer nodes in the model is reduced from tens of thousands of classes required for general speech recognition to single digits, significantly reducing the number of model parameters and computational load. Taking the small CNN acoustic model used in this embodiment as an example, combined with INT8 quantization inference technology, the quantized model size can be controlled within 5MB, perfectly adapting to the limited storage and computing resources of dedicated signal processing chips 13 such as STM32H743. Thus, low-power, high-efficiency, network-independent offline voice control is achieved on compact embedded hardware, providing a safe and real-time voice interaction foundation for puncture navigation surgery.

[0053] In conjunction with the first aspect, the gesture recognition module 2 includes an optical acquisition unit 21 and a gesture processing unit 22.

[0054] The optical acquisition unit 21 is an infrared depth camera used to capture three-dimensional spatial information of hand gestures remotely; the gesture processing unit 22 is used to extract gesture contour features, joint angle features and motion trajectory features from the three-dimensional spatial information, generate gesture feature vectors, and match the gesture feature vectors with a preset gesture command set.

[0055] The optical acquisition unit 21 (in this embodiment, an Intel RealSense D435i depth camera) is mounted above the operating table of the puncture navigation robot, with its lens facing the area of ​​the surgeon's hand movement. The working distance is set to 0.5 meters to 2.0 meters. This distance range ensures both the spatial resolution of the gestures and allows the surgeon to make gestures naturally in a sterile area without touching any equipment. The optical acquisition unit is used to capture the three-dimensional spatial information of the gestures remotely, including two-dimensional image information and depth point cloud information. It adopts active infrared structured light or stereo vision fusion technology and can stably output the spatial position, posture angle, and movement trajectory of the gestures under normal lighting conditions in the operating room.

[0056] The gesture processing unit 22 specifically adopts a field-programmable gate array (FPGA) processor, which is connected to the optical acquisition unit through a high-speed data bus to receive raw gesture data in real time.

[0057] The gesture processing unit 22 first performs noise reduction and normalization processing on the acquired signal to filter out ambient light and device interference; then it extracts gesture contour features, joint angle features, and motion trajectory features to generate a high-dimensional gesture feature vector; finally, it matches the feature vector with a preset gesture command set. The preset gesture command set corresponds one-to-one with the offline voice command set, and in this embodiment, it specifically includes: OK gesture for confirming the target point, left and right hand swing for adjusting the puncture angle, fist clenching and nodding for needle insertion, palm pushing backward for needle withdrawal, palm pressing down for pause, and hands crossing for emergency stop.

[0058] Through the aforementioned air-to-air depth imaging and real-time feature matching, the gesture recognition module 2 eliminates the need for operators to wear any markers or touch the touchscreen or keyboard, thus meeting the aseptic operation standards of the operating room from the outset. At the same time, the parallel processing architecture of the FPGA enables the gesture recognition response latency to reach the millisecond level. Combined with the anti-ambient light interference design of the infrared camera, it ensures that gesture commands can still be output stably and accurately in surgical environments where interference from equipment such as electrosurgical units and monitors occurs, providing reliable input for the subsequent modal fusion decision unit.

[0059] In conjunction with the first aspect, the modal fusion decision unit 3 includes: timestamp alignment subunit 31, priority scheduling subunit 32, and conflict early warning subunit 33.

[0060] The timestamp alignment subunit 31 is used to take the end time of the first instruction of the speech recognition result and the end time of the second instruction of the gesture recognition result as a unified time reference, and initiates consistency comparison only when the time difference between the end time of the first instruction and the end time of the second instruction is less than a preset value.

[0061] The priority scheduling subunit 32 is used to execute a scheduling strategy based on the operating stage of the puncture navigation system and the type of instruction, which may be based on voice instructions, gesture instructions, or only consistent instructions.

[0062] The conflict warning subunit 33 is used to generate a pop-up flashing prompt signal and a voice broadcast warning signal when there is a command conflict.

[0063] In this embodiment, the timestamp alignment subunit 31 is responsible for using the end time of the instruction from the speech recognition result and the end time of the instruction from the gesture recognition result as a unified time reference. Consistency comparison is only initiated when the time difference between the two end times is less than a preset value (set to 500 milliseconds in this embodiment).

[0064] In actual operation, the offline speech recognition module 1 and gesture recognition module 2 each record the end time timestamp of the instruction (e.g., system clock count) while outputting the recognition instruction.

[0065] The timestamp alignment subunit 31 receives two groups of instructions and their timestamps in real time, and calculates the time difference ΔT. If ΔT < T0, it is determined that the two instructions belong to the same interaction intention in terms of time, and the subsequent content consistency comparison is entered; if ΔT ≥ T0, it is determined that the two instructions are independent and occur at different times, and they are respectively sent to the priority scheduling subunit for processing. Among them, T0 is a preset duration threshold, and this value can be adjusted according to actual needs and is not limited here. In this embodiment, T0 = 500ms.

[0066] This mechanism effectively avoids the incorrect verification caused by the operator issuing different modality instructions successively (such as voice first and then gesture), and at the same time tolerates the normal physiological response delay between the two modalities (usually less than 300ms), improving the robustness of the fusion decision-making.

[0067] The priority scheduling subunit 32 executes a scheduling strategy that takes voice instructions as the main, gesture instructions as the main, or only outputs consistent instructions according to the operating stage of the puncture navigation system and the instruction type. The operating stage of the system is fed back to the modality fusion decision-making unit by the puncture navigation robot in real time, and is specifically divided into: the puncture path planning stage, the stage of waiting for puncture execution, and the puncture pose confirmation stage.

[0068] In this embodiment, the priority scheduling rules are as follows: In the puncture path planning stage, the system waits for the target point to be confirmed. At this time, safety and global confirmation are emphasized. Only when the voice instruction and the gesture instruction are consistent, the confirmation instruction is output and the calculation of the manipulator trajectory path is triggered; if there is an instruction conflict, the "withdraw the needle" or "stop" instruction in the voice instruction is preferentially output to avoid incorrect confirmation.

[0069] In the stage of waiting for puncture execution, the main control is to move the manipulator to the target point, insert the needle or make fine adjustments. At this time, the gesture instruction is more suitable for continuous spatial adjustment. Therefore, when there is an instruction conflict, the gesture instruction is preferentially output.

[0070] In the puncture pose confirmation stage, to ensure the absolute safety of the needle tip pose, only when the voice and gesture instructions are completely consistent is the fine adjustment of the manipulator angle allowed; if they are inconsistent, the "withdraw the needle" or "stop" instruction in the voice instruction is preferentially output.

[0071] In addition, the emergency stop instruction (regardless of voice or gesture) has the highest global priority. Once detected, it is immediately output and all current actions are interrupted. This priority scheduling subunit enables the dual-modal interaction to be adaptively adjusted according to the dynamic requirements of the surgical process, giving full play to the advantage of voice for global confirmation, making use of the intuitiveness of gesture for local fine adjustment, and ensuring a quick response in case of emergency.

[0072] The conflict warning subunit 33 is used to generate a pop-up window flashing prompt signal and a voice broadcast warning signal when there is an instruction conflict.

[0073] When the priority scheduling subunit 32 determines that the contents of two modal instructions are different and cannot be automatically resolved according to the rules of the current stage (for example, in the path planning stage, the voice output is "confirm target point" while the gesture output is "robotic arm retreat"), the conflict warning subunit 33 immediately triggers the warning mechanism, which is implemented as follows: A red flashing warning box pops up on the system operation screen, displaying the message "Command conflict, please re-enter"; at the same time, it drives the audio output unit of the offline speech recognition module to play the same voice prompt (e.g., the speaker says "Command conflict, please re-enter").

[0074] In this way, by alerting the surgical operator through both visual and auditory channels, the operator can immediately perceive any discrepancies and re-enter the correct voice or gesture commands, effectively avoiding surgical risks caused by misoperation. This early warning mechanism has been effectively validated in the puncture path planning and puncture position confirmation stages, for example, improving the safety and fault tolerance of the interaction.

[0075] Modal fusion decision unit 3 integrates offline speech and gesture recognition results into a single effective command through timestamp alignment, stage adaptive priority scheduling, and multi-channel conflict warning. This avoids command conflicts and misoperation problems caused by simple superposition, providing a reliable and safe human-computer interaction guarantee for puncture navigation surgery.

[0076] Secondly, this application provides a puncture navigation interactive control method based on the fusion of offline voice and gesture recognition, applied to the system described above. Combined with... Figure 2 As shown, the method includes: The S110 collects voice command signals through an offline voice recognition module and simultaneously collects gesture signals remotely through a gesture recognition module.

[0077] S120 performs front-end audio enhancement processing and endpoint detection on the voice command signal locally through the offline speech recognition module to generate an effective speech feature vector; and preprocesses the gesture action signal and extracts gesture features through the gesture recognition module to generate a gesture feature vector.

[0078] S130 uses an offline speech recognition module to infer and identify valid speech feature vectors and generate voice control commands; at the same time, it uses a gesture recognition module to match and process gesture feature vectors and generate gesture control commands.

[0079] S140 inputs voice control commands and gesture control commands into the modal fusion decision unit for consistency verification and priority scheduling, and outputs target control commands.

[0080] S150 sends the target control command to the control unit of the puncture navigation system, and the control unit drives the puncture navigation robot to perform the corresponding mechanical action.

[0081] In step S110, voice command signals are acquired through the offline voice recognition module 1, while gesture action signals are acquired remotely through the gesture recognition module 2. Specifically, the microphone array 11 (preferably a dual-microphone array or a ring microphone array) of the offline voice recognition module 1 is positioned within 30cm to 100cm in front of the operating table, facing the surgeon's usual position, to acquire voice commands (e.g., "confirm target point," "insert needle," "emergency stop," etc.). The analog voice signal output by the microphone array 11 is converted into a digital signal by the audio analog-to-digital converter 12 at a sampling rate of 16kHz, ensuring the integrity of the voice signal information. Simultaneously, the optical acquisition unit 21 of the gesture recognition module 2 (e.g., an infrared depth camera, mounted above the operating table of the puncture navigation robot 5, with a working distance of 0.5m to 2.0m) remotely captures the surgeon's gesture action signals (e.g., OK gesture, hand swaying left and right, hands crossed, etc.) using active infrared structured light or stereo vision, and outputs the two-dimensional image and three-dimensional depth point cloud information of the gesture area in real time. The two modules work synchronously and in parallel without interference, providing raw data for subsequent preprocessing.

[0082] Step S120 performs front-end audio enhancement processing and endpoint detection on the acquired voice command signal to extract effective voice feature vectors; performs denoising, normalization and feature extraction on the gesture action signal to generate gesture feature vectors.

[0083] In terms of speech preprocessing: First, an adaptive noise suppression algorithm is used. For steady-state noise of 50Hz to 500Hz generated by the laminar flow fan in the operating room, a Normalized Least Mean Square (NLMS) adaptive filter is used for filtering. For high-frequency pulse noise generated by the electrosurgical unit, a Kalman filter algorithm is used for dynamic prediction and suppression. For sudden non-steady-state noise such as alarm sounds from the monitor, a noise detection module is activated and adaptive threshold filtering is triggered. At the same time, a weighted prediction error (WPE) algorithm is used for dereverberation processing, with a preset reverberation time RT60 of 0.3 to 0.6 seconds. The enhanced speech signal is then processed by a speech activity detection (VAD) algorithm based on the fusion of short-time energy and short-time zero-crossing rate to determine the start and end points, and a background suppression logic is built in: a keyword library containing only core instruction keywords such as "target point," "needle insertion," "needle withdrawal," "pause," "emergency stop," and "angle adjustment" is preset. Speech segments containing these keywords are considered valid instructions and retained; otherwise, they are considered background conversation and discarded. Finally, the effective speech segments are converted into speech feature vectors using MFCC Mel-Cepstral Coefficients or Fbank filtering feature extraction methods.

[0084] In terms of gesture preprocessing: the raw data collected by the infrared depth camera is denoised (filtering ambient light and electromagnetic interference from the device), normalized (uniform scale) and feature extracted to extract gesture contour features, joint angle features and motion trajectory features, and generate high-dimensional gesture feature vectors.

[0085] Subsequently, in step S130, the offline speech recognition module 1 infers and recognizes the speech feature vector locally to obtain the speech control command; and the gesture recognition module 2 recognizes the gesture feature vector to obtain the gesture control command.

[0086] The offline speech recognition module 1 uses a dedicated signal processing chip 13 (such as an STM32H743 with a main frequency of 400MHz, or a TIC6000 series DSP) to call a pre-trained small CNN acoustic model (such as DS-CNN or MobileNetV3) and an instruction matching model in flash memory 14 to perform forward inference on the speech feature vectors and map them to the corresponding fixed speech instructions. Since the total number of instruction sets does not exceed 10 categories (specifically "confirm target point", "adjust puncture angle +5 degrees", "adjust puncture angle -5 degrees", "insert needle", "retract needle", "pause", "emergency stop"), the number of output layer nodes in the model is reduced from tens of thousands to single digits. Combined with INT8 quantization, the model size is controlled to within 5MB. The measured single-frame inference latency is ≤15ms, and the total latency for recognizing the entire instruction is ≤80ms.

[0087] The gesture recognition module 2 uses an FPGA processor (or a high-performance ARM processor) as the gesture processing unit. It matches gesture feature vectors with a preset gesture command set, which corresponds one-to-one with the voice command set: the OK gesture corresponds to "confirm target point", the left and right hand swing corresponds to "adjust puncture angle", the fist clenched and the head nod corresponds to "insert needle", the palm pushes backward corresponds to "withdraw needle", the palm presses down corresponds to "pause", and the crossed hands correspond to "emergency stop". The parallel architecture of the FPGA enables the matching process to be completed in milliseconds, outputting gesture control commands.

[0088] In conjunction with the first aspect, the step S120 of performing front-end audio enhancement processing on the voice command signal includes: S121, performs front-end audio enhancement processing on the digital speech signal output by the analog-to-digital converter. The front-end audio enhancement processing includes adaptive noise suppression and dereverberation processing to obtain the enhanced speech signal.

[0089] S122, perform endpoint detection on the enhanced speech signal and perform background suppression processing based on a preset instruction keyword library to extract effective speech feature vectors; wherein, during the background suppression processing, the enhanced speech signal is determined to be a valid speech instruction only if it contains keywords from the instruction keyword library.

[0090] S123, convert the valid voice command into a voice feature vector to obtain the valid voice feature vector.

[0091] Step S120 performs front-end audio enhancement processing and endpoint detection on the acquired voice command signal to extract effective voice feature vectors; performs denoising, normalization and feature extraction on the gesture action signal to generate gesture feature vectors.

[0092] Step S121 performs front-end audio enhancement processing on the digital voice signal output by analog-to-digital converter 12, specifically including adaptive noise suppression and dereverberation processing.

[0093] Adaptive noise suppression employs targeted suppression strategies for different noise sources in the operating room: For steady-state noise of 50Hz–500Hz generated by the laminar flow system fan, precise filtering is performed using an NLMS (Normalized Least Mean Square) adaptive filter, leveraging its fast convergence speed and small steady-state error to effectively suppress low-frequency steady-state interference; For high-frequency pulse noise generated by the electrosurgical unit, a Kalman filtering algorithm is used, dynamically adjusting filtering parameters by predicting noise signal characteristics in real time to achieve rapid suppression of pulse noise; For sudden non-steady-state noise such as monitor alarm tones, adaptive threshold filtering is triggered in conjunction with the noise detection module to avoid misfiltering valid speech signals.

[0094] Simultaneously, dereverberation processing is performed. Considering the actual operating room environment, the preset reverberation time RT60 range is 0.3–0.6 s. The WPE (Weighted Prediction Error) algorithm is used to estimate and compensate for the room impulse response, effectively eliminating room echoes and speech reverberation, thus improving speech signal clarity. Through this processing, an enhanced speech signal is obtained, providing low-noise, high-clarity input for subsequent endpoint detection.

[0095] Step S122 performs endpoint detection and background suppression processing on the enhanced speech signal. Endpoint detection uses a VAD (Voice Activity Detection) algorithm based on the fusion of short-time energy and short-time zero-crossing rate to determine the start and end positions of speech commands and extract candidate speech segments. Simultaneously, background suppression logic is added: a preset surgical operation command keyword library is used, which only contains core command keywords directly related to puncture navigation operations, such as "target point," "needle insertion," "needle withdrawal," "pause," "emergency stop," and "adjust angle." When VAD detects speech activity, keyword matching is performed on candidate speech segments: if the speech segment contains the above keywords (e.g., "confirm target point," "adjust angle 30 degrees," "pause needle insertion"), it is determined to be a valid speech command and retained; if the detected speech is conversation between the surgeon and the doctor (e.g., "patient's blood pressure is normal," "prepare sterilization instruments," "adjust surgical light angle"), it is determined to be invalid background speech because it does not contain any core command keywords from the keyword library and is directly filtered out. Through the above processing, accurate and filtered valid speech commands are obtained, eliminating interference from non-command background speech.

[0096] Step S123 converts valid speech commands into computer-recognizable speech feature vectors. Specifically, MFCC Mel-Cepstral Coefficients or Fbank filtering feature extraction methods are used to perform framing, windowing, Fourier transform, and Mel-Filter Bank mapping on the determined valid speech segments to generate fixed-dimensional feature vectors. This process preserves semantic information, discards redundant data, and obtains valid speech feature vectors. These feature vectors are then fed into the lightweight acoustic model in the flash memory 14 of the offline speech recognition module 1 for local inference and recognition, laying the foundation for subsequent command mapping.

[0097] In conjunction with the first aspect, step S120, which involves preprocessing the gesture action signal and extracting gesture features through the gesture recognition module to generate a gesture feature vector, includes: S125 performs noise reduction processing on the gesture signal to obtain the noise-reduced gesture signal.

[0098] S126, normalize the denoised gesture signal to obtain the normalized gesture signal.

[0099] S127, target category features are extracted based on the normalized gesture action signal to obtain gesture features for multiple target categories.

[0100] S128, combine all gesture features to generate a gesture feature vector.

[0101] In step S125, the gesture processing unit 22 of the gesture recognition module 2 (specifically executed by an FPGA processor) performs denoising processing on the raw gesture action signals acquired by the optical acquisition unit (infrared depth camera, such as Intel RealSense D435i, installed above the operating table of the puncture navigation robot, with a working distance of 0.5m to 2.0m). The raw signals include two-dimensional image sequences and three-dimensional depth point cloud data, but there are interferences in the operating room such as ambient light fluctuations, electromagnetic radiation from the monitor, and vibration of the robotic arm, which easily generate isolated noise points and false depth information. The gesture processing unit 22 adopts a denoising strategy that combines spatial and temporal domains: For two-dimensional images, mean filtering is applied to eliminate salt-and-pepper noise caused by sudden changes in ambient light.

[0102] For deep point clouds, the consistency of active infrared structured light is used to eliminate outliers caused by other infrared devices.

[0103] For motion blur caused by rapid hand movements, an adaptive deblurring algorithm is used to restore the edge contour.

[0104] The above denoising process yields a denoised gesture signal, providing a clean and accurate input for subsequent normalization.

[0105] Step S126 normalizes the denoised gesture signal to eliminate scale inconsistencies caused by individual operator differences and variations in acquisition distance. Specifically, the gesture processing unit 22 first calculates the bounding box of the hand region using depth point computing, translates the hand position to the origin of the coordinate system (e.g., using the palm or wrist as a reference point), and then uniformly scales the hand size to a preset standard range (e.g., normalizing the palm width to 80±10 pixels and scaling the finger length proportionally). This normalization process ensures consistent spatial representation of the same gesture under different operators or acquisition distances, thereby guaranteeing the robustness and cross-individual consistency of subsequent feature extraction. The normalized gesture signal is then obtained after processing.

[0106] Step S127 extracts three types of core gesture features from the normalized gesture action signal: gesture contour features, joint angle features, and motion trajectory features. The gesture processing unit 22 uses a convex hull algorithm and fingertip detection method to extract contour features such as palm area, number of fingers, and contour concavity / convexity from the binary contour of the hand; it uses depth point computing to calculate the three-dimensional coordinates of the wrist joint, metacarpophalangeal joints, and interphalangeal joints, thereby obtaining the flexion / extension angles of each joint and the angles between adjacent fingers as joint angle features; it tracks the spatial position changes of the palm or a specified fingertip over multiple consecutive frames (e.g., 5-10 frames), calculating motion speed, acceleration, orientation angle, and trajectory curvature to form motion trajectory features. Features of each category are extracted and saved separately, resulting in multiple non-redundant gesture features (e.g., contour feature vectors, angle feature vectors, and trajectory feature vectors), providing structured elements for subsequent fusion to generate a global feature vector.

[0107] Step S128 fuses and concatenates all gesture features (contours, joint angles, and motion trajectories) extracted in step S127 to generate a fixed-dimensional feature vector representing the complete attributes of the gesture. The processing unit connects various features in a preset order (e.g., contour features first, angle features in the middle, and trajectory features last) into a high-dimensional vector (usually 30-50 dimensions) and performs L2 normalization to balance the dimensional influence of different features. This feature vector retains both static shape information (such as the circle shape of the OK gesture) and dynamic motion information (such as the direction and amplitude of the hand's left and right swings), enabling the subsequent matching process with the preset gesture command set to accurately distinguish subtle differences in gestures (e.g., the clenched fist and nod for "inserting the needle" and the palm pushing backward for "removing the needle" have significantly different trajectory directions). The generated feature vector is sent to the modal fusion decision unit in real time for gesture command recognition. Due to the parallel pipeline architecture of the FPGA, the entire denoising, normalization, feature extraction, and fusion process can be completed within milliseconds (≤20ms), ensuring the real-time interactive requirements of puncture navigation surgery.

[0108] In conjunction with the first aspect, step S130 includes: S131, obtain the first instruction end time of the voice control command and the second instruction end time of the gesture control command.

[0109] S132, calculate the time difference based on the end time of the first instruction and the end time of the second instruction of the gesture control instruction.

[0110] S133, determine whether the time difference is greater than or equal to the preset duration threshold.

[0111] If so, proceed with steps S134-S135.

[0112] S134, determine that voice control commands and gesture control commands are independent commands.

[0113] S135 generates target control commands based on the priority scheduling rules of voice control commands and gesture control commands.

[0114] Step S131 is executed by the timestamp alignment subunit 31 of the modal fusion decision unit 3, and is used to obtain the time attributes of the voice control command and the gesture control command respectively. Specifically, when the offline voice recognition module 1 and the gesture recognition module 2 output the recognition result, they synchronously record the end time of the command and append it as a timestamp to the command data.

[0115] The timestamp alignment subunit 31 parses the end time of the first instruction (e.g., system clock count or absolute time) from the speech recognition result and the end time of the second instruction from the gesture recognition result. These two timestamps represent the actual time points when the two modal instructions were input, providing a reference for subsequent timing alignment.

[0116] Step S132 calculates the difference between the end time of the first instruction and the end time of the second instruction. The timestamp alignment subunit 31 subtracts the two timestamps and takes the absolute value to obtain the time difference (absolute value). For example, if the voice instruction ends at 1000ms on the system clock and the gesture instruction ends at 1350ms, the time difference is 350ms. This time difference reflects the time interval between the operator issuing the bimodal instruction and is a key indicator for determining whether the two instructions belong to the same interactive intent.

[0117] Step S133 compares the calculated time difference with a preset duration threshold. In this embodiment, the preset duration threshold is set to 500 milliseconds. This value is determined based on human physiological response characteristics (typically, the interval between bimodal commands is less than 300ms) and operational tolerance. If the time difference is ≥500ms, it indicates that the two commands are far apart in time, and are likely two independent commands issued sequentially by the operator (e.g., first saying "confirm target" verbally, then making the OK gesture 600ms later), rather than a bimodal expression of the same intent. If the time difference is <500ms, it is considered that the two commands are aligned in time and may point to the same operational intent.

[0118] When the time difference is greater than or equal to a preset duration threshold, step S134 determines the two commands as independent commands. This means that the modal fusion decision unit no longer requires their content to be consistent, but instead treats them as two unrelated control requests issued by the operator at different times. For example, if the operator first verbally issues "adjust angle +5 degrees" and then gestures "pause," with a time interval exceeding 500ms, the system will process these two commands separately without triggering a conflict warning due to content inconsistency. This mechanism effectively avoids false verification caused by the operator's natural reaction delay.

[0119] Subsequently, step S135 applies priority scheduling rules to the two instructions that have been determined to be independent, and generates the final output target control instruction.

[0120] The specific rules are as follows: First, check whether the two instructions contain an emergency stop instruction (voice "emergency stop" or gesture "cross hands"). If they do, the emergency stop instruction has the highest priority and is output immediately. If all commands are non-urgent, the priority of outputting a command is determined based on the current operational stage of the puncture navigation system (puncture path planning stage, puncture execution stage, or puncture position confirmation stage). For example, voice commands are the primary command in the planning stage, gesture commands are the primary command in the execution stage, and commands are only output when they match in the position confirmation stage.

[0121] Since the two instructions have been determined to be independent, priority scheduling will output them sequentially in chronological order (processing the instruction that occurs first), or select the most relevant instruction to output based on the stage requirements. Through the above processing, the modal fusion decision unit outputs a unique and reasonable target control instruction, avoiding system chaos caused by the irrelevance between independent instructions, while ensuring priority response to emergencies.

[0122] In conjunction with the first aspect, after step S133, the following also includes: If not, proceed to step S136.

[0123] S136, determine whether there is an emergency stop command among the voice control commands and gesture control commands.

[0124] If yes, proceed to step S1371; otherwise, proceed to step S1372.

[0125] S1371 immediately outputs an emergency stop command and terminates the subsequent verification process.

[0126] S1372, determine whether the recognition results of voice control commands and gesture control commands are the same; If yes, proceed to step S13721; otherwise, proceed to step S13722.

[0127] S13721 uses the identification result as a target control command.

[0128] S13722, determine the target control command based on the current operating stage of the puncture navigation system.

[0129] When step S133 determines that the time difference is less than the preset duration threshold (500ms), it indicates that the voice control command and the gesture control command are closely adjacent in time, and they are very likely to point to the same interaction intent. At this time, the modal fusion decision unit enters the dual-modal consistency verification process: Step S136 performs emergency stop command detection on the content of the voice control command and gesture control command. The emergency stop command is the highest priority control command in puncture-guided surgery, and its source may be the voice command "emergency stop" or the gesture command "cross your hands." The modal fusion decision unit 3 checks the semantic content of the two commands respectively: if at least one of the commands is an emergency stop command, it is determined that an emergency stop command exists, and the process proceeds to step S1371; if neither command is an emergency stop command, the process proceeds to step S1372. This determination ensures that any modal-triggered emergency stop operation can be responded to immediately, without waiting for bimodal consistency.

[0130] When step S136 determines that an emergency stop command exists, step S1371 immediately executes the emergency response. The modal fusion decision unit 3 ignores the content of the other command (even if inconsistent) and directly outputs the emergency stop command as the target control command, terminating all subsequent consistency checks and priority scheduling processes. The emergency control command has the highest priority; regardless of whether voice or gesture recognition detects the emergency stop command, it is output immediately. This mechanism ensures that in the event of a sudden surgical emergency (such as deviation from the puncture path or patient movement), the operator can quickly interrupt all robot actions through any modality, maximizing patient safety. After outputting the emergency stop command, the control unit 4 drives the puncture navigation robot 5 to immediately stop moving, completing the emergency response.

[0131] When step S136 determines that both commands are non-urgent commands, step S1372 performs a content consistency comparison on the recognition results of the two commands. The modal fusion decision unit 3 compares whether the voice control command (e.g., "confirm target point") and the gesture control command (e.g., "confirm target point" corresponding to the OK gesture) point to the same operation command. If they are the same, it means that the bimodal expressions are consistent, and the system can execute the command with high confidence, proceeding to step S13721; if they are different (e.g., the voice is "confirm target point" and the gesture is "robotic arm retract"), there is a command conflict, proceeding to step S13722.

[0132] When the recognition results of voice control commands and gesture control commands are the same, step S13721 directly outputs the consistent command as the target control command. When the recognition results of both correspond to the same control command, the command is confirmed to be valid and output. For example, if the operator simultaneously says "Confirm target point" and makes an OK gesture, the system determines that the two are consistent, outputs the "Confirm target point" command, and triggers the calculation of the robotic arm trajectory path. The dual-modal consistent output mode utilizes the principle of redundancy verification, which greatly reduces the risk of single-modal misrecognition (such as voice misrecognition or gesture jitter) and improves the safety of critical operations.

[0133] When the recognition results of voice control commands and gesture control commands are inconsistent, step S13722 performs priority scheduling based on the current operating stage of the puncture navigation system (provided in real-time by the puncture navigation robot to the modal fusion decision unit), and outputs target control commands suitable for the current surgical scenario. In this embodiment, scheduling rules for three stages are clearly defined: During the puncture path planning phase: The system awaits target confirmation, at which point safety and the accuracy of global commands are paramount. In case of conflicting commands, voice commands take precedence, only controlling needle retraction or stopping. For example, if the voice output is "Confirm target" while the gesture output is "Robotic arm retract," the system prioritizes the voice command "Retract needle" or "Stop" to avoid risks caused by misidentification of the target.

[0134] During the pre-puncture execution phase: the main control involves moving the robotic arm to the target point, inserting the needle, or making fine adjustments. Gesture commands are more suitable for continuous spatial adjustments. When commands conflict, gesture commands take precedence. For example, if the voice says "adjust angle +5 degrees" while the gesture indicates "insert needle," the system will prioritize the gesture command "insert needle," aligning with the operator's intuitive intention.

[0135] During the puncture position confirmation phase: To ensure the absolute safety of the needle tip position, the robotic arm is only allowed to make fine adjustments when the two instructions are consistent; when the instructions are inconsistent, the voice command to withdraw the needle or stop is given priority to ensure that no incorrect action is caused by misjudgment of gestures.

[0136] Meanwhile, when a command conflict is detected, the conflict warning subunit 33 of the modal fusion decision unit 3 is triggered simultaneously: a red flashing warning box pops up on the system operation screen and displays the words "Command conflict, please re-enter". At the same time, the same voice prompt is broadcast through the audio output unit of the offline speech recognition module 1, realizing dual visual and auditory reminders to ensure that the surgical operator corrects the misoperation in time.

[0137] Through the above steps S136 to S13722, the modal fusion decision unit 3 performs complete verification and scheduling of time-aligned dual-modal non-urgent instructions, which not only leverages the high reliability when the dual modes are consistent, but also improves the interaction fault tolerance and security by combining intelligent decision-making during the surgical stage when instructions conflict.

[0138] In conjunction with the second aspect, the method also includes: S210, when instruction recognition fails or execution is abnormal, the modal fusion decision unit immediately triggers an audible and visual alarm signal to prompt the surgical operator to reissue the instruction.

[0139] S220: When the command is executed successfully, the control unit displays a success status icon through the visual interface and drives the offline voice recognition module to broadcast a voice prompt.

[0140] When the offline speech recognition module 1 fails to map the speech feature vector to a valid instruction (e.g., the speech is unclear, contains noise, or is not in the instruction set), or the gesture recognition module 2 fails to match the preset gesture (e.g., the gesture is not standard, the trajectory is incomplete), or the control unit 4 detects an abnormality in the execution of the puncture navigation robot 5 (e.g., the robotic arm movement exceeds the limit, the sensor malfunctions), the modal fusion decision unit 3 immediately triggers an audible and visual alarm signal.

[0141] Specifically, the alarm signal is output through two channels: first, a red warning box pops up on the system operation screen, displaying text prompts such as "Invalid command, please re-enter" or "Execution error, please check the equipment"; second, the audio output unit driving the offline voice recognition module 1 plays the same voice warning content (e.g., the speaker emits "Invalid command, please re-enter"). This dual alarm mechanism ensures that the surgical operator can perceive the abnormality immediately and re-issue the correct voice or gesture command according to the prompts, avoiding misoperation or surgical delays caused by silent failure. The collaborative feedback between the visual interface and voice prompts improves the safety of the interaction.

[0142] When the modal fusion decision unit 3 outputs a valid control command, and the control unit 4 drives the puncture navigation robot 5 to successfully complete the corresponding mechanical action (e.g., target positioning completed, puncture angle adjusted, needle insertion stopped), the system executes a success feedback. Specifically, the control unit 4 sends the execution success status (e.g., "target confirmed", "needle insertion angle adjusted to +5 degrees") back to the modal fusion decision unit 3, and then displays a green success status icon or text prompt through the visual interface. At the same time, it drives the audio output unit of the offline speech recognition module 1 to broadcast a short voice prompt (e.g., "command executed successfully", "target confirmed"). This feedback allows the surgical operator to confirm in real time that the command has been accurately executed without turning their head to observe the robotic arm, forming a complete interactive closed loop of "command input - system response - execution confirmation", effectively reducing the operator's cognitive burden and improving the smoothness and reliability of the surgery.

[0143] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system and apparatus described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0144] Furthermore, in the description of the embodiments of the present invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in the present invention based on the specific circumstances.

[0145] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0146] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0147] Finally, it should be noted that the above embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A puncture navigation and interaction system based on offline voice and gesture recognition fusion, characterized in that, include: The offline speech recognition module is used to collect and recognize the surgeon's voice commands locally and offline, and generate speech recognition results. The gesture recognition module is used to remotely collect and recognize the hand gesture signals of the surgical operator and generate gesture recognition results; The modal fusion decision unit has its input terminals connected to the output terminals of the offline speech recognition module and the gesture recognition module, respectively. The modal fusion decision unit is used to receive the speech recognition results and the gesture recognition results, perform consistency verification and priority scheduling on the two, and output a single effective control command. A control unit, the input of which is connected to the output of the modal fusion decision unit, is used to receive the effective control command and convert it into a drive signal; A puncture navigation robot, the control end of which is connected to the output end of the control unit, the puncture navigation robot performs puncture navigation actions in response to the drive signal.

2. The system according to claim 1, characterized in that, The offline speech recognition module includes a microphone array, an analog-to-digital converter, a dedicated signal processing chip, and a flash memory. The flash memory stores a pre-trained lightweight acoustic model, an instruction matching model, and a fixed set of speech instructions. The dedicated signal processing chip is used to run the lightweight acoustic model and the instruction matching model to perform feature extraction and instruction mapping of the speech signal locally.

3. The system according to claim 1, characterized in that, The gesture recognition module includes an optical acquisition unit and a gesture processing unit; the optical acquisition unit is an infrared depth camera used to capture three-dimensional spatial information of gesture movements remotely; the gesture processing unit is used to extract gesture contour features, joint angle features and motion trajectory features from the three-dimensional spatial information, generate a gesture feature vector, and match the gesture feature vector with a preset gesture command set.

4. The system according to claim 1, characterized in that, The modality fusion decision unit includes: The timestamp alignment subunit is used to take the end time of the first instruction from the speech recognition result and the end time of the second instruction from the gesture recognition result as a unified time reference, and initiates consistency comparison only when the time difference between the end time of the first instruction and the end time of the second instruction is less than a preset value. The priority scheduling subunit is used to execute scheduling strategies based on the operating stage of the puncture navigation system and the type of instruction, such as voice instructions, gesture instructions, or only outputting consistent instructions. The conflict warning subunit is used to generate a pop-up flashing prompt signal and a voice broadcast warning signal when there is a command conflict.

5. A puncture navigation interactive control method based on offline voice and gesture recognition fusion, characterized in that, Applied to the system as described in any one of claims 1-4; the method comprises: Voice command signals are collected through an offline voice recognition module, and gesture action signals are collected remotely through a gesture recognition module. The offline speech recognition module performs front-end audio enhancement processing and endpoint detection on the speech command signal locally to generate an effective speech feature vector; the gesture recognition module preprocesses the gesture action signal and extracts gesture features to generate a gesture feature vector. The offline speech recognition module infers and identifies the effective speech feature vectors and generates voice control commands; simultaneously, the gesture recognition module performs matching processing on the gesture feature vectors to generate gesture control commands. The voice control command and the gesture control command are input into the modality fusion decision unit for consistency verification and priority scheduling, and the target control command is output. The target control command is sent to the control unit of the puncture navigation system, and the control unit drives the puncture navigation robot to perform the corresponding mechanical action.

6. The method according to claim 5, characterized in that, The steps of performing front-end audio enhancement processing on the voice command signal include: The digital speech signal output by the analog-to-digital converter is subjected to front-end audio enhancement processing, which includes adaptive noise suppression and dereverberation processing, to obtain an enhanced speech signal. Endpoint detection is performed on the enhanced speech signal, and background suppression processing is performed based on a preset instruction keyword library to extract effective speech feature vectors; wherein, during the background suppression process, the enhanced speech signal is determined to be a valid speech instruction only if it contains keywords from the instruction keyword library. The effective voice command is converted into a voice feature vector to obtain the effective voice feature vector.

7. The method according to claim 5, characterized in that, The steps of preprocessing the gesture action signal and extracting gesture features through the gesture recognition module to generate a gesture feature vector include: The gesture signal is denoised to obtain a denoised gesture signal. The denoised gesture signal is normalized to obtain a normalized gesture signal. Based on the normalized gesture signal, target category features are extracted to obtain gesture features for multiple target categories. The gesture feature vector is generated by combining all the gesture features.

8. The method according to claim 5, characterized in that, The steps of inputting the voice control command and the gesture control command into the modal fusion decision unit for consistency verification and priority scheduling, and outputting the target control command, include: Obtain the first instruction end time of the voice control command and the second instruction end time of the gesture control command; Calculate the time difference based on the end time of the first instruction and the end time of the second instruction of the gesture control instruction; Determine whether the time difference is greater than or equal to a preset duration threshold; If so, determine that the voice control command and the gesture control command are independent commands; The target control command is generated based on the priority scheduling rules of the voice control command and the gesture control command.

9. The method according to claim 8, characterized in that, After determining whether the time difference is greater than or equal to a preset duration threshold, the method further includes: If not, determine whether an emergency stop command exists in the voice control command and the gesture control command; If so, immediately output the emergency stop command and terminate the subsequent verification process; If not, determine whether the recognition results of the voice control command and the gesture control command are the same; If so, the recognition result shall be used as the target control command. If not, determine the target control command based on the current operating stage of the puncture navigation system.

10. The method according to claim 5, characterized in that, The method further includes: When instruction recognition fails or execution is abnormal, the modal fusion decision unit immediately triggers an audible and visual alarm signal to prompt the surgical operator to reissue the instruction. When the instruction is executed successfully, the control unit displays a success status icon through a visual interface and drives the offline speech recognition module to broadcast a voice prompt.

Images