Control methods, devices, electronic equipment and storage media

Abstract

This application provides a control method, device, electronic device, and storage medium that acquires target information of a user in a surgical scenario, including gesture information and / or voice information; determines target control instructions for a target device in the surgical scenario based on the target information; and controls the target device based on the target control instructions. This provides a simpler control method that meets aseptic operation requirements.

Description

Technical Field

[0001] This application belongs to the field of control technology for medical devices, and particularly relates to a control method, system, electronic device and storage medium. Background Technology

[0002] In a standard operating room setting, such as during digital subtraction angiography (DSA)-guided interventional procedures, surgeons must operate under sterile conditions. Due to the numerous and complex functions involved in the procedure, surgeons typically need to issue commands via a bedside control box or touchscreen to execute various functions. This requires multiple clicks and selections to complete a single function, a cumbersome and time-consuming process. Furthermore, the need for sterile gloves during surgery makes the use of touchscreen devices prone to contamination, compromising the sterile environment of the procedure. Summary of the Invention

[0003] Based on this, embodiments of this application provide a control method, system, electronic device, and storage medium that can provide a simpler control method that meets the requirements of aseptic operation.

[0004] This application provides a control system, including:

[0005] Acquire target information of the user in the surgical scenario, the target information including: gesture information and / or voice information;

[0006] Based on the target information, the target control command for the target device in the surgical scenario is determined;

[0007] The target device is controlled based on the target control command.

[0008] In some embodiments, determining the target control command for the target device in the surgical scenario based on the target information includes:

[0009] Based on the target information, a first control command for the target device in the surgical scenario is determined;

[0010] If there are at least two conflicts in the first control command, determine the identity information of the user who issued each first control command;

[0011] The first control commands are filtered based on at least the identity information of the user who issued each first control command to obtain the target control command.

[0012] In some embodiments, the step of filtering the first control commands based at least on the identity information of the user who issued each first control command to obtain the target control command includes:

[0013] The control permission information of each user is determined based on the identity information of the user who issues each first control command.

[0014] Based on the control permission information, each first control instruction is filtered to obtain a second control instruction, and the target control instruction is obtained based on the second control instruction.

[0015] In some embodiments, obtaining the target control command based on the second control command includes:

[0016] In the event that at least two of the second control commands conflict, determine the distance between the user who issued each second control command and the patient in the surgical scenario;

[0017] The priority of each second control command is determined based on the distance;

[0018] The target control instruction is obtained by filtering the second control instructions based on their priority.

[0019] In some embodiments, determining the identity information of the user who issued each first control command includes:

[0020] The image and / or voice information of the user who issues each first control command is input into the first neural network model to determine the identity information of the user who issues each first control command.

[0021] In some embodiments, where the target information includes gesture information and voice information, the method further includes:

[0022] Obtain the user's voice source location and gesture location corresponding to each first control command;

[0023] The sound source location and gesture location of each user are input into the second neural network model to obtain fused features;

[0024] Based on the fusion features, determine whether each first control command meets the requirements of the preset rules;

[0025] The first control commands are filtered based on whether each of the first control commands meets the requirements of preset rules.

[0026] In some embodiments, the method further includes:

[0027] Obtain the location of the user corresponding to each first control command;

[0028] Determine whether each user's location is within the command issuance area;

[0029] The first control command is filtered based on whether each user's location is within the command issuance area.

[0030] In some embodiments, determining the first control command for the target device in the surgical scenario based on the target information includes:

[0031] When the target information is gesture information, multiple consecutive frames of gesture images are input into the third neural network model to obtain the first control command of the target device in the surgical scenario, or, the first control command of the target device in the surgical scenario is determined based on the gesture information and the pre-established mapping relationship.

[0032] When the target information is gesture information and voice information, the gesture information and the voice information are input into the fourth neural network model to obtain the control command.

[0033] In some embodiments, the target control instructions are used to control the target device to perform one of the following functions: switching the displayed medical images, zooming in on the medical images, zooming out on the medical images, controlling the playback of the medical images, switching the image sequence, switching the signal source of the medical images, and controlling the movement of the medical device.

[0034] Secondly, embodiments of this application provide a control device, including:

[0035] The acquisition module is used to acquire target information of the user in the surgical scenario, the target information including: gesture information and / or voice information;

[0036] The first determining module is used to determine the target control command of the target device in the surgical scenario based on the target information.

[0037] The control module is used to control the target device based on the target control command.

[0038] This application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in any of the above-mentioned embodiments.

[0039] This application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in any of the above-described embodiments.

[0040] This application provides a computer program product that, when run on a terminal device, causes the electronic device to execute any of the methods described above.

[0041] This application provides a control method that acquires target information of a user in a surgical scenario, including gesture information and / or voice information; determines target control instructions for a target device in the surgical scenario based on the target information; and controls the target device based on the target control instructions. This method provides a simpler control method that meets the requirements of aseptic operation. Attached Figure Description

[0042] The present application will be described in more detail below with reference to embodiments and the accompanying drawings.

[0043] Figure 1 A schematic diagram of the structure of a control system provided in an embodiment of this application;

[0044] Figure 2 A schematic diagram illustrating the implementation process of a control method provided for the purposes of this application;

[0045] Figure 3 This is a schematic diagram of the structure of a control device provided in an embodiment of this application;

[0046] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0047] In the accompanying drawings, the same parts are referred to by the same reference numerals, and the drawings are not drawn to scale. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0049] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0050] If the application documents contain similar descriptions such as "first, second, third", the following explanation shall be added: In the following description, the terms "first, second, third" are used only to distinguish similar objects and do not represent a specific order of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0051] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0052] Based on the problems in related technologies, the embodiments of this application provide a control method that can be applied to electronic devices such as mobile phones, tablets, wearable devices, in-vehicle devices, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), and medical imaging equipment. The embodiments of this application do not impose any restrictions on the specific type of electronic device.

[0053] Figure 1 A schematic diagram of the structure of a control system provided in this application is shown below. Figure 1 As shown, the control system includes: electronic equipment, rack 1, catheter bed 2, display 3, foot valve 4, touch panel 5, bedside controller 6, computer vision component 7, and voice component 8. The electronic equipment is communicatively connected to rack 1, catheter bed 2, display 3, foot valve 4, touch panel 5, bedside controller 6, computer vision component 7, and voice component 8. The electronic equipment can acquire gesture information and voice information through computer vision component 7 and voice component 8, and generate target control commands based on the gesture information and voice information. The target control commands are used to control rack 1, catheter window 2, and display 3. The foot valve 4, touch panel 5, and bedside controller 6 can also control rack 1, catheter bed 2, and display 3.

[0054] The control method provided in this application can achieve its functions by having the processor of an electronic device call program code, wherein the program code can be stored in a computer storage medium.

[0055] This application provides a control method. Figure 2 A schematic diagram illustrating the implementation flow of a control method provided for the purposes of this application is shown below. Figure 2 As shown, the control methods include:

[0056] Step S101: Obtain the target information of the user in the surgical scenario. The target information includes: gesture information and / or voice information.

[0057] In this embodiment, the surgical scenario refers to the interaction environment between doctors, nurses, and other medical personnel and medical equipment (such as DSA equipment, operating tables, monitors, etc.) in the operating room environment. Surgical scenarios typically have aseptic requirements and are complex to operate and have many functions.

[0058] In this embodiment, the user's target information refers to the operational intent expressed by the user in a surgical scenario through gestures and / or voice. Gesture information refers to instructions expressed by the user through hand movements (such as pointing, sliding, pinching, etc.). Voice information refers to instructions expressed by the user through speech (such as "zoom in on the image" or "switch the signal source").

[0059] In this embodiment, user gestures are captured using a camera or depth sensor, and computer vision technologies (such as OpenPose and MediaPipe) are used to recognize and locate the gestures. User voice commands can be collected via a microphone array, and speech recognition technologies (such as deep learning-based speech recognition models) can be used to convert the speech into text commands.

[0060] Step S102: Determine the target control command for the target device in the surgical scenario based on the target information.

[0061] In this application embodiment, the target device refers to the medical equipment or system that needs to be controlled in the surgical scene, such as: the gantry motion control system of the DSA equipment, the image display system (such as switching medical images and adjusting image layout), the image post-processing system (such as image magnification, reduction, and rotation), the lifting control system of the operating table or gantry, etc.

[0062] In this embodiment, the target control command is a specific operation command generated based on the user's target information, used to control the target device. Examples include: switching the displayed medical images, adjusting the rack's movement angle, zooming in or out of the medical images, and switching the signal source layout.

[0063] In some embodiments, when the target information is gesture information, the recognized gesture information can be mapped to specific control commands. For example, a pinching gesture can be mapped to image shrinking, and a spreading gesture can be mapped to image magnification. A swipe gesture can be mapped to switching image sequences.

[0064] In some embodiments, when the target information is voice information, the recognized voice information can be mapped to specific control commands. For example, the voice command "zoom in image" can be mapped to an image zoom in command, and the voice command "switch signal source" can be mapped to a signal source switching command.

[0065] In some embodiments, when the target information includes gesture information and voice information, the gesture information and voice information can be fused to generate more precise control commands. For example, a user points to an image and says "zoom in" to generate a zoom-in command for that image.

[0066] In this embodiment, gesture information and language information can be input into a Transformer-based multimodal fusion model to generate joint features, and target control commands can be obtained based on the fused features.

[0067] Step S103: Control the target device based on the target control command.

[0068] In this embodiment, the generated target control commands are sent to the target device to drive the device to perform corresponding operations. For example, controlling the gantry of the digital subtraction angiography (DSA) device to move to a designated position, switching the medical images displayed on the screen, and adjusting the field of view of the images.

[0069] In this embodiment, the electronic device can send control commands to the target device via wired or wireless communication protocols. Upon receiving the commands, the target device executes the corresponding operations.

[0070] The method provided in this application embodiment obtains target information of the user in the surgical scene, including gesture information and / or voice information; determines target control instructions for the target device in the surgical scene based on the target information; and controls the target device based on the target control instructions, which can provide a simpler control method that meets the requirements of aseptic operation.

[0071] In some embodiments, step S102, determining the target control command for the target device in the surgical scenario based on the target information, can be achieved through the following steps:

[0072] Step S1021: Determine the first control command for the target device in the surgical scenario based on the target information.

[0073] In this embodiment of the application, the first control command may be a preliminary control command generated based on target information, which may be issued simultaneously by multiple users.

[0074] Step S1022: If there are at least two conflicts in the first control commands, determine the identity information of the user who issued each first control command.

[0075] In this embodiment, a conflict refers to a situation in which multiple users simultaneously issue contradictory or competing first control commands in a surgical scenario. For example, if user A issues a "zoom in" command and user B simultaneously issues a "zoom out" command, there is a conflict. Or, if user A issues a "move gantry to the left" command and user B simultaneously issues a "move gantry to the right" command, these two commands conflict.

[0076] In this embodiment, a conflict can be determined by analyzing the content of the first control instruction. For example, if two instructions operate on the same object but perform opposite operations (such as zooming in and zooming out), then a conflict is determined.

[0077] In this embodiment of the application, the user's identity information refers to the basic information of the user who issues the first control command, including: role, such as the role information of the surgeon, anesthesiologist, nurse, etc.

[0078] In this embodiment, user identity information can be determined using multimodal biometric recognition technologies (such as face recognition and voiceprint recognition). For example, a camera can be used to capture the user's facial image, and face recognition can be performed using a convolutional neural network (CNN). In some embodiments, a microphone can be used to collect the user's voice, and the user's identity can be determined using a voiceprint recognition algorithm.

[0079] Step S1023: The first control commands are filtered based at least on the identity information of the users who issued each first control command to obtain the target control command.

[0080] In this embodiment, the target control command is the final control command obtained after filtering the first control command based on the user's identity information in case of conflict. For example, when user A (the surgeon) and user B (the nurse) issue conflicting commands simultaneously, the first control command sent by user B is removed, while the first control command sent by user A is retained.

[0081] The method provided in this application, through conflict detection, user identification, and instruction filtering, can generate the final target control instruction based on the user's identity information when multiple users issue conflicting instructions simultaneously, thereby ensuring the safety and efficiency of operations in surgical scenarios.

[0082] In some embodiments, step S1023 includes:

[0083] Step S231: Determine the control permission information of each user based on the identity information of the user who issued each first control command.

[0084] In this embodiment, different identity information corresponds to different control permission information. The control permission information can have multiple permission levels, including high permission, medium permission, and low permission. Different permission levels correspond to different operations. For example, high permission: can perform all operations, such as a surgeon; medium permission: can perform some operations, such as an anesthesiologist; low permission: can only view, such as a nurse.

[0085] In this embodiment of the application, after obtaining the user's identity information, the user's permission level can be determined through the mapping relationship between identity and permission level, thereby determining the control permission information of each user.

[0086] In some embodiments, the roles of personnel in the operating room (such as surgeons, anesthesiologists, nurses, etc.) can be predicted in real time by multimodal biometric recognition (such as face recognition, voiceprint recognition) and / or behavioral pattern analysis, combined with deep learning algorithms (such as convolutional neural networks CNN, long short-term memory networks LSTM); different permission levels can be determined based on the predicted identity information.

[0087] Step S232: Filter each first control instruction based on the control authority information to obtain the second control instruction, and obtain the target control instruction based on the second control instruction.

[0088] In this embodiment of the application, in the event of a conflict, the second control instruction is an intermediate control instruction obtained by filtering the first control instruction based on the user's control permission information.

[0089] In this embodiment, the second control instruction can be the target control instruction. In some embodiments, the second control instruction can be further filtered to obtain the target control instruction.

[0090] In this embodiment, control permission information can be used to filter out second control instructions that have the authority to perform control, while first control instructions that do not have the authority are removed. For example, when user A (the surgeon) and user B (the nurse) issue conflicting instructions at the same time, user A has the authority, so the instruction is retained, while user B has the control authority, so user B's instruction is deleted. Here, the target control instruction is user A's instruction.

[0091] The method provided in this application embodiment, through user identity recognition, permission allocation and verification, and instruction filtering, can generate the final target control instruction based on the user's identity information when multiple users issue conflicting instructions at the same time, thereby ensuring the safety and efficiency of operation in surgical scenarios.

[0092] In some embodiments, step S232 can be implemented by the following steps:

[0093] Step S321: If there are at least two conflicts in the second control commands, determine the distance between the user who issued each second control command and the patient in the surgical scene.

[0094] In this embodiment, the distance between the user and the patient refers to the physical distance between the user issuing the second control command and the patient in the surgical scenario. For example, user A is 1 meter away from the patient, and user B is 2 meters away from the patient.

[0095] In this embodiment, the distance between the user and the patient can be measured using a camera, depth sensor, or positioning system (such as UWB or infrared positioning). For example, a depth sensor can be used to capture the user's location information and calculate the distance between the user and the patient. After determining the distance, the measured distance information is associated with the user's identity information to generate a user distance list.

[0096] Step S322: Determine the priority of each second control command based on distance.

[0097] In this embodiment, priority is used to indicate which user's second control command is preferentially selected. In this embodiment, the command of the user closer to the patient has a higher priority.

[0098] Step S323: Filter each second control instruction based on its priority to obtain the target control instruction.

[0099] In this embodiment, the second control commands are filtered based on the distance between the user and the patient. The command with the highest priority is selected as the target control command and sent to the target device for execution.

[0100] Continuing with the example above, if user A (1 meter away from the patient) and user B (2 meters away from the patient) issue commands simultaneously, then user A's second control command will be selected.

[0101] The method provided in this application embodiment can filter out the final target control command based on the distance between the user and the patient when multiple users issue conflicting commands at the same time, thereby ensuring the safety and efficiency of operation in the surgical scenario.

[0102] In some embodiments, determining the identity information of the user who issued each first control command may include:

[0103] The image and / or voice information of the user who issues each first control command is input into the first neural network model to determine the identity information of the user who issues each first control command.

[0104] In this embodiment, the user's image information refers to the facial image of the user captured by a camera. The audio information can be the user's voice commands collected through a microphone array. The first neural network model refers to a deep learning model used to identify user identity information, including: a face recognition model and / or a voiceprint recognition model.

[0105] In this embodiment, a convolutional neural network can be used to extract and match features from a user's facial image. For example, a pre-trained CNN model (such as ResNet or VGG) can be used to extract facial features. The extracted features are then matched with the facial features of a known user to determine the user's identity. Before inputting the facial image into the face recognition model, the image can be preprocessed, for example, by denoising, enhancing, or cropping.

[0106] In this embodiment, a deep learning model (such as a Transformer-based voiceprint recognition model) can be used to extract and match features from a user's speech information. Before processing the speech information using a deep learning model, the collected speech information can be preprocessed, for example, by performing noise reduction, enhancement, or segmentation. For example, a pre-trained voiceprint recognition model can be used to extract speech features. The extracted features are then matched with the voiceprint features of known users to determine the user's identity information.

[0107] In this embodiment, the face recognition module collects facial images of personnel in the operating room and uses a convolutional neural network (CNN) for feature extraction and matching; it also collects voice information of personnel in the operating room and uses a voiceprint recognition algorithm for identity verification; in some embodiments, it can also acquire user behavior patterns and gait characteristics, and use a first neural network model to analyze multi-dimensional data such as behavior patterns and gait characteristics of personnel in the operating room, combined with a long short-term memory network (LSTM) or a graph neural network (GNN) for identity reasoning and prediction. This invention, through the fusion and intelligent analysis of multimodal data, can effectively improve the accuracy and robustness of identity prediction in the operating room, and is applicable to scenarios such as operating room security management, personnel scheduling, and access control.

[0108] The method provided in this application, through the acquisition and processing of image and sound information, combined with the identity recognition technology of the first neural network model, enables the system to accurately determine the identity information of the user who issues the first control command, and dynamically allocate permission levels according to the identity information, thereby ensuring the safety and efficiency of operation in the surgical scenario.

[0109] In some embodiments, where the target information includes gesture information and voice information, after step S1021, the method further includes:

[0110] Step S1: Obtain the sound source location and gesture location of the user corresponding to each first control command.

[0111] In this embodiment, the sound source location refers to the physical location when the user issues a voice command. Sound signals can be collected using a microphone array, and the sound source location can be determined using beamforming technology or Time Difference of Origin (TDOA) algorithms. For example, a microphone array can be used to collect sound signals, and the time difference between the arrival time of the sound source at each microphone can be calculated to determine the sound source location.

[0112] In this embodiment, hand gestures can be captured using a camera or depth sensor, and computer vision technologies (such as OpenPose and MediaPipe) can be used to locate the gestures in real time. For example, a camera can be used to capture hand gestures, and the OpenPose algorithm can be used to identify key points on the hand and calculate the gesture position.

[0113] Step S2: Input the sound source location and gesture location of each user into the second neural network model to obtain fused features.

[0114] In this embodiment of the application, the second neural network model refers to a deep learning model used to fuse sound source location and gesture location information. The second neural network model can be a multimodal Transformer model or a graph neural network (GNN).

[0115] In this embodiment, the second neural network model performs spatiotemporal alignment and feature fusion between the sound source location and the gesture location to generate a joint feature representation. The fused feature refers to the joint feature representation generated by the second neural network model that combines information from both the sound source location and the gesture location.

[0116] Step S3: Determine whether each first control command meets the requirements of the preset rules based on the fusion features.

[0117] In this embodiment of the application, the preset rules include: the sound source position and the gesture position must meet the consistency requirement, otherwise the instruction is considered invalid.

[0118] Step S4: Filter the first control commands based on whether each first control command meets the requirements of the preset rules.

[0119] In this embodiment of the application, a first control instruction that does not meet the requirements of a preset rule is determined to be invalid.

[0120] The method provided in this application embodiment performs multimodal fusion and determination by sound source location and gesture location, which can effectively reduce the probability of false triggering of target devices and improve the reliability and security of voice and gesture interaction.

[0121] The method provided in this application embodiment, based on fusion features, combined with preset permission rules and contextual environment (such as role identity in the operating room and operation scenario), determines the legality of the first control command, and triggers the corresponding command only when the sound source position matches the gesture position and meets the permission requirements.

[0122] In this embodiment of the application, step S1022 can be executed after screening, and the first control instruction in step S1022 is the first control instruction after screening.

[0123] In some embodiments, gesture information can be input into a third neural network model to obtain control commands, and language information can be input into a fourth neural network model to obtain control commands. When the control commands obtained based on gesture information and the control commands obtained based on language information simultaneously meet the control conditions, the corresponding control commands are triggered.

[0124] In some embodiments, after step S1021, the method further includes:

[0125] Step S5: Obtain the location of the user corresponding to each first control command.

[0126] In this embodiment, the user's location refers to the physical location of the user who issued the first control command within the surgical scene. The user's location information can be obtained through a camera, depth sensor, or positioning system (such as UWB or infrared positioning).

[0127] Step S6: Determine whether each user's location is within the command issuance area.

[0128] In this embodiment, the command issuance area refers to a pre-defined area within the surgical scene, where only commands from users located within this area will be responded to. For example, the command issuance area is within a 2-meter radius around the operating table.

[0129] In this embodiment of the application, it can be determined whether the user is within the command issuance area based on the user's location information.

[0130] Step S7: Filter the first control commands based on whether each user's location is within the command issuance area.

[0131] In this embodiment, the first control command is filtered based on whether the user's location is within the command issuance area. First control commands issued by users outside the command issuance area are deleted, while first control commands issued by users within the command issuance area are retained.

[0132] In this embodiment of the application, step S1022 can be executed after screening, and the first control instruction in step S1022 is the first control instruction after screening.

[0133] In some embodiments, step S1021 can be implemented by the following steps:

[0134] Step S211: When the target information is gesture information, input multiple consecutive frames of gesture images into the third neural network model to obtain the first control command of the target device in the surgical scene. The gesture information includes: multiple frames of gesture images, or, determine the first control command of the target device in the surgical scene based on the gesture information and the pre-established mapping relationship, wherein the mapping relationship includes: the correspondence between the gesture information and the first control command of the target device.

[0135] In this embodiment, the third neural network model can be a network model based on the attention mechanism in the Transformer architecture. It can capture N consecutively acquired images from a camera as input and use the attention mechanism in the Transformer architecture to predict low-level instruction tokens end-to-end from image tokens. For example, for 12 input images, the encoder in the network model merges the patch sequence of each image into a global sequence. The Transformer's attention mechanism extracts features from these sequences, establishing dependencies between different frames and capturing temporal contextual information. In the decoding stage, the Transformer decoder interacts with the features output by the encoder, learning how to map image features to instruction tokens. The attention mechanism in the decoder dynamically focuses on the parts of the input image features most relevant to the currently predicted instruction token. Finally, the prediction head transforms the decoder's output into specific instruction tokens, thus achieving end-to-end mapping from image to instruction.

[0136] Network models based on the attention mechanism in the Transformer architecture fully leverage the advantages of the attention mechanism, efficiently utilizing information from contextual images to help accurately translate user intentions into underlying commands and effectively reduce accidental touches. For example, recognized up-and-down swipe gestures correspond to switching between image sequences, left-and-right swipe gestures correspond to switching between image frames, finger spread / pinch corresponds to image zooming, finger drag corresponds to image translation, and crossed hands correspond to closing pop-ups; finger spread and pinch correspond to zooming in and out of the FOV (field of view) of the tablet.

[0137] In this embodiment of the application, the correspondence between gesture information and the first control command can be pre-configured. For example, the correspondence between a hand-crossing gesture and the image pausing, ray lock closing, or motion lock closing, and the correspondence between a hand-raising gesture and the tablet raising, bed raising, or rack raising. After obtaining the gesture information, the first control command can be determined based on the mapping relationship.

[0138] In step S212, if the target information is gesture information and voice information, the gesture information and voice information are input into the fourth neural network model to obtain the first control command.

[0139] In this embodiment, the fourth neural network model can be an end-to-end network model for multimodal information fusion. Mapping speech and gesture information to control commands can be viewed as a sequence modeling problem, and a Transformer is used to learn this mapping. It can utilize large amounts of data and generalize effectively. It takes short sequences of images and natural speech commands at each time step as input and outputs the corresponding control commands.

[0140] The method provided in this application reduces the difficulty for users to memorize low-level commands. Users can directly map control commands to common everyday voice descriptions combined with gesture information observed in image sequences. This reduces the probability of accidental touches while improving generalization performance.

[0141] For example, taking the control system described above, a user can point to any location and say "Hello Xiao U, go here." The electronic device will then send the pointed location to the corresponding device and provide a visual interface prompt (to confirm whether the target location is correct). The user can then move to the desired location with a single click. The display has four signal sources: Live, Ref, 2D, 3D, and ECG signals. In addition to browsing Live real-time images, users may also browse 2D and 3D images. At this time, the user can use voice and gestures to point to the image, such as "Stop this image," "Play this image," "Next image," or "Previous image," and the electronic device will know which signal source to control. Since adjusting the position of the signal source is a high-frequency function for different users or surgical situations, the position of the signal source can be easily changed by pointing, for example, pointing and saying "Adjust these two positions."

[0142] In this embodiment of the application, the target control instruction is used to control the target device to perform one of the following functions: switching the displayed medical image, zooming in on the medical image, zooming out on the medical image, controlling the playback of the medical image, switching the image sequence, switching the signal source of the medical image, and controlling the movement of the medical device.

[0143] This application provides a control device. The various modules and units included in the control device can be implemented by a processor in a computer device; of course, they can also be implemented by specific logic circuits. In the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA), etc.

[0144] Figure 3 This is a schematic diagram of the structure of a control device provided in an embodiment of this application, such as... Figure 3 As shown, the control device 300 includes:

[0145] The acquisition module 301 is used to acquire target information of the user in the surgical scene, the target information including: gesture information and / or voice information;

[0146] The first determining module 302 is used to determine the target control command of the target device in the surgical scenario based on the target information.

[0147] The control module 303 is used to control the target device based on the target control command.

[0148] In some embodiments, the determining module 302 includes:

[0149] The first determining unit is configured to determine a first control command for the target device in the surgical scenario based on the target information.

[0150] The second determining unit is used to determine the identity information of the user who issued each of the first control commands when there are at least two conflicts in the first control commands.

[0151] The first filtering unit is used to filter the first control commands based at least on the identity information of the user who issued each first control command, so as to obtain the target control command.

[0152] In some embodiments, the first filtering unit includes:

[0153] The first determining subunit is used to determine the control permission information of each user based on the identity information of the user who issued each first control command.

[0154] The first filtering subunit is used to filter each first control instruction based on the control permission information to obtain a second control instruction, and to obtain the target control instruction based on the second control instruction.

[0155] In some embodiments, obtaining the target control command based on the second control command includes:

[0156] In the event that at least two of the second control commands conflict, determine the distance between the user who issued each second control command and the patient in the surgical scenario;

[0157] The priority of each second control command is determined based on the distance;

[0158] The target control instruction is obtained by filtering the second control instructions based on their priority.

[0159] In some embodiments, the second determining unit includes:

[0160] The second determining subunit is used to input the image information and / or voice information of the user who issued each first control command into the first neural network model to determine the identity information of the user who issued each first control command.

[0161] In some embodiments, where the target information includes gesture information and voice information, the control module further includes:

[0162] The information acquisition module is used to acquire the user's voice source location and gesture location corresponding to each first control command;

[0163] The fusion module is used to input the sound source location and gesture location of each user into the second neural network model to obtain fused features;

[0164] The second determining module is used to determine whether each first control instruction meets the requirements of a preset rule based on the fusion features;

[0165] The first filtering module is used to filter the first control instructions based on whether each first control instruction meets the requirements of a preset rule.

[0166] In some embodiments, the control module further includes:

[0167] The location acquisition module is used to acquire the location of the user corresponding to each first control command;

[0168] The third determination module is used to determine whether the location of each user is within the command issuance area;

[0169] The second filtering module is used to filter the first control commands based on whether each user's location is within the command issuance area.

[0170] In some embodiments, the first determining unit includes:

[0171] The first input subunit is used to input multiple consecutive frames of gesture images into the third neural network model when the target information is gesture information, so as to obtain the first control command of the target device in the surgical scenario. The gesture information includes: multiple frames of gesture images, or, the first control command of the target device in the surgical scenario is determined based on the gesture information and a pre-established mapping relationship, wherein the mapping relationship includes: the correspondence between the gesture information and the first control command of the target device.

[0172] The second input subunit is used to input the gesture information and the voice information into the fourth neural network model to obtain the first control command when the target information is gesture information and voice information.

[0173] In some embodiments, the target control instructions are used to control the target device to perform one of the following functions: switching the displayed medical images, zooming in on the medical images, zooming out on the medical images, controlling the playback of the medical images, switching the image sequence, switching the signal source of the medical images, and controlling the movement of the medical device.

[0174] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application.

[0175] Figure 4 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application, such as... Figure 4 As shown, the electronic device 300 of this embodiment may include: at least one processor 30 ( Figure 4Only one processor 30, memory 31, and computer program 32 stored in memory 31 and executable on at least one processor 30 are shown. When the processor 30 executes the computer program 32, it implements the steps in any of the above method embodiments, or the processor 30 executes the computer program 32 to implement the functions of each module / unit in the above system embodiments.

[0176] For example, computer program 32 may be divided into one or more modules / units, one or more of which are stored in memory 31 and executed by processor 30 to complete this application. One or more modules / units may be a series of computer program 32 instruction segments capable of performing a specific function, which describe the execution process of computer program 32 in electronic device 300.

[0177] This application also provides a computer-readable storage medium storing a computer program 32, which, when executed by a processor 30, implements the steps described in the above-described method embodiments.

[0178] This application provides a computer program product that, when run on an electronic device, enables the electronic device to perform the steps described in the various method embodiments above.

[0179] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program 32 instructing related hardware. The computer program 32 can be stored in a computer-readable storage medium, and when executed by the processor 30, it can implement the steps of the various method embodiments described above. The computer program 32 includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. A computer-readable medium can include at least: any entity or device capable of carrying computer program code to a terminal, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0180] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0181] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0182] In the embodiments provided in this application, it should be understood that the disclosed apparatus / network devices and methods can be implemented in other ways. For example, the apparatus / network device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0183] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0184] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A control method, characterized in that, include: Acquire target information of the user in the surgical scenario, the target information including: gesture information and / or voice information; Based on the target information, the target control command for the target device in the surgical scenario is determined; The target device is controlled based on the target control command.

2. The method according to claim 1, characterized in that, The step of determining the target control command for the target device in the surgical scenario based on the target information includes: Based on the target information, a first control command for the target device in the surgical scenario is determined; If there are at least two conflicts in the first control command, determine the identity information of the user who issued each first control command; The first control commands are filtered based on at least the identity information of the user who issued each first control command to obtain the target control command.

3. The method according to claim 2, characterized in that, The step of filtering the first control commands based at least on the identity information of the user who issued each first control command to obtain the target control command includes: The control permission information of each user is determined based on the identity information of the user who issues each first control command. Based on the control permission information, each first control instruction is filtered to obtain a second control instruction, and the target control instruction is obtained based on the second control instruction.

4. The method according to claim 3, characterized in that, The process of obtaining the target control command based on the second control command includes: In the event that at least two of the second control commands conflict, determine the distance between the user who issued each second control command and the patient in the surgical scenario; The priority of each second control command is determined based on the distance; The target control instruction is obtained by filtering the second control instructions based on their priority.

5. The method according to claim 2, characterized in that, The process of determining the identity information of the user who issued each first control command includes: The image and / or voice information of the user who issues each first control command is input into the first neural network model to determine the identity information of the user who issues each first control command.

6. The method according to claim 2, characterized in that, When the target information includes gesture information and voice information, the method further includes: Obtain the user's voice source location and gesture location corresponding to each first control command; The sound source location and gesture location of each user are input into the second neural network model to obtain fused features; Based on the fusion features, determine whether each first control command meets the requirements of the preset rules; The first control commands are filtered based on whether each of the first control commands meets the requirements of preset rules.

7. The method according to claim 2, characterized in that, The method further includes: Obtain the location of the user corresponding to each first control command; Determine whether each user's location is within the command issuance area; The first control command is filtered based on whether each user's location is within the command issuance area.

8. The method according to claim 2, characterized in that, The first control command for determining the target device in the surgical scenario based on the target information includes: When the target information is gesture information, multiple consecutive frames of gesture images are input into the third neural network model to obtain the first control command of the target device in the surgical scenario, or, the first control command of the target device in the surgical scenario is determined based on the gesture information and the pre-established mapping relationship. If the target information is gesture information and voice information, the gesture information and the voice information are input into the fourth neural network model to obtain the first control command.

9. The method according to any one of claims 1 to 8, characterized in that, The target control command is used to control the target device to perform one of the following functions: switch the displayed medical images, zoom in on the medical images, zoom out on the medical images, control the playback of the medical images, switch the image sequence, switch the signal source of the medical images, and control the movement of the medical device.

10. A control device, characterized in that, include: The acquisition module is used to acquire target information of the user in the surgical scenario, the target information including: gesture information and / or voice information; The first determining module is used to determine the target control command of the target device in the surgical scenario based on the target information. The control module is used to control the target device based on the target control command.

Images