Intelligent surgical exterior mirror system with voice navigation function

Abstract

The invention discloses an intelligent surgical exterior mirror system with a voice navigation function. The intelligent surgical exterior mirror system comprises a trolley, image display equipment, camera equipment, a mechanical arm, a voice acquisition device, a voice output device, an operation screen and a safety detection device. Voice of medical staff can be collected on site, the control instruction is generated through recognition, corresponding specific equipment is accessed to achieve control, the medical staff can operate the exterior mirror more conveniently in the operation process, and therefore the efficiency and smoothness of the operation are guaranteed. In the voice navigation process, the system has a safety detection function and a path planning function, the safety of a patient is ensured while an image is quickly obtained, and the safety and quality of an operation are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical devices and relates to an intelligent surgical exoscopic system with a voice navigation function. Background Art

[0002] Exoscopic technology is a powerful auxiliary means of surgery to magnify and image the surgical area during surgery. However, the surgical exoscopic systems currently available on the market require medical workers to manually adjust the equipment during the imaging process to ensure that the exoscopic device can produce a clear image on the display screen. However, manual operation during surgery will affect the smoothness and efficiency of the surgery. Therefore, there is an urgent need for an exoscopic system with intelligent automatic movement capabilities, such as motion control through voice navigation.

[0003] In addition to considering automatic movement, the safety of system movement must also be fully considered. During the movement, the safety of patients and equipment must not be affected.

[0004] Therefore, there is an urgent need for a surgical exoscopic system with voice navigation function to address the deficiencies in existing technologies, provide more intelligent services and a safer use environment, and meet the needs of modern surgical operations. Summary of the invention

[0005] Existing exoscopic systems often require manual operation by doctors during operation to achieve the purpose of obtaining an ideal image. During the operation, in order to meet the needs of the operation, it is often necessary to adjust the posture and field of view of the exoscopic scope. Therefore, frequent manual operation links bring inconvenience to the corresponding medical staff, and at the same time, the best auxiliary function of the exoscopic system cannot be brought into play. In order to solve the above problems, the present invention discloses an intelligent surgical exoscopic system with a voice navigation function. At the same time, in order to ensure the safe execution of the voice navigation system, a voice keyword database is established in the voice navigation, the voice recognition process is optimized, and invalid instructions are avoided from affecting the operation process. The external instrument tracking setting can obtain the position coordinates of the robot arm in real time, and can ensure the safe movement of the equipment during the voice navigation process.

[0006] The specific plan is as follows:

[0007] An intelligent surgical exoscopic system with a voice navigation function, characterized in that it includes a trolley, an image display device, a camera device, a robotic arm, a voice acquisition device, a voice output device, an operation screen and a safety detection device;

[0008] The trolley includes a box body and base casters arranged at the bottom of the box body, a mechanical arm control system, a light source control system, a voice recognition and processing system and an image processing system are arranged inside the box body, and an image display device is arranged on the top;

[0009] The robotic arm control system is used to receive control instructions issued, control the movement of the robotic arm through mechanical joints, and obtain clear images. The mechanical joints include planar joints, spherical joints, and annular joints.

[0010] The light source control system is used to provide light brightness adjustment, light source stability control and uniform lighting functions;

[0011] The speech recognition and processing system comprises a speech recognition unit, a keyword recognition unit, a control instruction unit and a feedback information receiving unit; the speech recognition unit is used to receive the speech instruction of the user obtained by the speech acquisition device and perform recognition; the keyword recognition unit is used to match and recognize the speech instruction with a keyword database; the control instruction unit is used to recognize the keyword obtained by the keyword recognition unit and generate a control instruction, and after the control instruction is generated, a specific controlled device is accessed; the feedback information receiving unit is used to receive the feedback signal of the safety detection device and assist the medical staff through voice prompts;

[0012] The image processing system is used to provide path planning functions and safety detection functions. The system behavior is controlled based on the analysis results of the voice command. If the movement of the robot arm needs to be controlled, a command to move to a specific position is issued through voice, and the system will automatically plan the path and provide a reasonable movement path of the robot arm. Before the specific execution of the robot arm movement command, the safety detection function module will simulate the movement trajectory of the robot arm and judge in real time whether the process will pose a danger to the surgical constraint area. If it poses a danger, a warning reminder will be issued to cancel the execution of the command, otherwise it will be executed according to the command.

[0013] The display device includes a polarized 4K 3D display screen and a head-mounted display device HMD. The polarized 4K 3D display screen is connected to a box body through a display rotating bracket for fixed-axis rotation. The head-mounted display device HMD includes VR glasses.

[0014] As a further improvement of the present invention, the keyword database is divided into a primary keyword database, a secondary keyword database and a tertiary keyword database according to priority; the primary keyword database includes manual mode and voice navigation mode, etc., the secondary keyword database includes specific operation objects of light source, camera equipment and robotic arm, etc., and the tertiary keyword database corresponds to the secondary keyword database, which is the specific action behavior controlled by voice navigation, including focusing, recording, robotic arm movement, etc.

[0015] As a further improvement of the present invention, the safety detection device includes a tracking device, an optical sensor, a processing unit and a feedback unit; the tracking device is divided into a marking paint coated on the robot arm and a tracking bracelet, the marking paint is sprayed on the robot forearm and the camera arm, and the tracking bracelet is worn by medical staff; the marking paint is used for the robot forearm and the camera arm, and when the robot arm receives a control instruction, the optical sensor obtains the real-time position coordinates of the robot arm's movement by sensing the marking paint; the processing unit stores the initial position coordinates, surgical constraint area, and global coordinates of the robot arm, the processing unit receives the real-time coordinates of the robot arm, the tracking bracelet sends the medical staff's position to the processing unit, the processing unit receives the coordinate information, and the feedback unit feeds the information back to the speech recognition and processing system.

[0016] As a further improvement of the present invention, the image display device is used to display the acquired image.

[0017] As a further improvement of the present invention, the camera device includes an exoscopic imaging optical path and an image sensor, which are used to obtain the image of the surgical area and transmit it to the image processing system.

[0018] As a further improvement of the present invention, the robotic arm includes a robotic main arm, a robotic small arm, and a camera arm, which are used to provide motion control and safety braking functions.

[0019] As a further improvement of the present invention, the voice output device includes a speaker for playing voice prompts and safety alarms.

[0020] As a further improvement of the present invention, the voice acquisition device includes a microphone for acquiring voice instructions issued by the doctor and transmitting the instructions to a voice recognition and processing system in the box.

[0021] As a further improvement of the present invention, the operating screen provides a manual operation mode. The intelligent exterior mirror system has the functions of manual operation and voice navigation. Medical personnel can select one of the modes according to actual conditions. In the manual operation mode, medical staff can manually adjust the light source, turn the video on and off, and move the robotic arm.

[0022] The beneficial effects of the present invention are:

[0023] 1. It can collect the voice of medical staff on site, generate control instructions through recognition, access the corresponding specific equipment to achieve control, and enable medical workers to operate the exoscopic scope more conveniently during the operation, thus ensuring the efficiency and smoothness of the operation.

[0024] 2. During the voice navigation process, it has safety detection and path planning functions, which can ensure the safety of patients while quickly obtaining images, thereby improving the safety and quality of surgery. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a schematic diagram of the three-dimensional structure of the present invention.

[0026] Figure 2 It is a schematic diagram of the voice navigation of the present invention.

[0027] Figure 3 Schematic diagram of the workflow of the voice navigation function in the present invention.

[0028] Figure 4 Schematic diagram of the workflow of the safety detection function in the present invention.

[0029] List of reference numerals:

[0030] 1. Image display device; 2. Operation screen; 3. Box; 4. Mechanical arm; 5. Camera arm; 6. Mechanical arm; 7. Camera equipment; 8. Fluorescent paint; 9. Optical sensor; 10. Processing and feedback unit; 11. Casters; 12. Tracking bracelet. DETAILED DESCRIPTION

[0031] The present invention will be further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0032] The present invention provides an intelligent surgical exoscopic system with a voice navigation function, comprising: a trolley, an image display device, a camera device, a mechanical arm, a voice acquisition device, a voice output device, an operation screen, and a safety detection device.

[0033] The trolley includes a housing 3 and base casters 11. The housing is provided with a mechanical arm control system, a light source control system, a voice recognition and processing system, an image processing system, and a display device control system. The mechanical arm includes a mechanical arm 6, a mechanical arm 4, and a camera arm 5. The mechanical arm is connected to the trolley, and the acquired image can be adjusted by adjusting the mechanical arm and the camera arm.

[0034] The voice recognition and processing system recognizes the voice commands acquired by the voice acquisition device, analyzes them and issues control commands to the controlled device.

[0035] The voice acquisition device includes a microphone, which is used to acquire the voice instructions issued by the doctor and transmit the instructions to the voice recognition and processing system in the box.

[0036] The safety detection device includes a tracking device, an optical sensor 9 , a processing unit and a feedback unit 10 .

[0037] The tracking device is divided into a mechanical arm externally coated with fluorescent paint 8 and a tracking bracelet 12. The optical sensor is used to obtain the position coordinates of the spraying equipment, and the processing unit receives and processes the position information fed back by the optical sensor and the tracking bracelet.

[0038] Example 1

[0039] refer to Figure 1 , Figure 2 , the intelligent exterior mirror system with voice navigation function proposed by the present invention will be described in conjunction with the accompanying drawings.

[0040] Specifically, the medical staff issues a voice command, the voice acquisition device collects the voice information of the medical staff, and transmits the information to the voice recognition and processing system, which recognizes and processes the voice information.

[0041] Specifically, the speech recognition and processing system performs keyword recognition on the acquired speech information, and pairs the acquired keywords with the keyword database. The keyword database is divided into a primary keyword database, a secondary keyword database, and a tertiary keyword database. The primary keyword database includes a manual mode and a voice navigation mode, allowing medical staff to select the appropriate operation mode according to the actual situation of the operation. The secondary keyword database includes light sources, cameras, and robotic arms, which specifically correspond to the controlled devices in the exoscopic system. The tertiary keyword database corresponds to the secondary keyword database, which specifically includes starting recording, stopping recording, brightness adjustment, moving the robotic arm forward, and other controls. During the implementation process, after system judgment, if a primary keyword appears, the operation mode is selected during the operation. After determining the operation mode, the secondary keyword screening begins, and the tertiary keywords continue to be identified after screening.

[0042] Specifically, when medical staff give instructions to the robotic arm to operate in order to obtain clear images, the image processing system automatically generates the optimal path for the exoscope to reach the target site based on the surgical objectives and anatomical structure. Combined with the navigation system, it guides the robotic arm to control the exoscope to move along the planned path, avoiding key blood vessels and nerve tissues.

[0043] Specifically, after the speech recognition and processing system recognizes the keyword, it issues a control instruction to the corresponding controlled unit and accesses the specific equipment to achieve management control.

[0044] The exterior mirror automatically focuses to acquire the image, and the medical staff determines whether the image meets the requirements and gives the next voice instruction.

[0045] Example 2

[0046] Combination Figure 1 , Figure 3 and Figure 4 , based on Example 1, an example is described. Figure 1The safety detection device shown includes a tracking device, an optical sensor 9, and a processing and feedback unit 10. The tracking device includes a fluorescent paint 8 and a tracking bracelet, wherein the fluorescent paint can be selectively applied to the mechanical arm 4 and the camera arm 5, and the tracking bracelet 12 is worn on the hand of the medical staff.

[0047] Reference Figure 3 In the safety detection process shown, the processing and feedback unit 10 first establishes the global spatial coordinates with the initial position of the robot arm, and divides the patient's position coordinates into the surgical constraint area. In combination with Example 1, the control instructions in the voice navigation are issued to the robot arm system, and the robot arm moves according to the path generated by the image processing system. The information obtained by the optical sensor 9 is transmitted to the processing and feedback unit 10 to obtain the position coordinates of the robot arm and the medical staff. Furthermore, the processing and feedback unit determines whether the coordinates conflict and thus affect the operation, and feeds back the information to the voice navigation system, gives a voice prompt and waits for the next instruction.

[0048] The safety detection process is further described. Fluorescent materials are optionally sprayed on the robotic arm 4 and the camera arm 5, and the optical sensor uses a wavelength filter suitable for the fluorescent material used. When the robotic arm or tracking bracelet moves, the processing unit obtains the real-time coordinates of the robotic arm or tracking bracelet based on the mark offset detected by the optical sensor and the initial coordinates. When there are more than two conflicts between the coordinates of the robotic arm, tracking bracelet, and surgical constraint area, the processing and feedback unit transmits the information to the voice recognition and processing system. The processing system issues a control command to the device to terminate the movement, and issues a voice prompt, waiting for the next instruction.

[0049] The technical means disclosed in the scheme of the present invention are not limited to the technical means disclosed in the above-mentioned implementation mode, but also include technical schemes composed of any combination of the above-mentioned technical features. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications are also regarded as the protection scope of the present invention.

Claims

1. An intelligent surgical exoscopic system with voice navigation function, characterized in that: It includes a trolley, an image display device, a camera device, a robotic arm, a voice acquisition device, a voice output device, an operation screen and a safety detection device; The trolley includes a box body and base casters arranged at the bottom of the box body, a mechanical arm control system, a light source control system, a voice recognition and processing system and an image processing system are arranged inside the box body, and an image display device is arranged on the top; The robotic arm control system is used to receive control instructions issued, control the movement of the robotic arm through mechanical joints, and obtain clear images. The mechanical joints include planar joints, spherical joints, and annular joints. The light source control system is used to provide light brightness adjustment, light source stability control and uniform lighting functions; The speech recognition and processing system comprises a speech recognition unit, a keyword recognition unit, a control instruction unit and a feedback information receiving unit; the speech recognition unit is used to receive the speech instruction of the user obtained by the speech acquisition device and perform recognition; the keyword recognition unit is used to match and recognize the speech instruction with a keyword database; the control instruction unit is used to recognize the keyword obtained by the keyword recognition unit and generate a control instruction, and after the control instruction is generated, a specific controlled device is accessed; the feedback information receiving unit is used to receive the feedback signal of the safety detection device and assist the medical staff through voice prompts; The image processing system is used to provide path planning functions and safety detection functions. The system behavior is controlled based on the analysis results of the voice command. If the movement of the robot arm needs to be controlled, a command to move to a specific position is issued through voice, and the system will automatically plan the path and provide a reasonable movement path of the robot arm. Before the specific execution of the robot arm movement command, the safety detection function module will simulate the movement trajectory of the robot arm and judge in real time whether the process will pose a danger to the surgical constraint area. If it poses a danger, a warning reminder will be issued to cancel the execution of the command, otherwise it will be executed according to the command. The display device includes a polarized 4K 3D display screen and a head-mounted display device HMD. The polarized 4K 3D display screen is connected to a box body through a display rotating bracket for fixed-axis rotation. The head-mounted display device HMD includes VR glasses.

2. The intelligent surgical exoscopic system with voice navigation function according to claim 1, characterized in that: The keyword database is divided into a primary keyword database, a secondary keyword database and a tertiary keyword database according to priority; the primary keyword database includes a manual mode and a voice navigation mode, the secondary keyword database includes specific operation objects of light sources, camera equipment and robotic arms, and the tertiary keyword database corresponds to the secondary keyword database, which is a specific action behavior controlled by voice navigation, including focusing, recording, and robotic arm movement.

3. The intelligent surgical exoscopic system with voice navigation function according to claim 1, characterized in that: The safety detection device includes a tracking device, an optical sensor, a processing unit and a feedback unit; the tracking device is divided into a marking paint coated on the robot arm and a tracking bracelet, the marking paint is sprayed on the robot forearm and the camera arm, and the tracking bracelet is worn by medical staff; the marking paint is used for the robot forearm and the camera arm, when the robot arm receives a control instruction, the optical sensor obtains the real-time position coordinates of the robot arm's movement by sensing the marking paint; the processing unit stores the initial position coordinates, surgical constraint area, and global coordinates of the robot arm, the processing unit receives the real-time coordinates of the robot arm, the tracking bracelet sends the medical staff's position to the processing unit, the processing unit receives the coordinate information, and the feedback unit feeds the information back to the speech recognition and processing system.

4. The intelligent surgical exoscopic system with voice navigation function according to claim 1, characterized in that: The image display device is used to display the acquired image.

5. The intelligent surgical exoscopic system with voice navigation function according to claim 1, characterized in that: The camera device is used to obtain images of the surgical area, and the camera device includes an exoscopic imaging optical path and an image sensor; the imaging optical path adopts a dual optical path, the image sensor replaces the eyepiece and is connected to the image processing system, and the image sensor is placed behind the exoscopic imaging optical path.

6. The intelligent surgical exoscopic system with voice navigation function according to claim 1, characterized in that: The robotic arm comprises a robotic main arm, a robotic small arm, and a camera arm, and is used to provide motion control and safety braking functions.

7. The intelligent surgical exoscopic system with voice navigation function according to claim 1, characterized in that: The voice output device is used to play voice operation prompts to remind the doctor to operate during the interaction process or to issue an alarm when the system determines that the movement of the robotic arm during the operation has safety hazards.

8. The intelligent surgical exoscopic system with voice navigation function according to claim 1, characterized in that: The voice acquisition device includes a microphone, which is used to acquire the voice instructions issued by the doctor and transmit the instructions to the voice recognition and processing system in the box.

9. The intelligent surgical exoscopic system with voice navigation function according to claim 1, characterized in that: The operation screen provides a manual operation mode. The intelligent exterior mirror system has the functions of manual operation and voice navigation. Medical personnel can select one of the modes according to actual conditions. In the manual operation mode, medical staff can manually adjust the light source, turn the video on and off, and move the robotic arm.

Images