Man-machine combined automatic camera system based on surgical instrument tail end mark guidance

Abstract

The invention provides a man-machine integrated automatic camera system based on surgical instrument tail end mark guidance. The man-machine integrated automatic camera system comprises a surgical instrument, a surgical instrument tail end mark, a camera module and a continuous automatic focusing control software module. A mark is established at the tail end of a surgical instrument, image feature information of the mark is collected and recognized by a camera module, and then the camera module is controlled by a focusing control software module to track and continuously focus on the mark at the tail end of the instrument, so that continuous and automatic focusing camera shooting of the mark is realized; a target seen by the eyes of an operator is converted into a target focused and clearly shot by a camera device through a mark at the tail end of a surgical instrument, and man-machine combination of automatic shooting in the surgical process is achieved. In the camera shooting target switching process, focusing control from non-automation to automation and from non-continuity to continuity is achieved, the advantages of being high in focusing accuracy, easy and convenient to operate, rapid in response and the like are achieved, whole-process continuous, high-definition and accurate video recording in an operation is facilitated, and teaching is facilitated.

Description

Technical Field

[0001] The present invention relates to the technical field of medical surgical imaging, and particularly to a human-machine combined surgical automatic imaging system based on guiding and focusing at the end of a surgical instrument. Background Art

[0002] Currently, medical surgical imaging is an important part of medical education, and is of great significance for the communication and dissemination of surgical techniques, teaching, recording of surgical procedures, and handling of medical disputes.

[0003] Currently, the specific process of surgical imaging is as follows: as Figure 1 shown, when a surgeon intends to image "Target A", the surgeon's eyes will fixate on this "Target A", and the surgeon's brain commands their hands / feet to adjust the focus control button of the imaging device. The imaging lens is focused until the image of "Target A" is clear, and then the brain commands the hands and feet to stop focusing, thus achieving surgical imaging. However, if intending to image another "Target B", the surgeon needs to repeat the above process again, that is: the surgeon's eyes will fixate on "Target B", and the surgeon's brain commands their hands / feet to re-adjust the focus control button of the imaging device. The imaging lens is focused until the image of "Target B" is clear, and then the brain commands the hands and feet to stop focusing, so as to achieve the switching of the imaging target from Target A to Target B during the surgical process.

[0004] During the process of switching imaging targets, the surgeon and the imaging device are completely separated. It is necessary to rely on the hands or feet of the surgeon, nurse or assistant to control the focus control button of the imaging device, so that the imaging device can refocus on the new target. The patent with the publication number CN 1801887A proposes a surgical imaging device with zoom control. Although this device controls the ultrasonic motor drive mechanism through a remote control button by a person to adjust the focal length of the camera and realizes the observation of details, this system only proposes a new zoom control scheme, and the imaging process still requires a person to adjust the remote control button to focus. Like the traditional zoom imaging system, essentially, they are both "a discontinuous focusing imaging method implemented by the hands and feet of a surgeon or assistant to control the focus button", and have inherent disadvantages such as cumbersome focus adjustment operations, slow response, and long time.

[0005] In addition, since there are often multiple targets in surgical imaging (such as: tumors, and the surrounding blood vessels, nerves or muscles, etc.), and the surgeon often switches imaging targets. Whether using a fixed-focus or zoom imaging device, the imaging device simply does not know and cannot "understand" which tissue or organ detail the surgeon wants to image, that is, it does not know where to focus; every time the target is changed, the surgeon or assistant needs to manually or foot-operatedly readjust the focal length again.

[0006] Therefore, to implement a surgical camera system that "does not require manual or foot adjustment of the focal length by surgeons, nurses, or other personnel, and the camera can quickly identify and track the surgeon's switching of the camera target" can solve the above-mentioned defects, achieve continuous and high-definition video recording throughout the operation, reduce the operation time, and has important value and significance for medical imaging. Summary of the Invention

[0007] In view of the above-mentioned disadvantages of the existing surgical camera technology, the purpose of the present invention is to provide a human-machine combined automatic camera system based on the guidance of the end marker of the surgical instrument to solve the above technical problems.

[0008] To achieve the above purpose and other related purposes, the present invention provides a human-machine combined automatic camera system based on the guidance of the end marker of the surgical instrument. The system includes: a surgical instrument, an end marker of the surgical instrument, a camera module, and a focus control software module; wherein, the end marker of the surgical instrument is established at the end of the surgical instrument and is set within the capture range of the camera module; the camera module is used to collect the image information of the end marker of the surgical instrument; the focus control software module is communicatively connected to the camera module and is used to continuously control the camera module to track and focus on the end marker of the surgical instrument after identifying the end marker of the surgical instrument.

[0009] In an embodiment of the present invention, the camera module includes one of a liquid lens module, a non-liquid lens module, and a fixed-focus lens module.

[0010] In an embodiment of the present invention, the focus control software module includes: an image storage unit for storing the image information collected during the focusing process; the image storage unit has at least two independent storage areas, one of which is used to store the unprocessed images, and the other is used to store the video information of the entire surgical process; an image processing unit connected to the image storage unit for identifying the collected image information and generating a corresponding focus control signal after identifying the end marker of the surgical instrument; a focus control driving unit communicatively connected to the image processing unit for receiving the focus control signal from the image processing unit and driving the camera module to move and focus in the Z-axis direction to control the camera module to track and focus on the end marker of the surgical instrument in the Z-axis direction.

[0011] In an embodiment of the present invention, the focus control driving unit includes: a focus control input end for inputting the set focus control parameters; a driving chip connected to the focus control input end for receiving the focus control signal and the focus control parameters and driving the camera module to perform corresponding focusing operations.

[0012] In an embodiment of the present invention, the image processing unit is configured to identify and compare the collected image information, and after identifying the mark at the end of the surgical instrument, generate a corresponding focusing control signal that can improve the image clarity according to the change in image features.

[0013] In an embodiment of the present invention, the image processing unit is configured to generate a corresponding focusing control signal according to the change in the clarity of the image after identifying the mark at the end of the surgical instrument.

[0014] In an embodiment of the present invention, after identifying the mark at the end of the surgical instrument, generating a corresponding focusing control signal according to the change in image clarity includes: after identifying the mark at the end of the surgical instrument, generating a primary focusing control signal to control the camera module to finely adjust the focal length forward or backward; after the camera module finely adjusts the focal length, generating a corresponding secondary focusing control signal according to the change in the clarity of the collected image information to control the camera module to coarsely adjust the focal length forward or backward and improve the clarity of the collected image information.

[0015] In an embodiment of the present invention, the focusing control parameters include: the feature information of the mark at the end of the surgical instrument to be tracked.

[0016] In an embodiment of the present invention, the feature information of the mark at the end of the surgical instrument includes one or more of: mark color, mark brightness, mark shape, mark graphics, and mark patterns.

[0017] In an embodiment of the present invention, the mark at the end of the surgical instrument is detachable or fixed to the end of the surgical instrument.

[0018] As described above, the present invention is a human-machine combined automatic imaging system guided by a mark at the end of a surgical instrument, including: a surgical instrument, a mark at the end of the surgical instrument, a camera module, and a continuous automatic focusing control software module. It has the following beneficial effects: By establishing a mark at the end of the surgical instrument, the camera module collects and identifies the image feature information of the mark, and then the focusing control software module controls the camera module to track and continuously focus on the mark at the end of the instrument, thereby realizing continuous automatic focusing imaging of the mark; Through the conversion of "what the surgeon sees with the eyes" to "what the end of the surgeon's instrument (mark) points to", and "what the end of the surgeon's instrument points to" to "what is recognized and focused by the imaging device", the human-machine combination of automatic imaging during the surgical process is realized. During the switching process of the imaging target, the present invention realizes the conversion of focusing control from non-automatic to automatic and from non-continuous to continuous, and has the advantages of high focusing accuracy, simple operation, and fast response, which is beneficial to continuous, high-definition, and accurate video recording during the operation and convenient for teaching. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1Shown is a schematic diagram of the implementation process of an existing medical surgical imaging solution.

[0020] Figure 2 Shown is a schematic diagram of the structure of a human-machine collaborative automatic imaging system based on the guidance of markings at the end of a surgical instrument in an embodiment of the present invention.

[0021] Figure 3 Shown is an effect diagram of guided focusing in an embodiment of the present invention.

[0022] Figure 4 Shown is a schematic diagram of the system hardware structure of a human-machine collaborative automatic imaging system based on the guidance of markings at the end of a surgical instrument in an embodiment of the present invention.

[0023] Figure 5 Shown is a schematic diagram of the implementation process of a human-machine collaborative automatic imaging system based on the guidance of markings at the end of a surgical instrument in an embodiment of the present invention. Detailed implementation manners

[0024] The following uses specific specific examples to illustrate the implementation manners of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific implementation manners. Various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, without conflict, the following embodiments and the features in the embodiments can be combined with each other.

[0025] It should be noted that in the following description, reference is made to the accompanying drawings, which describe several embodiments of the present invention. It should be understood that other embodiments can also be used, and mechanical composition, structure, electrical, and operational changes can be made without departing from the spirit and scope of the present invention. The following detailed description should not be considered restrictive, and the scope of the embodiments of the present invention is only defined by the claims of the published patent. The terms used here are only for describing specific embodiments and are not intended to limit the present invention. Spatially related terms, such as "upper", "lower", "left", "right", "below", "beneath", "lower part", "above", "upper part", etc., can be used in the text to facilitate the description of the relationship between one element or feature shown in the figure and another element or feature.

[0026] Throughout the specification, when it is said that a certain part is "connected" to another part, this includes not only the case of "direct connection", but also the case of "indirect connection" with other elements placed in between. In addition, when it is said that a certain part "includes" a certain constituent element, unless there is a particularly contrary record, it does not exclude other constituent elements, but means that other constituent elements can also be included.

[0027] The first, second, third, and other terms mentioned herein are used to illustrate various parts, components, regions, layers, and / or segments, but are not limited thereto. These terms are only used to distinguish one part, component, region, layer, or segment from other parts, components, regions, layers, or segments. Therefore, the first part, component, region, layer, or segment described below may refer to the second part, component, region, layer, or segment without departing from the scope of the present invention.

[0028] Furthermore, as used herein, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should be further understood that the terms "comprises" and "comprising" specify the presence of the stated features, operations, elements, components, items, kinds, and / or groups, but do not preclude the presence, occurrence, or addition of one or more other features, operations, elements, components, items, kinds, and / or groups. The terms "or" and "and / or" used herein are to be construed as inclusive, meaning either one or any combination. Thus, "A, B, or C" or "A, B, and / or C" means "any of the following: A; B; C; A and B; A and C; B and C; A, B, and C". An exception to this definition occurs only when the combination of elements, functions, or operations is inherently mutually exclusive in some manner.

[0029] The present invention provides a human-machine combined automatic camera system guided by a marker at the end of a surgical instrument, including: a surgical instrument, a marker at the end of the surgical instrument, a camera module, and a continuous autofocus control software module. By establishing a marker at the end of the surgical instrument, the camera module collects and identifies the image feature information of the marker, and then the focus control software module controls the camera module to track and continuously focus on the marker at the end of the instrument, so as to achieve continuous automatic focusing and imaging of the marker; through the marker at the end of the surgical instrument, the "target seen by the surgeon's eyes" is converted into the "target focused and clearly imaged by the imaging device", realizing the human-machine combination of automatic imaging during the surgical process. During the switching process of the imaging target, the present invention realizes the focus control from non-automatic to automatic, and from non-continuous to continuous, with advantages such as high focus accuracy, simple operation, and fast response, which is beneficial to continuous, high-definition, and accurate video recording during the operation and convenient for teaching.

[0030] The following will be described in detail with reference to the accompanying drawings for the embodiments of the present invention, so that those skilled in the technical field of the present invention can easily implement it. The present invention can be embodied in various different forms and is not limited to the embodiments described herein.

[0031] As Figure 2 shows a schematic structural diagram of a human-machine combined automatic camera system guided by a marker at the end of a surgical instrument in an embodiment of the present invention.

[0032] The system includes: a surgical instrument 1, a surgical instrument end marker 2, a camera module 3, and a focusing control software module 4;

[0033] The surgical instrument 1 is a commonly used surgical instrument. In addition to conventional surgical instruments, there are also some specialized instruments, such as those used in ophthalmology, orthopedics, urology, gynecology and obstetrics, burn department, plastic surgery, neurosurgery, cardiothoracic surgery, and general surgery, etc.

[0034] The surgical instrument end marker 2 is established at the end of the surgical instrument 1 and is set within the capture range of the camera module 3. For example, the surgical instrument end marker 2 established at the end of the surgical instrument 1 is located directly below the camera module 3, facilitating the camera module 3 to collect images. Specifically, the shape, brightness, color, size, and material of the surgical instrument end marker 2 can be set according to requirements, ensuring that the camera module 3 can identify it without affecting the surgery. The material of the surgical instrument end marker 2 uses sterile materials that meet medical and health standards. By determining the target object through the surgical instrument end marker 2 established at the end of the surgical instrument 1, the camera module 3 is further enabled to guide the focus on the target object. This solves the problem that during the surgical process, the imaging device cannot select the target to be focused on according to the surgeon's thinking.

[0035] The camera module 3 is used to collect the image information of the surgical instrument end marker 2; the camera module 3 is an auto - zoom lens with a focal length adjustment function and an image capture ability, and has a certain capture range, and performs focusing under the control of the focusing control software module 4.

[0036] The focusing control software module 4 is communicatively connected to the camera module 3. After identifying the surgical instrument end marker 2 in the image information collected by the camera module 3, it continuously controls the camera module 3 to focus on and track the surgical instrument end marker 2. The focusing control software module 4 can use any autofocus algorithm to perform focusing control on the camera module 3. The autofocus algorithm can be any focusing algorithm in the prior art.

[0037] In this solution, a three - dimensional rectangular coordinate system (XYZ axes) is established directly below the camera module. In this three - dimensional rectangular coordinate system, the Z - axis represents the depth, which is the focusing direction of the camera module 3; the X - axis and Y - axis perpendicular to the Z - axis represent the horizontal plane, which are the forward - backward and left - right movement directions of the camera module 3.

[0038] As Figure 3, the specific implementation process of this solution is as follows: Before autofocusing, a surgical instrument tip marker 2 is established at the tip of the surgical instrument 1. After the human eye fixes on the target A, the person holds the surgical instrument 1 with the surgical instrument tip marker 2 close to the target A. The camera module 3 captures and identifies the surgical instrument tip marker 2, and controls the camera module 3 to autofocus through the focusing control software module 4 until the image of the surgical instrument tip marker 2 is clear, achieving rapid focusing on the target A. If the human eye changes to fix on the target B and the tip of the surgical instrument 1 points to or moves to the target B at the same time, then the surgical instrument tip marker 2 approaches the target B, and the software module 4 controls the camera module 3 to automatically adjust the focal length (move in the Z-axis) until the image of the surgical instrument tip marker 2 is clear, achieving rapid focusing on the target B. By combining with the existing tracking software (move back and forth, left and right in the horizontal plane of the X and Y axes), this system can achieve tracking and focusing imaging in all directions of the X, Y, and Z axes.

[0039] In one embodiment, the surgical instrument tip marker 2 can be directly marked on the surgical instrument 1 or can be a separate marking structure. The marking forms of the surgical instrument tip marker 2 include one or more of graphic marking, brightness marking, color marking, morphological marking, and pattern marking. And the corresponding characteristic information includes one or more of marking color, marking brightness, marking morphology, marking graphics, and marking patterns.

[0040] The surgical instrument tip marker 2 can be marked with different graphics, that is, different shapes composed of drawn straight lines, circles, rectangles, and curves. The surgical instrument tip marker 2 can be marked with different colors, which can be one color or a mixture of different colors. For more convenient identification, a single color is generally selected, such as any color like blue, yellow, green, etc. The surgical instrument tip marker 2 can be marked with different morphologies, such as its own size, contour, shape, and structural features. The surgical instrument tip marker 2 can be marked with different patterns, and the pattern can be a specific pattern or a symbol marking.

[0041] In one embodiment, the surgical instrument tip marker 2 can be processed as a coating on the surface of the surgical instrument tip in a printing or spraying manner, and this coating has different colors, graphics, or patterns.

[0042] In one embodiment, the surgical instrument tip marker 2 can be an independent structure fixed to the tip of the surgical instrument 1. Regarding the structural shape of the surgical instrument tip marker, it can be spherical, square three-dimensional shape, sheet-like, and irregular three-dimensional shape, etc.

[0043] The fixing method adopted by the surgical instrument tip marker 2 can be set according to requirements, such as connection methods like bonding, bolt connection, welding, pin connection, and socket connection.

[0044] Specifically, there are many specific implementation methods for the surgical instrument end marker 2, and applicable methods should be selected according to the actual situation. For example: the surgical instrument end marker 2 can be a self-adhesive paper with different colors or patterns printed on the surface, which can be adhered to the end of the surgical instrument 1; the surgical instrument end marker 2 is a metal sheet or plastic sheet with different colors or patterns printed on the surface, and is fixed on the surface of the surgical instrument by snap or pasting; the surgical instrument end marker 2 can be a three-dimensional structure with a self-adhesive paper with different colors or patterns printed on the surface, which can be fixed to the end of the surgical instrument 1; the surgical instrument end marker 2 is a three-dimensional structure with a coating printed or sprayed, and the coating has different colors or patterns, and the surgical instrument end marker 2 can be fixed to the end of the surgical instrument 1.

[0045] In one embodiment, the surgical instrument end marker 2 is detachable or fixed to the end of the surgical instrument 1, facilitating the fixation of the surgical instrument end marker 2 to the ends of different types of surgical instruments 1.

[0046] In one embodiment, the camera module 3 includes: one of a liquid lens module, a non-liquid lens module, and a fixed-focus lens module.

[0047] Among them, the liquid lens in the liquid lens module uses the electro-wetting process to achieve excellent autofocus function. Two immiscible liquids are used, each with a different refractive index, to generate a variable-focus lens similar to a traditional high-quality optical lens. Of the two liquids, one is a conductive aqueous solution and the other is a non-conductive oil. These two liquids are contained in a small tubular container, forming a crescent-shaped curved surface equivalent to a glass lens inside the container, and the focal length of the liquid lens is adjusted by changing the curvature of the curved surface. The liquid lens focuses by controlling a layer of transparent liquid to form a nearly spherical lens under the action of an electric current between two electrodes, that is, by adjusting the DC voltage at both ends of the container, the shape of the crescent-shaped curved surface at the junction of the two different liquids can be changed.

[0048] The non-liquid lens module can use a common autofocus lens other than the liquid lens, such as a standard lens, a long and short focal length lens, etc. Among them, the traditional non-liquid lens achieves zooming by adjusting the relative position of the lenses inside the lens.

[0049] The fixed-focus lens module can use a lens with a fixed focal length, such as a standard lens, a wide-angle lens, etc. Among them, the fixed-focus lens changes the clarity of the output image by adjusting the relative position between the camera and the imaging target.

[0050] In one embodiment, as Figure 4 shown, the focus control software module includes:

[0051] An image storage unit 41 for storing image information collected during the focusing process; wherein, the image storage unit 41 has at least two independent storage areas, one storage area for storing unprocessed images and the other storage area for storing video information of the entire surgical process.

[0052] An image processing unit 42, connected to the image storage unit 41, for identifying the collected image information and generating a corresponding focusing control signal after identifying the mark at the end of the surgical instrument;

[0053] A focusing control driving unit 43, communicatively connected to the image processing unit 42, for receiving the focusing control signal from the image processing unit 42 and driving the camera module 3 to move and focus in the Z-axis direction to control the camera module 3 to track and focus on the mark 2 at the end of the surgical instrument in the Z-axis direction.

[0054] It should be noted that the units in this embodiment are only a logical function division. In actual implementation, they can be fully or partially integrated into a physical entity, or physically separated. And these units can all be implemented in the form of software called by processing elements; they can also all be implemented in the form of hardware; or some units can be implemented in the form of software called by processing elements and some units can be implemented in the form of hardware.

[0055] For the case where the image storage unit 41, the image processing unit 42, and the focusing control driving unit 43 are separated, the image processing unit 42 can be implemented by a computer device, and the focusing control driving unit 43 can be independently implemented by a chip connected to the camera module 3.

[0056] In one embodiment, the focusing control driving unit 43 includes:

[0057] A focusing control input terminal 431 for inputting set focusing control parameters; the focusing control input terminal 421 inputs the focusing control parameters in the form of an input control signal, and the input method can be automatic input or manual input, such as in the form of a keyboard, mouse, trackball, click gun, button, button, touchpad, or touch screen, etc.; the focusing control parameters are control parameters for controlling the camera module, such as feature information of the mark at the end of the surgical instrument, shooting interval time, focal length adjustment information, etc.

[0058] A driving chip 432, connected to the focusing control input terminal 431, for receiving the focusing control signal and the focusing control parameters and driving the camera module 3 to perform corresponding focusing operations.

[0059] In one embodiment, the focusing control parameters include: feature information of the mark at the end of the surgical instrument to be tracked.

[0060] The characteristic information of the end marker 2 of the surgical instrument to be tracked determines the target object to be tracked. If there are multiple types of end markers 2 of surgical instruments in the collected image, we only need to track the set type of end marker 2 of the surgical instrument to achieve classified tracking of the target object. That is to say, for objects with different characteristics, the system will try to separate them, thereby improving the accuracy of the image output. That is, only the set characteristics will be tracked, and the partial occlusion of other colored objects will not affect the tracking result of the system until the target object leaves the camera area.

[0061] In one embodiment, the characteristic information of the end marker 2 of the surgical instrument includes one or more of the following: marker color, marker brightness, marker shape, marker pattern, and marker graphic. Specifically, the marker color can be any color such as blue, yellow, green, etc.; the marker shape can be spherical, cubic three-dimensional shape, sheet-like, and irregular three-dimensional shape; the marker identification can be different graphics, characters, symbols, and codes, etc.

[0062] In one embodiment, the image processing unit 42 is connected to the driving chip 432. First, it identifies the end marker 2 of the surgical instrument in the collected image information. The characteristics of the end marker 2 of the surgical instrument can be determined according to the characteristic information of the end marker 2 of the surgical instrument to be tracked input through the focusing control input terminal 431 transmitted by the driving chip 432. After identifying the end marker 2 of the surgical instrument in the image, a corresponding focusing control signal is generated;

[0063] The focusing control method for generating the corresponding focusing control signal can be implemented by using existing autofocus algorithms. The existing autofocus algorithms mainly include the focus evaluation function and the peak search algorithm. The autofocus algorithm first uses the focus evaluation function to evaluate the sharpness information of the image on the focusing area, and then gives the search moving direction and step size of the sharpness extreme point through the search algorithm until the best clear image is obtained.

[0064] In one embodiment, the present invention can achieve corresponding focusing control according to the change of image characteristics; the image processing unit 42 is used to identify the collected image information, and after identifying the end marker 2 of the surgical instrument, generate a corresponding focusing control signal that can improve the image clarity according to the change of image characteristics.

[0065] In one embodiment, the image processing unit 42 is used to generate a corresponding focusing control signal according to the change of the image clarity after identifying the end marker of the surgical instrument.

[0066] Specifically, the image processing unit 42 is configured to identify the end marker of the surgical instrument in the acquired image information and to evaluate the clarity of the acquired image. The image processing unit 42 first evaluates the clarity of the image information for identifying the end marker 2 of the surgical instrument, and then evaluates the clarity of the subsequently acquired image information, detects the change in image clarity, and further generates a corresponding focusing control signal to control the automatic focusing of the camera module 3.

[0067] The evaluation of clarity can be determined by the change in contrast between the end marker 2 of the surgical instrument and the background in the image information; and not only gradient information can be used, but also various other effective information in the image can be extracted for judgment, such as frequency, phase, etc.

[0068] In a specific embodiment, after identifying the end marker of the surgical instrument, generating a corresponding focusing control signal according to the change in image clarity includes:

[0069] After identifying the end marker 2 of the surgical instrument, a primary focusing control signal is generated to control the camera module 3 to finely adjust the focal length forward or backward;

[0070] After the camera module 3 finely adjusts the focal length, a corresponding secondary focusing control signal is generated according to the change in image clarity to control the camera module 3 to coarsely adjust the focal length forward or backward to improve the clarity of the acquired image.

[0071] Specifically, after identifying the end marker 2 of the surgical instrument, a primary focusing control signal for controlling the camera module 3 to finely adjust the focal length forward (i.e., slightly increase the focal length) is generated. After the camera module 3 adjusts the focal length forward, if the clarity of the acquired image information becomes higher, a corresponding secondary focusing control signal for continuing to control the camera module 3 to coarsely adjust the focal length forward (i.e., significantly increase the focal length) is generated to control the camera module to continue to significantly increase the focal length to improve the clarity of the acquired image until the clarity reaches the standard; if the clarity of the acquired image becomes lower, a corresponding secondary focusing control signal for controlling the camera module 3 to coarsely adjust the focal length backward (i.e., significantly decrease the focal length) is generated to control the camera module to significantly decrease the focal length to improve the clarity of the acquired image until the clarity reaches the standard.

[0072] After identifying the end marker 2 of the surgical instrument, a primary focusing control signal for controlling the camera module 3 to finely adjust the focal length in the reverse direction (i.e., slightly decrease the focal length) is generated. After the camera module 3 adjusts the focal length in the reverse direction, if the clarity of the collected image information becomes higher, a secondary focusing control signal for continuously controlling the camera module 3 to coarsely adjust the focal length in the reverse direction (i.e., significantly decrease the focal length) is generated, and the camera module is controlled to continuously and significantly decrease the focal length to improve the clarity of the collected image until the clarity reaches the standard; if the clarity of the collected image information becomes lower, a secondary focusing control signal for controlling the camera module 3 to coarsely adjust the focal length in the forward direction (i.e., significantly increase the focal length) is generated, and the camera module is controlled to significantly increase the focal length to improve the clarity of the collected image until the clarity reaches the standard.

[0073] The image processing unit provided in the embodiments of the present invention can be implemented on the terminal side or the server side. The terminal can be a mobile phone, a computer device, a tablet device, a personal digital processing device, a factory background processing device, etc. An optional hardware structure of the terminal includes: at least one processor and a memory.

[0074] The memory can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM, Read Only Memory), a programmable read-only memory (PROM, Programmable Read-Only Memory), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM, StaticRandom AccessMemory), synchronous static random access memory (SSRAM, Synchronous Static RandomAccess Memory). The memory described in the embodiments of the present invention is intended to include but not be limited to these and any other suitable categories of memories.

[0075] The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above-mentioned focusing control and image processing method can be completed by the integrated logic circuit in the hardware of the processor or the instructions in the form of software. The above-mentioned processor can be a general-purpose processor, a digital signal processor (DSP, Digital SignalProcessor), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0076] In an exemplary embodiment, the terminal can be one or more application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), or complex programmable logic devices (CPLDs) for performing the foregoing focusing control method.

[0077] Those of ordinary skill in the art can understand that all or part of the steps for implementing the above embodiments of the image processing unit can be completed by hardware related to a computer program. The foregoing computer program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps included in the above method embodiments; and the foregoing storage medium includes: ROM, RAM, magnetic disk, or optical disc and other media that can store program codes.

[0078] In the embodiments provided in the present application, the computer-readable and writable storage medium may include a read-only memory, a random access memory, an EEPROM, a CD-ROM, or other optical disc storage devices, a magnetic disk storage device, or other magnetic storage devices, a flash memory, a USB flash drive, a mobile hard disk, or any other medium that can be used to store the desired program code in the form of instructions or data structures and can be accessed by a computer.

[0079] Through the above solution content, the present invention has the following innovative points compared with the prior art:

[0080] 1. The present invention converts the "fixation point and focusing point of the human eye" into the "marking point at the end of the surgical instrument held by the human hand", and through the marking, it converts the "direction pointed by the end of the operator's instrument" into the "target focused and clearly imaged by the imaging device", realizing the combination of man and machine for automatic imaging during the surgical process.

[0081] 2. The present invention converts the non-automatic control of the "focusing button of the imaging device by the human hand or foot" into the software automatic focusing control of the "marking guidance at the end of the instrument held by the human hand", without relying on the focusing button.

[0082] 3. During the switching process of the imaging target, the present invention converts the discontinuous control process of the "imaging target - human eye - human brain - hand and foot control the focusing button" into the continuous control process of "identifying, tracking, and focusing on the end of the surgical instrument by software".

[0083] In summary, the human-machine combined automatic camera system based on the end marker guidance of surgical instruments of the present invention includes: surgical instruments, end markers of surgical instruments, a camera module, and a continuous autofocus control software module. By establishing a marker at the end of the surgical instrument, the camera module collects and identifies the image feature information of the marker, and then the focus control software module controls the camera module to track and continuously focus on the marker at the end of the instrument, so as to achieve continuous automatic focusing and imaging of the marker; through the conversion of "what the surgeon sees with the eyes" to "what the end of the surgeon's instrument (marker) points to", and "what the end of the surgeon's instrument points to" to "what is recognized and focused by the imaging device", the human-machine combination of automatic imaging during the surgical process is realized. Since the end marker and the imaging target are within the same depth of field, the conversion from "clear marker image" to "clear imaging target" is achieved: "clear marker image" means "clear imaging target", as Figure 5 shown. During the switching process of the imaging target, the present invention realizes the focus control from non-automatic to automatic and from non-continuous to continuous, and has the advantages of high focusing accuracy, simple operation, fast response, etc., which is beneficial to continuous, high-definition and accurate video recording during the operation and convenient for teaching. Therefore, the present invention effectively overcomes various disadvantages in the prior art and has high industrial application value.

[0084] The above embodiments are only used to exemplarily illustrate the principles and effects of the present invention, rather than to limit the present invention. Any person familiar with this technology can modify or change the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or changes made by those with ordinary knowledge in the technical field without departing from the spirit and technical ideas disclosed by the present invention should still be covered by the claims of the present invention.

Claims

1. A human-machine combined automatic camera system based on surgical instrument end mark guidance, characterized in that: The system includes: a surgical instrument, a surgical instrument end mark, a camera module and a focus control software module; wherein, The surgical instrument end mark is established at the end of the surgical instrument and is set within the capture range of the camera module; The camera module is used to collect image information of the end mark of the surgical instrument; The focus control software module is in communication with the camera module and is used to continuously control the camera module to track and focus on the end mark of the surgical instrument after identifying the end mark of the surgical instrument.

2. The human-machine combined automatic camera system based on surgical instrument end mark guidance according to claim 1, characterized in that: The camera module comprises: one of a liquid lens module, a non-liquid lens module and a fixed-focus lens module.

3. The human-machine combined automatic camera system based on surgical instrument end mark guidance according to claim 1, characterized in that: The focus control software module includes: An image storage unit is used to store image information collected during the focusing process; wherein the image storage unit has at least two independent storage areas, one of which is used to store unprocessed images, and the other is used to store video information of the entire surgical process; An image processing unit, connected to the image storage unit, for identifying the acquired image information and generating a corresponding focus control signal after identifying the end mark of the surgical instrument; The focus control drive unit is communicatively connected with the image processing unit, and is used to receive a focus control signal from the image processing unit, and drive the camera module to move and focus in the Z-axis direction, so as to control the camera module to track and focus on the end mark of the surgical instrument in the Z-axis direction.

4. The human-machine combined automatic camera system based on surgical instrument end mark guidance according to claim 3, characterized in that: The focus control driving unit comprises: A focus control input terminal, used to input set focus control parameters; The driving chip is connected to the focus control input terminal and is used to receive the focus control signal and the focus control parameter to drive the camera module to perform corresponding focusing operation.

5. The human-machine combined automatic camera system based on surgical instrument end mark guidance according to claim 2, characterized in that: The image processing unit is used to identify and compare the collected image information, and after identifying the end mark of the surgical instrument, generate a corresponding focus control signal that can improve the image clarity according to the change of image features.

6. The human-machine combined automatic camera system based on surgical instrument end mark guidance according to claim 5, characterized in that: The image processing unit is used to generate a corresponding focus control signal according to the change in image clarity after identifying the end mark of the surgical instrument.

7. The human-machine combined automatic camera system based on surgical instrument end mark guidance according to claim 6, characterized in that: After identifying the end mark of the surgical instrument, the corresponding focus control signal is generated according to the change in image clarity, including: After identifying the end mark of the surgical instrument, a primary focus control signal is generated to control the camera module to fine-tune the focus in a forward or reverse direction; After the camera module fine-tunes the focus, a corresponding secondary focus control signal is generated according to the change in the clarity of the collected image information to control the camera module to coarsely adjust the focus in a forward or reverse direction to improve the clarity of the collected image information.

8. The human-machine combined automatic camera system based on surgical instrument end mark guidance according to claim 4, characterized in that: The focus control parameters include: characteristic information of the end mark of the surgical instrument to be tracked.

9. The human-machine combined automatic camera system based on surgical instrument end mark guidance according to claim 8, characterized in that: The characteristic information of the surgical instrument end mark includes: one or more of mark color, mark brightness, mark shape, mark graphic and mark pattern.

10. The human-machine combined automatic camera system based on surgical instrument end mark guidance according to claim 1, characterized in that: The surgical instrument end marker can be detachable or fixed to the surgical instrument end.

Images