Image processing of endoscopic video

Abstract

In an endoscope system, a processor is programmed to receive video image data from an image sensor at a distal end of an endoscope and display the image data to a surgeon in real-time. The processor is programmed to process image data received from the image sensor via a machine learning model trained to simultaneously upsample the image data to a higher resolution than captured by the image sensor, sharpen edges, and enhance local contrast. The video image data has a frame rate at which image data is generated by the image sensor. The processor is programmed to control the image sensor and / or an illumination source designed to illuminate a scene viewed by the image sensor, the control being programmed to underexposure or overexposure every other frame of the video image data. The processor is programmed to process the image data received from the image sensor to combine successive frame pairs of the image data to adjust the dynamic range to enhance over-bright or over-dark portions of the image to expose details, and to generate combined frames at a full frame rate of the video generated by the image sensor.

Description

Cross - Reference to Related Applications

[0001] This application claims priority from the following applications: U.S. Provisional Application No. 63 / 538,485, entitled "Endoscope", filed on September 14, 2023; U.S. Provisional Application No. 63 / 534,855, entitled "Endoscope", filed on August 27, 2023; U.S. Provisional Application No. 63 / 531,239, entitled "Endoscope", filed on August 7, 2023; U.S. Provisional Application No. 63 / 437,115, entitled "Endoscope with Identification and Configuration Information", filed on January 4, 2023; U.S. Application No. 17 / 954,893, entitled "Illumination for Endoscope", filed on September 28, 2022; U.S. Provisional Application No. 63 / 376,432, entitled "Super Resolution for Endoscope Visualization", filed on September 20, 2022. Technical Field

[0002] This application relates to endoscopes, laparoscopes, arthroscopes, colonoscopes, and similar surgical devices or instruments, which are particularly suitable for or intended for the evaluation, examination, measurement, monitoring, research, or testing of living or dead human and animal bodies for medical purposes, or for performing surgery on the human body or preparing for surgery in conjunction with devices designed to assist in surgery. Background Art

[0003] Endoscopes can be arthroscopes (for joint surgery), laparoscopes (for abdominal surgery), colonoscopes (rectum, colon, and lower small intestine), cystoscopes (bladder and urethra), electroencephaloscopes (brain), hysteroscopes (vagina, cervix, uterus, and fallopian tubes), rhinoscopes (ear, nose, throat), thoracoscopes (extra - pulmonary chest), bronchoscopes (trachea and bronchi), esophagoscopes (esophagus and stomach), etc. Endoscopes can have a rigid shaft or a flexible insertion tube. Summary of the Invention

[0004] Generally, in a first embodiment, the invention features an apparatus including a computer processor and a memory. The processor is programmed to receive video image data from an image sensor at a distal end of an endoscope and display the image data in real time to a surgeon. The processor is programmed to process the image data received from the image sensor via a machine learning model that is trained to simultaneously upsample the image data to a higher resolution than that captured by the image sensor, sharpen edges, and enhance local contrast.

[0005] Generally, in a second aspect, the invention features an apparatus including a computer processor and a memory. The processor is programmed to receive video image data from an image sensor at a distal end of an endoscope and display the image data in real time to a surgeon. The video image data has a frame rate at which the image sensor generates the image data. The processor is programmed to control the image sensor and / or an illumination source designed to illuminate a scene viewed by the image sensor, and the control is programmed to underexpose or overexpose every other frame of the video image data. The processor is programmed to process the image data received from the image sensor to combine consecutive frame pairs of the image data, thereby adjusting the dynamic range to enhance overbright or overdark portions of the image to expose details and generate combined frames at the full frame rate of the video generated by the image sensor.

[0006] Generally, in a third aspect, the invention features an apparatus including a computer processor and a memory. The processor is programmed to receive video image data from an image sensor at a distal end of an endoscope and display the image data in real time to a surgeon. The processor is programmed to sum an error of an intensity of an image relative to a setpoint intensity. The processor is programmed to simultaneously control at least two of gain, exposure, and illumination via a PID control algorithm to achieve image display at the setpoint intensity, and a maximum change for each step of the PID control is damped to prevent oscillation.

[0007] Embodiments may include one or more of the following features, either alone or in any combination. The processor may further be programmed to control an image sensor and / or an illumination source designed to illuminate a scene viewed by the image sensor. The control may be programmed to underexpose or overexpose every other frame of video image data. The processor may further be programmed to process image data received from the image sensor to combine successive frame pairs of the image data, thereby adjusting the dynamic range to enhance overbright or overdark portions of the image to expose details. The processor may further be programmed to generate combined frames at the full frame rate of the video generated by the image sensor. The processor may further be programmed to sum the error of the intensity of the image relative to a setpoint intensity. The processor may further be programmed to simultaneously control at least two of gain, exposure, and illumination via a PID control algorithm to achieve image display at the setpoint intensity. The maximum change of each step of the PID control may be damped to prevent oscillation. The processor may further be programmed to process image data received from the image sensor via a machine learning model that is trained to simultaneously upsample the image data to a higher resolution than that captured by the image sensor, sharpen edges, and enhance local contrast. The processor may further be programmed to enhance video image data via dynamic range compensation. The processor may further be programmed to adjust the exposure time, illumination intensity, and / or gain in image capture to adjust exposure saturation. The processor may further be programmed to enhance video image data via noise reduction. The processor may further be programmed to enhance video image data via lens correction. The processor may further be programmed to enhance at least two of dynamic range compensation, noise reduction, and lens correction, in addition to resolution. The processor may further be programmed to rotate the image display to compensate for rotation of the endoscope. The processor may further be programmed to adjust the exposure time, illumination intensity, and / or gain in image capture to adjust exposure saturation.

[0008] The above advantages and features are only representative embodiments and are presented only to aid in understanding the invention. However, it should be understood that they should not be considered as limitations of the invention as defined by the claims. Additional features and advantages of embodiments of the invention will become apparent in the following description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1A 、 2A Figures 3A, 3C, 3D, 4C through 4I, 5, 6, 7, 9, 10A, 10D through 10O, 10Q are perspective views or perspective cross-sectional views of an endoscope and / or an endoscope-related device.

[0010] Figure 3B 、 4A, 4B, 8, 10B, 10C, 10P, 11E are plan views, plan sectional views or plan partial sectional views of an endoscope and / or endoscope-related devices.

[0011] Figure 1B , 11A to 11D and 11G are block diagrams of a computer or a processor.

[0012] Figure 11F is a time series of video frames. Detailed Description of the Invention

[0013] The structure of the present detailed description is as follows. I. Overview I.A. Endoscopic Surgery I.B. Overall Architecture I.C. Integrated Sterile Packaging I.D. Disposable Handpiece II. Additional Features of the Endoscope III. Endoscope Tip III.A. Molding and Assembly of Components of the Endoscope Tip III.B. Irradiation III.C. Diffusion Terminal Surface IV. Endoscope Tip IV.A. Molding and Assembly of Components of the Endoscope Tip V. Image Processing Unit V.A. Image Processing V.B. HDR Exposure Fusion to Maintain Frame Rate V.C. Automatic Exposure V.D. Video Processing with Super Resolution V.E. Diagnosis and Lesion Detection V.F. Endoscope Control V.G. Flexible Boards and Electronics in the Endoscope Handle V.H. Cable V.I. Wireless Communication Instead of Cable V.J. Isolation V.K. Other Peripherals V.K.1. Monitor V.K.2. USB Port V.K.3. Connection to Cloud Storage V.K.4. USB Connection for Keyboard and Mouse V.K.5. Microphone V.K.6. Insufflation Tube VI. Electronic Serial Number VI.A. Electronic Serial Number VI.B. Using the Electronic Serial Number to Reduce Errors and Ensure Sterile Single-Use VI.C. Using the Electronic Serial Number for Inventory Control, Location Tracking, Reordering, and Stock Management VI.D. Using the Electronic Serial Number to Transfer Patient Data into the Electronic Medical Record VII. Examples I. General Overview I.A. Endoscopic Surgery

[0014] Referring to Figure 1A and 2A , the endoscope 100 can be used for arthroscopic surgery, arthroscopic access, or other minimally invasive surgeries. The features of the endoscope can provide cost reduction and disposability. Various endoscope tip designs ( Figures 4A - 4G and Figures 10A through 10Q ) can have the following characteristics. The entire tip may be small enough to meet the dimensions of the endoscope, typically the dimensions in the table in paragraph

[0021] below. In some cases, the diameter of the tip may be slightly larger or smaller than the shaft. The tip can mechanically hold the camera 410, illumination, fluid injection or evacuation ports, procedural tools, etc. stable within that diameter. The tip can be sealed to resist the elevated pressures typically used to move tissue out of the field of view of the endoscope to prevent the intrusion of body tissues and fluids as well as insufflation fluids. The tip can deliver or allow the delivery of illumination light via an LED 418 mounted in the tip or using an optical fiber 430 to transmit light from the handle or controller. The opaque portion of the tip assembly can exclude stray light from unwanted optical paths within the tip from the illumination fiber / LED / light guide. The tip can be manufactured in the desired quantity and at the desired cost. The tip can have a configuration that causes no damage to surrounding tissue, e.g., no sharp points or edges. The endoscope can be formed from biocompatible materials such as stainless steel and / or certain plastics. In some cases, the tip may have a piercing point. The tip can be designed to resist fogging or fouling. The tip can allow cleaning, preferably at the in-situ surgical site. I.B. Overall Architecture

[0015] Referring to FIGS. 1B, 1C, 2, and 10A, the endoscope 100 can be part of an integrated system designed to deliver high-definition video for endoscopic surgery. The system can provide real-time high-definition video for display on a video monitor and for capture as stored video and still images; illumination of the surgical cavity, irrigation and / or insufflation (blowing) of the surgical site, and image refinement such as zooming, rotation, removal or reduction of hotspots and other artifacts.

[0016] The system may include an endoscope that includes a blow tube, a communication / control / power / illumination cable, a cannula, and an obturator. The image processing unit (IPU) or the main controller may be reused in multiple procedures. If illumination is provided via an optical fiber, there may also be a light box, typically near the IPU, to align the optical fiber with other necessary cords and hoses. One or more of the endoscope, the tube, the cable, the cannula, and the obturator may be designed for single use and sold together as an integrated kit.

[0017] Referring to Figure 11A and 11B , the endoscope may have electronics in the handle to control the camera and illumination (LED or optical fiber). The IPU may have a computer processor for various image processing functions, as well as a controller for electromechanical devices in the endoscope, WiFi or similar radio communication, USB, and cloud storage, etc. Since the endoscope is for single use, it is easy to provide sterility. The connecting cable may also be for single use so that it can be delivered in a sterile package. The IPU is costly and not easily disinfected, so it is not in the sterile area.

[0018] Referring to Figure 11G , various isolation couplers may provide electrical isolation between the patient and the wall voltage components of the IPU. I.C. Integrated sterile package

[0019] Referring again to Figure 1C and 10A , the endoscope, the tube, and the cable may be designed for single use and packaged and sold together as an integrated kit. Additionally, one or more obturators and cannulas may be packaged and sold with the kit. The kit may be sold in a sterile package. The package may be designed to be opened within the sterile area around the patient during surgery. The lid on the package may be made of or some similar film that is transparent to ethylene oxide or similar disinfectants so that the package and the components can be disinfected together during manufacturing. The film covering will remain in place until shortly before surgery. This eliminates the need to disinfect or sterilize the endoscope immediately before surgery. The tray containing the components may be transparent so that the contents of the tray can be seen before opening the Tyvek lid.

[0020] Since the components are sold together, they can be calibrated with respect to each other. The various characteristics of the illumination, image sensor, lens, filter, etc. can be calibrated with respect to each other as a group at the manufacturing plant. White balance can be one of the parameters calibrated at the factory - since these components are single - use and sold as an integrated package, they can be internally calibrated at the factory and this joint calibration will be followed throughout the life cycle of the product. In contrast, for a conventional endoscope, the light source and the endoscope are separate, and the color temperature or balance of the illumination source varies from light source to light source, and the color sensitivity of the pixels of the image sensor also varies from endoscope to endoscope, so the user must perform white balance as part of the preparation for each procedure. In the configuration where the endoscope is sold as a single - use configuration, its electronic serial number is associated with the calibration factors measured at the factory (see §VI.A and paragraphs

[0123] to

[0130] below), and the endoscope can be calibrated by imaging a white surface that provides equal portions of red, green, and blue pigments for the test surface, where the illumination results in medium - level, non - saturated pixel values in the image sensor, and a correction coefficient matrix can be calculated to adjust the color balance of the pixels of the signal of the image sensor. I.D. Disposable handpiece

[0021] The endoscope itself can be designed for single - use. The image sensor, lens, filter, and cover window, as well as the illumination emitter (the distal end of LED 418 or the distal end of the fiber optic illumination fiber or waveguide) can be located at the distal end of the insertion shaft. The sensor, lens, filter, cover window, and illumination emitter can be designed to interoperate with each other to allow for a small - diameter insertion shaft. Single - use ensures sterility, even for components with complex geometries and materials that cannot be autoclaved (such as the electronics of the endoscope). The endoscope can have an electronic tracking function to ensure single - use (see §VI.B and paragraphs

[0131] to

[0137] below). The typical dimensions for various surgical specialties may be as follows (measured in millimeters): II. Additional functions of the endoscope

[0022] Illumination can be provided by LED 418 at or near the distal end, or via fiber optic 430 from an illumination source in the handle, or illumination can be provided at an external controller.

[0023] Refer again to Figure 1A and 2A, the endoscope may have handles 112, 114, 120 and a shaft 110 for insertion into the body. At or near the distal end 116 of the shaft 110 may be a lens, an electronic image sensor, a filter, or other optical component 410. The orientation of the camera may be fixed within the endoscope or may be translatable. The camera 410 may be located at the tip 116, looking outwards from the shaft, or may be recessed a short distance behind the structural tip of the shaft. At or near the tip may also be an illumination source, such as an LED 418. The tip 116 may have a rigid pointed trocar tip, or may have a spoon-shaped portion that extends beyond the distal surface of the window in the tip 116, or may be flexible (in the manner of the tip of a colonoscope), in each case extending slightly beyond the distal surface of the window in the tip 116 to provide physical protection for the tip 410 during insertion, or to protect the camera 410 from surgical cutting devices.

[0024] The illumination may be visible light, infrared, and / or ultraviolet light. In some cases, the illumination LED (light-emitting diode) or other illumination source may be placed in the reusable handles 112, 114 or in a docking station / controller, and the disposable shaft may have an optical fiber 430 to transmit light to the tip, and the adapter 130 may have an optical coupler. In other cases, the illumination LED 418 may be placed in the tip 116 to directly illuminate the surgical cavity; in this case, the adapter 130 may have a power connector. In some cases, the LED 418 may be recessed from the tip, or placed somewhere along the shaft, or may be in an external controller, and the optical fiber 430 may carry the illumination light to the tip. The optical fiber 430 may be configured to have, for example, a split so that the light will be arranged in a desired pattern around the image sensor to better distribute the light into the surgical cavity around the camera.

[0025] The shaft 110 itself may be rigid, made of a non-corrosive metal such as stainless steel or coated aluminum. In some cases, the surgical cavity around the endoscope tip 400 may be insufflated with gas (usually carbon dioxide) or flushed with saline. In either case, the flow of fluid in and out may be affected by channels passing through the shaft.

[0026] The shaft 110 can also carry power cables to the illumination LED 418 and the camera 410, and carry signal cables for the electronics that bring the video signal back from the camera 410 to the reusable portions 112, 114 of the handle. The power for the camera 410 can be provided through conductors in a flexible cable or on a printed circuit board (flexible or rigid), and can be insulated with a conformal and insulating coating such as parylene. The same flexible circuit board 416 can have signal conductors for the video signal from the image sensor 410. The video signal can be transmitted from the image sensor 410 to the handle using any video signal protocol, such as MIPI-CSI2 (Mobile Industry Processor Interface - Camera Serial Interface 2) or HDMI. In some cases, the parylene coating can improve biocompatibility.

[0027] The shaft 110 can also carry cables or other mechanical elements to control the panning of the camera 410.

[0028] Referring Figure 3A and 3C , the rotating collar can have various features that make rotation easy. For example, the recess 302 can provide a good grip for the fingers to obtain a small rolling torque. The fin 304 can provide a greater leverage for a greater rolling torque and can also provide a fixed rotation reference point.

[0029] The button 310 can perform various functions, such as turning on or off the illumination LED 418 or the fiber optic illumination driver, taking a photo, starting and stopping video, etc. Depending on the nature of the press, a single button can perform all of these functions. For example, holding for 3 seconds can turn the illumination on and off. A quick press can capture a single-frame still picture. A double tap can start and stop video recording. The push button can have a magnet at the bottom of the button and a Hall effect sensor on the handle board. This can provide a button with no physical contact that can fail due to the penetration of liquid or biological material.

[0030] If the camera 410 at the end 116 of the shaft 110 is translatable or has other controllable features, there may be controls (e.g., a lever or a touch slider panel, etc.) near the button 310 to control the adjustment of the camera 410.

[0031] One or more ultraviolet LEDs or other illumination sources can be placed inside the handle 112, 114, inside the shaft 110, or near the end 116 to help ensure the sterility of the internal components of the device or the water passing through the device.

[0032] Referring Figure 3A 、 3CFor 3C and 3D, the irrigation / insufflation hoses 160, 162 can enter at various points through the handle. For example, the irrigation / insufflation hoses 160, 162 can enter laterally somewhere near the distal end of the handle, such as through the fin 304. Alternatively, as shown in 3C and 3D, the irrigation / insufflation fluid / gas hoses 160, 162 can enter through the proximal end of the handle 114. The hose can then be disconnected via the fluid disconnect fitting 320 within the fitting 130.

[0033] Referring Figure 3D , electrical connectors 150, 152, such as USB-A, USB-C, or mini HDMI connectors, can be used to connect the camera 410 to a circuit board inside the handle 114.

[0034] Referring Figure 2A , 3A For 3B and 3B, rotation between the fixed portion 114 of the handle and the rotating collar 112 can be provided by a rotary bearing at the fitting 128.

[0035] The proximal handle 114 can include a rotation sensor so that the angular orientation of the camera 410 can be determined. For example, one or more magnets 320 can be mounted on the inner surface of the proximal handle 114, and the printed circuit board 322 (which rotates with the rotating collar 112 and the disposable cover 120) can have Hall effect sensors 324 that detect the magnets. This can be used to calculate the rotational orientation, which in turn can be used to "correct" the image from the camera 410 on a video display.

[0036] The distal end of the shaft, the camera 410 mounted therein, and the component mounting within the shaft 110 can be designed to be robust. Occasionally, during a surgical procedure, the tip of the endoscope may come into contact with a scraper, ablation probe, or cautery probe, and it may be desirable to make the tip robust to such contact. To reduce the risk that components may become detached and remain inside the patient, the disposable shaft and its components can be designed to avoid joints with a higher risk of mechanical failure. The disposable optical system can prevent image degradation that occurs when non-disposable optical devices are reused during multiple surgical procedures.

[0037] Endoscopes as a genus include arthroscopes, laparoscopes, colonoscopes, and other specialized mirrors for various body cavities. For arthroscopes used in arthroscopic surgery, the shaft can be as small as 6 mm, 5 mm, 4.5 mm, 4 mm, 3.6 mm, 3.3 mm, 3 mm, 2.8 mm, 2.6 mm, 2.4 mm, 2.2 mm, 2 mm, or 1.8 mm and have high rigidity. For other endoscopes, such as colonoscopes, the diameter may be larger, and the shaft may be flexible.

[0038] Referring Figure 3D, the hoses 160, 162 for flushing / blowing in fluids / gases and the electrical connection cord 164 can be permanently fixed 340, 342 to the disposable cover 120. Such an arrangement can allow the hose 162 that carries water out of the surgical cavity and is thus contaminated to be disposable, and no fluid will contact the reusable part 114 of the handle. The hoses and cords 160, 162 can be routed through the passage 354 that extends the length of the reusable handle 112, 114. The inner diameter of the passage 344 can be large enough to allow the hoses and cords 160, 162, 164 and the connectors 350, 352 to pass through easily, and has a continuous smooth wall that allows for easy disinfection to permit the ready replacement of replaceable parts at any time. The passage 354 can be offset from the central axis to allow the printed circuit board 322 to be located on the central axis. The connectors 350, 352 at the ends of the hoses and cords 160, 162 can be small enough to pass through the passage 354. Thus, the replacement of the shafts 110, the cover 120, and the hoses and cords 160, 162 can be achieved by passing the connectors 350, 352 and the hoses and cords 160, 162 through the passage 344. The wire 164 can have a connector 354 at or near the junction 130. The hose 160 for flushing / blowing in the fluid / gas flowing into the surgical cavity can also have a connector at the junction 130 to allow the hose to be reusable, or can be permanently fixed 340 to reduce the likelihood of leakage. Aligning the hoses and cables 160, 162 substantially on the axis reduces unwanted cable flipping when using the endoscope and reduces unwanted torque on the cover 120. Forming the shaft 120, the cover 120, and the hoses 160, 162 as an integral unit for replacement reduces the likelihood of leakage and improves the sterility of the replacement operation. III. Endoscope Tip

[0039] The components of the endoscope tip 400 can be designed to allow the image sensor 410, lenses, filters, illumination emission sources, and windows to be mounted within a confined space having a diameter of 6 mm or less, 5.5 mm or less, 5 mm or less, 4.5 mm or less, or 4 mm or less, such as an endoscope or arthroscope for arthroscopic surgery. In some cases, fluid management can be carried out in the same space. In some cases, the shaft can have the strength and stiffness common in arthroscopes. In some cases, the illumination emission can be emitted by one or more LEDs 418 located at or near the endoscope tip. In other cases, the illumination emission can be emitted via the optical fiber 430 and / or the light guide 450 that conduct the illumination light around the image sensor 410 within the diameter of the shaft 110. III.A. Molding and Assembly of Components of the Endoscope Tip

[0040] Refer toFigure 4A and 4B ,the endoscope tip 400 can be formed by a chassis and a flexible circuit board 416. The structural components can be formed from an opaque biocompatible plastic, such as Lustran 348. The image sensor 410 can be mounted on one side of the flexible circuit board 416, and the LED 418 can be mounted on the other side. The transparent window 420 can protect the image sensor 410 from the external environment, such as the tissue and body fluids of an endoscopic procedure, as well as pressurized insufflation fluid. The entire assembly can be locked together via overmolding, fusion welding, plastic welding caps, biocompatible adhesives, etc.

[0041] Referring Figure 4A and 4B ,for assembling the components, one end of the flexible circuit board 416, i.e., the end with the LED 418 mounted, can be slotted into a slot or channel in the top bracket portion 412, which holds the LED 418 in place. Then, the board 416 can be folded around the curved portion in the top bracket 412 so that the camera 410 is positioned through its hole in the top bracket 412. The folding and rotation bring the LED 418 close to the camera 410, which allows the assembly to fit within the 5mm diameter of the tip 400. Then, the bottom bracket 414 can be placed in position, which holds the top bracket 412, bottom bracket 414, circuit board 416, LED 418, and camera 410 in aligned positions with each other. Locking notches and clips, or ultrasonic welding can hold the assembly together for a period of time. Then, overmolding or a similar step can lock everything together.

[0042] Referencing Figure 4C 、 4D, 4E, 4F, and 4G, the transparent window 420 can cover the camera 410 to protect it. The window 420 can have two thicknesses, a thicker region above the camera 410, and a thinner region at the portion for mounting the illumination emitter (the end of an LED, optical fiber, or light pipe, etc.) and through which illumination passes. The window can have embedded features such as grooves, inserts, or other opaque materials that isolate the illumination path outside the window from the imaging path of the illumination reflected from the object of interest into the window. Alternatively, two separate windows can be used to isolate the illumination path outside the window from the imaging path of the illumination reflected from the object of interest into the window, one on the camera, and one on the illumination. The peripheral ridge of the endoscope tip 400 can extend a small portion beyond the window 420. The top bracket 412 can include an opaque wall around the LED 418, optical fiber, or light pipe. These shapes, alone or in combination, can provide one or more of the following advantages. First, these shapes can reduce the stray light from the LED 418 (or other illumination) being internally reflected into the image sensor 410. Second, when the edges of the field of view of the image sensor image are occluded or the lens 460 gathers less light towards its edges, the thickness of the window 420 on the lens side can reduce vignetting artifacts. Similarly, the shape of the lens can be used to reduce distortion, such as fisheye distortion. Third, the ridge can tend to keep tissue away from the lens 460, thus reducing occlusion and improving the field of view. Alternatively, the window can be placed only on the camera element, and the illumination emitter can have a separate window, or the light emitter can project through an opaque retainer to the same plane as the outer surface of the camera window and be sealed to the opaque light emitter retainer with an adhesive.

[0043] Figure 4E and 4F The window 420 can be placed on the surface of the assembly with the circuit board 416 having the LED 418 and the camera 410, the top bracket 412, the bottom bracket 414, and the window 420 ( Figure 4B 、 4C ). Then, the assembly with the window 420 can be locked together via overmolding of the covering sheath ( Figure 4C 、 4D, 4G). This overmolding can provide watertightness for the entire end component. Then, the overmolded component can be installed on the end of the endoscopic insertion shaft. The cost of the plastic window may be lower than that of glass, which reduces the cost and allows the endoscope to be discarded after single use. The refractive index of the plastic can be very high, exceeding 1.5, with high transparency and high plasticity. The co-molded transparent plastic window can be molded on the opaque structural components. The window can be applied in a two-shot mold, where the opaque structural components (brackets / chassis 412, 414, 438) are first injected at a high temperature and allowed to cool, and then the window 420 can be injected at a lower temperature. The components of the bracket / chassis, lens, and flexible PCB can be ultrasonically welded, laser welded, fusion welded, or fixed by an adhesive. This welding or adhesive can provide a watertight seal to prevent fluid from reaching the sensor and LED 418.

[0044] In other cases, in an alternative, the transparent window 422 can be overmolded onto a partial component of the end 400. When the window 422 is overmolded, a flat plate can be placed to project through the hole of the camera 410 to provide a mold surface, thus providing an optically smooth rear surface of the window 422. The mold can be flat (planar), or can have the desired curvature to form a convex lens or a concave lens in the overmolded window 422. The circumferential edge of the internal components of the end 400 can be shaped to provide a safety lock that engages with the overmolded window 422. Then, the circuit board 416 with the LED 418 can be inserted into the slot, folded around the top bracket 412, and then the bottom bracket 414 can be snapped into place and welded by ultrasonic, laser, or fusion welding.

[0045] Generally speaking, these features can provide an endoscope tip 400 with a very small diameter, such as 4 mm or less, 5 mm or less, 4.5 mm or less, or 4 mm or less, 3.6 mm or less, 3.3 mm or less; 3 mm or less, 2.8 mm or less, 2.6 mm or less, 2.4 mm or less, 2.2 mm or less; 2 mm or less, 1.8 mm or less, or a tip 400 slightly larger than the endoscope shaft, with all components fitting inside the tip diameter. Mounting the LED 418 and the camera 410 on opposite sides of the flexible circuit board 416 can help make the entire assembly easier to manufacture. This manufacturing can involve inserting the ends of the flexible circuit board 416 into slots and wrapping the board 416 around the molded parts, or wrapping the board 416 into the channels between the molded parts to place the various components in their preferred operating orientations. This positioning of the board 416, including bending and wrapping, may create some additional slack in the positioning of the board 416, which may result in some stress relief and increased reliability. The components can be welded together by ultrasonic welding. Overmolding can be used to hold the components together structurally and provide a watertight seal. Overmolding of the transparent windows 420, 422 onto the structural parts 412, 414, 438, or structural parts molded onto the transparent windows, can also help with the watertight seal.

[0046] This overall design concept can allow for the reconfiguration and reuse of much of the engineering for endoscopes of different sizes, and can be scaled according to the size of the sensor and the needs of a particular procedure (in contrast, for rod-lens endoscopes, many design decisions are dedicated to a single design). Features that contribute to scalability include the use of a single flexible board, top and bottom brackets or chassis 412, 414, 438, and overmolded windows 420. III.B. Illumination

[0047] Refer to Figure 5, a disposable endoscope 100 or a disposable tip for a reusable handle may have an image sensor 410 at the tip. The disposable endoscope 100 may use an optical fiber 430 to transmit illumination light. The plastic optical fiber 430 may provide an attractive combination of properties for disposable or single-use endoscope applications, including cost, flexibility to bend around curves and for movement during a surgical procedure, numerical aperture (the cone of light radiated by the fiber and the cone of light received), low heat radiation at the surgical site, and manufacturing resilience. Fiber optic illumination may transmit sufficient illumination for applications such as laparoscopy, where the objective surface distance from the camera 410 may be 200 or 300 mm, while avoiding the heat dissipation issues that may occur when placing the LED 418 at the tip. Fiber optic illumination may reduce the complexity of the chips on the tip circuitry in the confined space of the endoscope tip 400. Fiber optic illumination may allow the use of multiple illumination sources of different wavelengths coupled at the collection end of the optical fiber to vary the illumination at the endoscope tip 400.

[0048] Referring to Figure 5 , one or more illumination sources 432 may be located in the reusable endoscope handle or the base station / IPU / master controller. The illumination source 432 may be one or more of a monochromatic LED, white light source, tri-color white LED, infrared or ultraviolet light, etc. The illumination source 432 may be an LED 418, a combination of LEDs, a flash lamp, an incandescent lamp, a laser, or other illuminator. The optical fiber 430 may be coupled to the illumination source 432 at the collection end through a butt-joint adjacent or other collection mechanism. In some cases, when multiple illumination sources 432 are provided, they may be located on a rotating turntable, a sliding multiplexer, or other switch that butt-joint couples the successive illumination sources among the multiple illumination light sources to the coupling end of the optical fiber 430. A light source device 432 having the same size or slightly larger than the collection end of the optical fiber 430 provides the most efficient butt-joint coupling.

[0049] The plastic optical fiber 430 may be commercially available under the trade name Raytela TMThe fluorinated polymer optical fibers are obtained from Toray Industries, Inc. of Japan or other plastic optical fiber suppliers. The plastic optical fiber 430 can reduce costs relative to the glass optical fiber 430, which may be a particularly important consideration in disposable or discardable endoscope designs. The plastic optical fiber 430 can be formed of two different plastic resins having two different refractive indices, with the higher refractive index resin serving as the core and the lower refractive index resin serving as the cladding. The boundary between the layers can provide total internal reflection to conduct light along the optical fiber 430. The diameter of the optical fiber 430 can be selected to optimize several simultaneous characteristics. The amount of light that each optical fiber can carry is roughly proportional to the cross-sectional area. The cost of the optical fiber 430 is mainly proportional to the length, with a smaller increase in cost as the diameter increases. Similarly, the manufacturing cost generally increases with the number of optical fibers and with the number of optical fibers that are broken or damaged during the manufacturing process, so the fewer the larger diameter optical fibers, the lower the cost tends to be. On the other hand, it is usually more difficult to install the camera 410 and any working channel devices, and it is easier to install them in small spaces if the diameter of the optical fiber 430 is smaller, which tends to favor more small diameter optical fibers 430. To optimize among these trade-offs, in some cases, at least one optical fiber, at least two optical fibers, at least three optical fibers, at least four optical fibers, at least six optical fibers, at least eight optical fibers, at least nine optical fibers, at least twelve optical fibers, or at least fifteen optical fibers can be used. The diameter of the optical fibers can be about 0.4 mm, 0.5 mm, 0.6 mm, 0.75 mm, or about 1 mm. They can be placed around the working end 400 of the endoscope 100. In other cases, particularly for endoscopes with a larger diameter, fewer larger diameter optical fibers can be used, or the optical fibers can be fed into a light guide 450 to irradiate around the image sensor 410 in the area of the end 400.

[0050] Refer to Figure 5, in some cases, the optical fibers 430 may be relatively evenly spaced around the 360° periphery of the end 400. The placement of the illuminating optical fibers 430 is more uniform, and centered on the camera 460, it is possible to reduce illumination variations, shadows, and other unwanted artifacts on the imaging field. In other cases, the distal end faces of the optical fibers 430 or the light guides 450 may be distributed over an arc less than 360°, such as at least about 180°, at least about 240°, at least about 250°, at least about 260°, at least about 270°, or at least about 300°. In some cases, the endoscope may be used very close to the anatomical structure where the surgery is being performed, and thus distributing the illumination emission around the periphery can reduce glare and hot spots. In some cases, larger optical fibers 430 may be used for a portion of the periphery, and smaller optical fibers 430 may be used for the portion of the end of the endoscope that is crowded with other mechanical components. The closer the optical fibers 430 are to a uniform 360° distribution, the more uniform the illumination, and thus the better the image. Using optical fibers 430 with a larger numerical aperture or other dispersion at the end can also improve the dispersion, thereby improving the uniformity of illumination and the image quality. Non-circular optical fibers 430 can be used to allow for a larger surface area at the illuminating end of the optical fiber, thus providing better illumination.

[0051] Referring to Figure 4H and 4I , the image sensor 410 may be mounted on the flexible circuit board 416. The lens and filter 434 may be held in place by the inner end portion 436, and these parts may be assembled into the lens subassembly 460.

[0052] Referring to Figure 6 , 7 and 8, the lens assembly 460 may be formed in the tube 462, which surrounds the end cap 464, the first lens 466, the spacer / aperture 468, and the second lens 470 as well as the filter. The diameters of the circular parts 464, 466, 468, 470 may be about 1 mm or 1.2 mm. To facilitate reliable assembly, the shapes 474 embody the principle of error prevention so that they can only be stacked in one way. For example, the cone angle, straight-vs-curvature, etc. 474 may be different at different joints so that the parts cannot be assembled in the wrong order. For the two lens parts 466, 470, the lens itself is only the central circular portion 472 (which looks similar to Figure 6 and Figure 8 the cornea of the eye in Figure 8Shown in the front lens as a recess 472 and in the rear lens as a raised bubble 472. The central spacer 468 can have an exact lateral depth to ensure the correct spacing between the two lenses 466, 470, and have a relatively small central aperture to block excess light. The outer cover 464 can be used as a positioning spacer, as a flange for capturing other parts, and / or for blocking excess light. The excess light to be blocked can be light leaking from the light guide 450, or light indirectly reflected from within the surgical cavity but outside the image area. The excess light can be blocked so that it does not degrade the image quality.

[0053] Referring Figure 9 , the image sensor 410 can be mounted on the flexible circuit board 416. The end can be formed using a chassis, which in turn holds the camera 410 and the cover window in place. The chassis parts can be molded as a single part of opaque plastic or made of machined aluminum. The sides of the chassis can have channels for holding the light guide to the periphery. The chassis can have features at the joints that mate in only one way (e.g., a circular protrusion at the front of the chassis can mate with a circular groove in the rear chassis and make the mating parts square to ensure angular reproducibility). The chassis has a stepped conical hole 486 to reduce the interference of stray light reaching the camera. The rear chassis 484 can have an opening so that it does not contact the light guide 450 in the narrowing area, since the internal reflection angle of the optical fiber component is higher when against air than when against plastic. The lens assembly ( Figure 6 and Figure 7 460 in) can be mounted in the front chassis 482. Then, the rear chassis 484 can slide along the length of the circuit board 416 so that the circuit board 416 extends through the central hole of the rear chassis part 484. Then, the image sensor 410 / circuit board 416 can be mounted to the front chassis 482. Then, the rear chassis 484 can mate with the front chassis 482, and the front chassis 482 holds the lens assembly 460, the camera 410, and the board 416 in place relative to the two chassis parts 482, 484. This method can reduce the bending of the board 416, which can reduce the risk of strain to the flexible board 416 and its electronics, but still creates some slack and stress relief in the assembly.

[0054] The lens assembly can include an IR cut-off filter to remove unwanted IR from entering the image sensor.

[0055] Alternatively, the lens and filter elements can be directly adhered to the image sensor. The spacing between the image sensor and the lens and filter elements can be controlled via glass beads with a diameter matching the required spacing.

[0056] The chassis 480, 482, 484 can sequentially mount the transparent windows. The window 420 can be molded and bonded in place, or can be overmolded last as a molding step to hold the other components together. Light can be transmitted from the optical fiber via the light guide 450 to the face of the endoscope.

[0057] Then, the front and rear chassis 480, 482, 484 hold the lens and filter assemblies 460, the image sensor 410, and the flexible board 416 and hold them in the proper spatial relationship within the shaft 110. This reduces the number of parts. The chassis 480 can hold all the components in an assembly that can be installed in the shaft 110 in a single operation, which can simplify manufacturing. The parts 474, 489 can use error-proofing design techniques so that the configuration of the parts allows only one-way assembly and draws attention before an error propagates. III.C. Diffusion Terminal Surface

[0058] In some cases, the distal surface 490 of the optical fiber 430 or the light guide 450 can be roughened or coated with a diffusive coating, similar to the coating used to coat the interior of a soft white light bulb. By diffusing the light at the emission end 490 of the optical fiber 430 or the light guide 450, the dispersion angle can be increased, which increases the illumination cone and the field of view width and can reduce unwanted shadows and other artifacts. In some cases, the dispersion can be achieved by a holographic diffuser in the optical fiber 430 or the light guide 450. In other cases, the diffuser can be applied by a random process, such as sandblasting, molding on a sandblasted surface, or by some similar random process. In other cases, one or more texture patterns can be lithographed in the die steel for the tip of the optical fiber or the light guide 450. An example texture can be a series of micro-domes, small circular features, each having a lens profile designed to diffuse light. The micro-domes can be randomly placed and have random sizes to avoid collimation or diffraction in a particular direction, which can cause cold spots. In some cases, the distal surface 490 can be roughened by a rough grinding process, similar to the early stages of grinding a lens. Opal glass can be embedded in the distal end 490 of the light guide 450. The distal end 490 can be textured with other diffusion patterns, such as circles, lines, or hexagons. IV. Endoscope Tip

[0059] The components of the endoscope tip 400 can be designed to allow the image sensor 410, lens, filter, illumination source 418, and window to be mounted within a confined space having a diameter of 6 mm or less, 5.5 mm or less, 5 mm or less, 4.5 mm or less, or 4 mm or less, such as an endoscope or arthroscope for arthroscopic surgery. In some cases, fluid management can be managed in the same space. In some cases, the shaft can have the strength and stiffness common in arthroscopes. In some cases, the illumination emission can be emitted by one or more LEDs located at or near the endoscope tip. In other cases, the illumination emission can be emitted via the optical fiber 430 and / or the light guide 450, which conduct the illumination light around the image sensor 410 within the diameter of the shaft 110. IV.A. Molding and Assembly of Components of the Endoscope Tip

[0060] Referring Figure 10A , the endoscope tip 400 can be formed by a spacer clip 1020 that holds the flexible circuit board 416, which in turn mounts the camera 410 and the LED 418. The spacer clip 1020 and the camera housing 1012 can be formed of an opaque biocompatible plastic, such as Lustran 348. The camera 410 can be mounted on one side of the flexible circuit board 416, and the LED 418 can be mounted on the other side. The transparent window 420 can protect the image sensor 410 from the external environment, such as the tissues and body fluids of arthroscopic surgery, and the pressurized insufflation fluid.

[0061] Referring Figure 10B and 10C , the components of the tip can be mounted on the flexible circuit board. The flexible circuit board 416 can be bent into a bracket, chassis, or the channel of the spacer clip 1020 to position the illumination emitter (the emission end of the LED or the optical fiber or the light guide). The flexible circuit board 416 can have multilayer printed lines on the surfaces of multiple planes of the board. To provide signal integrity, anti-interference shielding, impedance control, and mechanical flexibility, the ground plane can be laid on the board as a patch pattern (as opposed to a traditional solid ground plane). The layers with signal lines can be alternated between the layers filled with the ground plane. Different parts of the board plane can be alternated for signal or ground planes to provide the required electrical performance. Various spacings and geometric characteristics can be tuned and adjusted to provide the required impedance matching and signal shielding, and improve the manufacturability for a given manufacturing tolerance.

[0062] Referring Figure 10D and 10E, the lens and filter element can be held in the camera housing 1010. The camera housing 1010 can be molded around the lens element. Alternatively, the lens and filter elements can be assembled and then the camera housing 1010 can be lowered onto the image sensor and fused together. The camera housing 1010 and the lens assembly can be fixed to the terminals of the flexible board 416. The attachment can be achieved by adhesive, thermal welding or sonic welding. The lens assembly can include an IR cut-off filter to remove unwanted IR from entering the image sensor. The combination of the flat and beveled surfaces can be customized to match the interior 1042 of the end sheath 1040 to ensure that the camera 416 is precisely located within the end 400. The front plane 1044 of the lens barrel of the camera housing 1010 can be positioned to press against the window 420 to position the camera 410.

[0063] Alternatively, the lens and filter elements can be directly adhered to the image sensor. The spacing between the image sensor and the lens and filter elements can be controlled via glass beads whose diameter matches the required spacing.

[0064] Referring to Figure 10F , 10G , 10H and 10I, the spacer clip 1020 can have a mounting surface 1022 for the camera 410, a holding pocket 1024 for the LED 418, and a curved channel 1026 into which the flexible board 416 is inserted. The mounting surface 1022 can be slightly recessed to accommodate flexible boards of different thicknesses or to allow the use of a pressure-sensitive adhesive. The positioning of the camera must be very precise, which is achieved between the spring thrust of the flexible board 416 against the window 420, as described in paragraph

[0069] below. The pocket 1024 can allow the final position of the LED 418 to float slightly forward so that it is pressed against the rear surface of the window 420, as described in paragraphs

[0069] and

[0070] below.

[0065] Referring to Figure 10J , 10K and 10L, the end can be assembled by connecting the LED 418 to one side of the flexible board 416 and the camera 410 to the other side. In both cases, the electrical connection can be welded and the structural components can be glued. The attachment can secure two of the four sides (e.g., the long sides) of the camera housing 410, 1010, while the other two sides are not fixed. Leaving two sides unsealed can prevent gas from being trapped inside the camera housing 1010 during the gluing process and can provide a release for thermal expansion. The flexible board 416 can be threaded through the channel 1026 of the spacer clip 1020. Then, the ends of the LED 418 and the flexible board 416 can pass through the hole 1028.

[0066] Referring to Figure 10M, the ends of the LED 418 and the flexible board 416 can be inserted into the retaining pocket 1024 so that the LED 418 faces outward.

[0067] Referring to Figure 10A and 10N , the shaft 110 can be inserted into the plastic deflector at the end of the trocar. The insertion portion 1036 of the spacer clip 1020 can have an asymmetrical octagonal shape to engage with the mating asymmetrical octagonal opening of the plastic deflector. The asymmetrical shape (or keying) ensures the correct orientation. The deflector can have a tongue 1032, and the spacer clip 1020 can have a mating recess 1034 that locks together to ensure that the two parts are assembled with respect to each other in the proper orientation and prevent the user from twisting.

[0068] Referring to Figure 10O , the end outer sheath 1040 with the transparent window 420 can slide over the spacer sheath. The spacer clip 1020 can have a profile (such as a trapezoid) that matches the mating profile of the hole 1042 of the end outer sheath 1040 to ensure a single assembly orientation only.

[0069] Referring to Figure 10P , the flexibility of the board 416 may tend to push the camera 410 forward against the window 420 at the flush contact 1044 and push the LED 418 forward against the window 420 at the flush contact 1045. The end outer sheath 1040 can have internal features 1042 that engage with the faces of the camera housing 410, 1010 and the face of the LED 418 to hold the camera 410 and the LED 418 in an exact orientation. For example, the bevel angle of the camera housing 1010 can mate with the beveled internal feature 1042 of the end sheath 1040 to ensure precise positioning of the camera relative to the end sheath 1040.

[0070] The bending of the flexible board 416 through the channel 1026 and the elasticity behind the window 420 can urge the flat surfaces 1044 of the LED 418 and the camera housing 1010 against the inner surface of the window 420, which keeps the LED 418 and the camera 410 in precise angular alignment. This may tend to hold the camera 410 precisely perpendicular to the window 420 to reduce refractive distortion. The LED 418 can have a light distribution cone 1046 of about 30°. At the outer surface of the window 420, a few percent of the light can be reflected 1047 back into the interior of the endoscope. The spacing between the LED 416 and the camera aperture 1048 can be large enough so that the back reflection 1047 does not enter the camera aperture 1048.

[0071] Referring again to Figure 10A, the spacer clip 1020 holds the LED 418 in position in front of the lens of the camera 410. Since most of the light from the LED is emitted forward, keeping the camera behind the light cone ( Figure 10P of 1046) can reduce light leakage into the camera 410.

[0072] Referring to Figure 10Q , the components at this time can be designed to engage with each other sufficiently via a sliding / pressure fit to maintain integrity without adhesives. The outer sheath 1040 can be designed to fit precisely over the spacer clip 1020 such that the outer sheath 1040 can be very thin to operate within the very limited space inside the tip 400, while the combination of the outer sheath 1040, spacer clip 1020, and flexible board 416 can fit tightly together to provide structural integrity. This fit may leave a groove 1049. When the shaft assembly 120 rotates, an adhesive (such as an ultraviolet curable adhesive) can be placed into the groove 1049 via a needle. This adhesive can be cured to seal, prevent fluid intrusion, and provide a final structural lock.

[0073] This overall design concept can allow for the reconfiguration and reuse of most of the engineering for endoscopes of different sizes, and can be extended according to the size of the sensor and the needs of a particular procedure (in contrast, for rod lens endoscopes, many design decisions are dedicated to a single design). Features that contribute to scalability include the use of a single flexible board, top and bottom brackets or chassis 412, 414, 438, and the overmolded window 420. Error-proofing design principles can be applied to ensure that each assembly step allows only one direction. V. Image Processing Unit

[0074] Referring to Figure 11A and 11B , the Image Processing Unit (IPU) can drive and receive signals from the endoscope via an interface board, a cable, and a custom or off-the-shelf motherboard. In some cases, the motherboard can be an off-the-shelf motherboard with an Intel CPU and an Nvidia GPU. The motherboard provides most of the external interface ports. The patient can be isolated from the line voltage (110 or 120V 60Hz in the United States, 240V 50Hz in Europe) through a medical-grade AC / DC power supply and a separate interface board (referred to as the patient interface board). The patient interface board processes the signals, converting them between the form of the signals used inside the IPU and the form of the signals going to and from the endoscope. V.A. Image Processing

[0075] An image processing computer can perform image processing. The GPU provides well-documented APIs that can be used to accelerate graphics processing, while the software running on the motherboard can have internal APIs that allow combining software processing components for image enhancement. A series of video chips in the endoscope handle and the IPU (Image Processing Unit) box can convert very small high-speed video signals from the sensor (such as the MIPI-CSI2 interface in Bayer format) into signals suitable for transmission over distances of more than a few centimeters and into a protocol that is easier to process and store at the various stages of the imaging pipeline (such as YCbCr422 or MPEG). The IPU processor can receive data (which can be video data, still images, telemetry data, etc.) from the endoscope via the handle board, cable, and patient interface board. The IPU can capture still images from the video and / or process the video through image correction and enhancement software to deliver high-quality images on a monitor or store them on some storage medium or in the patient record.

[0076] A variety of video signal processing chips, image signal processors (ISPs), and graphics processing units (GPUs) can perform multiple video transformations on the video data received from the endoscope before the data is displayed on a monitor or saved to an output device. The IPU box can have multiple processors, including a dedicated image signal processor (ISP), a general-purpose CPU (such as an Intel Pentium), a graphics accelerator (GPU), a field-programmable gate array (FPGA), custom accelerator hardware, and perhaps other processors. The video conversion can be performed in one or another of these processors, or in software, or in some combination of hardware and software. The sum of the processing capabilities can be selected to ensure that the image processing can be performed within the requirements of image latency. The following conversions can be performed: · Receive the raw image data in the MIPI-CSI2 stream in Bayer format from the image sensor of the endoscope and re-encode it into a YCbCr422 or h.264 MPEG stream to improve processability · Convert the MIPI-CSI2 video stream into a USB 3.0 video stream compliant with the UVC specification via a video stream processor such as Cypress CX3. · HDR or WDR processing (High Dynamic Range or Wide Dynamic Range) - software (a) to extend the dynamic range of the captured image by avoiding overexposed or underexposed areas of the video. This is achieved by combining consecutive overexposed and underexposed image frames from the image sensor and reducing the display intensity of abnormally bright pixels to reduce hot spots and increasing the display brightness of abnormally dark pixels in the frame to improve visibility of the image. See Figure 11F . HDR / WDR processing can use the Mertens exposure fusion algorithm. ·Perform rotation and image correction based on the rotation sensor of the handle (see the discussion in Figure 3A , 3B and 3C). Include displaying and rotating position indicators around the perimeter of the circular mask on the user interface, which can be displayed as arrows. ·Distortion correction (systematic distortion caused by fisheye distortion or similar distortion in lens specifications, or specific distortion measured within a specific range, corrected in the IPU through inverse transformation) to remove artifacts. ·Crop the rectangular image received from the endoscope into a rectangle that can be rotated around the center point of the display. Apply a circular mask to this rectangular crop to provide the user with a circular image display. This can replicate the view used by surgeons with dozens of rod-lens endoscopes. Additionally, outside the field-of-view cone provided by the lens, the outer edges of the image may be distorted or blocked by the edges of the lens housing, thus conveying more interference than information. ·Perform automatic exposure by adjusting the exposure time and gain in the image capture pipeline to achieve the desired average image brightness. ·Demosaic ·Black Level correction ·Gain adjustment ·Shadow correction ·Defect correction ·Noise reduction ·Tone mapping ·Color correction and white balance correction ·Zoom in / out within the target image ·Lens resolution correction ·Local contrast enhancement ·Edge enhancement ·Image magnification (magnify the circle displayed on the monitor, and the upper and lower limbs of the circular display may be lost) ·Reformat and compress the video data for storage on the storage device, and decompress the stored video data for display. ·Control the transmission over the network connection for storage on the storage device in the cloud or locally in the IPU, or other non-cloud storage ·Super-resolution is discussed in Sections V.D

[0085] to

[0094] below - this upsamples from a lower resolution (e.g., 1280x720) to 2160x2160 ("4K") resolution ·The frame writer is the last stage, which puts the video into the frame buffer of the video system for display or storage. The fully processed video stream can be displayed on the video monitor, or can be sent to the storage device or network interface.

[0077] Dividing the pipeline into multiple stages enables parallelism. For example, each stage can be assigned to a core in a multi-core CPU or a different functional unit of a GPU. V.B. HDR Exposure Fusion to Maintain Frame Rate

[0078] Refer to Figure 11F , HDR exposure fusion can be performed on pairs of frames captured simultaneously by two different cameras, and then the images are merged in pairs. The exposure fusion algorithm includes Mertens-Kautz-Van Reeth or Hugin / Enfuse.

[0079] In other cases, a single image sensor can be programmed to overexpose frame n, then underexpose frame n+1, then overexpose frame n+2, and so on. This can be controlled by strobing the LED 418 at the frame rate or by controlling the exposure time of the image sensor. The short exposure time frames can present details of the overexposed parts ("hot spots") in the image, while the overexposed frames can present details of the underexposed parts ("dark areas") in the image. By merging the frames, both the hot spots and the dark areas are captured in the output image, thereby increasing the dynamic range that can be captured.

[0080] Then, in addition to being applied to overlapping frame pairs, the same class of HDR exposure fusion algorithm can be used to merge the frames in pairs, to merge frame n with frame n+1, then frame n+1 with frame n+2, then frame n+2 with frame n+3, and so on. This maintains the output frame rate at the input frame rate. V.C. Auto Exposure

[0081] Auto exposure algorithms can be used to adjust for fluctuations in the light intensity level of the scene being captured by the image sensor to reach a target brightness level. If the camera moves closer to an object with static gain, exposure, and illumination intensity, the entire scene becomes brighter, and thus, the exposure time, gain, and / or illumination intensity for each frame should be reduced to capture less light. Conversely, if the camera moves farther away from the object, the entire scene becomes darker, and the exposure time, gain, and / or illumination intensity should be increased to capture more light.

[0082] Auto exposure implementations can control both the exposure time and the gain to achieve a target intensity set point. The gain control can be analog gain in the units of the pixels of the image sensor or digital gain applied in the image sensor or the digital image processing pipeline. The brightness set point can be set via the user "brightness" control or can be set automatically. The auto exposure algorithm can perform the following steps: 1. Divide the frame into n x n pixel blocks. 2. Calculate the average intensity of each block. 3. Compare the calculated intensity for each block with an intensity set point, which can be set for each block or for the image as a whole, to obtain an error value for each block. Each block can be assigned a weight to scale its calculated error value. This weight allows some blocks to be more important than others (i.e., blocks in the middle of the grid are weighted higher than those further out). 4. Sum all the weighted block errors to obtain a total error value. 5. Evaluate the change: a. If the total error value is below a defined change threshold, no change is made. b. If the total error value is above the defined change threshold, scale the change for an update period and update the change threshold relative to the magnitude of the total error by the following equation. Maximum change threshold = Maximum change threshold + (Total error × Multiplier), where the multiplier < 1 to allow a damped response. c. The maximum threshold is set to minimize the user's perception of discrete light level changes in a similar usage environment, but allows for rapid updates when changing quickly from a dark to a bright or from a bright to a dark environment. The multiplier is used to adjust this response to achieve the fastest response time to large changes in environmental conditions while preventing oscillation of the light level perceived by the user. 6. Input the total error into an exposure or gain PID control: a. If the scene is too bright: (i) If the gain is at its minimum value, the exposure PID control runs (ii) Otherwise, the gain PID control runs b. If the scene is too dark: (i) If the exposure reaches its maximum value, the gain PID control runs. (ii) Otherwise, the exposure PID control runs c. Depending on the implementation, any two or more parameters can replace gain and exposure, including illumination intensity, exposure time, etc. 7. Write the resulting exposure and gain to the ISP.

[0083] The automatic exposure algorithm can be downstream of the WDR algorithm, perhaps the immediately following stage. This reduces the sensitivity of the automatic exposure algorithm to frame-by-frame variations in the exposure time used by the WDR algorithm. The automatic exposure algorithm can be run once every several frames (instead of every frame) to reduce the processing bandwidth. The per-block intensity calculation can be parallelized to run on the GPU.

[0084] The software can specify that many parameters of the algorithm can be adjusted via a configuration file loaded as part of system startup, including the number of frames allowed to run between recalculations of the auto-exposure parameters, the block size of step 1, the average intensity set point of step 3, the block weight map of step 3, and the PID coefficients for the PID calculation of step 5. Video Processing for V.D. Super-Resolution

[0085] Reference Figure 11C and 11D , the input to the super-resolution block can be a low-resolution video (e.g., 720×720 pixels (“720p”) or 1280×720 image), and the output can be an enhanced-quality 2160×2160 pixel (“4K”) image. The “super-resolution” block can in turn have a block diagram as Figure 11D shown. Machine learning models can be used to combine noise reduction, lens resolution correction, edge enhancement, local contrast enhancement, and magnification as an integrated module. When these functions are performed individually, each function makes various trade-offs, and image enhancement in one stage may interfere with and degrade enhancement in another stage. For example, many noise reduction algorithms tend to cause image blurring. Traditional edge sharpening tends to amplify noise. By combining all these functions in a single machine learning model, these trade-offs can be reduced.

[0086] Various types of machine learning models can be used with the systems disclosed with respect to Figure 11C and Figure 11D , including fully convolutional neural networks, generative adversarial networks, cyclic generative adversarial networks, or deep convolutional networks. Convolutional neural networks (CNNs) are particularly useful for image processing. The super-resolution CNN block can be formed by combining: · A CNN magnification module from NexOptic Technology Corporation in Vancouver, British Columbia. This can allow the processor to infer pixel interpolation based on local information as well as previous and next frame information to improve the apparent resolution. ● A noise reduction module from NexOptic. This can reduce noise and stray photons from the image sensor and electronics ● A lens resolution correction module from NexOptic. This step can enhance the performance of the lens by understanding the transfer function of the fixed image through the lens. · A local contrast enhancement module from NexOptic. This can assist the surgeon by increasing the contrast between light and dark, various shades of red, etc. ● Dynamic range compensation - Image portions washed out due to overexposure can be balanced relative to those washed out due to darkness. The overall dynamic range can be adjusted to increase contrast and extract details lost in overexposed or underexposed portions (see Figure 11F ). ● Edge enhancement module from NexOptic. This can reduce resolution loss (blurring) that may be introduced by the lens system (e.g., due to limitations in lens size or complexity) or by camera motion of objects in the scene, and can improve edge extraction to assist the surgeon by making structures more apparent at the surgical site. ● High-entropy random noise interference data compression. The CNN can be trained to recognize and remove random pixel noise, which can improve data compression.

[0087] By combining all these functions into a single CNN, local contrast, edge enhancement, and noise reduction can all be improved simultaneously. Similar to the human neural network's ability to optimize multiple parameters skillfully at the same time, the computer CNN can be trained to optimize several characteristics simultaneously. Hardware contrast and edge enhancement can be disabled. In some cases, degradation and training can involve at least two of the parameters in the above list, e.g., resolution and edge enhancement, or resolution and local contrast. In some cases, any three of these types of image degradation can be trained into the model, e.g., resolution, local contrast, and edge enhancement, or resolution, image sensor noise, and lens correction. In some cases, the model can be trained on any four of these parameters. In some cases, all five can be trained.

[0088] In one exemplary implementation, given an input sequence of low-resolution frames High-resolution frame I i The sequence corresponding to the low-resolution frames. The super-resolution frame can be calculated where T is the radius of the temporal neighborhood F i is the warping operator from frame i to the current frame S i is the decimation for frame i

[0089] Video super-resolution models can be executed in two steps: a motion estimation and compensation process, followed by an upsampling process. Alternatively, instead of explicitly computing and compensating for the motion between input frames, motion information can be implicitly utilized to generate a dynamic upsampling filter, and the super-resolution frame can be directly constructed by locally filtering the frames constructed at the center of the computation window. A machine learning model can be trained by capturing a reference video at normal resolution and then degrading the reference video via simulated resolution loss, introducing noise, lens aberration, and lens-like noise, reducing contrast, and / or applying a transformation that degrades the edges. The machine learning model can be trained to recover the full-resolution original reference video. The same training may be sufficient to allow a video captured at normal resolution to be upsampled to a higher resolution. A lens model can be created from a combination of images captured from design data and standard test patterns (such as a checkerboard or a Cartesian line array) to detect and measure lens defects specific to the lens design or each endoscope and create a general transformation or store registration corrections for a particular endoscope. In some cases, high-quality reference data can be displayed on a physical display and viewed via an endoscope camera. A machine learning model can be trained to reconstruct the reference data from the camera video. The training can utilize an l1 loss with total variation (TV) regularization to reduce visual artifacts

[0090] The lens correction model can address defects that still exist in the lens system after balancing all constraints, e.g., by creating a lens model and passing a large set of ultra-high-resolution images captured with a camera having a very high-quality lens (to establish a baseline "perfect" image) through the lens model, and then training a CNN to correct the set of images passed through the lens model to transform each image into a "perfect" image.

[0091] The super-resolution CNN may produce better overall image quality (compared to the raw data directly from the camera and compared to using all classical blocks independently). Combining classical enhancement algorithms with an enhancement CNN can provide an opportunity to adjust the parameters of the classical algorithms in parallel based on CNN training, where the classical algorithms require serial parameter adjustment. The super-resolution CNN can allow adjustable runtime performance via architecture selection, thus trading off between overall image quality and speed.

[0092] In some cases, a CNN can retrain itself on the fly. For example, at moments when the camera and the image are stationary relative to each other, frames can be intentionally captured alternately under underexposed (too dark) illumination and normal illumination. The CNN can be retrained to identify hotspots where details are lost due to overexposure, as well as hotspots where details are lost in the dark areas of underexposed frames. In some cases, several machine learning systems can be chained together, for example, one for enhancing the dynamic range, one for reducing blur and edge sharpening, one for identifying frame-to-frame motion, one for increasing contrast, and one for upsampling for super resolution.

[0093] In some cases, a bypass feature can disable the super resolution neural network and instead upsample the image to 2160x2160 resolution via conventional means such as bicubic interpolation.

[0094] NexOptic components can be obtained under the product name Super Resolution, as described in U.S. Patent No. 11076103, Gordon, Photographic Underexposure Correction Using a Neural Network, and U.S. Publication No. 2021 / 0337098A1, Gordon, Neural Network Supported Camera Image or Video Processing Pipelines, both of which are incorporated by reference. V.E. Diagnosis and Lesion Detection

[0095] In some cases, Figure 11B the image processing pipeline can include processing for detecting various lesions. For example, during a colonoscopy, the image processing pipeline can have a processor for detecting polyps. During an esophagoscopy, the image processing pipeline can have a processor for detecting Barrett's esophagus. V.F. Endoscope Control

[0096] The endoscope can have several controls, including buttons on the endoscope, a touch screen on the surface of the IPU, and a graphical user interface with a touch screen that can be accessed from an external computer over the Internet.

[0097] One button on the endoscope can control three things: (a) still frame capture, (b) video recording on / off, (c) LED adjustment, high beam / low beam. For example, a single press can capture the current view as a still frame. A double press can start or stop video recording. A triple press or a press for three seconds can adjust the LED brightness.

[0098] The IPU may have front panel controls for the endoscope, including image adjustment, color, brightness, zoom, etc. In the user-visible or system setup / test mode, the controls on the IPU front panel or via a computer accessed over the Internet can control: ● LED illumination - Since the endoscope on button is just a single momentary connection switch, it cannot provide fine control, only a rough on / off control. Another user interface can provide more precise illumination control ● Sensor control - Adjusting hue or color balance, zoom, etc. ● Controlling image and video storage in the IPU non-volatile memory - Which part of which video to store, etc.

[0099] The adjustment of LED brightness needs to be carefully integrated with the image sensor. If the brightness is controlled by traditional pulse width modulation (PWM) that is not synchronized with the frame synchronization of the image sensor, stripes may appear in the image. Alternatively, a constant current source or voltage-controlled current source can be used to adjust the LED brightness and avoid stripes. Flexible board and electronics in the V.G. endoscope handle

[0100] The flexible circuit board 416 can transfer signals and power from the handle to the components at the tip. At the tip, molded plastic parts (brackets or chassis 412, 414, 438) can hold all the components in the appropriate orientation. Components (image sensor, lens, filter, window, and mount) can be selected to ensure the required offset angle (usually 0°, 30°, 45°, or 60° on-axis) and the required field of view (usually 50°, 60°, 70°, 80°, 90°, 100°, 130°, or 180°).

[0101] The distance from the image sensor at the tip to the receiver on the circuit board in the handle is about 115 mm to 330 mm, which is relatively long for MIPI-CSI2 video connections. The flexible circuit board can select the circuit layout and shielding to create an impedance-matched signal path for the video data from the video sensor, with low radiation emission, low loss, and low sensitivity to external interference. By allowing the video signal from the image sensor to float with respect to the RF application energy, the connection of the isolated reference potential from the inner insertion shaft to the handle circuit board can prevent interference from RF ablation or coagulation devices, thus minimizing the interference caused on the signal conductors that transfer the MIPI-CSI2 signal from the image sensor to the handle board.

[0102] The rigid circuit board in the handle (HB PCBA - "Handle Board Printed Circuit Board Assembly") can have a microprocessor, a magnetic sensor, and a transmitter chip. The transmitter chip can receive low-power, high-bandwidth, high-speed signals that can be transmitted using MIPI-CSI2 from an image sensor received on a flexible board, and convert the video signal into a serial signal suitable for transmission through a 3-meter cable to the IPU. Since 3 meters is a relatively long distance, the cable can be carefully impedance-matched with low insertion loss to ensure signal integrity. The serial signal is received on the IPU, converted back to the MIPI-CSI2 interface, and passed to an Image Signal Processor (ISP) for processing. V.H. Cable

[0103] Reference Figure 11E and 11F The IPU can be connected to the endoscope via a custom cable. The length of the cable can be approximately 3 meters (10 feet) - long enough for the surgeon to move freely and keep the non-sterilized IPU at an acceptable distance from the patient. The connector can be customized to ensure that the endoscope cannot be connected to other devices that do not provide the necessary patient isolation.

[0104] The cable can use a USB Type A or C connector, as the connector has good shielding and physical insertion characteristics, so even in this application, the cable does not carry USB signals or use the USB protocol. The cable can have a protective cover that extends a few millimeters beyond the end of the USB connector (alternatively, the USB connector can be recessed below the end of the protective cover). When the cable is disconnected from the IPU, the cover can provide insulation around the connector, which provides the creepage distance and clearance distance required for patient electrical isolation, for example, in the case where the end of the cable happens to touch something charged or grounded. The cover can be keyed, so it will only connect to the correct port on the IPU, will not (easily) be inserted into a general-purpose USB connector, and ensures that the cable is only connected to the connector on the IPU in the correct way. The cable end and the plug on the IPU box can be color-coded with each other.

[0105] The cable can power the endoscope, send command signals to the endoscope, obtain configuration information stored in the endoscope's on-board memory, and bring the video signal from the endoscope back to the IPU. The cable can also support a scheme for detecting when the endoscope is connected to the IPU. This is achieved by sensing a voltage change on the pins of the endoscope cable, which are pulled to a logic high voltage when the cable is disconnected and forced to a logic low voltage when the cable is connected. The pins on the cable can be connected to a pull-up resistor on the IPU side and pulled to GND on the handle board side, so when the handpiece is connected to the IPU, the handle board will pull down the pins, and the processor may detect that the handpiece has been connected. V.I. Wireless Communication Instead of Cable

[0106] The cable connection between the IPU and the handpiece can be replaced by wireless transmission such as Bluetooth, Wi-Fi or other wireless protocols. In these cases, the handpiece can have a battery, the capacity of which can drive the handpiece during the longest duration of the operation. As required by the IEC 60601-1 standard, the wireless connection can provide an alternative architecture to achieve electrical isolation of the patient. V.J. Isolation

[0107] Refer to Figure 11G , the patient interface board can interrupt the copper signal path by providing an optical connection or a transformer, thereby electrically isolating the main board from the patient-facing cable and the endoscope. Isolation of data can be provided between the video stream processor (such as Cypress CX3) and the main board via an optical fiber cable, which is driven by USB 3.0 transceivers at both ends of the cable and has no power conductors, allowing the copper conductors to be interrupted while communicating via the USB 3.0 communication protocol.

[0108] The physical interface between the endoscope and the IPU can be a USB 3.0 cable consisting of three pairs of twisted wires, a ground conductor, and a pair of power lines, although the physical layer communication is not USB 3.0. The patient interface board can interrupt the copper signal path by providing an optical connection or a transformer, thereby electrically isolating the processing circuit from the patient-facing cable and the endoscope. The isolation mechanism can isolate the patient from the possibility of electric shock and prevent excessive leakage current.

[0109] The IPU box can include a transformer 1170 that steps down the 120 / 220V AC voltage to a secondary voltage for operating the processing circuit 1172 inside the IPU box, and a second transformer 1180 can isolate the secondary circuit 1172 from the patient and the patient-facing circuit.

[0110] Two safety capacitors 1182 and 1184 can be serially arranged across the primary and secondary of the isolation transformer. The purpose of capacitors 1182 and 1184 is to create a shunt for the common-mode current generated in the isolated switched-mode power supply using transformer 1180. With respect to the parasitic capacitance between the patient isolation island 1174 (including the endoscope) and ground, the lower impedance of these capacitors can attract most of the common-mode current, thereby reducing the common-mode current propagating between the patient isolation island 1170 (including the endoscope) and ground, and thus reducing the radiated emissions. These two capacitors can be surface-mounted ceramic capacitors to minimize their impedance at higher frequencies. Capacitor 1186 can be differentially placed across the secondary of transformer 1180 to create a low impedance at high frequencies across the secondary of the transformer. This low impedance allows the common-mode current traveling on the positive output of the transformer to travel to the negative output of the transformer, through capacitor 1186 and back to the transformer through capacitors 1182 and 1184. The two capacitors 1182 and 1184 can be serially arranged and can be UL-listed safety capacitors to meet the requirements of IEC 60601.

[0111] A second pair of two capacitors 1192, 1194 in series can connect the USB connector housing (the metal shield sleeve on the female side of the USB connector) to two mounting holes that are tied to the grounded IPU chassis to provide a grounded short circuit for the common-mode current injected into the endoscope and / or the endoscope cable. The capacitor pair 1192, 1194 can be symmetrically placed on each side of the USB connector housing, both connected to the grounded chassis mounting points (e.g., connected to the housing 1196 of the IPU 1100) to improve the shielding effect against the common-mode current injected into the endoscope cable.

[0112] The values of capacitors 1182, 1184, 1186, 1188, 1192, 1194 are selected to provide sufficient reduction of the common-mode current and to meet the leakage requirements of IEC 60601-1.

[0113] According to the IEC 60601-1 patient isolation requirements, a pure fiber optic cable can be used to transmit high-speed video data from the patient isolation circuit 1174 to the secondary circuit 1172. The fiber optic cable can include USB 3.0 transceivers at each end of the cable. The high-speed video from the endoscope can be converted from the MIPI-CSI2 protocol used by the image sensor to the USB 3.0 protocol through an integrated circuit. The USB 3.0 ultra-high-speed RX and TX data pairs can be converted to optical signals transmitted through the fiber optic cable via an optical transceiver. The optical transceivers at each end of the cable can be locally powered to avoid the need to run power and copper wires on the fiber optic cable, thus keeping the cable compliant with the IEC 60601-1 isolation requirements.

[0114] The patient interface board can provide an endoscope interface, including isolation circuitry and BF type patient isolation as required by IEC 60601-1. This includes isolation of the power supply, as well as isolation of any other interfaces (such as USB interfaces) with copper wires that can conduct electricity. V.K. Other Peripherals V.K.1. Monitor

[0115] The IPU can drive a video monitor so that the surgeon can have a real-time display of the surgery. V.K.2. USB Ports

[0116] USB ports can be provided at the front of the unit for use with a USB flash drive, which can be cable-connected to the main board. Four USB ports can be provided at the rear of the unit for use with a USB mouse and keyboard. Ethernet and Wi-Fi interfaces can be provided from the main board for network connection to cloud storage (see Sections VI.C and VI.D below, paragraphs

[0138] to

[0143] ). An analog microphone input can be provided at the rear of the unit, as well as a Bluetooth interface that can be used for annotation during the procedure. A speaker can be provided in the IPU. The AC power plug can supply power to the IPU. The AC power can be controlled by a power switch. V.K.3. Connection to Cloud Storage

[0117] The IPU and programming can allow for the capture and saving of videos, images, metadata, and other data. Programs on the IPU can allow for the updating of the IPU's software. This data can be uploaded or backed up, for example, via Wi-Fi, Bluetooth, or a similar wireless connection to the cloud, or can be stored on an external removable USB flash drive connected to the USB port. The flash drive can then be used to transfer the data to the patient record or upload it to cloud storage from an external PC as needed by the facility (see Sections VI.C and VI.D below, paragraphs

[0138] to

[0143] ).

[0118] Videos can be stored in two-minute increments. If a write error occurs, the length of the lost video can be kept within that limit. The stored videos and still images can be annotated with date, time, and location metadata, as well as the serial numbers of the endoscope and IPU. In the cloud, the serial numbers can be used to link the videos and images to the correct patient case.

[0119] At the end of each surgical day, the data for the day's cases can be stored in the cloud server or on the USB drive. If the connection to the cloud fails, the USB storage can provide an easily accessible backup. The surgeon can later access the cloud storage or USB data to transfer it to the patient's case and annotate it with the doctor's notes. V.K.4. USB Connection for Keyboard and Mouse

[0120] During normal operation, the endoscope buttons are the only available user input. A USB keyboard and mouse can be connected to the system to perform system configuration. The keyboard and mouse can allow access to service or configuration screens. V.K.5. Microphone

[0121] The IPU can have a connector for a wired microphone and can allow a wireless microphone to be connected. This can allow for real-time annotation of the video captured by the surgeon. The system settings can allow the user to specify whether they wish to enable audio and then connect a microphone with a 3.5 mm jack or a Bluetooth interface. V.K.6. Blowpipe

[0122] Refer to Figure 1C , for irrigation or insufflation (blowing), the endoscope can include a short-tail irrigation tube with a three-way stopcock at the end, allowing the user to connect an external irrigation pump and vacuum. The tail can also include a wire clip. The endoscope can be packaged with a disposable tubing set, with its proximal end connected to a fluid source such as saline and its distal end having a luer lock. The tubing set can have a pinch-proof transparent tube with a stopcock valve to select inflow or suction, and a tubing clip that can be used to stop irrigation at the endoscope. The transparent tube supports the flow of fluid through the endoscope handle to the front molding of the endoscope, where the fluid passes through the cannula cover between the inner tube and the cannula of the insertion shaft. The transparent tube is fixed to the endoscope front molding using barb fittings and retaining clips. VI. Electronic Serial Number VI.A. Electronic Serial Number

[0123] Each endoscope shipped can have one or more endoscope-specific data encoded in machine-readable and scannable and / or human-readable form. The data can include one or more of the serial number of the endoscope, configuration data, manufacturing calibration data, tracking data, etc. This data can be used for a variety of purposes.

[0124] Information can be encoded on the box, embedded in the packaging, or embedded in the endoscope as a scannable code. The scannable code can be any form of matrix (2D) or linear bar or machine vision code that can be scanned by a smartphone. Examples include QR codes, Code 39, Code 49, Code 93, Code 128, Aztec codes, Han Xin barcodes, Data Matrix codes, JAB codes, Maxi codes, PDF417 codes, SPARQ codes, and other codes. The scannable code can be an RFID or a similar tag that can be scanned by a sensor in the phone. The scanning can be optical or can use any IEEE 802 or related communication protocol, including Bluetooth, RFID (ISO 14443), or NFC (ISO 18092). The scannable code can be encoded on the packaging, on the endoscope handle, or on the nose cap of the replaceable endoscope insertion tip. Alternatively, it can be stored in the EEPROM memory in a handheld device and connected via SPI (Serial Peripheral Interface), I2C (Inter-Integrated Circuit), USB, or a single-wire protocol for reading when the endoscope is inserted into the Image Processing Unit (IPU). The endoscope can have a small amount of non-volatile memory that can be read and written by the IPU during initial device manufacturing. This memory can store an electronically readable serial number written to the memory during the manufacturing process. This memory can also store configuration information for each endoscope, such as endoscope model, serial number, white balance coefficients, lens properties that can be corrected in the IPU, focus parameters, etc. This memory can also be used to store usage information, such as timestamps or usage times determined by the IPU, to prevent reuse of the endoscope after 24 hours. To ensure tamper resistance, the information written to the handle memory can be written under a secure or encrypted protocol used between the IPU and the handle microprocessor.

[0125] The information can be stored as a single piece of data (essentially a serial number, or some other data that semantically equivalently uniquely identifies the endoscope), which can be used as an index key for a database at a server, and the database at the server in turn has complete data about the endoscope. In some cases, various operating parameters of the endoscope can be stored in the server's database, and the model or serial number can be used as a lookup key to retrieve this set of configuration data and parameters. In other cases, the operating parameters can be individualized for each separate endoscope. For example, at the start of a shoulder arthroscopic surgery, the IPU can confirm that the endoscope to be used is indeed an arthroscope with the appropriate diameter, length, and optical capabilities. These two methods can be combined so that some parameters are stored based on the model, while others can be stored individually according to the endoscope.

[0126] The data stored in the on-board memory or remotely accessible database can include: ● A unique serial number or database lookup key ● Model and version number (integer or ASCII) ● Text description of the part model / model identifier that can be displayed on the control display screen (usually a 32-character ASCII string) ● Calibration / standardization data ● Complete configuration specification - for example: o Manufacturer part number of the image sensor, which can allow looking up many additional attributes of the image sensor in a table in the IPU, including: o Dimensions of the image sensor (rows x columns) o Frame rate and frame reporting rate supported by the sensor o Minimum / maximum integration time and integration time configuration resolution (e.g., 0.1 to 100 milliseconds, increment of 1 millisecond) o Identifier of the illumination source on the endoscope - white, infrared, ultraviolet, each color, etc. o Identifier of the sensor in the image plane - for example, one bit on / off for each of red, green, blue, ICG infrared, and other colors to be extended in future software updates o Information for establishing white balance, color correction gamma curve, distortion correction coefficients, etc. o (Boolean) Whether the illumination source is provided or not in the handpiece o (Boolean) Whether the defogging heater is provided or not in the handpiece o (Boolean) Whether the rotation sensor is provided or not in the handpiece o (Boolean) Whether focus control is supported or not in the handpiece ● Calibration / normalization data - for example o Correction data for lens focal length variation o Correction coefficients for compensating image sensor color sensitivity, illumination color, white balance, distortion correction o LED illumination brightness coefficient. ● Identifier for enabling / disabling certain image enhancement parameters based on the hardware image configuration - This can be used to pre-configure image processing settings based on the expected imaging applications of this endoscope. For example, a bit vector can enable or disable optimizations for resolution, contrast, smoothing, and other optical characteristics ● Various size and length characteristics, which may be important for controlling water pressure, etc. ● Manufacturing date ● Date and time of first use ● Duration of the process

[0127] Storing data in on-board memory (instead of a non-on-board database) can improve on-site adaptability. On-board data storage can reduce the need for IPU software updates and can improve robustness if the endoscope is used in parts of a hospital or facility without reliable internet access.

[0128] Data stored in the handle can be encrypted with a decryption key stored in the IPU. Encryption can enhance security by preventing malicious actors from corrupting the memory contents or otherwise interfering with the normal operation of the system.

[0129] Data can be communicated in a fixed-field binary protocol or a "keyword=" protocol (similar to the JSON protocol for web pages).

[0130] The connector can be a standard connector (such as USB-A) or a proprietary connector. A proprietary connector can ensure that mismatched devices cannot be plugged together. A proprietary connector can allow for additional pins to support all the required signals and video, such as video signals over twisted pairs, higher current to power a heater in the handle, and an optical connector for illuminating the fiber optic. VI.B. Using Electronic Serial Numbers to Reduce Errors and Ensure Sterile Single-Use

[0131] The stored data can allow a single IPU to be used with multiple endoscope configurations, reducing the complexity of storing, supplying, and using different endoscopes for different purposes.

[0132] The database can store information tracking the endoscope's history. If the serial number is remotely scannable (e.g., in an RFID tag), the location of the endoscope can be tracked through the distribution channel and storage at the buyer's hospital. This information can be used to ensure that the endoscope has not exceeded any time limits, is not stored in a location known to exceed temperature limits, etc. For example, the IPU can enforce a 24-hour limit after first use by reading the time of first use from non-volatile memory on the handle board PCBA. At the start of the procedure, the IPU can query the internet to confirm that the endoscope has not exceeded the manufacturer's date, that the endoscope is still within specifications, and that it is not subject to any safety recalls.

[0133] When the endoscope is about to be used, the serial number can be scanned, which can be a two-dimensional optical barcode on the box enclosed in the packaging, or on the endoscope itself, or via remote sensing (e.g., RFID tag), or the serial number can be read from the EEPROM memory when the endoscope is inserted into the IPU. As an alternative, the box or packaging can be printed with information such as the product model, lot number, and serial number, which allows for redundancy in cases where electronically readable information cannot be read.

[0134] The serial number can be used to check any usage restrictions. For example, the endoscope can be sold as a single-use item to ensure sterility, reliability, and to meet all expiration dates. This single use can be recorded on the manufacturer's server or in the memory of the endoscope itself. This single use can be recorded as a single binary flag that, once set, will prohibit further use. Alternatively, the first use can be marked with a timestamp and / or location such that the endoscope cannot be reused for a period of time (e.g., two to four hours). This will allow for multiple insertions of the endoscope during a single procedure (e.g., to untangle cables or reset after a power failure), but still be sufficient to prevent reuse.

[0135] If the endoscope is refurbished, the flag can be cleared to allow reuse.

[0136] The electronic serial number can check if this endoscope has been assigned to a facility / location where use has already started.

[0137] When the procedure starts, or when the endoscope is inserted into the IPU, the IPU can run through a dialog to confirm that the endoscope and the procedure are suitable for each other. For example, the IPU can query the patient's electronic medical record to confirm the procedure to be performed and confirm that the attached endoscope is suitable for that procedure. If a mismatch is detected, the IPU can issue a warning and request confirmation and override. If there are audit issues, the serial number of the exact endoscope used can be stored in the medical record. VI.C. Using the Electronic Serial Number for Inventory Control, Location Tracking, Reordering, and Stock Management

[0138] The purchaser / hospital can interact with the database to set minimum inventory levels. Alternatively, a computer system accessible to the manufacturer can determine average usage rates, location-based delivery times, and any pending or in-transit inventory to calculate reorder inventory levels. As each endoscope is used, one or more computers can decrement the existing inventory levels, and if the decremented level suggests reordering compared to the reorder stock level, the computer can automatically enter a reorder to maintain the inventory at an appropriate level.

[0139] The location can be scanned when necessary, typically when the endoscope arrives at the hospital / purchaser site so that the inventory can be registered, and when the inventory is moved from one internal location to another (e.g., storage rooms on different floors or wings). Additionally, the system can use tracking information from UPS or Fedex or other shippers / logistics managers to determine the location of the in-transit inventory from the manufacturer, through the distribution chain, to the final hospital / purchaser. The system can use the tracking proof of delivery as a signal that the product has been received at the customer site.

[0140] If the endoscope appears to be missing, the system can issue a warning. For example, the system can calculate the typical inventory time at a given location (e.g., perhaps two weeks), and can note if an endoscope has not been scanned or moved within some multiple of that time. Similarly, the system can issue a warning for unexpected inventory changes. The system can be programmed to eliminate false alarms and over-reporting - for example, movement to a shipping center, or movement via the hospital's internal distribution system may take the endoscope on an unexpected route, but should be suppressed to avoid over-reporting.

[0141] This tracking can improve utilization and inventory management by ensuring "just-in-time" ordering, VI.D. Transferring patient data to the electronic medical record using an electronic serial number

[0142] During the procedure, the surgeon or assistant can mark all or a marked portion of the video for permanent storage in the patient's electronic medical record, or in another database maintained by the hospital / customer or the endoscope manufacturer. In some cases, the IPU can compute voice-to-text for the doctor's narration during the procedure. The IPU can be connected to a cloud application via Wi-Fi or Ethernet. Images and videos can be sent to this cloud application in real-time after each procedure, or can be stored on a USB memory. The images and videos can be sent as a live stream to the cloud application, or can be collected in the IPU's storage for periodic upload, e.g., at the end of the day.

[0143] As described in patent application serial number 16 / 278,112, filed on February 17, 2019 (incorporated by reference), this video can be edited and delivered to the patient, perhaps via voice dictation. The video can improve the patient's post-operative recovery and can provide a report tailored to the patient. VII. Examples

[0144] Embodiments of the present invention can include any one or more of the following features, alone or in any combination.

[0145] The endoscope 100 can have a handle and an insertion shaft, with a camera at its distal end. The insertion shaft can have solid-state illumination and imaging circuitry at or near its tip, which is designed to provide illumination and imaging of the interior of the body cavity to the surgeon during surgery. The proximal portion of the handle can have electronics for driving the illumination circuitry and receiving imaging signals from the imaging circuitry. The proximal handle portion can be designed to allow for disinfection between uses. The junction between the proximal handle portion and the insertion shaft can be designed to detachably connect the insertion shaft to the proximal handle portion. When it is detached, the junction can allow removal of the insertion shaft for disposal and replacement. The junction can be designed such that, when connected, it can transfer mechanical force from the surgeon's hand to the insertion shaft and provide an electrical connection between the proximal handle circuitry and the illumination and imaging circuitry. The handle can have proximal and distal portions. The distal portion can be located between the insertion shaft and the proximal handle portion. The insertion shaft can be rigidly fixed to the distal handle portion. The junction can be arranged to connect and disconnect the distal and proximal portions of the handle. The distal handle portion can be designed to indirectly transfer mechanical force between the surgeon's hand and the insertion shaft and provide an indirect electrical connection between the proximal handle circuitry and the illumination and imaging circuitry. The handle can have a rotary collar with surface features designed to assist the surgeon in rotating the insertion shaft relative to the proximal handle portion about the axis of the insertion shaft in the roll dimension. The electronics within the proximal handle portion can be designed to sense the roll of the insertion shaft and provide an angular rotation signal, which is designed to allow for the orthosis of the display image received from the imaging circuitry. The mounting for the image sensor can be designed to allow the image sensor to pan about a pitch or yaw axis perpendicular to the central axis of the insertion shaft. One or more ultraviolet LEDs within the endoscope can be designed to disinfect the interior area of the endoscope. A hose for blowing in fluid or gas can be designed to be located on or near the central axis of the proximal handle portion. Two or more insertion shafts having different sizes from each other can each be connected to the proximal handle portion at the junction to allow the proximal handle to be used in surgeries with different requirements for the insertion shaft. A disinfection cabinet can be designed to disinfect the components of the endoscope. The insertion shaft at the tip of the endoscope has a rigid proximal portion and a distal portion. The distal portion can be bent to direct the field of view of the imaging circuitry in a desired direction. The illuminator and the solid-state imaging circuitry are located at or near the distal end of the articulable distal portion. The illuminator is designed to illuminate the interior of the body cavity to the surgeon during surgery, and the imaging circuitry is designed to capture imaging of the interior of the body cavity to the surgeon during surgery. The coupling for the replaceable endoscope tip is designed to detachably connect the insertion shaft to the handle portion at the junction and disconnect the junction. The coupling has a mechanical connector. When the junction is detached, the mechanical connector allows removal of the insertion shaft from the handle for disposal and replacement.When the engagement member is connected, the engagement member is designed to provide mechanical force transmission between the surgeon's hand and the insertion shaft. The electrical connector is designed to connect the insertion shaft to the electronics in the handle. The handle electronics are designed to drive the illuminator and receive imaging signals from the imaging circuit, and the handle is designed to allow for sterilization between uses. The control force transmission element is designed to allow the surgeon to direct the imaging circuit by transmitting a mechanical force guided by the surgeon to the articulable distal portion. The distal flexible portion includes a series of articulated rigid segments. The sheath or cover on the articulated rigid segments is designed to reduce intrusion or compression. The distal flexible portion is formed from a solid member that is flexible in the lateral and height dimensions and relatively inextensible in compression in the longitudinal dimension. The distal flexible portion can extend from and retract into the solid sheath. The distal flexible portion can be bent in one dimension. The distal flexible portion can be bent in two orthogonal dimensions. The imaging circuit is mounted at or near the distal end of the articulable distal portion via a pannable mount. The pannable mount is designed as two sides of a parallelogram. The imaging circuit is mounted on structural segments hinged to the two sides of the parallelogram. The channels and holes are designed to allow flushing fluid to pass through to improve the field of view on the imaging circuit from the lens or window. The channels and holes are designed to allow insufflation fluid to pass through to expand the cavity for the surgery. The mechanical connector of the coupling includes a twist lock that is designed to secure the endoscope insertion shaft to the handle portion. A plurality of endoscope heads are bundled and packaged together through the handle. The handle has electronics designed to drive the illuminator and receive imaging signals from the imaging circuit. The plurality of heads and the handle are packaged for bulk shipping and sale. The illuminator is an illumination LED mounted at or near the distal end. The illuminator is the emitting end of an optical fiber driven by an illumination source in the handle. The camera 410 can be encapsulated within a plastic housing. The plastic housing can be formed as a overmolded sheath that is designed to protect the camera 410 from bodily fluids and to structurally hold the components of the head in an operating configuration. The overmolded sheath can be designed to retain a transparent window in the operating configuration with the camera 410. The overmolded component can be formed from a transparent plastic. The overmolded component can be designed to serve as a lens for the image sensor 410. The image sensor 410 can be mounted on a flexible circuit board. The flexible circuit board 416 can mount the illumination LED 418. The LED 418 and the image sensor can be mounted on opposite sides of the flexible circuit board 416. The image sensor 410 can be protected behind a transparent window. The window can be molded in two thicknesses, with the thinner portion designed for mounting and to allow illumination light to pass through, and the thicker portion located above the camera 410. The handle can contain a circuit board that has circuitry for controlling and receiving signals from the camera 410. The handle and its components can be designed to be free of metal fasteners and adhesives, except for those captured by overmolding.The control buttons of the endoscope can be molded with protrusions that serve as return springs. The protrusions can be adhered to the endoscope handle via melting. The circuit board can be overmolded with plastic, which encapsulates the circuit board and keeps it out of contact with water. The circuit board can be mounted to the handle via melting. The components of the handle can be connected to each other via melting to form an integral structure. The components of the handle can be connected by an elastic clip, which is designed to hold two components against each other before being connected to form an integral structure via melting. The handle can be formed by two concentric housings. The rotation of the two housings relative to each other can be controlled via one or more O-rings that frictionally engage with the two respective housings. The handle can be overmolded with a layer of high-friction elastomer. The insertion shaft can be connected to the handle via a separable joint. The water joint of the separable joint can be molded for an interference seal without an O-ring. The water chamber of the separable joint can be designed to apply a vortex to the water flowing from the handle to the insertion shaft. The insertion shaft can be formed of stainless steel and connected to the handle via a separable joint. The plastic components of the endoscope can be joined to the insertion shaft without an adhesive by overmolding plastic into slots aligned at an oblique angle in the wall of the insertion shaft. The water joint can be formed as two cones in an interference fit. The cones may interfere at the large diameter. The interference of the cones can be via ridges raised on the lip of the inner male cone. The obturator 104 can be designed to pierce tissue to introduce the endoscope. The features for twist-locking the obturator 104 to the trocar 102 can be compatible with the features for twist-locking the endoscope to the trocar.

[0146] The endoscope can have a handle and an insertion shaft. The insertion shaft has solid-state illumination and imaging circuitry at or near its distal end, which is designed to provide illumination and imaging of the interior of the body cavity to the surgeon during the surgery. The proximal portion of the handle has electronics for driving the illumination circuit and receiving video signals from the image sensor, and the proximal handle portion is designed to allow for disinfection between uses. The joint between the proximal handle portion and the insertion shaft is designed to detachably connect the insertion shaft to the proximal handle portion. When the joint is detached, the joint allows the removal of the insertion shaft for disposal and replacement. The joint can be designed such that, when connected, it can transfer mechanical force from the surgeon's hand to the insertion shaft and provide an electrical connection between the proximal handle circuitry and the illumination and imaging circuitry.

[0147] An endoscope may have a handle and an insertion shaft. The insertion shaft has solid-state illumination and imaging circuitry at or near its distal end, which is designed to provide illumination and imaging of the interior of a body cavity during a surgical procedure. The proximal portion of the handle has electronics for driving the illumination circuitry and receiving video signals from an image sensor. The proximal handle portion is designed to allow for disinfection between uses. And a coupling between the proximal handle portion and the insertion shaft is designed to detachably connect the insertion shaft to the proximal handle portion. The coupling is disengaged to allow removal of the insertion shaft for disposal and replacement. The coupling is reconnected to a new insertion shaft, and the connection is designed to provide mechanical force transfer between the surgeon's hand and the insertion shaft and electrical connection between the proximal handle circuitry and the illumination and imaging circuitry.

[0148] Embodiments of the invention may include one or more of the following features. The handle may have proximal and distal portions. The distal portion may be located between the insertion shaft and the proximal handle portion. The insertion shaft may be rigidly fixed to the distal handle portion. The coupling may be arranged to connect and disconnect the distal and proximal portions of the handle. The distal handle portion may be designed to indirectly transfer mechanical force between the surgeon's hand and the insertion shaft and provide an indirect electrical connection between the proximal handle circuitry and the illumination and imaging circuitry. The handle may have a rotating collar with surface features designed to assist the surgeon in rotating the insertion shaft about its axis relative to the proximal handle portion in a rolling dimension. Electronics inside the proximal handle portion may be designed to sense the rolling of the insertion shaft and provide an angular rotation signal designed to allow for orthosis of the display image received from the image sensor. The mounting for the image sensor may be designed to allow the image sensor to pan about a pitch or yaw axis perpendicular to the central axis of the insertion shaft. One or more ultraviolet LEDs inside the endoscope may be designed to disinfect the interior region of the endoscope. A hose for blowing in fluid or gas may be designed to be located on or near the central axis of the proximal handle portion. Two or more insertion shafts having different sizes from each other may each be connected to the proximal handle portion at the coupling to allow the proximal handle to be used in procedures with different requirements for the insertion shaft. A sterilizer cabinet may be designed to disinfect components of the endoscope.

[0149] The endoscope can have a handle and an insertion shaft. The insertion shaft can have a camera at its distal end. The camera 410 can be encapsulated in a plastic housing with an overmolded sheath that is designed to protect the camera 410 from damage by body fluids and to structurally hold the components at the tip in an operating configuration. The camera 410 can be protected behind a transparent window. The window can be molded in two thicknesses. The thinner portion is designed for mounting and to allow illumination light to pass through, and the thicker portion is located above the camera 410. The handle can hold a circuit board that has circuitry for controlling and receiving signals from the camera 410. The handle and its components can be designed to have no metal fasteners and no adhesives, except for those captured by overmolding. The handle can be formed by two concentric housings. The rotation of the two housings relative to each other can be controlled via one or more O-rings that frictionally engage the two respective housings. The handle can have an overmolded layer of high-friction elastomer. The insertion shaft can be connected to the handle via a separable fitting, and the water fitting of the separable fitting can be molded as an interference seal, without an O-ring. The insertion shaft can be connected to the handle via a separable fitting. The water chamber of the separable fitting can be designed to apply a vortex to the water flowing from the handle to the insertion shaft. The insertion shaft can be formed of stainless steel and connected to the handle via a separable fitting. The plastic components of the endoscope can be joined to the insertion shaft by overmolding plastic into slots angled in the wall of the insertion shaft, without an adhesive. The insertion shaft can be connected to the handle via a separable fitting. The obturator 104 can be designed to pierce tissue to introduce the endoscope. The features for twist-locking the obturator 104 into the trocar 102 can be compatible with the features for twist-locking the endoscope into the trocar 102.

[0150] The overmolded sheath can be designed to retain the transparent window in the operating configuration with the camera 410. The overmolded component can be formed of transparent plastic and designed to function as a lens for the camera 410. The camera 410 can be mounted on a flexible circuit board: an illumination LED 418 can be mounted on the flexible circuit board 416. The LED and the camera 410 can be mounted on opposite sides of the flexible circuit board 416. The control buttons of the endoscope can be molded with protrusions that function as reset springs and that are adhered to the inside of the endoscope handle by melting. The circuit board can be overmolded with plastic that encapsulates the circuit board so that it is not in contact with water. The circuit board can be mounted to the handle by melting. The components of the handle can be connected to each other by melting into an integral structure. The components of the handle can also be connected by an elastic clip that is designed to hold the two components fixed to each other before being connected into an integral structure by melting. The fitting can be formed as two frustoconical bodies with an interference fit. The two frustoconical bodies can interfere at their large diameters. The frustoconical bodies can interfere via ridges that protrude on the lip of the male frustoconical body.

[0151] An endoscope can have a handle and an insertion shaft. The insertion shaft has solid-state illumination and imaging circuitry at or near its distal end, which is designed to provide illumination and imaging of the interior of a body cavity to a surgeon during a surgical procedure. The proximal portion of the handle has electronics for driving the illumination circuitry and receiving imaging signals from the imaging circuitry, and the proximal handle portion can be designed to allow for disinfection between uses. A coupling between the proximal handle portion and the insertion shaft is designed to detachably connect the insertion shaft to the proximal handle portion. When the coupling is detached, the coupling allows removal of the insertion shaft for disposal and replacement. The coupling is designed such that when connected, the coupling can transfer mechanical force from the surgeon's hand to the insertion shaft and provide an electrical connection between the proximal handle circuitry and the illumination and imaging circuitry.

[0152] An endoscope can have a handle and an insertion shaft, the insertion shaft having solid-state illumination and imaging circuitry at or near its distal end, which is designed to provide illumination and imaging of the interior of a body cavity to a surgeon during a surgical procedure. The proximal portion of the handle can have electronics for driving the illumination circuitry and receiving imaging signals from the imaging circuitry. The proximal handle portion can be designed to allow for disinfection between uses. A coupling between the proximal handle portion and the insertion shaft is designed to detachably connect the insertion shaft to the proximal handle portion. The coupling can be detached to allow removal of the insertion shaft for disposal and replacement. The coupling can be reconnected to a new insertion shaft, the connection being designed to provide transfer of mechanical force between the surgeon's hand and the insertion shaft and an electrical connection between the proximal handle circuitry and the illumination and imaging circuitry.

[0153] Embodiments of the present invention may include one or more of the following features. The handle may have proximal and distal portions. The distal portion may be located between the insertion shaft and the proximal handle portion. The insertion shaft may be rigidly fixed to the distal handle portion. A coupling member may be provided to connect and disconnect the distal and proximal portions of the handle. The distal handle portion may be designed to indirectly transfer mechanical forces between the surgeon's hand and the insertion shaft and provide an indirect electrical connection between the proximal handle circuitry and the illumination and imaging circuitry. The handle may have a rotating collar having surface features designed to assist the surgeon in rotating the insertion shaft about the axis of the insertion shaft relative to the proximal handle portion in a rolling dimension. Electronics within the proximal handle portion may be designed to sense the roll of the insertion shaft and provide an angular rotation signal designed to permit orthosis of the display image received from the image sensor. The mounting for the image sensor may be designed to permit the image sensor to pan about a pitch or yaw axis perpendicular to the central axis of the insertion shaft. One or more ultraviolet LEDs within the endoscope may be designed to disinfect the area within the endoscope. The hose for blowing in fluid or gas may be designed to be located on or near the central axis of the proximal handle portion. Two or more insertion shafts having different sizes from each other may each be connected to the proximal handle portion at the coupling member to permit use of the proximal handle in surgeries having different requirements for the insertion shaft. A sterilizer cabinet may be designed to disinfect the components of the endoscope.

[0154] A replaceable endoscope tip for an endoscope may have a rigid proximal portion and a distal portion. The distal portion may be bendable to direct the field of view of the imaging circuitry to a desired direction. The illuminator and the image sensor may be located distally or near the distal portion of the articulable distal portion. The illuminator may be designed to illuminate the interior of the body cavity for the surgeon during the surgery, and the image sensor may be designed to capture imaging of the interior of the body cavity for the surgeon during the surgery. A coupling member is designed to separably connect the replaceable endoscope tip to the handle portion at the coupling and disconnect the coupling. The coupling member has a mechanical connector designed to: (a) when separated, the mechanical connector permits removal of the replaceable endoscope tip from the handle for discard and replacement; and (b) when connected, the coupling is designed to provide mechanical force transfer between the surgeon's hand and the insertion shaft. An electrical connector is designed to connect the replaceable endoscope tip to the electronics in the handle, the handle electronics being designed to drive the illuminator and receive the video signal from the image sensor, and the handle may be designed to permit disinfection between uses. A control force transfer element is designed to permit the surgeon to direct the direction of the imaging circuitry by transferring mechanical forces directed by the surgeon to the bendable distal portion.

[0155] An optical prism can be designed to shift the field of view offset angle of an endoscope. A connector is designed to fix the optical prism to the end of the endoscope. The field of view of the endoscope is offset from the axis of the endoscope at an initial offset angle and the optical prism is held against displacement forces during insertion of the endoscope into a body cavity. The optical prism and the connector are designed to reduce the offset angle of the field of view of the endoscope towards the axis relative to the initial offset when the prism and the connector are fixed to the optical end of the endoscope. The endoscope can be inserted into a body cavity. The endoscope has a field of view that is offset from the axis of the endoscope at an initial offset angle. The endoscope has an optical prism fixed to its distal end, which is designed to reduce the offset angle of the field of view of the endoscope towards the axis relative to the initial offset. The prism is fixed to the distal end of the endoscope by a connector, which is designed to hold the optical prism against displacement forces during insertion of the endoscope into a body cavity. The endoscope is withdrawn from the body with the prism fixed. The prism is removed from the endoscope. The endoscope is reinserted back into the body cavity with its field of view at the initial offset angle. The optical prism can be designed to reduce the offset angle of the field of view of the endoscope to no more than 10°, or no more than 5°, or no more than 3°. The optical prism can be optically convex to magnify the image. The optical prism can be optically concave to widen the field of view of the endoscope. The connector can be designed to be fixed to the endoscope by mechanical force. A filter can be coupled with the prism. The endoscope can have a wetting surface, which is designed to entrain an anti-stick lubricant in a layer on the lens or window of the endoscope. The wetting surface can be a porous solid. The porous solid can be formed by sintering or other heating of particles. The optical prism and the connector can be fixed to the endoscope for transportation and are designed to keep the anti-stick lubricant in contact with the lens or window of the endoscope during transportation. A vial, hole or cavity can have a lid with a seal to seal around the axis of the endoscope. The anti-stick lubricant can include silicone oil or a mixture thereof. The anti-stick lubricant can include a mixture of silicone oils of different viscosities. The vial or cavity can include an optical prism designed to shift the field of view of the endoscope.

[0156] A packaging for an endoscope can have mechanical features designed to hold components of the endoscope and protect the endoscope for transportation and / or delivery. The packaging has a vial, hole or cavity designed to keep an anti-stick lubricant in contact with the lens or window of the endoscope.

[0157] The distal flexible portion may include a series of articulated rigid segments. A sheath or cover may cover the articulated rigid segments, which is designed to reduce intrusion or squeezing. The distal flexible portion may be formed of a solid member that is flexible in the lateral and height dimensions and relatively incompressible in the longitudinal dimension when compressed. The distal flexible portion may extend from and be retractable into the solid sheath. The distal flexible portion may be bendable in one dimension. The distal flexible portion may be bendable in two orthogonal dimensions. A camera may be mounted at or near the distal end of the bendable distal portion via a movable mount. The movable mount may be designed as two sides of a parallelogram, and the camera may be mounted on a structural segment that is hinged to the two sides of the parallelogram. Channels and holes may be designed to allow flushing fluid to pass through to improve the field of view of the imaging circuit from the lens or window. Channels and holes may be designed to allow insufflation fluid to pass through to expand the cavity for surgery. The mechanical connector of the coupling may include a twist lock that is designed to fix the endoscope-replaceable endoscope head to the handle portion. A plurality of endoscope-replaceable endoscope heads may be packaged with the reusable handle for unitary shipping and sale, the handle having electronics designed to drive an illuminator and receive imaging signals from the imaging circuit. The illuminator may be an illumination LED mounted at or near the distal end. The illuminator may be the emitting end of an optical fiber driven by an illumination source in the handle.

[0158] An arthroscope may have a handle and an insertion shaft. The insertion shaft may have a solid-state camera near its distal end. The shaft may have a light conductor encapsulated therein, which is designed to conduct illumination light to the distal end. The shaft may have an outer diameter not greater than 6 mm. The shaft may have rigidity and strength to insert the camera into a joint for arthroscopic surgery. The light conductor in the area of the camera may be designed to conduct illumination light from the optical fiber through the space between the camera and the inner surface of the insertion shaft to the distal end.

[0159] The optical fiber may have a flat area that is shaped to be located between the inner surface of the outer wall of the endoscope camera and the endoscope shaft and is shaped to conduct illumination light to the distal end of the endoscope shaft for illuminating the surgical cavity to be observed by the camera. The diameter of the shaft may not be greater than 6 mm. The flat area is formed by heating an area of the plastic optical fiber and squeezing the heated area in a polished mold.

[0160] Embodiments of the present invention may include one or more of the following features. One or more light guides may be designed to conduct illumination light from an optical fiber to a distal end. The light guide may have a cross-section other than circular. The light guide may have a coupler to receive illumination light from a circular cross-section optical fiber. If the size of the optical fiber in the light guide corresponds to the radius of the insertion axis, the cross-section of the light guide in the region of the camera may be narrower than the diameter. At least one of the inner and outer surfaces of one or more light guides may be longitudinally grooved. The distal surface or flat region of one or more light guides may be designed to diffuse the emitted light. The distal surface of one or more light guides may have surface micro-domes designed to diffuse the emitted light, or may otherwise be configured to improve the illumination uniformity entering the surgical cavity into which the arthroscope enters. One or more light conductors in the region of the camera may be formed as a flat region of an optical fiber. The flat region may be shaped to be located between the inner surface of the outer wall of the endoscopic camera and the endoscopic shaft. The flat region may be shaped to conduct illumination light to the distal end of the endoscopic shaft for illuminating the surgical cavity to be observed by the camera. The outer diameter of the shaft may not be greater than 6 mm. The flat region may be formed by heating a region of a plastic optical fiber. The flat region may be formed by extruding the optical fiber in a polished mold. The components for mounting near the distal end of the endoscope may be shaped using error-proofing design principles to ensure correct assembly. The components for mounting the lens assembly near the distal end may be shaped using error-proofing design principles to ensure correct assembly. The components near the distal end may be shaped to allow focusing adjustment of the lens assembly during manufacturing. The endoscope may have a terminal window that is designed to be sealed with the shaft to prevent intrusion of body fluids, body tissues, and / or insufflated fluids. The terminal window may be designed to reduce optical artifacts. The artifacts that may be reduced may be reflections, light leakage in the endoscope, contamination by body fluids and / or body tissues, and fogging. The light conductor in the region of the camera may include at least one optical fiber of substantially continuous diameter from a light source, the diameter of the optical fiber not being greater than about 0.5 mm, and arranged around or partially around the circumference of the distal end of the endoscope. The arthroscopic insertion shaft may have a camera near its distal end. The shaft may have a light conductor encapsulated therein, which is designed to conduct illumination light to the distal end. The shaft may have rigidity and strength to insert the camera into a joint for arthroscopic surgery. The size of the flat region may be designed to conduct illumination light from the optical fiber through the space between the camera and the inner surface of the insertion shaft to the distal end.

[0161] An apparatus may include a computer processor and a memory. The processor is programmed to receive video image data from an image sensor at the distal end of the endoscope and display the image data in real time to a surgeon. The processor is programmed to process the image data received from the image sensor via a machine learning model, the machine learning model being trained to simultaneously upsample the image data to a higher resolution than that captured by the image sensor, sharpen edges, and enhance local contrast.

[0162] An apparatus may include a computer processor and a memory. The processor is programmed to receive video image data from an image sensor at a distal end of an endoscope and display the image data in real time to a surgeon. The video image data has a frame rate at which the image sensor generates the image data. The processor is programmed to control the image sensor and / or an illumination source designed to illuminate a scene viewed by the image sensor, the control being programmed to underexpose or overexpose every other frame of the video image data. The processor is programmed to process the image data received from the image sensor to combine consecutive frame pairs of the image data so as to adjust the dynamic range to enhance overbright or overdark portions of the image to expose details, and generate combined frames at the full frame rate of the video generated by the image sensor.

[0163] An apparatus may include a computer processor and a memory. The processor is programmed to receive video image data from an image sensor at a distal end of an endoscope and display the image data in real time to a surgeon. The processor is programmed to sum errors of an intensity of an image relative to a set point intensity. The processor is programmed to simultaneously control at least two of gain, exposure, and illumination via a PID control algorithm to achieve image display at the set point intensity, with the maximum change per step of the PID control damped to prevent oscillation.

[0164] Embodiments may include one or more of the following features, either alone or in any combination. The processor may further be programmed to control an image sensor and / or an illumination source designed to illuminate a scene viewed by the image sensor. The control may be programmed to underexpose or overexpose every other frame of video image data. The processor may further be programmed to process image data received from the image sensor to combine consecutive frame pairs of the image data, thereby adjusting the dynamic range to enhance over-bright or over-dark portions of the image for exposure details. The processor may further be programmed to generate combined frames at the full frame rate of the video generated by the image sensor. The processor may further be programmed to sum the error of the intensity of the image relative to a set point intensity. The processor may further be programmed to simultaneously control at least two of gain, exposure, and illumination via a PID control algorithm to achieve image display at the set point intensity. The maximum change of each step of the PID control may be damped to prevent oscillation. The processor may further be programmed to process image data received from the image sensor via a machine learning model, the machine learning model being trained to simultaneously upsample the image data to a higher resolution than that captured by the image sensor, sharpen edges, and enhance local contrast. The processor may further be programmed to enhance video image data via dynamic range compensation. The processor may further be programmed to adjust exposure time, illumination intensity, and / or gain in image capture to adjust exposure saturation. The processor may further be programmed to enhance video image data via noise reduction. The processor may further be programmed to enhance video image data via lens correction. The processor may further be programmed to enhance at least two of dynamic range compensation, noise reduction, and lens correction, in addition to resolution. The processor may further be programmed to rotate the image display to compensate for rotation of the endoscope. The processor may further be programmed to adjust exposure time, illumination intensity, and / or gain in image capture to adjust exposure saturation.

[0165] The various processes described herein can be implemented by a suitably programmed general-purpose computer, a special-purpose computer, and computing devices. Generally, a processor (e.g., one or more microprocessors, one or more microcontrollers, one or more digital signal processors) will receive instructions (e.g., from a memory or similar device) and execute those instructions to perform one or more processes defined by those instructions. The instructions can be embodied in one or more computer programs, one or more scripts, or in other forms. The processing can be executed on one or more microprocessors, central processing units (CPUs), computing devices, microcontrollers, digital signal processors, graphics processing units (GPUs), field-programmable gate arrays (FPGAs), or similar devices, or any combination thereof. The programs implementing the processing and the data on which it operates can be stored and transmitted using a variety of media. In some cases, hardwired circuitry or custom hardware can be used in place of some or all of the software instructions that can implement these processes, or the hardwired circuitry or custom hardware can be used in combination with some or all of these software instructions. Algorithms other than those described can be used.

[0166] The programs and data can be stored in a variety of media suitable for that purpose, or in a combination of different types of media that can be read and / or written by a computer, a processor, or a similar device. The media can include non-volatile media, volatile media, optical or magnetic media, dynamic random access memory (DRAM), static RAM, floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, DVDs, any other optical media, punch cards, paper tapes, any other physical media with a hole pattern, RAM, PROM, EPROM, FLASH-EEPROM, other non-volatile memories, any other memory chips or cartridges, or other storage technologies.

[0167] The database can be implemented using a database management system or an ad-hoc storage organization scheme. Alternative database structures to those described can be readily adopted. The database can be stored locally or remotely from the device accessing the data in such a database.

[0168] In some cases, processing can be performed in a network environment that includes a computer communicating (e.g., via a communication network) with one or more devices. The computer can communicate directly or indirectly with the devices via any wired or wireless medium (e.g., the Internet, LAN, WAN, or Ethernet, token ring, telephone lines, cable lines, wireless channels, optical communication lines, commercial online service providers, bulletin board systems, satellite communication links, any combination of the foregoing). The transmission medium includes coaxial cables, copper wires, and optical fiber 430, including lines that comprise a system bus coupled to a processor. Transmissions can occur over the transmission medium or via electromagnetic waves at various frequencies using a variety of protocols, such as via infrared, Wi-Fi, Bluetooth, etc. Each device itself can include a computer or other computing device, such as those based on or Centrino TM processors that are adapted to communicate with the computer. Any number and type of devices can communicate with the computer.

[0169] A server computer or central authorization authority may or may not be necessary or desirable. In various cases, the network may or may not include a central authorization device. Various processing functions can be performed on a central authorization server, one of several distributed servers, or other distributed devices.

[0170] The following applications are incorporated by reference: U.S. Provisional Application No. 63 / 538,485, filed on September 14, 2023, entitled "Endoscope"; U.S. Provisional Application No. 63 / 534,855, filed on August 27, 2023, entitled "Endoscope"; U.S. Provisional Application No. 63 / 531,239, filed on August 7, 2023, entitled "Endoscope"; U.S. Provisional Application No. 63 / 437,115, filed on January 4, 2023, entitled "Endoscope with Identification and Configuration Information"; U.S. Application No. 17 / 954,893, filed on September 28, 2022, entitled "Illumination for Endoscope"; U.S. Provisional Application No. 63 / 376,432, filed on September 20, 2022, entitled "Super Resolution for Endoscope Visualization"; U.S. Application No. 17 / 896,770, filed on August 26, 2022, entitled "Endoscope"; U.S. Provisional Application No. 63 / 400,961, filed on August 25, 2022, entitled "Endoscope"; U.S. Application No. 17 / 824,857, filed on May 25, 2022, entitled "Endoscope"; U.S. Provisional Application No. 63 / 249,479, filed on September 28, 2021, entitled "Endoscope"; U.S. Provisional Application No. 63 / 237,906, filed on August 27, 2021, entitled "Endoscope"; U.S. Application No. 17 / 361,711, filed on June 29, 2021, entitled "Endoscope with Bendable Camera Shaft"; U.S. Provisional Application No. 63 / 214,296, filed on June 24, 2021, entitled "Endoscope with Bendable Camera Shaft"; U.S. Provisional Application No. 63 / 193,387, entitled "Anti-adhesive Window or Lens for Endoscope Tip".U.S. Provisional Application No. 63 / 067,781, filed on August 19, 2020, entitled "Endoscope with Articulated Camera Shaft"; U.S. Provisional Application No. 63 / 047,588, filed on July 2, 2020, entitled "Endoscope with Articulated Camera Shaft"; U.S. Provisional Application No. 63 / 046,665, filed on June 30, 2020, entitled "Endoscope with Articulated Camera Shaft"; U.S. Application No. 16 / 434,766, filed on June 7, 2019, entitled "Endoscope with Disposable Camera Shaft and Reusable Handle"; U.S. Provisional Application No. 62 / 850,326, filed on May 20, 2019, entitled "Endoscope with Disposable Camera Shaft"; U.S. Application No. 16 / 069,220, filed on October 24, 2018, entitled "Anti-Fouling Endoscopes and Uses Thereof"; U.S. Provisional Application No. 62 / 722,150, filed on August 23, 2018, entitled "Endoscope with Disposable Camera Shaft"; U.S. Provisional Application No. 62 / 682,585, filed on June 8, 2018, entitled "Endoscope with Disposable Camera Shaft".;

[0171] For clarity of explanation, the above description focuses on a representative sample of all possible embodiments, teaching the principles of the present invention and conveying a sample of the best mode contemplated for carrying out the present invention. The present invention is not limited to the described embodiments. The formal definition of the property rights protected exclusively is set forth in the claims of exclusive control. This specification does not attempt to exhaust all possible variations. Other variations or modifications not described are also possible. In cases where multiple alternative embodiments are described, in many instances, it will be possible to combine the elements of different embodiments, or to combine the elements of the embodiments described herein with other modifications or variations not explicitly described. Unless otherwise explicitly stated, a list of items does not imply that any or all of the items are mutually exclusive, nor does it mean that any or all of the items are comprehensive in any category. In many cases, a feature or group of features can be used separately from the entire device or method described. Many alternatives, variations, modifications, and equivalents not described are within the literal scope of the following claims, and others are equivalent. The claims may be practiced without some or all of the specific details described in the specification. In many cases, the method steps described in this specification can be performed in an order different from the order presented in this specification, or in parallel rather than sequentially.

Claims

1. An apparatus, comprising: a computer processor and a memory; the processor being programmed to: receive video image data from an image sensor at a distal end of an endoscope and display the image data in real time to a surgeon, and process the image data received from the image sensor via a machine learning model, the machine learning model being trained to simultaneously upsample the image data to a higher resolution than that captured by the image sensor, sharpen edges, and enhance local contrast.

2. The apparatus according to claim 1, wherein the video image data has a frame rate at which the image sensor generates the image data; the processor being further programmed to: control the image sensor and / or an illumination source, the illumination source being designed to illuminate a scene viewed by the image sensor, the control being programmed to underexpose or overexpose every other frame of the video image data; and process the image data received from the image sensor to combine consecutive frame pairs of the image data, thereby adjusting the dynamic range to enhance over-bright or over-dark portions of the image to expose details and generating a combined frame at the full frame rate of the video generated by the image sensor.

3. The apparatus according to claim 2, the processor being further programmed to: sum the error of the intensity of the image relative to a setpoint intensity; and simultaneously control at least two of gain, exposure, and illumination via a PID control algorithm to achieve image display at the setpoint intensity, with the maximum change per step of the PID control being damped to prevent oscillation.

4. The apparatus according to claim 1, the processor being further programmed to: sum the error of the intensity of the image relative to a setpoint intensity; and simultaneously control at least two of gain, exposure, and illumination via a PID control algorithm to achieve image display at the setpoint intensity, with the maximum change per step of the PID control being damped to prevent oscillation.

5. An apparatus, comprising: a computer processor and a memory; the processor being programmed to: receive video image data from an image sensor at a distal end of an endoscope and display the image data in real time to a surgeon, the video image data having a frame rate at which the image sensor generates the image data; control the image sensor and / or an illumination source, the illumination source being designed to illuminate a scene viewed by the image sensor, the control being programmed to underexpose or overexpose every other frame of the video image data; and process the image data received from the image sensor to combine consecutive frame pairs of the image data, thereby adjusting the dynamic range to enhance over-bright or over-dark portions of the image to expose details and generating a combined frame at the full frame rate of the video generated by the image sensor.

6. The apparatus according to claim 5, the processor being further programmed to: Process the image data received from the image sensor via a machine learning model, the machine learning model being trained to simultaneously upsample the image data to a higher resolution than that captured by the image sensor, sharpen edges, and enhance local contrast.

7. The apparatus according to claim 5, wherein the processor is further programmed to: Sum the error of the intensity of the image relative to a setpoint intensity; and Simultaneously control at least two of gain, exposure, and illumination via a PID control algorithm to achieve image display at the setpoint intensity, with the maximum change per step of the PID control damped to prevent oscillation.

8. An apparatus comprising: A computer processor and a memory; The processor is programmed to: Receive video image data from an image sensor at a distal end of an endoscope and display the image data in real time to a surgeon; Sum the error of the intensity of the image relative to a setpoint intensity; And Simultaneously control at least two of gain, exposure, and illumination via a PID control algorithm to achieve image display at the setpoint intensity, with the maximum change per step of the PID control damped to prevent oscillation.

9. The apparatus according to claim 8, wherein the processor is further programmed to: Process the image data received from the image sensor via a machine learning model, the machine learning model being trained to simultaneously upsample the image data to a higher resolution than that captured by the image sensor, sharpen edges, and enhance local contrast.

10. The apparatus according to claim 8, wherein the processor is further programmed to: The video image data has a frame rate at which the image sensor generates the image data; The processor is further programmed to: Control the image sensor and / or an illumination source, the illumination source being designed to illuminate a scene viewed by the image sensor, the control being programmed to underexpose or overexpose every other frame of the video image data; and Process the image data received from the image sensor to combine consecutive frame pairs of the image data, thereby adjusting the dynamic range to enhance over-bright or over-dark portions of the image to expose details and generate combined frames at the full frame rate of the video generated by the image sensor.

11. The apparatus according to any one of claims 1-10, wherein the processor is further programmed to: enhance the video image data by adjusting exposure saturation via adjusting exposure time, illumination intensity, and / or gain in image capture.

12. The apparatus according to any one of claims 1-10, wherein the processor is further programmed to: enhance the video image data by adjusting dynamic range compensation.

13. The apparatus according to any one of claims 1-10, wherein the processor is further programmed to: enhance the video image data via noise reduction.

14. The apparatus according to any one of claims 1-10, wherein the processor is further programmed to: enhance the video image data via lens correction.

15. The apparatus according to any one of claims 1-10, wherein the processor is further programmed to enhance the video image data by dynamically adjusting at least two of dynamic range compensation, noise reduction, and lens correction.

16. The apparatus according to any one of claims 1-10, wherein the processor is further programmed to rotate the image display to compensate for rotation of the endoscope.

Images