Information management and inventory tracking for medical devices

Abstract

During manufacture of a medical device, parameters if the device are measured, and results of that measurement are stored in a database. For example, parameters of an endoscope's image sensor may be measured and stored in the database. The medical device has a machine-readable serial number. A computer reads the serial number and records a reduction of an on-hand inventory of the medical devices. When the on-hand inventory warrants reordering to replenish the on-hand inventory. When warranted, the computer places an order with a supplier of medical devices of the class to replenish inventory. A computer controlling the medical device may read the database to obtain the measured properties. The obtained data may be used to calibrate or adjust the device's function. For example, white balance and color correction of an endoscope may be used to compute normalized video.

Description

Information Management and Inventory Tracking for Medical DevicesBACKGROUND

[0001] This application claims priority from U.S. application Ser. No. 18 / 370,375, filed Sep.19, 2023, titled Image Enhancement for Endoscope; from U.S. Provisional application Ser. No. 63 / 538,485, filed Sep. 14, 2023, titled Endoscope; from U.S. Provisional application Ser. No. 63 / 534,855, filed Aug. 27, 2023, titled Endoscope; from U.S. Provisional application Ser. No. 63 / 531,239, filed Aug. 7, 2023, titled Endoscope; from U.S. Provisional application Ser. No. 63 / 437,115, filed Jan. 4, 2023, titled Endoscope with Identification and Configuration Information. The priority applications are incorporated by reference.

[0002] This application relates to endoscopes, laparoscopes, arthroscopes, colonoscopes, and similar surgical devices or appliances specially adapted or intended to be used for evaluating, examining, measuring, monitoring, studying, or testing living or dead human and animal bodies for medical purposes, or for use in operative surgery upon the body or in preparation for operative surgery, together with devices designed to assist in operative surgery.

[0003] An endoscope may be an arthroscope (for joint surgery), a laparoscope (for abdominal surgery), colonoscope (rectum, colon, and lower small intestine), cystoscope (bladder and urethra), encephaloscope (brain), hysteroscope (vagina, cervix, uterus, and fallopian tubes), sinuscope (ear, nose, throat), thoracoscope (chest outside the lungs), tracheoscope (trachea and bronchi), esophageoscope (esophagus and stomach), etc. An endoscope may have a rigid shaft or a flexible insertion tube.SUMMARY

[0004] In general, in a first aspect, the invention features a method. A computer reads a machine-readable serial number of a medical device as the medical device is put into use. Based on the reading of the serial number, a computer records a reduction of an on-hand inventory level of a class of medical devices of which the medical device is a member. A computer computes when the reduction of on-hand inventory of the class indicates a level that warrants reordering to replenish the on-hand inventory. When warranted, a computer places an order with a supplier of medical devices of the class to replenish on-hand inventory of medical devices of the class.

[0005] In general, in a second aspect, the invention features a computer system. The system has one or more processors and a machine readable, non-transitory memory having stored therein instructions. The instructions are programmed to cause the processor(s) to read a machine-readable serial number of a medical device as the medical device is put into use, and based on the reading of the serial number, to record a reduction of an on-hand inventory level of a class of medical devices of which the medical device is a member. The instructions are programmed to cause the processor(s) tocompute when the reduction of on-hand inventory of the class indicates a level that warrants reordering to replenish the on-hand inventory. The instructions are programmed to cause the processor(s) to place an order with a supplier of medical devices of the class to replenish on-hand inventory of medical devices of the class.

[0006] In general, in a third aspect, the invention features an endoscope. The endoscope has an image sensor designed to gather photons to produce video. The image sensor is of a type that has properties that vary from one sensor to another within manufacturing tolerances. The endoscope has a connector to connect the endoscope to a computer image processor. The image processor is programmed to obtain data from a database designed to store properties of the endoscope’s image sensor and / or the image sensor’s behavior, the database storing properties specific to specific sensors or a specific class of sensors. The obtained data describes the image sensor’s properties from the database. The image processor is programmed to compute normalized video based on the obtained properties and the video from the image sensor.

[0007] In general, in a fourth aspect, the invention features a method. A computer processor programmed to receive image date from an image sensor of an endoscope, obtains data from a database designed to store properties of endoscopes’ image sensors and / or the image sensors’ behavior. The image sensor is of a type that has properties that vary from one sensor to another, the database storing properties specific to specific sensors or a specific class of sensors. A computer computes normalized video based on the obtained properties and the video from the image sensor.

[0008] In general, in a fifth aspect, the invention features a method. A database stores information relating to specific individual medical devices, including physical location and / or ownership. An optical reader reads 2D identification codes on packaging of the medical devices. Database records are updated show current physical location and / or ownership based on the read 2D identification codes.

[0009] In general, in a sixth aspect, the invention features a computer system. The system has one or more processors, and a machine readable, non-transitory memory having stored therein instructions. The instructions are programmed to cause the processor(s) to store information relating to specific individual medical devices, including physical location and / or ownership. An optical reader reads 2D identification codes on packaging of the medical devices. Database records are updated to show current physical location and / or ownership based on the read 2D identification codes.

[0010] Embodiments may include one or more of the following features. A computer may read a machine -readable serial number of a endoscope as the endoscope is put into use. Based on the reading of the serial number, an on-hand inventory level of a class of endoscopes may be reduced and recorded. A reorder level may be computed based on the reduction in on-hand inventory level. A computer may compute when on-hand inventory level warrants reordering to replenish the on-hand inventory. Based on the computed warrant to reorder, a computer may place an order with a supplierof endoscopes of the class to replenish on-hand inventory of endoscopes of the class. The medical device may be an endoscope. The endoscope may have an image sensor. The image sensor may be designed to gather photons to produce video. The image sensor may be of a type that has properties that vary from one sensor to another within manufacturing tolerances. The programs may be programmed to connect a one of the medical devices or endoscopes to a computer processor. The processor may be programmed to obtain properties of the connected medical device from a database. The database may be designed to store information relating to specific individual medical devices, endoscopes, or image sensors. The database may be accessed based on the read serial number. The database may store properties of the endoscope’s image sensor. The database may store properties of the image sensor’s behavior in a database. The database may store properties specific to specific sensors. The database may provide properties of the one endoscope’s image sensor from the database, and to compute normalized video based on the obtained properties and the video from the image sensor. The database may store physical location of individual medical devices, endoscopes, or components. The database may store physical location and / or ownership of individual medical devices, endoscopes, or components. An optical reader may read identification codes on packaging of the medical devices. Database records of physical location and / or ownership may be updated to show current physical location and / or ownership based on the read identification codes. When a medical device or endoscope may be connected to its control computer, the computer may be programmed to obtain properties of the connected device from a database. The identification code may be a machine- readable 2D optical code. The identification code may be stored in machine -readable electronic memory. The database stores information relating to a device model of a class of medical devices manufactured to be interchangeable. The database may be designed to index to the device model information based on the read identification code of an individual medical device. The database may be designed to store data describing a color spectrum capability of the image sensor. The database may be designed to store data describing an image plane resolution of the image sensor. The database may be designed to store calibration properties of the endoscope’s image sensor specific to the specific image sensor of the specific endoscope. The image processor may be programmed to compute normalized video based on the obtained calibration properties and the video from the image sensor. The calibration properties may describe a white balance of the image sensor, or a color correction gamma curve of the image sensor, or a distortion correction of the image sensor, or any two or more, or any three or more of these properties. The computer may compute a prediction of delivery date based on the read 2D identification codes. The computer may update the physical location and / or ownership database records to show current physical location based on the read 2D identification codes or read of the machine-readable memory.

[0011] The above advantages and features are of representative embodiments only, and are presented only to assist in understanding the invention. It should be understood that they are not to beconsidered limitations on the invention as defined by the claims. Additional features and advantages of embodiments of the invention will become apparent in the following description, from the drawings, and from the claims.DESCRIPTION OF THE DRAWINGS

[0012] FIGS. 1A, 2, 3, 4, are perspective or perspective cutaway views of endoscopes and / or endoscope related apparatus.

[0013] FIGS. IB, 5A to 5D, 5F, 6E, and 6H are block diagrams of computers or processors.

[0014] FIG. 5E is a time sequence of video frames.

[0015] FIGS. 6A-6D, 6F, and 6G are screen shots.DESCRIPTION

[0016] The Description is organized as follows.I. OverviewI.A. Endoscopic surgeryI.B. Overall architectureI.C. Integrated sterile packagingI.D. Single-use handpieceII. An endoscope that is partially-reusable, partially disposable / replaceable, and a coupling joint betweenIII. Extendable, bendable, or articulated camera tipIV. Additional features of an endoscopeV. Endoscope tipV.A. Molding and assembly of components of the endoscope tipV.B. Fiber optic illumination for a single -use use scope tipV.C. A tip design with light guidesV.D. Diffusion terminal surfaceVI. Antifouling and antifoggingVI.A. HeatingVLB. For endoscope’s delivery packaging, vial of fluid to protective a coating on the endoscope’s lens / windowVII. Lens cap with optical correction for use during insertion of offset-view endoscopeVIII. Avoiding fasteners, springs, and other small componentsVIII.A. A button with embedded springsVIII.B. Overmolding of case over circuit boardVIII.C. Avoiding internal fastenersVIII.D. Rotational resistance via O-ringsVIII.E. Ultrasonic welding of the two halves of the outer handle shellVIII.F. Thermoplastic elastomer coating handleIX. Liquid flowIX.A. Liquid-tight sealIX.B. Inducing spiral flowX. Molding and JoiningX.A. Diagonal slots to connect dissimilar materialsX.B. Joining component parts of obturatorX.C. Twist-locking the parts togetherXI. Endoscope tipXI.A. Molding and assembly of components of the endoscope tipXII. Image processing unitXII.A. Image processingXII.B. HDR exposure fusion to preserve frame rateXII.C. Auto ExposureXII.D. Video processing for superresolutionXII.E. Diagnosis and lesion detectionXII.F. Scope controlXII.G. Flexboard and electronics in the endoscope handleXII.H. CableXII.I. Wireless communication in place of cableXII. J. IsolationXII.K. Other peripheralsXII.K.l. MonitorXII.K.2. USB portXII.K.3. Connections to cloud storageXII.K.4. USB connection for keyboard and mouseXII.K.5. MicrophonesXII.K.6. Insufflation tubingXIII. Machine -readable serial numberXIII.A. Machine -readable serial numberXIII.B. Use of machine -readable serial number to reduce errors, ensure non-expiration and ensure sterile single -useXIII.C. Use of machine -readable serial number to communicate patient data into electronic medical recordXIII.D. Use of machine -readable serial number for inventory control, location tracking, reordering, and stock managementXIII.E. Configuration / capability informationXIII.F. Calibration informationXIII.G. ImplementationXIV. EmbodimentsI. OverviewI.A. Endoscopic surgery

[0017] Referring to FIG. 1A, endoscope 100 (such as an arthroscope, laparoscope, or other), trocar 102, and obturator 104 may be used for joint surgery, joint access, or other minimally-invasive surgery

[0018] Various endoscope tip designs may have the following properties. The overall tip may be small enough to meet the dimensions of the endoscope, typically the dimensions in the table of paragraph

[0026] below. In some cases, the tip may be slightly larger or smaller in diameter than shaft The tip may hold camera 410, illumination, fluid injection or evacuation ports, procedural tools, etc. mechanically stable, within that diameter. The tip may seal against elevated pressures that are typically used to distract tissues out of the view of the scope, to prevent intrusion of bodily tissues and fluids and insufflation fluid. The tip may deliver or allow delivery of illumination light, either via an LED 418 mounted in the tip, or using fiber optics 430 to convey light from the handle or a controller. Opaque parts of the tip assembly may exclude stray light, from non-desirable lights paths within the tip from the illumination fibers / LEDs / light guides, and reflected light from the surgical cavity. The tip may be manufacturable at desired quantities and cost. The tip may have a configuration that is atraumatic to surrounding tissue, for instance, without sharp points or edges. The scope may be formed of biocompatible materials, such as stainless steel and / or certain plastics. In some cases, the tip may have a piercing point. The tip may be designed to resist fogging or fouling. The tip may permit cleaning, preferably while in situ the surgical site.LB. Overall architecture

[0019] Referring to FIG. IB, endoscope 100 may be part of an overall system designed to deliver high-definition video for use in endoscopic surgeries. The system may provide live high- definition video to be displayed on a video monitor, and to be captured as stored video and still images; illumination of the surgical cavity, irrigation and / or inflation (insufflation) of the surgical site, and image refinement such as zoom, rotation, removal or reduction of hotspots and other artifacts, etc.

[0020] The system may include an endoscope, including insufflation tubing, a communications / control / power / illumination cable, a cannula, and an obturator. An image processing unit (IPU) or master controller may be reusable over multiple procedures. If illumination is provided via fiber optics, there may in addition be a light box, typically near the IPU so that the fiber optics fibers are aligned with the other necessary cords and hoses. One or more of the endoscope, tubing, cable, cannula, and obturator may be designed for disposable single use, and sold together as an integrated kit.

[0021] Referring to FIG. 5 A and 5B, the endoscope may have electronics in the handle that controls the camera and illumination (LED or fiber optics). The IPU may have a computer processor for various image processing functions, and controllers for the electromechanical devices in the endoscope, Wi-Fi or similar radio communication, USB and cloud storage, and the like. Because the scope is single -use, sterility is easily provided. The connecting cable may be single-use as well, so that it can be delivered in the sterile packaging. The IPU is higher cost, and cannot easily be sterilized, so it is outside the sterile field.

[0022] Referring to FIG. 5F, various isolation couplers may provide electrical isolation between the wall-voltage components for the IPU and the patient.I.C. Integrated sterile packaging

[0023] The endoscope, tubing, and cable may be designed for disposable single use, and packaged and sold together as an integrated kit. Additionally, one or more of the obturator and cannula may be packaged and sold together with the kit. The kit may be sold in sterile packaging. The packaging may be designed to be opened at the surgery, within the sterile field surrounding the patient. The cover on the packaging may be made of Tyvek® or some similar film that is transparent to ethylene oxide or a similar sterilant, so that the packaging and components may be sterilized together at the time of manufacture. The film covering stays in place until shortly before the surgery. This eliminates the need to disinfect or sterilize the scope immediately before surgery. The tray holding the components may be transparent, so that the contents of the tray are visible before the Tyvek cover is opened.

[0024] Because the components are sold together, they can be calibrated to each other. Various properties of the illumination, image sensor, lens, filter, and the like can be calibrated to each other as a set at the manufacturing plant. White balance may be one of the parameters calibrated atthe factory — because the components are single use and sold as an integrated package, they can be inter-calibrated at the factory, and that co-calibration follows them for the life of the product. In contrast, for conventional endoscopes, the light source and the endoscope are independent and the color temperature or balance of the illumination source varies from light source to light source, and the color sensitivity of the pixels of the image sensor vary scope-to-scope, so white balance must be performed by the user as part of the prep for each procedure. In a configuration where the scope is sold as a disposable single-use configuration, with an electronic serial number that ties back to calibration factors measured at the factory (see § IV. A and H

[0088] to

[0095] , below), the scope may be calibrated by imaging a white surface, which provides a test surface with equal parts red, green, and blue pigment, with illumination that results in mid-level, non-saturated pixel values from the image sensor and an matrix of correction coefficients may be computed adjust color balance of the pixels of the image sensor’s signal.I.D. Single-use handpiece

[0025] The endoscope itself may be designed for disposable single use. The image sensor, a lens , a filter, and cover window, and illumination emitter (either an LED 418 or the distal end of fiber optic lighting fibers or wave guides) may be located at the distal end of an insertion shaft. The sensor, lens, filter, cover window, and illumination emitter may be designed to interoperate with each other to allow insertion in a small diameter insertion shaft. Single use ensures sterility, even of components with complex geometric forms and materials that cannot be autoclaved (like the electronics of endoscopes). The endoscope may have electronic tracking to ensure single use (see § IV.B and H

[0096] to

[0107] , below).

[0026] Typical dimensions for various surgical specialties may be as follows (measured in millimeters):II. Additional features of an endoscope

[0027] Referring to FIG. 2, disposable shaft portion 110, 120 may in turn be separable into an outer cannula 132 for protection and strength, and an inner shaft portion 134 carrying variousillumination, optical, and fluid-carrying componentry. Illumination may be provided by LED 418 at or near the distal tip, or via fiber optics 430 from an illumination source in the handle, or illumination at an external controller.

[0028] The endoscope may have a handle 112, 114, 120, and a shaft 110 for insertion into a body. At or near distal tip 116 of the shaft 110 may be a lens, electronic image sensor, filter, or other optical component 410. The camera’s orientation may be fixed in the scope, or may be pannable. Camera 410 may be at tip 116, looking out from the shaft, or may be recessed a short distance behind the structural tip of the shaft. Also at or near the tip may be an illumination source, such as LED 418. Tip 116 may have a rigid pointed tocar tip, or may have a spoon-shaped portion that reaches past the distal surface of the window in tip 116, or may be flexible (in the manner of the tip of a colonoscope), in each case extending a little beyond the distal surface of the window in tip 116 to provide physical protection to the tip 410 during insertion or to protect the camera 410 from a surgical cutting device.

[0029] Illumination may be in visible light, infrared, and / or ultraviolet. In some cases, an illumination LED (light emitting diode) or other illumination source may be placed in reusable handle 112, 114 or in a docking station / controller, and the disposable shaft may have fiber optics 430 to transmit light to the tip, and joint 130 may have an optical coupler. In other cases, illumination LED 418 may be placed in tip 116 to illuminate the surgical cavity directly; in such cases, joint 130 may have a power connector. In some cases, LED 418 may be recessed from the tip, or placed somewhere in the shaft, or may be in an external controller, and optical fiber 430 may carry illumination light to the tip. Optical fiber 430 may be configured, for example, with a split, so that light will be arrayed in a desired pattern around the image sensor to better distribute the light into the surgical cavity around the camera.

[0030] Shaft 110 itself may be rigid, made of a nonbioreactive metal such as stainless steel or coated aluminum. In some cases, a surgical cavity around endoscope tip 400 may be insufflated by gas (typically carbon dioxide), or irrigated by saline solution. In either case, fluid inflow and outflow may be effected by channels through the shaft.

[0031] Shaft 110 may also carry power wires to illumination LED 418 and camera 410, and carry signal wires that carry a video signal back from camera 410 to electronics in the reusable portion 112, 114 of the handle. Electrical power to camera 410 may be supplied over conductors in a flexible cable or on a printed circuit board (flexible or rigid), and may be insulated with a conformal and insulating coating such as parylene. This same flexible circuit board 416 may have signal conductors for the video signal from image sensor 410. The video signal may be transmitted from image sensor 410 to the handle using any video signal protocol, for example, MIPI-CSI2 (Mobile Industry Processor Interface - Camera Serial Interface2) or HDMI. In some cases, a parylene coating may improve biocompatibility.

[0032] Shaft 110 may also carry cables or other mechanical elements to control panning of camera 410.

[0033] Referring again to FIG. 2, the handle may have a rotation collar. The rotation collar may have various features that make rotation easy. For example, depressions 302 may provide a good grip for fingers for light roll torque. A fin may provide greater leverage for greater roll torque, and may also provide a fixed rotational point of reference.

[0034] Proximal handle 114 may include rotational sensors so that an angular orientation of camera 410 may be ascertained. For example, the inner surface of proximal handle 114 may mount one or more magnets 320, and printed circuit board 322 (which rotates with rotation collar 112 and disposable cap 120) may have Hall effect sensors 324 that detect the magnets. This may be used to compute a rotational orientation, which may in turn be used to “right” the image from camera 410 on a video display screen.

[0035] The distal tip of the shaft, camera 410 mounted therein, and the mounting of componentry within shaft 110 may be designed to be robust. Occasionally, during surgery, the tip of the endoscope may come into contact with a shaver, ablation probe, or cauterization probe, and it may be desirable to have the tip be robust to such contacts. To reduce risk that componentry may be dislodged and left in the patient, the disposable shaft and its componentry may be designed to avoid joints that are at high risk of mechanical failure. A disposable optical system may prevent the image degradation that occurs when nondisposable optics are reused in multiple surgical procedures.

[0036] Endoscopes as a genus include arthroscopes, laparoscopes, colonoscopes, and other specialized scopes for various body cavities. For an arthroscope for joint surgery, the shaft may be as small as 6mm, 5mm, 4.5mm, 4mm, 3.6mm, 3.3mm, 3mm, 2.8mm, 2.6mm, 2.4mm, 2.2mm, 2mm, or 1.8mm, and highly rigid. For other endoscopes, such as a colonoscope, the diameter may be larger, and the shaft may be flexible.

[0037] The endoscope may be delivered as a handle and multiple tips, each tip individually sealed for sterility.

[0038] Referring to FIG. 3, hoses 160, 162 for irrigation / insufflation fluid / gas in, irrigation / insufflation fluid / gas out, and electrical connection cord 164 may be permanently affixed 340, 342 to disposable cap 120. This arrangement may allow that hose 162 that carries water out of the surgical cavity, and which is therefore contaminated, may be disposable, and no fluid will come into contact with the reusable part 114 of the handle. Hoses and cord 160, 162 may be routed through channel 354 running the length of reusable handle 112, 114. Channel 344 may be of inner diameter large enough to permit easy passage of hoses and cord 160, 162, 164, and connectors 350, 352, and have a continuous smooth wall that permits easy sterilization, to permit ready replacement of the replaceable components. Channel 354 may be off the central axis, to allow printed circuit board 322 to lie on the central axis. Connectors 350, 352 at the end of hoses and cords 160, 162 may be smallenough to pass through channel 354. Thus, replacement of shaft 110, cap 120, hoses and cords 160, 162 may be effected by threading connectors 350, 352 and hoses and cord 160, 162 through channel 344. Electrical cord 164 may have a connector 354 at or near joint 130, and hose(s) 160 for irrigation / insufflation fluid / gas flowing into the surgical cavity may likewise have a connector at joint 130 to allow this hose(s) to be reusable, or may be permanently affixed 340 to reduce possibility of leaking. Having hoses and cable 160, 162 roughly on-axis reduces undesirable cable flop as the scope is in use, and reduces undesirable torque on cap 120. Forming shaft 120, cap 120, and hoses 160, 162 as an integral unit for replacement reduces possibility of leaking, and improves sterility of the replacement operation.ILA. A tip design with light guides

[0039] Light fibers 430 may be extruded in shapes that improve light delivery, such as rectangular, or a U-shaped light guide 450 that would extend the length of the endoscope from the illumination source to the U-shaped emission surface at the distal end of the endoscope. In some cases, individual fibers 430 may be replaced, for at least some portion of their length, by a shaped light guide 450, such as a circular or U-shaped ring of clear light guide around the periphery of the tip chassis 438, 480. Light guide 450 may be a two-component structure, with two different indices of refraction for internal reflection analogous to optical fiber. In other cases, light guide 450 may be formed of a clear light transmission medium coated by a reflective coating, such as aluminum, gold, or silver. In some cases, the shaped light guide 450 may extend only a short distance, for example, the length of the inner tip part, and traditional circular fibers may be used to bring light from the illumination source to the proximal end of light guide 450 at the tip chassis 438, 480.III. Image processing unit

[0040] Referring to FIGS. 5 A and 5B, the image processing unit (IPU) may use an interface board to drive and receive signals from the scope via the cable and a custom or off-the-shelf motherboard. In some cases, the motherboard may be an off-the-shelf motherboard with an Intel CPU and an Nvidia GPU. The motherboard provides most of the external interface ports. The patient may be isolated from the line voltage (110 or 120V 60Hz in the U.S., 240V 50Hz in Europe) by a medical grade AC / DC power supply and a separate interface board, called the patient interface board The patient interface board processes signals to convert them between signal forms used internally to the IPU and the signal forms that travel to and from the scope.III.A. Image processing

[0041] An image processing computer may perform image processing. GPUs provide a well-documented API that can be exploited for acceleration of graphical processing, and the software running on the motherboard may in turn have internal APIs that permit combining software processing components for image enhancement. A series of video chips in the scope handle and theIPU (Image Processing Unit) box may convert the very small, high speed video signals from the sensor (such as a Bayer formatted MIPI-CSI2 interface) to a signal suited for transmission distances longer than a few centimeters, and to a protocol more easily processed by various stages of an imaging pipeline and for storage (such as YCbCr422 or MPEG). The IPU processor may receive data from the scope (which may be video data, still images, telemetric data, etc.) via the handle board, cable, and patient interface board. The IPU may capture still images out of the video, and / or process the video through image correction and enhancement software to deliver an high-quality image on the monitor or for storage on some storage medium or in the patient record.

[0042] Various video signal processing chips, an image signal processor (ISP) and a graphics processing unit (GPU) may perform a number of video transformations on the video data received from the scope before the data are displayed on the monitor or saved to an output device. The IPU box may have multiple processors, including a specialized image signal processor (ISP), a general purpose CPU such as an Intel Pentium, a graphics accelerator (GPU), a field programmable gate array (FPGA), a custom accelerator hardware, and perhaps others. Video transformations may be performed in one or another of these processors, or in software, or some combination of hardware and software. The sum total of processing power may be chosen to ensure that the image processing may be performed within requirements for image latency. The following transforms may be performed:• Receive raw image data from the image sensor of the endoscope in a Bayer Formatted MIPI- CSI2 stream and re -encode to a YCbCr422 or h.264 MPEG stream to improve processability• Translate a MIPI-CSI2 video stream into a UVC complaint USB 3.0 video stream via a video stream processor such as the Cypress CX3.• HDR or WDR processing (High Dynamic range or Wide Dynamic range)-software (a) to expand the dynamic range of the captured image by avoiding over or under-exposed areas of the video. This is achieved by combining sequential over and under exposed frames of images from the image sensor and reducing the displayed intensity of exceptionally bright pixels to reduce hot-spotting, and increasing the displayed intensity of exceptionally dim pixels in a frame to improve visibility images. See FIG. 5E. HDR / WDR processing may use the Mertens exposure fusion algorithm.• Rotate and image righting based on the handle’s rotation sensor. Including the display of and rotation of a position indicator, which may be displayed as an arrow, around the perimeter of the circular mask on the user interface.• Correction of distortion (either systemic because of fish-eye distortion or similar distortion in the lens specification, or specific distortions measured in specific scopes to be corrected in the IPU by a reverse transform), removal of artifacts.• Crop the rectangular image received from the scope to a rectangle that can be rotated around a central point in the display. A circular mask is applied over this rectangular crop to provide a circular image display to the user. This may replicate the view surgeons are used to fromdecades of rod lens scopes. Also, outside the field of view cone provided by the lens, the outer edges of the image may be so distorted or obscured by the edges of the lens housing, that it communicates more distraction than information.• Auto-exposure to target a desired average image brightness by adjusting exposure times and gains in the image capture pipeline.• De-mosaic• Black Level Correction• Gain Adjustment• Shading Correction• Defect Correction• Noise Reduction• Tone Mapping• Color correction and white balance correction• Zoom in / zoom out within the target image• Lens resolution correction• Local Contrast Enhancement• Edge Enhancement• Image enlargement (enlarge the circle displayed on the monitor, perhaps losing the upper and lower limb of the circular display)• Reformatting and compressing the video data for storage on a storage device, and decompressing stored video data for display.• Controlling transmission over network connections for storage in the cloud or on storage devices local to the IPU, or other non-cloud storage• Super-Resolution is discussed below in § III.D at H

[0051] to

[0060] — this upsamples from a lower resolution (for example 1280x720) resolution to 2160x2160 (“4K”) resolution• Frame Writer is the last stage, putting the video into the video system’s frame buffer for display or to storage. The fully-processed video stream may be displayed on a video monitor, or may be sent to a storage device or network interface.

[0043] Dividing the pipeline into phases allows parallelism. For example, each phase may be assigned to one core of a multi-core CPU or different functional units of a GPU.III.B. HDR exposure fusion to preserve frame rate

[0044] Referring to FIG. 5E, HDR exposure fusion may be performed on pairs of frames taken simultaneously by two different cameras, and then the images are merged pairwise. Exposure fusion algorithms include Mertens-Kautz-Van Reeth, or Hugin / Enfuse.

[0045] In other cases, a single image sensor may be programmed to overexpose frame n, then underexpose frame n+1, then overexpose frame n+2, etc. This may be controlled by strobing illumination LED 418 at the frame rate, or by controlling the exposure time of the image sensor. The short exposure time frames may bring out detail in overexposed parts (“hot spots”) if the image, and the overexposed frames may bring out detail in underexposed parts of the image (“dark areas”). By merging the frames, both hot spots and dark areas are captured in the output image, increasing the dynamic range that can be captured.

[0046] The frames may then be merged pairwise using HDR exposure fusion algorithms of the same class, except applied to overlapping pairs of frames, to merge frame n with frame n+1, then merge frame n+1 with frame n+2, then merge frame n+2 merged with frame n+3, etc. This maintains the output frame rate at the input frame rate.III.C. Auto Exposure

[0047] An auto exposure algorithm may be used to adjust for fluctuations in light intensity level of a scene the image sensor is capturing to a target brightness level. If the camera is moved close to an object with static gain, exposure, and illumination intensity, the overall scene becomes brighter and therefore the exposure times, gain, and / or illumination intensity per frame should be reduced to capture less light. Conversely, if the camera moves farther away from an object, the overall scene becomes darker and exposure times, gain, and / or illumination intensity should be increased to capture more light.

[0048] An auto exposure implementation may control both the exposure time and gain to achieve a target intensity setpoint. The gain control may be either analog gain in the cell of the pixel of the image sensor, or digital gain applied in the image sensor or digital image processing pipeline. The brightness setpoint may be set via a user “brightness” control, or may be set automatically. The auto exposure algorithm may perform the following steps:1. Divide the frame into zzx / z- ixel blocks.2. Compute the average intensity for each block.3. Compare the computed intensity for each block to an intensity setpoint (which may be set for each block, or for the image as a whole) to get an error value for each block. Each block may be assigned a weight to scale its computed error value. This weight allows for certain blocks to be more important than others (i.e., blocks in the middle of the grid weighted higher than those further out).4. Sum all weighted block errors for an overall error value.5. Evaluate the change: a. If the overall error value is below the defined change threshold, no changes are made. b. If the overall error value is above the defined change threshold, scale the change for one update cycle, update change threshold relative to a size of the overall error by the equation below.Max Change Threshold = Max Change Threshold + (Overall Error x Multiplier) where Multiplier is less than 1 to allow a damped response. c. The max threshold is set to minimize the perception of a discrete light level change by the user in similar use environments, but allow fast updates when quickly changing from dark to light or light to dark environments. The multiplier is used to tune this response to achieve the fastest response time to large changes in environmental conditions while preventing oscillations in the light levels perceived by the user.6. Input overall error into either the exposure or gain PID control: a. If the scene is too bright:(i) If the gain is at its minimum, the exposure PID control runs(ii) Otherwise, the gain PID control runs b. If the scene is too dark:(i) If the exposure is maxed out, the gain PID control runs.(ii) Otherwise, the exposure PID control runs c. Depending on implementation, any two or more parameters may be substituted for gain and exposure, including illumination intensity, exposure time, etc.7. Write resulting exposure and gain to ISP.

[0049] The auto exposure algorithm may be downstream from the WDR algorithm, perhaps the immediately following stage. This reduces sensitivity of the auto exposure algorithm to frame to frame changes in exposure time used by the WDR algorithm. The auto exposure algorithm may run every several frames (rather than every frame) to reduce processing bandwidth. The per-block intensity computation may be parallelized to run on the GPU.

[0050] Software may provide that many of the parameters of this algorithm may be tunable via a config file loaded as part of system startup, including the number of frames allowed to run between recalculation of the autoexposure parameters, the block size for step 1, the mean intensity setpoint of step 3, a map of block weights for step 3, the PID coefficients for the PID calculation of Step 5.III.D. Video processing for superresolution

[0051] Referring to FIGS. 5C and 5D, the input to the Super Resolution block may be low resolution video (for example, 720x720 pixel (“720p”) or 1280x720 image, and the output may be an enhanced quality 2160x2160 pixel (“4K”) image. The “Super Resolution” box may in turn have a block diagram as shown in FIG. 5D. A machine learning model may be used to combine noise reduction, lens resolution correction, edge enhancement, local contrast enhancement, and upscaling as an integrated module. When these functions are performed singly, each is subject to various tradeoffs, and image enhancements by one stage may interfere with and degrade enhancements from another stage. For example, many noise reduction algorithms tend to result in blurred images. Traditional edge sharpening tends to amplify noise. By combining all these functions in a single machine learning model, those tradeoffs may be reduced.

[0052] Various types of machine learning models can be used with the systems disclosed with respect to FIGS. 5C and 5D, including fully convolutional neural networks, generative adversarial networks, recurrent generative adversarial networks, or deep convolutional networks. Convolutional neural networks (CNNs) are particularly useful for image processing. The Super Resolution CNN block may be formed by combining:• A CNN upscaling module from NexOptic Technology Corp, of Vancouver, B.C. This may allow a processor to infer inter-pixel interpolations based on local information, and previous and next frame information, to improve apparent resolution.• A noise reduction module from NexOptic. This may reduce noise from the image sensor, electronics, and stray light photons• A lens resolution correction module from NexOptic. This step may enhance the performance of the lens by understanding the transfer function of a fixed image through the lens.• A local contrast enhancement module from NexOptic. This may assist the surgeon by increasing contrast between light and dark, various shades of red, etc.• Dynamic range compensation — portions of the image that are washed out because of overexposure may be balanced out against parts of the image that are washed out because of darkness. The total dynamic range may be adjusted to improve contrast and to draw out detail that is lost in the over- or under-exposed portions (see FIG. 5E).• An edge enhancement module from NexOptic. This may reduces loss of resolution (blurring) that may have been introduced by the lens system (e.g., due to limitations of lens size or complexity) or by motion of the camera of objects in the scene, and may improve edge extraction to assist a surgeon by making structures more apparent at the surgical site.• High entropy random noise interferes with data compression. The CNN may be trained to recognize and remove random pixel noise, which may improve data compression.

[0053] By combining all these functions into a single CNN, local contrast, edge enhancement, and noise reduction may all be simultaneously improved. Much like human neuralnetworks skillfully optimize for multiple parameters simultaneously, a computer CNN may be trained to simultaneously optimize for several characteristics. Hardware contrast and edge enhancement may be disabled. In some cases, the degradation and training may involve at least two of the parameters in above list, for example, resolution and edge enhancement, or resolution and local contrast. In some cases, any three of these types of image degradation may be trained into the model, for example, resolution, local contrast, and edge enhancement, or resolution, image sensor noise, and lens correction. In some cases, the model may be trained on any four of these parameters. In some cases, it may be trained for all five.

[0054] In one example implementation, given an input sequence of low resolution framesa sequence of high resolution frames Z, corresponding to the corresponding to the low resolution frames. The super resolution frames may be computed1: For all except the reference frame2: Compute warping F from3: Compute4: EnwhereT is the radius of the temporal neighborhoodF, is the warping operator for frame i to the current frameSi is the decimation for frame i

[0055] The video super resolution model may execute in two steps: a motion estimation and compensation procedure followed by an upsampling process. Alternatively, instead of explicitly computing and compensating for motion between input frames, the motion information may be implicitly utilized to generate dynamic upsampling filters, and the super resolution frames may be directly constructed by local filtering to a frame being constructed in the center of a computation window. The machine learning model may be trained by capturing reference video at normal resolution, and then degrading the reference video via transforms that simulate loss of resolution, introduction of noise, lens aberration and similar lens noise, degrading contrast, and / or degrading edges. The machine learning model may be trained to recover the full resolution original reference video. That same training may be sufficient to allow video captured at normal resolution to be upsampled to higher resolution. A lens model may be created from a combination of design data andimages captured of standard test patterns (for instance a checkerboard or array of Cartesian lines) to detect and measure lens imperfections for a lens design or specific to each scope, and create a generalized transform or store registration correction for a specific scope. In some cases, high quality reference data may be displayed on a physical display, and viewed via an endoscope camera. The machine learning model may be trained to recreate the reference data from the camera video. The training may exploit the fl loss with total variation (TV) regularization to reduce visual artifacts

[0056] The lens correction model may address the imperfections in a lens system that remain after balancing for all constraints, for example, by creating a lens model and passing a large set of ultra-high resolution images captured with a camera with a very high quality lens (to establish a baseline “perfect” image) through the lens model, then training the CNN to correct that the image set passed through the lens model to transform each image into the “perfect” image.

[0057] The Super Resolution CNN may yield better overall image quality (compared to the raw data directly out of the camera, and compared to using all classical blocks independently). Combining classical enhancement algorithms with the enhancement CNN may provide opportunities to tune parameters of the classical algorithms in parallel based on the CNN training where classical algorithms require tuning parameters in series. The Super Resolution CNN may allow tunable runtime performance via architecture choice allowing for tradeoffs between overall image quality and speed.

[0058] In some cases, the CNN may retrain itself on the fly. For example, at moments when the camera and image are stationary relative to each other, alternating frames may be taken at deliberately underexposed (too dark) illumination and normal illumination. The CNN may be retrained to recognize hot spots where detail is lost because of overexposure, and where detail is lost in the dark regions of the underexposed frame. In some cases, several machine learning systems may be chained together, for example, one to enhance dynamic range, one to reduce blur and for edge sharpening, one to recognize frame-to-frame motion, one to improve contrast, and one to upsample for super resolution.

[0059] In some cases, a bypass feature may disable the Super Resolution neural network, and instead upsample the image to 2160x2160 resolution via conventional means such as bicubic interpolation.

[0060] The NexOptic components may be obtained under the product name Super Resolution, as described in U.S. Pat. No. 11,076,103, Gordon, Photographic Underexposure Correction Using a Neural Network, and U.S. Publication No. 2021 / 0337098 Al, Gordon, Neural Network Supported Camera Image or Video Processing Pipelines, both incorporated by reference.III.E. Diagnosis and lesion detection

[0061] In some cases, the image processing pipeline of FIG. 5B may include processing to detect various lesions. For example, during colonoscopy, the image processing pipeline may have aprocessor to detect polyps. During esophageoscopy, the image processing pipeline may have a processor to detect Barrett’s esophagus.III.F. Scope control

[0062] The scope may have several controls, including a pushbutton on the scope, a touch screen on the face of the IPU, and a graphical user interface with a touchscreen that may be accessed over the internet from an external computer.

[0063] One pushbutton on the scope may control three things: (a) still frame capture, (b) video record on / off, (c) LED adjustment, high beam / low beam. For example, one press may capture the current view as a still frame. A doublepress may start or stop the video recording. A triplepress or a press for three seconds may adjust the LED brightness.

[0064] The IPU may have front panel controls for the scope, including image adjustment, color, brightness, zoom, and the like. In either a user-visible or a system set-up / testing mode, controls on the front panel of the IPU or accessible via a computer over the internet may control:• LED illumination — because the on-scope button is only a single momentary connection switch, it cannot provide fine control, only gross on / off control. Another user interface may provide finer lighting control• Sensor control — adjust tone or color balance, zoom, etc.• Control image and video storage in the IPU’s nonvolatile memory — which portion of which video to store, etc.

[0065] Adjustment of LED brightness requires careful integration with the image sensor. If brightness is controlled by conventional pulse width modulation (PWM) that is not synchronized with the frame sync of the image sensor, banding can occur in the image. Alternatively, a constant current source or voltage controlled current source may be used to adjust the LED brightness and avoid banding.III.G. Flexboard and electronics in the endoscope handle

[0066] Flex circuit board 416 may carry signal and power from the handle to the components at the tip. At the tip, molded plastic parts (brace or chassis 412, 414, 438) may hold all the component parts in proper orientation. The components (image sensor, lens, filter, window, and mounting) may be selected to assure a desired offset angle (typically 0° on-axis, 30°, 45°, or 60°) and a desired field of view (typically 50°, 60°, 70°, 80°, 90°, 100°, 130°, or 180°).

[0067] The distance from the image sensor at the tip to the receiver on the circuit board in the handle may be about 115mm to 330mm, relatively long for a MIPI-CSI2 video connection. The flex circuit board may have circuit layout and shielding chosen to create an impedance matched signal path for the video data from the video sensor, with low radiated emissions, low loss, and low sensitivity to external interference. A connection from the inner insertion shaft to the handle circuitboard’ s isolated reference potential may protect against interference from RF ablation or coagulation devices by allowing the video signals from the image sensor to float relative to the RF application energy, minimizing the interference induced on the signal conductors transporting the MIPI-CSI2 signaling from the image sensor to the handle board..

[0068] A rigid circuit board in the handle (HB PCBA — “handle board printed circuit board assembly”) may have a microprocessor, magnetic sensors, and a transmitter chip. The transmitter chip may receive the low-power, high-bandwidth, high speed signals, which may be transported using a MIPI-CSI2 stream, from the image sensor received over the flexboard, and convert the video signals into serialized signals suitable for transmission over a 3-meter cable to the IPU. Because 3 meters is a relatively long distance, the cable may be carefully impedance matched with low insertion loss to ensure signal integrity. The serialized signals are received on the IPU, converted back into a MIPI- CSI2 interface, and passed to the image signal processor (ISP) for processing.III.H. Cable

[0069] The IPU may be connected to the scope via a custom cable. The cable may be about 3 meters (10 feet) long — long enough to give the surgeon freedom of movement, and to keep the nonsterile IPU acceptably distant from the patient. The connector may be customized to ensure that the scope cannot be connected to other devices that would not supply the necessary patient isolation.

[0070] The cable may use a USB Type A or C connector, because the connector has good shielding and physical insertion characteristics, even though in this application, the cable does not carry USB signals or utilize the USB protocols. The cable may have a protective hood that extends several millimeters beyond the end of the USB connector (alternatively, the USB connector may be recessed below the end of the hood). The hood may provide insulation around the connector when the cable is disconnected from the IPU, which provides the creepage and clearance distances required for electrical isolation of the patient, for example, if the end of the cable should happen to touch something electrically live or earthed. The hood may be keyed so it will only connect to the correct port on IPU, and won’t (easily) plug to a generic USB connector, and ensures right-way-only connection of the cable to the connector on the IPU. The cable end and plug on the IPU box may be color coded to each other.

[0071] The cable may supply power to the scope, communicate command signals to the scope, obtain configuration information that was stored in the scope’s on-board memory, and carry video signals back from the scope back to the IPU. The cable may also support a scheme for detecting that a scope is connected to the IPU. This is achieved by sensing a voltage change on a pin of the scope cable, which is pulled to a logic -high voltage when the cable is disconnected and forced to a logic -low when the cable is connected. A pin on the cable may be connected to a pull-up resistor on the IPU side and pulled to GND on the handle board side, so when the handpiece is connected tothe IPU, the handle board pulls the pin down and a processor may detect that a handpiece is connected.III.I. Wireless communication in place of cable

[0072] The cable connection between the IPU and the handpiece may be replaced by wireless transmission such as Bluetooth, Wi-Fi, or some other wireless protocol. In these cases, the handpiece may have a battery whose capacity can drive the handpiece for the longest length of a surgery. A wireless connection may provide an alternative architecture to implement electrical isolation of the patient, as required by the IEC 60601-1 standard.III.J. Isolation

[0073] Referring to FIG. 5F, the patient interface board may electrically isolate the motherboard from the patient-facing cable and scope by providing an optical connection or transformer to interrupt the copper signal path. The isolation of the data may be provided between the video stream processor (such as the Cypress CX3) and the motherboard via a fiber optic cable driven by USB 3.0 transceivers on each end of the cable, without power conductors, that allow an interruption of copper conductors, while communicating via the USB 3.0 communication protocol.

[0074] The physical interface between the scope and the IPU may be a USB 3.0 cable consisting of three twisted pairs, a ground conductor, and a power pair of wires, although the physical layer communication is not USB 3.0. The patient interface board may electrically isolate the processing circuitry from the patient-facing cable and scope by providing an optical connection or transformer to interrupt the copper signal path. The isolation mechanism may isolate the patient from the possibility of electric shock and prevent excessive leakage currents.

[0075] The IPU box may include a transformer 1170 that steps down 120 / 220 V AC voltage to a secondary voltage used internally to operate the processing circuitry 1172 of the IPU box, and a second transformer 1180 may isolate the secondary circuitry 1172 from the patient and patient-facing circuitry.

[0076] Two safety capacitors 1182 and 1184 may be provided in series across the primary and secondary of the isolation transformer. The purpose of capacitors 1182 and 1184 is to create a current divider for common mode current created in the isolated switching supply that utilizes the transformer 1180. The lower impedance of these capacitors, relative to the parasitic capacitance between the patient isolated island 1174, including the scope, and the earth, may attract the majority of the common mode current reducing the common mode currents that travel between the patient isolation island 1174, including the scope, and earth, thereby to reduce radiated emissions. The two capacitors may be surface mount ceramic capacitors, to minimize their impedance at higher frequencies. Capacitor 1186 may be placed differentially across the secondary of transformer 1180 creating a low impedance at high frequencies across the secondary of the transformer. This low- impedance allows common mode currents traveling on the positive output of the transformer to travelto the negative output of the transformer, through capacitor 1186 and back across the transformer through capacitors 1182 and 1184. The two capacitors 1182 and 1184 may be placed in series and may be UL listed safety capacitors to comply with the requirements of IEC 60601.

[0077] A second pair of two capacitors in series 1192, 1194 may connect the USB connector shell (the metal shielding jacket on the female side of a USB connector) to two mounting holes which tie to the earth connected IPU chassis to provide a short return path to earth for common mode currents injected into the scope and / or scope cable. The capacitor pairs 1192, 1194 may be placed symmetrically on each side of the USB connector shell, both connecting to a chassis mounting point that is earth connected (for example, to the housing 1196 of IPU 1100), to improve the shielding effectiveness to common mode currents injected onto the scope cable.

[0078] The value of capacitors 1182, 1184, 1186, 1188, 1192, 1194 is selected to provide sufficient reduction of common mode currents and comply with the leakage requirements for IEC 60601-1.

[0079] A fiber-only optical cable may be utilized to transport high speed video data from the patient isolated circuits 1174 to the secondary circuits 1172 in compliance with the IEC 60601-1 patient isolation requirements. The fiber optic cable may contain USB 3.0 transceivers on each end of the cable. The high-speed video from the scope may be translated from a MIPI-CSI2 protocol used by the image sensor to a USB 3.0 protocol through an integrated circuit. The USB 3.0 superspeed RX and TX data pairs may be converted to optical signals transported over the optical cable via optical transceivers. The optical transceivers on each end of the cable may be powered locally to avoid the need to run power, and copper wires, over the optical cable allowing the cable to maintain compliance with IEC 60601-1 isolation requirements.

[0080] The patient interface board may provide the scope interface, including the BF type patient isolation required per the isolation diagram and IEC 60601-1. This includes the isolated power supply, and isolation of any other interfaces with copper wire that may conduct electricity (USB interfaces, etc.).III.K. Other peripheralsIII.K.l. Monitor

[0081] The IPU may drive a video monitor so that the surgeon can have a real-time display of the surgery.III.K.2. USB port

[0082] A USB port may be provided on the front of the unit for use with a USB flash drive, which may be cabled to the motherboard. Four USB ports may be provided on the rear of the unit for use with a USB mouse and keyboard. Ethernet and Wi-Fi interfaces may be provided from the motherboard for network connectivity to cloud storage (see §§ IV.C and IV.D, H

[0108] to

[0115] ,below). An analog microphone input may be provided on the rear of the unit as well as a Bluetooth interface that can be used for annotating during procedures. A speaker may be provided in the IPU. An AC mains plug may provide power for the IPU. The AC mains may be controlled by a power switch.III.K.3. Connections to cloud storage

[0083] The IPU and programming may allow videos, images, metadata, and other data to be captured and saved. Programs on the IPU may allow update of software of the IPU. This data may be uploaded or backed up, for example over Wi-Fi, Bluetooth, or a similar wireless connection to the cloud, or may be stored to an external removable USB flash drive connected at the USB port. This flash drive may then be used to transfer the data to patient records as needed by the facility or uploaded to cloud storage from an external PC (see§§ IV.C and IV.D, H

[0108] to

[0115] , below).

[0084] Video may be stored in two-minute increments. If there’s a write error, the length of video lost may be kept to that limit. The stored video and still images may be annotated with date, time, and location metadata, and the serial number of scope and IPU. In the cloud, the serial number may be used to connect the video and images to the right patient’ s medical record.

[0085] At end of each surgical day, data for the day’s cases may be stored either in a cloud server or on the USB drive. If connections to the cloud fail, the USB storage may provide an easily- accessed backup. The surgeon may later access the cloud storage or USB data to transfer into the patient’s medical record, and annotate with physician’s notes.III.K.4. USB connection for keyboard and mouse

[0086] During normal operation, the scope pushbutton is the only user input available. A USB keyboard and mouse may be connected to the system to perform system configuration. A keyboard and mouse may allow entry to a service or configuration screen.III.K.5. Microphones

[0087] The IPU may have a connector for a wired microphone and may allow the connection of a Wireless microphone. This may allow real-time annotation of videos captured by the surgeon. The system setup may allow the user to specify if they wish audio to be enabled and then to either connect a microphone with a 3.5mm jack or a Bluetooth Interface.IV. Machine-readable serial numberIV.A. Machine-readable serial number

[0088] Each scope as shipped may have one or more scope-specific data encoded in machine-readable and scannable, and / or human-readable form. The data may include one or more of the scope’s serial number, configuration data, manufacturing calibration data, tracking data, etc. These data may be used for multiple purposes.

[0089] Information may be encoded in a machine-readable code on packaging for the scope, embedded in packaging, or embedded in the scope as a scannable code. The scannable code may be any form of Matrix (2D) or linear bar or machine vision code that can be scanned by a smartphone. Examples include any variant of QR code, Code 39, Code 49, Code 93, Code 128, Aztec code, Han Xin Barcode, Data Matrix code, JAB Code, MaxiCode, PDF417 code, SPARQCode, and others. The scannable code may be an RFID or similar tag that can be scanned by a sensor in a phone. The scan may be optical, or may use any IEEE 802 or related communications protocol, including Bluetooth, RFID (ISO 14443) or NFC (ISO 18092). The scannable code may be coded on packaging, in the scope’s handle, or in the nose cap of a replaceable scope insertion tip. Alternatively, it may be stored in an EEPROM memory in the handset, connected by an SPI (Serial Peripheral Interface), I2C (Inter-Integrated Circuit), USB, or a one -wire protocol, to be read when the scope is plugged into the image processing unit (IPU). The scope may have a small amount of non-volatile memory that can be read and written during initial device manufacture and by the IPU. That memory may store an electronically-readable serial number written into the memory during manufacture. This memory may also store per-scope configuration information, such as scope model, serial number, white balance coefficients, lens properties that can be corrected in the IPU, focus parameters, etc. This memory may also be used to store usage information such as timestamps or usage time as determined by the IPU, to prevent reuse of the scope after 24 hours. To ensure tamper resistance, information written into the handle memory may be written under a secure or encrypted protocol used between the IPU and the handle’s microprocessor.

[0090] The information may be stored as s single datum (essentially a serial number, or some other datum that semantically-equivalently uniquely identifies the scope), which may be used as an index key it a database at a server, which in turn has the full data about the scope. In some cases, various operating parameters of the scope may be stored in a database of a server, and either the model number or serial number may be used as a lookup key to retrieve this configuration data and collection of parameters. In other cases, the operating parameters may be separately individualized to each individual scope. For example, at the beginning of an arthroscopic surgery on a shoulder, the IPU may confirm that the scope to be used is indeed an arthroscope of suitable diameter, length, and optical capabilities. The two approaches may be combined, so that some parameters are stored based on model number, and others are stored individually per scope.

[0091] Data stored in on-board memory or in a remotely-accessible database may include:• A unique serial number or database lookup key• A model number and version number (integer or ASCII)• A text description of the component’ Model Number / Model identifier that can be displayed on the control display screen (typically a 32-charactarer ASCII string)• Calibration / normalization data• Full configuration specifications — for example: o The manufacturer’s part number for the image sensor, which may allow many additional properties of the image sensor to be looked up in a table in the IPU including: o The size (in rows x columns) of the image sensor o Supported frame rates for the sensor and frame reporting rates o Minimum / maximum integration time and integration time configuration resolution (for example,. 0.1 to 100 ms in increments of 1 ms) o An identifier for illumination sources on board the scope — white, infrared, ultraviolet, individual colors, image plane resolution (720x480, 1280x720, etc.) o An identifier for what sensors are in the image plane — for example, one bit on / off for each of red, green, blue, ICG infrared, and other colors as extended in future software updates o Information to establish white balance, color correction gamma curves, coefficients for distortion correction, sensor color sensitivity, illumination color, etc. o (Boolean) Does / does not provide an illumination source in the handpiece o (Boolean) Does / does not provide a de-fogging heater in the handpiece o (Boolean) Does / does not provide a rotation sensor in the handpiece o (Boolean) Does / does not support focus control in the handpiece• Calibration / normalization data — for example o Corrective data for variations in lens focus o Correction coefficients to compensate for image sensor color sensitivity, illumination color, white balance, distortion correction o LED illumination brightness coefficients• An identifier to enable / disable certain image enhancement parameters based on the hardware image configuration — this may be used to pre-configure image processing settings based on anticipated imaging application for this scope. For example, a bit vector may enable or disable optimization of resolution, contrast, smoothing, and other optical properties• Various size and length properties, which may be important to control water pressure and the like• Manufacturing date• Date and time of first use• Date and time of initial connection and disconnections• Length of scope use during the procedure

[0092] Storing the data in an on-board memory (rather than in an off-board database) may improve field-adaptability. On-board data storage may reduce the need for software updates to theIPU, and may improve robustness if scopes are used in parts of a hospital or facility that do not have reliable internet access. In some cases, especially where the endoscopes use interchangeable replaceable components (for example, insertion shafts that are interchangeable on a reuseable handle), each individual replaceable component may have a machine-readable serial number.

[0093] Data stored in the handset may be encrypted, with a decryption key stored in the IPU. Encryption may improve safety and security, by preventing a malicious actor from corrupting the memory contents or otherwise interfering with proper operation of the system.

[0094] Data may be communicated either in a fixed-field binary protocol or in a “keyword=” protocol (analogous to JSON protocols for web pages).

[0095] The connector may be a standard connector (e.g. USB-A) or a special purpose connector. A special purpose connector may ensure that mismatched devices are not plugged together. A special-purpose connector may allow additional pins to support all required signals and video, for example, video signals over twisted pair, higher current to power to a heater in the handset, and an optical connector for illumination light fibers.IV.B. Use of machine-readable serial number to reduce errors, ensure nonexpiration and ensure sterile single-use

[0096] The machine-readable serial number may be used to improve efficiency of initiating a new procedure. When the scope is about to be used, the serial number may be scanned, either as a 2D optical bar code on the box, enclosed in packaging, or the scope itself, or via a remote sensing (for example, an RFID tag), or it may be read from EEPROM or similar nonvolatile memory onboard the scope itself, for example, as the scope is plugged into the IPU. As an alternative, the box or packaging may have printed information such as product model number, lot, manufacture date, expiration date and serial number, that allows redundancy in case the machine -readable information cannot be read.

[0100] Referring to FIGS. 6 A and 6B, as one example of pairing a scope with a specific patient and procedure, a member of surgical staff may request a computer display of a set of patient names, for example, the patients scheduled to be seen during the next few hours. The staff member may click on a button associated with the specific patient to be paired to the scope. The computer may respond by turning on a camera and asking the staff member to present a Matrix (2D) barcode on scope packaging to the camera. When the camera finds the Matrix (2D) barcode, the computer may pair the scope associated with that Matrix (2D) barcode with that patient and the patient’s medical record. Once the human patient, patient electronic medical record, and scope are paired to each other, one or more of the following operations may occur:• Database records for the patient and scope may be linked to each other.• The database record for the scope may be marked to indicate that the scope is about to be used.• The database may be queried to ensure that the scope has not been previously paired to a different patient. (In some cases, software may provide an emergency override so long as the scope has not been placed in actual use with another patient.)• The computer may make a number of validation checks and use constraints: o Based on the machine -readable serial number and the database, a computer may check whether this scope is assigned to the facility / location at which use has been initiated. o Based on the machine -readable serial number and the database, a computer may check whether the scope to be used is appropriate for the procedure for which the patient is scheduled. For example, if the scope is an arthroscope for joints and the surgeon is a GI physician or the scheduled procedure is a colonoscopy, the computer may flag a warning to the human user and seek confirmation. o The computer may check that all expiration dates are satisfied, that the scope is not subject to a recall or other product actions..• The database record for the scope may be updated to indicate a storage location for video, still images, or voice dictation to be stored in association with the patient’s medical record.

[0101] In another example, the computer may read the 2D barcode or other machine- readable optical code first, and based on that code, present a list of patients that are scheduled for procedures using that form of scope, or may request a patient identifier such as name, patient record number, or the like.

[0102] When the scope is plugged into the IPU for use, the steps and cross-checks listed above may be performed or repeated. If the scope was not previously paired to the patient, then the checks may be performed for the first time, and the computer may prompt a human user to provide the patient identity. In addition, in the database record for the scope, initial connection to the IPU may cause the following updates to a database of scope data:• A flag may be set to indicate that the scope has been removed from its packaging and is no longer sterile.• If the scope is sold for single use, to ensure sterility, reliability, the fact that use has been initiated may be recorded at the manufacturer’s database and / or in the memory of the scope itself. That single use may be recorded as a single binary flag, that, when set, forbids further use.• The first use may be marked as a timestamp and / or location, so that in some period of time (for example two or four hours), the scope cannot be reused. This would allow for the scope to be plugged in multiple times during a single procedure (for example to untangle a cable, or to reset after a power failure), but still be sufficient to prevent reuse.

[0103] As a procedure begins, or when a scope is plugged into the IPU, the IPU may run through a dialog to confirm that the scope and procedure are appropriate for each other. For example, the IPU may query the patient’s electronic medical record to confirm the procedure to be performed, and confirm that the attached scope is appropriate for the procedure. If a mismatch is detected, the IPU may offer a warning and request a confirmation and override. The serial number of the exact scope used may be stored in the medical record in case of an audit issue.

[0104] During a procedure, the scope or IPU may encounter an error or failure. In such event, the IPU or scope may communicate with a computer or database to initiate complaint handling or feedback process with manufacturer, hospital or appropriate reporting body. Alternatively, software may provide an ability for a user to initiate a feedback event. An error code may be stored in the medical record in case of audit.

[0105] The stored data may allow a single IPU to be useable with multiple scope configurations, reducing complexity of stocking, supplying, and using different scopes for different purposes.

[0106] If the scope is refurbished, that flag can be cleared to allow reuse.

[0107] The machine-readable serial number may provide an index to the manufacturer’s literature library. This can provide near instant access to information should it be required during a surgical procedure.IV.C. Use of machine-readable serial number to communicate patient data into electronic medical record

[0108] Referring to FIG. 6C, the pairing of the scope to the patient may improve efficiency of getting information into the patient’s electronic medical record. During the procedure, the surgeon or an assistant may mark the entirety or marked portions of the video for permanent storage in the patient’s electronic medical record, or into another database maintained by the hospital / customer or the scope manufacturer. In some cases, the IPU may compute voice-to-text of the physician’s narration during the procedure. The IPU may connect to a cloud application via Wi-Fi or Ethernet. Images and videos may be sent to this cloud application in real time, after each procedure, or stored on the USB memory. The images and video may be sent to the cloud application as a live stream, or may be collected in storage in the IPU for periodic uploading, such as at the end of the day. Images and video may be stored with the patient’s electronic medical record, either in raw form, or after some selection and editing. They may be annotated with text or dictation by a physician or assistant. The video or images from the procedure may be automatically routed to storage associated with a specific patient (such as the patient’s electronic medical record, or a patient portal hosted via a database separate from the hospital’s or surgeon’s electronic medical record) based on the pairing established by means of the machine-readable serial number.

[0109] This video may be anonymized, edited and / or delivered to the patient, perhaps with voice over dictation, as described in Patent App. Ser. No. 16 / 278,112, filed Feb. 17, 2019, incorporated by reference. This video may improve the patient’s post-operative rehab, and may provide patient-specific reporting.IV.D. Use of machine-readable serial number for inventory control, location tracking, reordering, and stock management

[0110] Referring to FIG. 6D and 6E, the purchaser / hospital may interact with the database to set a minimum inventory level. Alternatively, a computer system accessible to the manufacturer may ascertain an average rate of use, time for delivery based on location, and any pending or in-transit inventory, to compute a reorder inventory level. As each scope is used, one or more computers may decrement the existing stock level, and if that decremented level, compared against the reorder stock level, suggests reorder, the computer may automatically enter a reorder to maintain the stock at a proper level. The computer may integrate via an API or similar EDI (electronic data interchange) connection with a distribution center or 3rdparty to place reordered quantity. A given user ID may have one or more billing locations and one or more delivery addresses.

[0111] Scope packages may be scanned as necessary, typically as scopes arrive at the hospital / purchaser site, so that inventory can be checked in, and as inventory is moved from one internal location to another (for example, store rooms on different floors or wings). Additionally, the system may use tracking information from UPS or FedEx or another shipper / logistics manager to determine location of in-transit inventory from manufacturer, though the distribution chain to the final hospital / purchaser. The system may use tracking proof of delivery as a signal that a product was received by the customer site.

[0112] The system may issue a warning if it detects that a scope seems to have gotten lost. For example, the system may compute a typical inventory time for a given location (for example, perhaps two weeks), and may notice if one scope has not been scanned or moved for some multiple of that time. Similarly, the system may warn for unexpected inventory movement. The system may be programmed to eliminate false positives and over-reporting — for example, movement to a shipping hub, or movement via a hospital’ s internal distribution system may take a scope on an unexpected route, but should be suppressed to avoid over-reporting.

[0113] Referring to FIGS. 6F and 6G, the system may provide dashboards to provide realtime data to surgeons, facility managers and staff, and the supply chain, to provide ready information about inventory levels and flow through the supply chain. For example, dashboards may give orders placed and not yet received, shipment tracking information, delivery estimates, current on-hand inventor levels, and recent use patterns. FIG. 6F shows an inventory list of current inventory of scopes on hand ready for use, including model name, model description, lot number (typically useful in case of a product recall), serial number, and expiration date. FIG. 6G shows recently usedscopes — the serial number, and where, when, and by whom used. This information may be presented at a total or statistical level, or as a list of individual scopes and their current status.

[0114] This tracking may improve utilization and inventory management by ensuring “just in time” ordering,.

[0115] A database may store information that tracks the history of the scope. If the serial number is remotely scannable (for example, in an RFID tag), then the location of the scope may be tracked through the distribution channel and storage at the purchaser hospital. This information may be used to ensure that the scope has not exceeded any time limits, that it has not been stored in locations that were known to go over temperature limits, etc. For example, the 24-hour limit after first use may be enforced by the IPU by reading the time of first use from the non-volatile memory on the handle board PCBA.. As a procedure begins, the IPU may do a query over the internet to confirm the scope has not exceeded a manufacturer’s date, and that the scope remains within specification and is not subject to any safety recall.IV.E. Configuration / capability information

[0116] Each machine -readable serial number may be related to information describing specific configuration and / or capability information relating to the scope. The serial number may be used to trace the device back to the records of how it was built and the components used in the build. In some cases, a reusable scope handle may be designed to be used with interchangeable parts, and each interchangeable or replacement component may have a machine -readable serial number, which, in turn, may be related to a database record with a specific model number, and that model number’ s database record may have a list of specific configuration and / or capability information. This information can be checked as the scope is paired to a patient or a similar time when specific capabilities of a specific scope or component should be compared to the specific needs of a patient. The serial number may be used to associate with a patient and to track inventory.IV.F. Calibration information

[0117] During manufacture, specific measurements and calibration data may be measured. For example, white balance, general level setting, color correction gamma curves, coefficients for distortion correction, sensor color sensitivity, illumination color, any anomalies of focus in different parts of the visual field, and the like. These data may be stored in a database indexed by the scope’s serial number, so that it may be available to the image processing pipeline for compensation.IV.G. Implementation

[0118] A database for implementation of the above features may include tables that track information such as the following:

[0119] The database may have tables describing scopes. One table may describe properties of models of scope, properties that are common to a product ID class of scopes. A second table may describe individual scopes.

[0120] The database may have tables describing procedures. One table may describe properties of a class of procedures. A second table may describe individual procedures on individual patients.

[0121] The database may have tables describing orders and inventory states.

[0122] The database may have tables describing patients.

[0123] The database may have tables that set up the relations between scopes, procedures, patients, surgeons, inventory states, and the like.

[0124] Other tables may represent surgeons, surgical specialties, surgical offices, surgical staff members, system administrators, addresses, passwords and similar security information; links to link video, voice dictation, and images into patient medical records; procedure checklists, access permissions so that patients are able to see only the patient information to which each is entitled, and the like.

[0125] Referring to FIG. 6H, an architecture of a system may be arranged around clientserver principles. Users may interact with user-device client software, such as browser HTML or a downloaded app on either a phone or a desktop computer. On the server side, the front-end may be provided by off-the-shelf components such as Amazon AWS, ALB (application load balancer), and / or CloudFront content delivery network. The application server may include components for patient management, inventory management, management of user IDs, permissions, authentication, and authorization; facility management; procedure management; order management; scope management; pairing between scopes and patients and procedures, and other administrative functions. A relational / SQL database may provide storage and lookup.V. Embodiments

[0126] Embodiments of the invention may include any one or more of the following features, singly or in any combination.

[0127] Endoscope 100 may have a handle, and an insertion shaft, the insertion shaft having at its distal end a camera. The insertion shaft may have solid state illumination and imaging circuitry at or near a tip designed to provide illumination and imaging of the interior of a body cavity for a surgeon during surgery. The proximal portion of the handle may have electronics for drive of the illumination circuitry and to receive imaging signal from the imaging circuitry. The proximal handle portion may be designed to permit sterilization between uses. A joint between the proximal handle portion and the insertion shaft may designed to separably connect the insertion shaft to the proximal handle portion. When it is separated, the joint may permit removal of the insertion shaft for disposal and replacement. The joint may be designed so that, when connected, the joint can transfer mechanical force from a surgeon’s hand to the insertion shaft, and provides electrical connectivity between the proximal handle circuitry and the illumination and imaging circuitry. The handle may have proximal and distal portions. The distal portion may lie between the insertion shaft and proximal handle portion. The insertion shaft may be rigidly affixed to the distal handle portion. The joint may be disposed to connect and disconnect the distal and proximal portions of the handle. The distal handle portion may be designed to indirectly transfer mechanical force between a surgeon’s hand to the insertion shaft, and provide indirect electrical connectivity between the proximal handle circuitry and the illumination and imaging circuitry. The handle may have a rotation collar having surface features designed to assist the surgeon in rotating the insertion shaft in the roll dimension about the axis of the insertion shaft relative to the proximal handle portion. The electronics inside the proximal handle portion may be designed to sense roll of the insertion shaft, and provide an angular rotation signal designed to permit righting of a displayed image received from the imaging circuitry. A mounting for the image sensor may be designed to permit panning of the image sensor about a pitch or yaw axis perpendicular to the central axis of the insertion shaft. One or more ultraviolet LEDs internal to the endoscope may be designed to sterilize a region of the interior of the endoscope. Hoses for insufflation fluid or gas may be designed on lie on or near a central axis of proximal handle portion. Two or more insertion shafts each having dimensions different than the others, may each be connectable to the proximal handle portion at the joint, to permit use of the proximal handle in surgery with different requirements for insertion shaft. A sterilization cabinet may be designed to sterilize components of the endoscope. An insertion shaft of an endoscope tip has a rigid proximal portion and a distal portion. The distal portion is bendable to direct a field of view of imaging circuitry in a desired direction. An illuminator and solid state imaging circuitry are at or near a distal tip of the articulable distal portion. The illuminator is designed to illuminate, and the imaging circuitry being designed to capture imaging of, an interior of a body cavity for a surgeon duringsurgery. A coupling of the replaceable endoscope tip is designed to separably connect the insertion shaft at a joint to a handle portion, and to disconnect the joint. The coupling has mechanical connectors. When the joint is separated, the mechanical connectors permit removal of the insertion shaft from the handle for disposal and replacement. When the joint is connected, the joint is designed to provide mechanical force transfer between a surgeon’s hand to the insertion shaft. Electrical connectors are designed to connect the insertion shaft to electronics in the handle. The handle electronics are designed for drive of the illuminator and to receive imaging signal from the imaging circuitry, the handle being designed to permit sterilization between uses. Control force transfer elements are designed to permit a surgeon to direct a direction of the imaging circuitry by transfer of mechanical force directed by a surgeon to the articulable distal portion. The distal bendable portion includes a series of articulated rigid segments. A sheath or cover over the articulated rigid segments is designed to reduce intrusion or pinching. The distal bendable portion is formed of a solid component, bendable in its lateral and elevation dimensions, and relatively incompressible in compression in its longitudinal dimension. The distal bendable portion is extendable from and retractable into a solid sheath. The distal bendable portion is bendable in one dimension. The distal bendable portion is bendable in two orthogonal dimensions. The imaging circuitry is mounted within at or near a distal tip of the articulable distal portion via a pannable mounting. The pannable mounting is designed as two sides of a parallelogram. The imaging circuitry is mounted on a structural segment hinged to the two parallelogram sides. Passages and apertures are designed to pass irrigation fluid to improve view from a lens or window over the imaging circuitry. Passages and apertures are designed to pass inflation fluid to enlarge a cavity for surgery. Mechanical connectors of the coupling include a twist-lock designed to affix the endoscope insertion shaft to the handle portion. A plurality of the endoscope tips are bundled and packaged together with a handle. The handle has electronics designed for drive of the illuminator and to receive imaging signal from the imaging circuitry. The plurality of tips and handle are packaged for integrated shipment and sale. The illuminator is an illumination LED mounted at or near the distal tip. The illuminator is an emission end of a fiber optic fiber driven by an illumination source in the handle. Camera 410 may be enclosed within a plastic casing. The plastic casing may be formed as an overmolded jacket that is designed to protect camera 410 from bodily fluids and to structurally hold components of the tip in an operating configuration. The overmolded jacket may be designed to retain a transparent window in operating configuration with camera 410. The overmolded component may be formed of transparent plastic. The overmolded component may be designed to function as a lens for image sensor 410. Image sensor 410 may be mounted on a flexible circuit board. Flexible circuit board 416 may mount an illumination LED 418. LED 418 and image sensor 410 may be mounted on opposite sides of flexible circuit board 416. Image sensor 410 may be protected behind a transparent window. The window may be molded in two thicknesses, a thinner portion designed for mounting and to allowpassage of illumination light, a thicker portion over camera 410. The handle may contain a circuit board with circuitry for control of and receipt of signals from camera 410. The handle and its components may be designed with no metal fasteners, and no adhesives, except those captured by overmolding. Control buttons of the endoscope may be molded with projections that function as return springs. The projections may be adhered into the endoscope handle via melting. The circuit board may be overmolded by plastic that encapsulate the circuit board from contact with water. The circuit board may be mounted into the handle via melting. Components of the handle may be joined to each other into a unitary structure via melting. Components of the handle may be joined by resilient clips designed to held the two components to each other before joining into unitary structure via melting. The handle may be formed of two shells concentric with each other. Rotation of the two shells relative to each other may be controlled via one or more O-rings frictionally engaged with the two respective shells. The handle may have overmolded a layer of a high-friction elastomer. The insertion shaft may be connected to the handle via a separable joint. A water joint of the separable joint may be molded for an interference seal without O-rings. A water cavity of the separable joint may be designed to impart swirl to water flowing from the handle to the insertion shaft. The insertion shaft may be formed of stainless steel and connected to the handle via a separable joint. Plastic components of the endoscope may be joined to the insertion shaft via overmolding of plastic into slots aligned at an oblique angle in the wall of the insertion shaft, without adhesives. The water joint may be formed as two cones in interference fit. The cones may interfere at a large diameter. The cones may interfere via a ridge raised on a lip of the inner male cone. Obturator 104 may be designed to pierce tissue for introduction of the endoscope. Features for twist-locking obturator 104 into trocar 102 may be compatible with features for twist-locking the endoscope into trocar.

[0128] An apparatus may include a computer processor and a memory. The processor is programmed to receive video image data from an image sensor at the distal end of an endoscope and to display the image data to a surgeon in real time. The processor is programmed to process the image data received from the image sensor via a machine learning model, the machine learning model trained to simultaneously upsample the image data to a resolution higher than that captured by the image sensor, to sharpen edges, and to enhance local contrast.

[0129] An apparatus may include a computer processor and a memory. The processor is programmed to receive video image data from an image sensor at the distal end of an endoscope and to display the image data to a surgeon in real time. The video image data have a frame rate at which the image data are generated by the image sensor. The processor is programmed to control the image sensor and / or an illumination source designed to illuminate a scene viewed by the image sensor, the controlling programmed to underexpose or overexpose every other frame of the video image data. The processor is programmed to process the image data received from the image sensor to combine successive pairs of frames of the image data to adjust dynamic range to enhance over-bright or over-dark portions of the image to expose detail, and to generate combined frames at the full frame rate of the video as generated by the image sensor.

[0130] An apparatus may include a computer processor and a memory. An apparatus may include a computer processor and a memory. The processor is programmed to receive video image data from an image sensor at the distal end of an endoscope and to display the image data to a surgeon in real time. The processor is programmed to sum an error for an intensity of the image relative to a setpoint intensity. The processor is programmed to simultaneously control at least two of gain, exposure, and illumination via a PID control algorithm to achieve image display at the setpoint intensity, maximum change per step of the PID control damped to prevent oscillation.

[0131] Embodiments may include one or more of the following features, singly or in any combination. The processor may be further programmed to control the image sensor and / or an illumination source designed to illuminate a scene viewed by the image sensor. The controlling may be programmed to underexpose or overexpose every other frame of the video image data. The processor may be further programmed to process the image data received from the image sensor to combine successive pairs of frames of the image data to adjust dynamic range to enhance over-bright or over-dark portions of the image to expose detail. The processor may be further programmed to generate combined frames at the full frame rate of the video as generated by the image sensor. The processor may be further programmed to sum an error for an intensity of the image relative to a setpoint intensity. The processor may be further programmed to simultaneously control at least two of gain, exposure, and illumination via a PID control algorithm to achieve image display at the setpoint intensity. A maximum change per step of the PID control may be damped to prevent oscillation. The processor may be further programmed to process the image data received from the image sensor via a machine learning model, the machine learning model trained to simultaneously upsample the image data to a resolution higher than that captured by the image sensor, to sharpen edges, and to enhance local contrast. The processor may be further programmed to enhance the video image data via dynamic range compensation. The processor may be further programmed to adjust exposure time, illumination intensity, and / or gain in image capture to adjust exposure saturation. The processor may be further programmed to enhance the video image data via noise reduction. The processor may be further programmed to enhance the video image data via lens correction. The processor may be further programmed to in addition to resolution, enhance at least two of dynamic range compensation, noise reduction, and lens correction. The processor may be further programmed to rotate the image display to compensate for rotation of the endoscope. The processor may be further programmed to adjust exposure time, illumination intensity, and / or gain in image capture to adjust exposure saturation.

[0097] A computer reads a machine-readable serial number of a medical device as the medical device is put into use. Based on the reading of the serial number, a computer records areduction of an on-hand inventory level of a class of medical devices of which the medical device is a member. A computer computes when the reduction of on-hand inventory of the class indicates a level that warrants reordering to replenish the on-hand inventory. When warranted, a computer places an order with a supplier of medical devices of the class to replenish on-hand inventory of medical devices of the class.

[0098] A computer system one or more processors and a machine readable, non-transitory memory having stored therein instructions. The instructions are programmed to cause the processor(s) to read a machine-readable serial number of a medical device as the medical device is put into use, and based on the reading of the serial number, to record a reduction of an on-hand inventory level of a class of medical devices of which the medical device is a member. The instructions are programmed to cause the processor(s) to compute when the reduction of on-hand inventory of the class indicates a level that warrants reordering to replenish the on-hand inventory. The instructions are programmed to cause the processor(s) to place an order with a supplier of medical devices of the class to replenish on-hand inventory of medical devices of the class.

[0099] An endoscope has an image sensor designed to gather photons to produce video. The image sensor is of a type that has properties that vary from one sensor to another within manufacturing tolerances. The endoscope has a connector to connect the endoscope to a computer image processor. The image processor is programmed to obtain data from a database designed to store properties of the endoscope’s image sensor and / or the image sensor’s behavior, the database storing properties specific to specific sensors or a specific class of sensors. The obtained data describes the image sensor’s properties from the database. The image processor is programmed to compute normalized video based on the obtained properties and the video from the image sensor.

[0100] A computer processor is programmed to receive image date from an image sensor of an endoscope, and to obtain data from a database designed to store properties of endoscopes’ image sensors and / or the image sensors’ behavior. The image sensor is of a type that has properties that vary from one sensor to another, the database storing properties specific to specific sensors or a specific class of sensors. A computer computes normalized video based on the obtained properties and the video from the image sensor.

[0101] A database stores information relating to specific individual medical devices, including physical location and / or ownership. An optical reader reads 2D identification codes on packaging of the medical devices. Database records are updated show current physical location and / or ownership based on the read 2D identification codes.

[0102] A computer system has one or more processors, and a machine readable, non- transitory memory having stored therein instructions. The instructions are programmed to cause the processor(s) to store information relating to specific individual medical devices, including physical location and / or ownership. An optical reader reads 2D identification codes on packaging of themedical devices. Database records are updated to show current physical location and / or ownership based on the read 2D identification codes.

[0103] A computer may read a machine-readable serial number of a endoscope as the endoscope is put into use. Based on the reading of the serial number, an on-hand inventory level of a class of endoscopes may be reduced and recorded. A reorder level may be computed based on the reduction in on-hand inventory level. A computer may compute when on-hand inventory level warrants reordering to replenish the on-hand inventory. Based on the computed warrant to reorder, a computer may place an order with a supplier of endoscopes of the class to replenish on-hand inventory of endoscopes of the class. The medical device may be an endoscope. The endoscope may have an image sensor. The image sensor may be designed to gather photons to produce video. The image sensor may be of a type that has properties that vary from one sensor to another within manufacturing tolerances. The programs may be programmed to connect a one of the medical devices or endoscopes to a computer processor. The processor may be programmed to obtain properties of the connected medical device from a database. The database may be designed to store information relating to specific individual medical devices, endoscopes, or image sensors. The database may be accessed based on the read serial number. The database may store properties of the endoscope’s image sensor. The database may store properties of the image sensor’s behavior in a database. The database may store properties specific to specific sensors. The database may provide properties of the one endoscope’s image sensor from the database, and to compute normalized video based on the obtained properties and the video from the image sensor. The database may store physical location of individual medical devices, endoscopes, or components. The database may store physical location and / or ownership of individual medical devices, endoscopes, or components. An optical reader may read identification codes on packaging of the medical devices. Database records of physical location and / or ownership may be updated to show current physical location and / or ownership based on the read identification codes. When a medical device or endoscope may be connected to its control computer, the computer may be programmed to obtain properties of the connected device from a database. The identification code may be a machine-readable 2D optical code. The identification code may be stored in machine-readable electronic memory. The database stores information relating to a device model of a class of medical devices manufactured to be interchangeable. The database may be designed to index to the device model information based on the read identification code of an individual medical device. The database may be designed to store data describing a color spectrum capability of the image sensor. The database may be designed to store data describing an image plane resolution of the image sensor. The database may be designed to store calibration properties of the endoscope’s image sensor specific to the specific image sensor of the specific endoscope. The image processor may be programmed to compute normalized video based on the obtained calibration properties and the video from the image sensor. The calibration properties may describe a whitebalance of the image sensor, or a color correction gamma curve of the image sensor, or a distortion correction of the image sensor, or any two or more, or any three or more of these properties. The computer may compute a prediction of delivery date based on the read 2D identification codes. The computer may update the physical location and / or ownership database records to show current physical location based on the read 2D identification codes or read of the machine -readable memory.

[0132] Various processes described herein may be implemented by appropriately programmed general purpose computers, special purpose computers, and computing devices. Typically a processor (e.g., one or more microprocessors, one or more microcontrollers, one or more digital signal processors) will receive instructions (e.g., from a memory or like device), and execute those instructions, thereby performing one or more processes defined by those instructions. Instructions may be embodied in one or more computer programs, one or more scripts, or in other forms. The processing may be performed on one or more microprocessors, central processing units (CPUs), computing devices, microcontrollers, digital signal processors, graphics processing units (GPUs), field programmable gate arrays (FPGAs), or like devices or any combination thereof. Programs that implement the processing, and the data operated on, may be stored and transmitted using a variety of media. In some cases, hard-wired circuitry or custom hardware may be used in place of, or in combination with, some or all of the software instructions that can implement the processes. Algorithms other than those described may be used.

[0133] Programs and data may be stored in various media appropriate to the purpose, or a combination of heterogeneous media that may be read and / or written by a computer, a processor or a like device. The media may include non-volatile media, volatile media, optical or magnetic media, dynamic random access memory (DRAM), static ram, a floppy disk, a flexible disk, hard disk, magnetic tape, any other magnetic medium, a CD-ROM, DVD, any other optical medium, punch cards, paper tape, any other physical medium with patterns of holes, a RAM, a PROM, an EPROM, a FLASH-EEPROM, other non-volatile memories, any other memory chip or cartridge or other memory technologies.

[0134] Databases may be implemented using database management systems or ad hoc memory organization schemes. Alternative database structures to those described may be readily employed. Databases may be stored locally or remotely from a device which accesses data in such a database.

[0135] In some cases, the processing may be performed in a network environment including a computer that is in communication (e.g., via a communications network) with one or more devices. The computer may communicate with the devices directly or indirectly, via any wired or wireless medium (e.g. the Internet, LAN, WAN or Ethernet, Token Ring, a telephone line, a cable line, a radio channel, an optical communications line, commercial on-line service providers, bulletin board systems, a satellite communications link, a combination of any of the above). Transmission mediainclude coaxial cables, copper wire and fiber optics 430, including the wires that comprise a system bus coupled to the processor. Transmission may occur over transmission media, or over electromagnetic waves, such as via infrared, Wi-Fi, Bluetooth, and the like, at various frequencies using various protocols. Each of the devices may themselves comprise computers or other computing devices, such as those based on the Intel® Pentium® or Centrino™ processor, that are adapted to communicate with the computer. Any number and type of devices may be in communication with the computer.

[0136] A server computer or centralized authority may or may not be necessary or desirable. In various cases, the network may or may not include a central authority device. Various processing functions may be performed on a central authority server, one of several distributed servers, or other distributed devices.

[0137] The following applications are incorporated by reference. U.S. Provisional application Ser. No. 63 / 544,608, filed Oct. 17, 2023, titled Artificial Intelligence Agent for Preparing Physician’s Note; U.S. application Ser. No. 18 / 370,375, filed Sep. 19, 2023, titled Image Enhancement for Endoscope; U.S. Provisional application Ser. No. 63 / 538,485, filed Sep. 14, 2023, titled Endoscope; U.S. Provisional application Ser. No. 63 / 534,855, filed Aug. 27, 2023, titled Endoscope; U.S. Provisional application Ser. No. 63 / 531,239, filed August 7, 2023, titled Endoscope; U.S. Provisional application Ser. No. 63 / 437,115, filed January 4, 2023, titled Endoscope with Identification and Configuration Information; U.S. application Ser. No. 17 / 954,893, filed Sep. 28, 2022, titled Illumination for Endoscope; U.S. Provisional App. Ser. No. 63 / 376,432, filed Sep. 20, 2022, titled Super Resolution for Endoscope Visualization; U.S. application Ser. No. 17 / 896,770, filed Aug. 26, 2022, titled Endoscope; U.S. Provisional App. Ser. No. 63 / 400,961, filed Aug. 25, 2022, titled Endoscope; U.S. App. Ser. No. 17 / 824,857, filed May 25, 2022, titled Endoscope; U.S. Prov. App. Ser. No. 63 / 249,479, filed Sept. 28, 2021, titled Endoscope; U.S. Prov. App. Ser. No. 63 / 237,906, fled Aug. 27, 2021, titled Endoscope; U.S. App. Ser. No. 17 / 361,711, filed Jun. 29, 2021, titled Endoscope with Bendable Camera Shaft; U.S. Prov. App. Ser. No. 63 / 214,296, filed Jun. 24, 2021, titled Endoscope with Bendable Camera Shaft; U.S. Provisional App. Ser No. 63 / 193,387 titled Anti-adhesive Window or Lens for Endoscope Tip; U.S. Provisional App. Ser. No. 63 / 067,781, filed Aug. 19, 2020, titled Endoscope with Articulated Camera Shaft; U.S. Provisional Application Ser. No. 63 / 047,588, filed Jul. 2, 2020, titled Endoscope with Articulated Camera Shaft; U.S. Provisional App. Ser. No. 63 / 046,665, filed Jun. 30, 2020, titled Endoscope with Articulated Camera Shaft; U.S. App. Ser. No. 16 / 434,766, filed Jun. 7, 2019, titled Endoscope with Disposable Camera Shaft and Reusable Handle; U.S. Provisional App. Ser. No. 62 / 850,326, filed May 20, 2019, titled Endoscope with Disposable Camera Shaft; U.S. App. Ser. No. 16 / 069,220, filed Oct. 24, 2018, titled AntiFouling Endoscopes and Uses Thereof; U.S. Provisional App. Ser. No. 62 / 722,150, filed August 23,2018, titled Endoscope with Disposable Camera Shaft; U.S. Provisional App. Ser. No. 62 / 682,585 filed June 8, 2018, titled Endoscope with Disposable Camera Shaft.

[0138] For clarity of explanation, the above description has focused on a representative sample of all possible embodiments, a sample that teaches the principles of the invention and conveys the best mode contemplated for carrying it out. The invention is not limited to the described embodiments. Well known features may not have been described in detail to avoid unnecessarily obscuring the principles relevant to the claimed invention. Throughout this application and its associated file history, when the term “invention” is used, it refers to the entire collection of ideas and principles described; in contrast, the formal definition of the exclusive protected property right is set forth in the claims, which exclusively control. The description has not attempted to exhaustively enumerate all possible variations. Other undescribed variations or modifications may be possible. Where multiple alternative embodiments are described, in many cases it will be possible to combine elements of different embodiments, or to combine elements of the embodiments described here with other modifications or variations that are not expressly described. A list of items does not imply that any or all of the items are mutually exclusive, nor that any or all of the items are comprehensive of any category, unless expressly specified otherwise. In many cases, one feature or group of features may be used separately from the entire apparatus or methods described. Many of those undescribed alternatives, variations, modifications, and equivalents are within the literal scope of the following claims, and others are equivalent. The claims may be practiced without some or all of the specific details described in the specification. In many cases, method steps described in this specification can be performed in different orders than that presented in this specification, or in parallel rather than sequentially.

Claims

CLAIMSThe invention claimed is:

1. A method, comprising the steps of: by computer, reading a machine -readable serial number of a medical device as the medical device is put into use, and based on the reading of the serial number, recording a reduction of an on-hand inventory level of a class of medical devices of which the medical device is a member; by computer, computing when the reduction of on-hand inventory of the class indicates a level that warrants reordering to replenish the on-hand inventory; and by computer, based on the computed warrant to reorder, placing an order with a supplier of medical devices of the class to replenish on-hand inventory of medical devices of the class.

2. The method of claim 1 : connecting a one of the medical devices to a computer processor, the processor programmed to obtain properties of the connected medical device from a database, the database being designed to store information relating to specific individual medical devices, the database being accessed based on the read serial number.

3. The method of claim 2: the medical device being an endoscope having an image sensor, the image sensor designed to gather photons to produce video, the image sensor being of a type that has properties that vary from one sensor to another within manufacturing tolerances; and further comprising the steps of: storing properties of the endoscope’s image sensor and / or the image sensor’s behavior in a database, the database storing properties specific to specific sensors; and the image processor programmed to obtain the properties of the one endoscope’s image sensor from the database, and to compute normalized video based on the obtained properties and the video from the image sensor.

4. The method of claim 2: the database being further designed to store physical location and / or ownership of individual medical devices; and further comprising the steps of: by an optical reader, reading identification codes on packaging of the medical devices; and updating the physical location and / or ownership database records to show current physical location and / or ownership based on the read identification codes.

5. The method of claim 4: the medical device being an endoscope having an image sensor, the image sensor designed to gather photons to produce video, the image sensor being of a type that has properties that vary from one sensor to another within manufacturing tolerances; and further comprising the steps of: storing properties of the endoscope’s image sensor and / or the image sensor’s behavior in a database, the database storing properties specific to specific sensors; and the image processor programmed to obtain the properties of the one endoscope’s image sensor from the database, and to compute normalized video based on the obtained properties and the video from the image sensor.

6. The method of claim 2, wherein: the identification code is a machine-readable 2D optical code.

7. The method of claim 2, wherein: the identification code is stored in machine -readable electronic memory.

8. The method of claim 7, wherein: the identification code is stored in machine -readable nonvolatile memory.

9. The method of claim 2, wherein: the database stores information relating to a device model of a class of medical devices manufactured to be interchangeable, the database designed to index to the device model information based on the read identification code of an individual medical device.

10. A computer system, comprising: one or more processors; and a machine readable, non-transitory memory having stored therein instructions programmed to cause the processor(s) to: read a machine -readable serial number of a medical device as the medical device is put into use, and based on the reading of the serial number, to record a reduction of an on-hand inventory level of a class of medical devices of which the medical device is a member; compute when the reduction of on-hand inventory of the class indicates a level that warrants reordering to replenish the on-hand inventory; and based on the computed warrant to reorder, place an order with a supplier of medical devices of the class to replenish on-hand inventory of medical devices of the class.

11. The computer system of claim 10: the medical device being an endoscope having an image sensor, the image sensor designed to gather photons to produce video, the image sensor being of a type that has properties that vary from one sensor to another within manufacturing tolerances; the instructions being further programmed to cause the processor(s) to: connect a one of the medical devices to a computer processor, the processor programmed to obtain properties of the connected medical device from a database, the database being designed to store information relating to specific individual medical devices, the database being accessed based on the read serial number; store properties of the endoscope’s image sensor and / or the image sensor’s behavior in a database, the database storing properties specific to specific sensors; and obtain the properties of the one endoscope’s image sensor from the database, and to compute normalized video based on the obtained properties and the video from the image sensor.

12. The computer system of claim 10: the database being further designed to store physical location and / or ownership of individual medical devices; the instructions being further programmed to cause the processor(s) to: direct an optical reader to read identification codes on packaging of the medical devices and connect a one of the medical devices to a computer processor, the processor programmed to obtain properties of the connected medical device from a database, the database being designed to store information relating to specific individual medical devices, the database being accessed based on the read serial number.update the physical location and / or ownership database records to show current physical location and / or ownership based on the read identification codes.

13. An endoscope, comprising: an image sensor designed to gather photons to produce video, the image sensor being of a type that has properties that vary from one sensor to another within manufacturing tolerances; a connector to connect the endoscope to a computer image processor, the image processor programmed to: obtain data from a database designed to store properties of the endoscope’s image sensor and / or the image sensor’s behavior, the database storing properties specific to specific sensors or a specific class of sensors; the obtained data to describe the image sensor’s properties from the database, and compute normalized video based on the obtained properties and the video from the image sensor.

14. The endoscope of claim 13, further designed to connect to a computer processor programmed to: read a machine -readable serial number of a endoscope as the endoscope is put into use, and based on the reading of the serial number, record a reduction of an on-hand inventory level of a class of endoscopes of which the endoscope is a member; compute when the reduction of on-hand inventory of the class indicates a level that warrants reordering to replenish the on-hand inventory; and based on the computed warrant to reorder, to place an order with a supplier of endoscopes of the class to replenish on-hand inventory of endoscopes of the class.

15. The endoscope of claim 13: the database being further designed to store information relating to specific individual medical devices, including physical location and / or ownership; by an optical reader, reading 2D identification codes on packaging of the medical devices; and updating the physical location and / or ownership database records to show current physical location and / or ownership based on the read 2D identification codes.

16. The endoscope of claim 13, the database designed to store data describing a color spectrum capability of the image sensor.

17. The endoscope of claim 13, the database designed to store data describing an image plane resolution of the image sensor.

18. The endoscope of claim 13: obtain data from a database designed to store calibration properties of the endoscope’s image sensor specific to the specific image sensor of the specific endoscope; the image processor being programmed to compute normalized video based on the obtained calibration properties and the video from the image sensor.

19. The endoscope of claim 18, wherein the calibration properties describe a white balance of the image sensor.

20. The endoscope of claim 18, wherein the calibration properties describe a color correction gamma curve of the image sensor.

21. The endoscope of claim 18, wherein the calibration properties describe a distortion correction of the image sensor.

22. A method comprising the steps of: at a computer processor programmed to receive image date from an image sensor of an endoscope, obtaining data from a database designed to store properties of endoscopes’ image sensors and / or the image sensors’ behavior, the image sensor being of a type that has properties that vary from one sensor to another, the database storing properties specific to specific sensors or a specific class of sensors; at the computer, compute normalized video based on the obtained properties and the video from the image sensor.

23. The method of claim 22, further comprising the steps of: at a computer, reading a machine -readable serial number of a endoscope as the endoscope is put into use, and based on the reading of the serial number, record a reduction of an on-hand inventory level of a class of endoscopes of which the endoscope is a member; computing when the reduction of on-hand inventory of the class indicates a level that warrants reordering to replenish the on-hand inventory; andby computer, based on the computed warrant to reorder, placing an order with a supplier of endoscopes of the class to replenish on-hand inventory of endoscopes of the class.

24. The method of claim 22: the database being further designed to store information relating to specific individual medical devices, including physical location and / or ownership; by an optical reader, reading 2D identification codes on packaging of the medical devices; and updating the physical location and / or ownership database records to show current physical location and / or ownership based on the read 2D identification codes.

25. The method of claim 22, further comprising the steps of: obtain data from a database designed to store calibration properties of the endoscope’s image sensor specific to the specific image sensor of the specific endoscope; the image processor being programmed to compute normalized video based on the obtained calibration properties and the video from the image sensor.

26. The method of claim 25, wherein the calibration properties describe a white balance of the image sensor.

27. A method, comprising the steps of: in a database, storing information relating to specific individual medical devices, including physical location and / or ownership; by an optical reader, reading 2D identification codes on packaging of the medical devices; and updating the physical location and / or ownership database records to show current physical location and / or ownership based on the read 2D identification codes.

28. The method of claim 27, further comprising the steps of: by computer, reading a machine -readable serial number of a medical device as the medical device is put into use, and based on the reading of the serial number, recording a reduction of an on-hand inventory level of a class of medical devices of which the medical device is a member; by computer, computing when the reduction of on-hand inventory of the class indicates a level that warrants reordering to replenish the on-hand inventory; andby computer, based on the computed warrant to reorder, placing an order with a supplier of medical devices of the class to replenish on-hand inventory of medical devices of the class.

29. The method of claim 27 : the medical devices being endoscopes having respective image sensors, the image sensors designed to gather photons to produce video, the image sensors being of a type that has properties that vary from one sensor to another within manufacturing tolerances; and further comprising the steps of: storing properties of the endoscope’s image sensor and / or the image sensor’s behavior in a database, the database storing properties specific to specific sensors; and connecting a one of the endoscopes to a computer image processor, the image processor programmed to obtain the properties of the one endoscope’s image sensor from the database, and to compute normalized video based on the obtained properties and the video from the image sensor.

30. The method of claim 27, further comprising the step of: for medical devices between the time of a sale and the time of delivery, computing a prediction of delivery date based on the read 2D identification codes.

31. The method of claim 27, further comprising the step of: in a database, storing information relating to specific individual medical devices, including physical location; by an optical reader, reading 2D identification codes on packaging of the medical devices; and updating the physical location and / or ownership database records to show current physical location based on the read 2D identification codes.

32. The method of claim 27, further comprising the step of: in a database, storing information relating to specific individual medical devices, including ownership; by an optical reader, reading 2D identification codes on packaging of the medical devices; and updating the physical location and / or ownership database records to show current ownership based on the read 2D identification codes.

33. A computer system, comprising: one or more processors; and a machine readable, non-transitory memory having stored therein instructions programmed to cause the processor(s) to: in a database, storing information relating to specific individual medical devices, including physical location and / or ownership; by an optical reader, reading 2D identification codes on packaging of the medical devices; and updating the physical location and / or ownership database records to show current physical location and / or ownership based on the read 2D identification codes.

34. The computer system of claim 33, the instructions being further programmed to cause the processor(s) to: read a machine -readable serial number of a medical device as the medical device is put into use, and based on the reading of the serial number, recording a reduction of an on-hand inventory level of a class of medical devices of which the medical device is a member; compute when the reduction of on-hand inventory of the class indicates a level that warrants reordering to replenish the on-hand inventory; and based on the computed warrant to reorder, place an order with a supplier of medical devices of the class to replenish on-hand inventory of medical devices of the class.

35. The computer system of claim 33: the medical devices being endoscopes having respective image sensors, the image sensors designed to gather photons to produce video, the image sensors being of a type that has properties that vary from one sensor to another within manufacturing tolerances; the instructions being further programmed to cause the processor(s) to: store properties of the endoscope’s image sensor and / or the image sensor’s behavior in a database, the database storing properties specific to specific sensors; and connect a one of the endoscopes to a computer image processor, the image processor programmed to obtain the properties of the one endoscope’s image sensor from the database, and to compute normalized video based on the obtained properties and the video from the image sensor.

36. A method, comprising the steps of: in a database, storing information relating to specific individual endoscopes, the endoscopes having respective image sensors of a type that has properties that vary from one sensor to another within manufacturing tolerances, the stored information including: information recording properties of individual endoscopes’ image sensors and / or the image sensors’ behavior; and physical location and / or ownership of individual endoscopes; by an optical reader, reading 2D identification codes on packaging of the endoscopes, and updating the physical location and / or ownership database records to show current physical location and / or ownership based on the read 2D identification codes. as respective endoscopes are put into use, by computer, reading a machine -readable serial number of the endoscope, and based on the reading of the serial number, recording a reduction of an on-hand inventory level of a class of endoscopes of which the endoscope is a member; based on the read serial number, recording a relation between the serial number’ s endoscope and a patent for a procedure in which the endoscope is to be used; by computer, computing when the reduction of on-hand inventory of the class indicates a level that warrants reordering to replenish the on-hand inventory; by computer, based on the computed warrant to reorder, placing an order with a supplier of endoscopes of the class to replenish on-hand inventory of endoscopes of the class; based on the read serial number, obtaining from the database the recorded properties of the endoscope’s image sensor, and computing normalized video based on the obtained properties and the video from the image sensor.

Images