Image processing system and method of using the same

Abstract

A method for generating a color image using a monochrome image sensor. The method includes sequentially illuminating a surface with a plurality of colors, one color at a time. The monochrome image sensor captures a plurality of image frames of the surface based on the plurality of colors. The plurality of image frames are identified, and at least one feature of a target in the plurality of image frames is highlighted. Color intensities of the plurality of image frames are normalized. A color intensity map of the target is generated for each of the plurality of image frames. A correlation score is determined by comparing each color intensity map of the plurality of image frames. A color image is generated based on the correlation score.

Description

[0001] priority

[0002] This application claims priority to provisional application No. 63 / 073,126, filed on September 1, 2020, the entire contents of which are incorporated herein by reference. Technical Field

[0003] Various aspects of this disclosure generally relate to image processing systems, apparatuses, and related methods. Examples of this disclosure relate to systems, apparatuses, and related methods for estimating motion and coloring images captured by a monochrome sensor, among others. Background Technology

[0004] Technological advancements have enabled users of medical systems, devices, and methods to perform increasingly complex medical procedures on a wide range of patients. However, in the field of minimally invasive surgery, accurately observing the target treatment site within the patient's body, such as tumors or lesions located in the gastrointestinal tract, remains a known challenge. While treatment site images captured by monochrome sensors can provide high-quality contrast definition and spectral flexibility, limitations in imaging methods and devices used for target treatment site image colorization can overload image processors, leading to image processing delays and / or limiting their effectiveness. Summary of the Invention

[0005] This disclosure relates to systems, apparatuses, and methods for providing image processing systems and target extraction logic, intensity normalization logic, and intensity correlation logic. Each aspect disclosed herein may include one or more features described in connection with any other disclosed aspect.

[0006] According to one aspect, a method for generating a color image using a monochrome image sensor is provided. The method includes illuminating a surface sequentially with multiple colors, one color at a time. The monochrome image sensor captures multiple image frames of the surface based on the multiple colors. The multiple image frames are identified, and at least one feature of a target in the multiple image frames is highlighted. The color intensities of the multiple image frames are normalized. A color intensity map of the target is generated for each of the multiple image frames. A correlation score is determined by comparing each color intensity map of the multiple image frames. A color image is generated based on the correlation score.

[0007] Any method described herein may include any of the following steps: A plurality of image frames include a first image frame of at least a first color and a second image frame of a second color. The color intensities of the plurality of image frames are normalized by illuminating a surface with the first color of the plurality of colors. The intensity of the illuminated first color is determined. A normalized value is assigned to the intensity of the first color. The surface is illuminated with the second color of the plurality of colors. The intensity of the illuminated second color is determined. A normalized intensity value for the second color is generated based on the normalized value. The plurality of colors include at least one of red, green, or blue. At least one feature in the target is highlighted by applying an edge filter to extract at least one edge of at least one feature. The color intensity map includes a plurality of pixels, each of which has a color intensity value. Based on the color intensity maps of the plurality of image frames, targets in each of the plurality of image frames are compared. When a correlation score is higher than a predetermined threshold, matching pixels of the target between the first color intensity map and the second color intensity map are determined. The plurality of image frames are downsampled. After downsampling the plurality of frames, a correlation score is determined. The plurality of frames are downsampled by a factor of two at least twice. Peak intensity clusters in the color intensity map are determined. Target motion is estimated by comparing peak intensity clusters in a first color intensity map and a second color intensity map. A correlation score is determined within 150 milliseconds. The surface includes gastrointestinal tissue.

[0008] According to one aspect, a medical device includes an axis, a monochrome image sensor coupled to a distal end of the axis, and at least one illumination device coupled to the distal end of the axis. The at least one illumination device is configured to emit multiple colors, one color at a time. The medical device further includes one or more computer-readable media storing instructions for performing image processing using the monochrome image sensor, and one or more processors configured to execute the instructions to perform the image processing. The one or more processors are configured to sequentially illuminate a surface with the multiple colors. The monochrome image sensor captures multiple image frames of the surface based on the multiple colors. The one or more processors identify targets in the multiple image frames. The one or more processors highlight at least one feature of the targets in the multiple image frames. The one or more processors normalize the color intensities of the multiple image frames. The one or more processors generate a color intensity map of the target for each of the multiple image frames. The one or more processors determine a correlation score by comparing each color intensity map of the multiple image frames. The one or more processors generate a color image based on the correlation score.

[0009] Any medical device described herein may include any of the following features: At least one illumination device is configured to selectively emit at least one color among red, blue, and green. One or more processors normalize color intensity by illuminating a surface with a first color from a plurality of colors, determining the intensity of the illuminated first color, and assigning a normalized value to the intensity of the first color. One or more processors normalize the color intensity of a plurality of image frames by illuminating a surface with a second color from a plurality of colors, determining the intensity of the illuminated second color, and generating a normalized intensity value for the second color based on the normalized value.

[0010] According to one aspect, a non-transitory computer-readable medium stores instructions for performing image processing using a monochrome image sensor. When executed by one or more processors, the instructions cause the one or more processors to perform operations. The one or more processors illuminate a surface sequentially with multiple colors, one color at a time. The monochrome image sensor captures multiple image frames of the surface based on the multiple colors. The one or more processors identify targets in the multiple image frames. The one or more processors highlight at least one feature of the targets in the multiple image frames. The one or more processors normalize the color intensities of the multiple image frames. The one or more processors generate a color intensity map of the target for each of the multiple image frames. The one or more processors determine a correlation score by comparing each color intensity map of the multiple image frames. The one or more processors generate a color image based on the correlation score.

[0011] It is understood that the foregoing general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the scope of the claimed invention. Attached Figure Description

[0012] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate exemplary aspects of this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0013] Figure 1 This is a schematic diagram of an exemplary medical system based on various aspects of this disclosure.

[0014] Figure 2 This explains the use of various aspects of this disclosure. Figure 1 An exemplary process by which a medical system images a target area.

[0015] Figure 3 This explains the various aspects of the use of this disclosure. Figure 1 An exemplary process for detecting motion in images captured by a medical system.

[0016] Figures 4A-4B This explains the various aspects of the use of this disclosure. Figure 1Another exemplary process for detecting motion in graphics captured by a medical system.

[0017] Figure 5 This describes the use of various aspects of this disclosure. Figure 1 A flowchart of an exemplary method for imaging a target area using a medical system. Detailed Implementation

[0018] Examples of this disclosure include systems, apparatus, and methods for enhancing images of one or more target treatment sites within a subject (e.g., a patient) by capturing images using one or more monochrome sensors and identifying one or more features of the target site (e.g., blood vessels, other features of the vascular system, tissue features, abnormalities, etc.) to precisely colorize the captured images. Reference will now be made in detail to various aspects of this disclosure, examples of which are illustrated in the accompanying drawings. Wherever possible, the same or similar reference numerals will be used in the drawings to refer to the same or similar parts. The term “distal” refers to the part furthest from the user when the device is introduced into the patient. In contrast, the term “proximal” refers to the part closest to the user when the device is placed into the subject. As used herein, the terms “comprising,” “including,” or any other variation thereof are intended to cover non-exclusive inclusion, and thus a process, method, article, or apparatus that comprises a list of elements does not necessarily include only those elements, but may include other elements not expressly listed or inherent to such a process, method, article, or apparatus. The term “exemplary” is used in the sense of “example” rather than “ideal.” As used herein, the terms “about,” “substantially,” and “approximately” indicate a value within a range of + / -10% of the stated value.

[0019] Examples of this disclosure can be used to identify target sites within a subject by generating a processed image based on multiple image frames of a multimodal spectrum captured by one or more monochrome image sensors of a medical system. In some embodiments, the medical device may include an image processing apparatus comprising a processor and a memory storing one or more executable instructions and an algorithm for detecting motion of features of the target site. Furthermore, the processor and memory may generate relative pixel blocks for coloring based on motion of features of the target site detected in multiple image frames of a multimodal spectrum captured by one or more monochrome image sensors. In embodiments, the memory may include programmable and executable instructions based on imaging logic, target extraction logic, intensity normalization logic, and intensity correlation logic. Additionally, the image processing apparatus may include a user interface operable to receive user input thereon. The processed image generated by the image processing apparatus of the medical device may include color-resolution frames of pixel values ​​that can be output to a display device.

[0020] Examples of this disclosure may relate to systems, devices, and methods for performing various medical procedures and / or treating the large intestine (colon), small intestine, cecum, esophagus, any other part of the gastrointestinal tract, and / or any other suitable patient anatomy (collectively, “target treatment site”). The various examples described herein include single-use or disposable medical devices. Reference will now be made in detail to the examples of this disclosure described above and illustrated in the accompanying drawings. Wherever possible, the same reference numerals will be used throughout the drawings to refer to the same or similar parts.

[0021] Figure 1 A schematic description of an exemplary medical system 100 according to one example of this disclosure is shown. The medical system 100 may include one or more light sources 130, an image processing device 101, a medical device 110, and a medical device 140. The image processing device 101 may be communicatively coupled to the medical device 110 via, for example, a wired connection, a wireless connection, etc. In this example, the image processing device 101 is a computer system comprising multiple hardware components that allow the image processing device 101 to receive data (e.g., image sensor data), process information (e.g., intensity, motion, or spectral data), and / or generate processed images for output to a user of the medical system 100. The illustrative hardware components of the image processing device 101 may include at least one processor 102, at least one memory 103, at least one user interface 108, and at least one display 109.

[0022] The processor 102 of the image processing device 101 may include any computing device capable of executing machine-readable instructions, which may be stored on a non-transitory computer-readable medium, such as the memory 103 of the image processing device 101. For example, the processor 102 may include a controller, integrated circuit, microchip, computer, and / or any other computer processing unit operable to perform computational and logical operations required to execute a program. As described in more detail herein, the processor 102 is configured to perform one or more operations according to instructions stored in the memory 103.

[0023] Still referencing Figure 1The memory 103 of the image processing device 101 may include a non-transitory computer-readable medium on which machine-readable instructions, such as imaging logic 104, target extraction logic 105, intensity normalization logic 106, and intensity correlation logic 107, are stored. The imaging logic 104 may include executable instructions or algorithms that allow the medical system 100 to capture digital images (e.g., raw digital images) by actuating one or more components of the medical device 110, such as one or more image sensors 150 and one or more light sources 130. The one or more image sensors 150 may include, for example, one or more monochrome image sensors. The one or more light sources 130 may be configured to emit white light, colored light (e.g., red, blue, and green), ultraviolet light, near-infrared (NIR) light, and / or various other wavelengths of light within or outside the visible spectrum. The one or more light sources 130 may be one or more light-emitting diodes (hereinafter referred to as LEDs). Furthermore, the image sensors 150 (or one or more image sensors) of the medical device 110 may be communicatively coupled to the image processing device 101 of the medical system 100, for example, via wired connections, wireless connections, and / or similar means. The image sensor 150 of the medical device 110 can be configured and operable to capture raw images (e.g., digital images) of the environment surrounding the tip 122 of the axis 120.

[0024] In one embodiment, the image sensor 150 may include a photoelectric sensor array (not shown) configured and operable to convert a light beam received by the photoelectric sensor array into an electric current. For example, an electric current may be generated by the photoelectric sensor array arranged on the image sensor 150 when photons from the received light are absorbed by a plurality of photosensitive elements (not shown) arranged on the photoelectric sensor array. Furthermore, each of the plurality of photosensitive elements is operable to receive, capture, and absorb incident light of different wavelengths at photosensitive element locations along the surface of the photoelectric sensor array. Thus, the plurality of photosensitive elements can capture the incident light and generate an electrical signal, which is quantized and stored as a value in the resulting processed image file. It should be understood that the photoelectric sensor array may include various suitable shapes, sizes, and / or configurations.

[0025] Still referencing Figure 1The target extraction logic 105 may include executable instructions or algorithms that allow the medical system 100, for example, to identify features of a target site on a subject. The intensity normalization logic 106 may include executable instructions or algorithms that allow the medical system 100, for example, to normalize image data obtained from multiple frames of a multimodal spectrum. The intensity correlation logic 107 may include executable instructions or algorithms that allow the medical system 100, for example, to generate pixel-to-pixel intensity maps and identify the best-fit match between the generated intensity maps to obtain relative pixel blocks for coloring an image captured by a monochrome image sensor.

[0026] In some embodiments, imaging logic 104, target extraction logic 105, intensity normalization logic 106, and / or spatial correlation logic 107 may include executable instructions and algorithms that allow the medical system 100 to automatically perform periodic image processing of the target site without requiring user input. In other embodiments, the image processing device 101 may be configured, for example, to receive user input from a user interface 108 of the image processing device 101 to initiate image processing of the target site. It should be understood that in some embodiments, the user interface 108 may be a device integrated with the image processing device 101, while in other embodiments, the user interface 108 may be a remote device communicating with the image processing device 101 (e.g., wirelessly, wired, etc.), including switches, buttons, or other inputs on the medical instrument 110.

[0027] It should be understood that various programming algorithms and data supporting the operation of the medical system 100 may reside wholly or partially in the memory 103. The memory 103 may include any type of computer-readable medium suitable for storing data and algorithms, such as, for example, random access memory (RAM), read-only memory (ROM), flash memory, hard disk, and / or any device capable of storing machine-readable instructions. The memory 103 may include one or more datasets, including but not limited to image data from one or more components of the medical system 100 (e.g., medical device 110, medical equipment 140, etc.).

[0028] Still referencing Figure 1The medical device 110 may be configured to facilitate the positioning of one or more components of the medical system 100, such as, for example, medical device 140, relative to a subject (e.g., a patient). In some embodiments, the medical device 110 may be any type of endoscope, duodenoscope, gastroscope, colonoscope, ureteroscope, bronchoscope, catheter, or other delivery system, and may include a handle 112, an actuation mechanism 114, at least one port 116, and a shaft 120. The handle 112 of the medical device 110 may have one or more cavities (not shown) in communicative communication with cavities of one or more other components of the medical system 100. The handle 112 further includes at least one port 116 that opens into one or more cavities of the handle 112. As described in further detail herein, the at least one port 116 is sized and shaped to receive one or more instruments, such as medical device 140 of the medical system 100.

[0029] The shaft 120 of the medical device 110 may include a sufficiently flexible tube such that the shaft 120 is configured to selectively bend, rotate, and / or twist when inserted into and / or through the tortuous anatomy of a subject to a target treatment site. The shaft 120 may have one or more cavities (not shown) extending therethrough, including, for example, a working cavity for receiving instruments (e.g., medical device 140). In other examples, the shaft 120 may include additional cavities, such as a control line cavity for receiving one or more control lines for driving one or more distal components / tools (e.g., articulated joints, lifters, etc.), a fluid cavity for delivering fluid, an illumination cavity for receiving at least a portion (not shown) of an illumination assembly, and / or an imaging cavity for receiving at least a portion (not shown) of an imaging assembly.

[0030] Still refer to Figure 1 The medical device 110 may further include a tip 122 located at the distal end of the shaft 120. In some embodiments, the tip 122 may be attached to the distal end of the shaft 120, while in other embodiments, the tip 122 may be integral with the shaft 120. For example, the tip 122 may include a cap configured to receive the distal end of the shaft 120 therein. The tip 122 may include one or more openings that communicate with one or more cavities of the shaft 120. For example, the tip 122 may include a working opening 123 through which the medical device 140 can be withdrawn from the working cavity of the shaft 120. It should be understood that one or more other openings at the tip 122 of the shaft 120 are not shown. An actuation mechanism 114 of the medical device 110 is disposed on a handle 112 and may include one or more knobs, buttons, levers, switches and / or other suitable actuators. The actuation mechanism 114 is configured to at least control the deflection of the shaft 120 (e.g., actuation via a control line).

[0031] The medical device 140 of the medical system 100 may include a catheter having a longitudinal body 142 between a proximal end 141 and a distal end 144 of the medical device 140. The longitudinal body 142 of the medical device 140 may be flexible, such that the medical device 140 is configured to bend, rotate, and / or twist when inserted into the working cavity of the medical instrument 110. The medical device 140 may include a handle at the proximal end 141 of the longitudinal body 142, which may be configured to move, rotate, and / or bend the longitudinal body 142. Furthermore, the handle at the proximal end 141 of the medical device 140 may define one or more ports (not shown) sized to receive one or more tools through the longitudinal body 142 of the medical device 140.

[0032] Still referencing Figure 1 The medical device 110 can be configured to receive the medical device 140 via at least one port 116, through a working cavity passing through the shaft 120, and reaching a working opening 123 at a tip 122. In this configuration, the medical device 140 can extend distally from the working opening 123 and into the surrounding environment of the tip 122, for example, at a target treatment site on a subject, as described in further detail below. A distal end 144 of the medical device 140 can extend distally from the tip 122 in response to translation of the longitudinal body 142 through the working cavity of the shaft 120. The medical device 140 may include one or more end effectors (not shown) at the distal end 144 of the longitudinal body 142 for performing one or more operations at the target treatment site.

[0033] In one embodiment, the medical device 110 may be further configured to receive one or more light sources 130 via at least one in a cavity of the medical device 110 through an axis 120 for connection to an optical fiber 146. In this example, the one or more light sources 130 are shown as components separate from the image processing device 101, such that the light sources 130 are separately coupled to the medical device 101 (e.g., via cable). It should be understood that in other embodiments, the one or more light sources 130 may be included on the image processing device 101, such that the light sources 130 are communicatively coupled to the medical device 110 with respect to the image processing device 101.

[0034] Still referencing Figure 1The tip 122 of the medical device 110 may include an optical fiber 146 and an image sensor 150 at the tip 122. In one embodiment, the optical fiber 146 may be coupled to one or more light sources 130 of the medical system 100, such that each of the one or more light sources 130 can transmit light through a single optical fiber 146. Although not shown, it should be understood that multiple light sources 130 may be coupled to the optical fiber 146 via an optical fiber splitter / combiner. The optical fiber 146 of the medical device 110 may be configured and operable to transmit amplitudes of various light from one or more light sources 130 distally from the tip 122 of the axis 120. In some embodiments, the optical fiber 146 may be configured to transmit white light, ultraviolet light, near-infrared (NIR) light, and / or various other wavelengths within or outside the visible spectrum.

[0035] In other embodiments, the medical device 110 may include (though not shown) a multicolor LED assembly at the tip 122 of the shaft 120. For example, the multicolor LED assembly may include one or more LEDs arranged in a ring array surrounding the image sensor 150. Each LED may be configured and operable to transmit different wavelengths and / or amplitudes of light relative to each other. It should be understood that different illumination sources can produce different spectra (e.g., red, green, and blue colors).

[0036] In other embodiments, as further described herein, the image sensor 150 may be configured and operable to fully capture all incident light at each individual pixel location of the image sensor 150, regardless of the color of the incident light.

[0037] Still referencing Figure 1 The medical device 110 of the medical system 100 can be inserted into the body of a subject (not shown) to position the tip 122 adjacent to the target site 201 (to be discussed later). Figure 2 (As shown in the diagram). For example, shaft 120 can be guided through the digestive tract of a subject (e.g., a patient) by inserting tip 122 into the nose or mouth (or other suitable natural orifice) of the subject's body, and through the gastrointestinal tract (e.g., esophagus, stomach, small intestine, etc.) until reaching the target site. It should be understood that the length of shaft 120 may be sufficient such that the proximal end of medical device 110 (including handle 112) is outside the subject, while tip 122 of medical device 110 is inside the subject's body. While this disclosure relates to the use of medical system 100 in the digestive tract of a subject, it should be understood that the features of this disclosure can be used in various other locations within the subject's body (e.g., other organs, tissues, etc.).

[0038] Figure 2 It shows how to use according to Figure 1A schematic diagram of the multimodal image capture and relative pixel block generation process 200 of one or more image sensors 150 (hereinafter referred to as monochrome image sensors 150) in a disclosed medical system 100. Compared to image sensors with color filters, monochrome image sensors can achieve higher quantum efficiency (e.g., sensitivity) and have the potential to produce better contrast sharpness and spectral flexibility when converting images captured by monochrome image sensors 150 into color images. Still referring to... Figure 2 In step 210, the monochrome image sensor 150 can capture one or more images of a target site 201 (e.g., esophagus, stomach, small intestine, other organs, tissues, polyps, etc.) of a subject (e.g., a patient), which can be illuminated by a light source 130 (or a multicolor LED assembly) at the tip 122 of axis 120. For example, the target site 201 of the subject can be illuminated sequentially by the light source 130 (or the multicolor LED assembly) in different colors, such as red, green, and blue (or cyan, magenta, or any other color suitable for generating a color image). Accordingly, the monochrome image sensor 150 can capture images at a higher frame rate (e.g., three times or more) than conventional color imagers to mimic conventional color imagers using full-spectrum illumination.

[0039] exist Figure 2 In one exemplary embodiment, in step 210, when the light source 130 (or multi-color LED assembly) illuminates the target region 201 with, for example, red light during the first frame 202, blue light during the second frame 204, and green light during the third frame 206, the monochrome image sensor 150 can initially capture at least three multi-mode spectral image frames at the target region 201. Each image frame captured during the three frames 202, 204, 206 can be identified by an M x N pixel block. In this case, the image processing device 101 can adjust and adapt the three frames 202, 204, 206 with any motion artifacts of the multi-modal spectrum to provide clear color image reproduction. That is, the target feature 203 defined by the K x L pixel block in the first frame 202 can be identified and tracked in the second frame 204 and the third frame 206 based on the spatial displacement (e.g., motion) of the target feature 203 relative to the tip of the medical instrument 110, as shown by the relative positions of the target features 205 and 207 with respect to the position of the target feature 203. Accordingly, each of the three frames 202, 204 and 206, illuminated with different colors (e.g., red, green, blue, etc.), can be identified by matching K x L pixel blocks of target features 203, 205 and 207, respectively.

[0040] In a given frame of M×N pixels (e.g., frame 202), identifying matching pixel data blocks of a predetermined size (e.g., a K×L pixel block of target feature 203) and corresponding matching pixel data blocks in neighboring frames (e.g., frames 204 or 206) (e.g., K×L pixel blocks of target features 205 or 207) can be used to perform motion estimation and feature extraction, such as tracking relevant key features moving within a given frame while the background remains the same. Conventional motion estimation techniques for imaging target areas typically assume constant intensity across frames. However, in the case of colorized images captured by the monochrome image sensor 150, each captured image frame (e.g., frames 202, 204, or 206) has variable intensity due to the order of polychromatic light (e.g., red, green, blue, etc.). Therefore, applying conventional motion estimation techniques that assume constant intensity to images captured by the monochrome image sensor 150 may not produce an accurate representation of target feature movement / motion (e.g., the spatial displacement of target feature 203 represented by target features 205 and 207).

[0041] In one exemplary embodiment of this disclosure, each of the target features 203, 205, and 207 can be identified according to executable instructions or algorithms stored on the target extraction logic 105. For example, the target extraction logic 105 may include executable instructions and logarithmic, hierarchical, and / or exhaustive block matching algorithms. Furthermore, the target extraction logic 105 may include executable instructions or algorithms for performing edge detection filtering on the target features 203, 205, and 207 to highlight key features in the K x L pixel blocks of the target features 203, 205, and 207. Therefore, in step 220, the image processing device 101 can extract the relevance features 222, 224, and 226 of the K x L pixel blocks by performing edge detection filtering on the K x L pixel blocks of the target features 203, 205, and 207 according to the executable instructions and algorithms stored on the target extraction logic 105. In one embodiment, the extracted correlation feature 222 may include the red color intensity data of the target feature 203, the correlation feature 224 may include the blue color intensity data of the target feature 205, and the correlation feature 226 may include the green color intensity data of the target feature 207.

[0042] Still referencing Figure 2In step 230, the processor 102 can generate inter-block pixel-to-pixel intensity relationship maps 232, 234, and 236 for each of the K x L pixel block correlation features 222, 224, and 226 for edge transformation, based on the executable instructions and algorithms in the intensity correlation logic 107. For example, intensity relationship map 232 may include pixel-by-pixel intensity data of correlation feature 222 including red intensity pixel data, intensity relationship map 234 may include pixel-by-pixel intensity data of correlation feature 224 including green intensity pixel data, and intensity relationship map 236 may include pixel-by-pixel intensity data of correlation feature 226 including blue intensity pixel data. Therefore, the processor 102 can generate, by running intensity relationship maps 232, 234, and 236 and using the spatial correlation filter provided by intensity correlation logic 107, the best-matching relative pixel blocks that may represent each of the target features 203, 205, and 207, according to the executable instructions and algorithms of intensity correlation logic 107. For clarity, the intensity pixel data for each pixel in intensity relationship graphs 232, 234, and 236 are shown in... Figure 2 The intensity is represented in black and white. However, it is understandable that each pixel in intensity relationship diagrams 232, 234, and 236 can be represented in various grayscale values, depending on the intensity detected by each pixel of the monochrome image sensor 150.

[0043] In one embodiment, intensity relationship maps 232, 234, and 236 can be substantially similar (e.g., slightly varied due to the multimodal nature of frames 202, 204, and 206). Furthermore, each of intensity relationship maps 232, 234, and 236 can provide registration. That is, each relative pixel block generated based on the similarity in each intensity relationship map 232, 234, and 236 can be used to align the estimated motion of the frames with new changes in subsequent frames by cropping and overlaying information from prior frames. In one embodiment, processor 102 can use the generated relative pixel blocks to colorize an image of target location 201 captured by monochrome image sensor 150.

[0044] In some embodiments, the processor 102 can accelerate the process by applying a correlation matrix threshold scoring method. Figure 2 The disclosed relative pixel block generation process. According to embodiments of this disclosure, edge filtering is performed on target features 203, 205, and 207, allowing for... Figure 2The intensity relationship graphs 232, 234, and 236 shown create a more efficient dataset. However, associating two sets of data from these intensity relationship graphs 232, 234, and 236 may require searching for similar intensity distribution peaks. Due to the multimodal nature of frames 202, 204, and 206, a perfect positive correlation (e.g., +1) may be difficult to achieve. Accordingly, creating a threshold score, such as a correlation score > +0.75, can ensure faster motion registration matching of target features 203, 205, and 207 while maintaining low video stream latency (e.g., less than 150 ms).

[0045] In one exemplary embodiment according to this disclosure, the processor 102 may generate relative pixel blocks using the following formula:

[0046] Correlation matrix score > +0.75

[0047] For example, processor 102 can generate or assign a correlation score to each of the intensity distribution peaks in intensity relationship graphs 232, 234, and 236, and can then determine any correlation score higher than +0.75 between two or more frames as a matching score. Then, processor 102 can... Figure 2 The disclosed process 200 generates relative pixel blocks. Furthermore, the processor 102 can determine the optimal match for target features 203, 205, and 207 based on intensity relationship maps 232, 234, and 236 and the correlation matrix scoring equation.

[0048] In some embodiments, processor 102 may perform relative intensity matching for generating relative pixel blocks according to embodiments of the present disclosure by normalizing the intensity of different colors (e.g., red, green, blue, etc.) detected in image frames 202, 204, 206, based on executable instructions or algorithms of intensity normalization logic 107. The relative intensity of target features (e.g., target features 203, 205, or 207) within image frames (e.g., frames 202, 204, or 206) may depend on the specific color available for illumination. Therefore, to provide an estimate of the relative intensity between different colors illuminated during frames 202, 204, 206, the response of each individual color channel to healthy tissue can be used. That is, while ignoring the influence of blood vessels and other prominent features in healthy tissue, the relative intensity between color channels can be estimated by applying, for example, the following algorithm:

[0049] The value that normalizes the first color channel (e.g., red) to 1.

[0050] Using the same healthy tissue, processor 102 can estimate the responses of other color channels (e.g., green, blue, etc.) by comparing the responses of other color channels with the response of the first color channel (e.g., red). Processor 102 can then normalize the responses of the other color channels to the same value as the response of the first color channel. Once each color is normalized based on the healthy tissue, normalization coefficients (e.g., normalized color intensity values) can be used for different anatomical structures to estimate the relative intensities of, for example, red, green, and blue channels. Therefore, the relative intensities between frames 202, 204, and 206 can be estimated more accurately to generate relative pixel blocks according to embodiments of this disclosure.

[0051] Figure 3 An exemplary image transformation process for generating relevant pixel blocks using a hierarchical block matching algorithm according to embodiments of the present disclosure is illustrated. In one embodiment, the monochrome imaging sensor 150 can... Figure 2 The disclosed process 200 captures anchor frames 300 and target frames 310 of target sites on a subject (e.g., a patient). In this embodiment, processor 102 can utilize executable instructions and algorithms (e.g., imaging logic 104, target extraction logic 105, intensity normalization logic 106, and / or intensity correlation logic 107) stored in memory to perform operations according to... Figure 3 The image conversion process described above. For example, the processor 101 of the image processing device 102 can determine or identify target features 303 at a first position in the anchor image frame 300 and a corresponding first position in the target frame 310. Further, the processor 102 can determine the motion / spatial displacement of the target features 303 identified in the anchor frame 300 by determining or identifying the displacement target features 315 according to executable instructions or algorithms stored in the target extraction logic 105. Furthermore, the processor 101 can... Figure 2 The process disclosed in the paper generates pixel intensity relationship diagrams 302 and 312 between blocks.

[0052] In an exemplary embodiment, processor 102 may downsample anchor frame 300 and target frame 310, for example, downsample twice, to obtain a first downsampled anchor frame 304 and a first downsampled target frame 314. Processor 101 may further downsample the first downsampled anchor frame 304 and the first downsampled target frame 314 again, for example, downsample twice, to obtain a second downsampled anchor frame 306 and a second downsampled target frame 316. In this case, processor 102 may proceed according to... Figure 2 The disclosed process 200 generates an inter-block pixel and pixel intensity relationship map 302 for each of the anchor frames 300, 304, and 306, representing the target feature 303. Furthermore, in each of the target frames 310, 314, and 316, the processor 102 can identify the displaced target feature 315 and, according to... Figure 2 The process disclosed in section 200 generates a pixel-to-pixel intensity relationship diagram 312 for the target feature 315 of the displacement. (Compared to...) Figure 2 Similarly, for clarity, the intensity pixel data for each pixel in intensity relationship maps 302 and 312 are shown in... Figure 3 The intensity is represented in black and white. However, it is understandable that each pixel in intensity relationship diagrams 302 and 312 can be represented in various grayscale values, depending on the intensity detected by each pixel of the monochrome image sensor 150.

[0053] Still referencing Figure 3 The processor 101 can be based on Figure 2 The disclosed process 200 generates relative pixel blocks based on intensity relationship graphs 302 and 312 at each hierarchical level (e.g., frames 300, 304, and 306, and frames 310, 314, and 316). In one embodiment, a modified relevance matching score may be used at each downsampled hierarchical level. Downsampling of anchor frame 300 and target frame 310, and zooming in to the region of interest to identify features of interest (e.g., target features 303 and 315), reduces the search time within each frame at each level.

[0054] In one embodiment, relative pixel blocks can be generated for each hierarchical level (e.g., frames 300, 304, and 306, and frames 310, 314, and 316). Different relative pixel blocks at each hierarchical level may require the application of [a specific method / approach]. Figure 3 The described hierarchical block matching algorithm. For example, highlight blocks at each level in a relative pixel block can vary from one level to another based on downsampling. That is, at one hierarchical level, quadrants (e.g., frames 306 and 316) of a feature of interest (e.g., 303 and 315) can be searched and located. At another hierarchical level, refinement techniques can be applied to the region of interest (e.g., frames 304 and 314), including the feature of interest (e.g., 303 and 315). At yet another hierarchical level, the actual image frames (e.g., frames 300 and 310) can be obtained based on the region of interest determined at the previous hierarchical level. Therefore, compared to another hierarchical level (e.g., frames 304 and 316), fewer intensity blocks may need to be identified to determine the relative pixel block relevance at one hierarchical level (e.g., frames 306 and 316). Furthermore, as the hierarchical level gets closer to the actual image (e.g., frames 300 and 310), more detailed relative pixel blocks may be generated. Accordingly, the features of interest (e.g., 303, 315) can be quickly determined by downsampling twice (or more) and creating different relative pixel blocks of the quadrants of the features of interest (303 and 315) at each hierarchical level (303, 304, and 306).

[0055] Figure 4A and 4B An exemplary method for determining relative pixel blocks using indirect motion estimation via pixel projection is illustrated. In this embodiment, prior to performing the indirect pixel projection method according to this disclosure, it is possible to... Figure 2 The process 200 generates an intensity relationship map of the target feature image captured by the monochrome image sensor 150. For example, as... Figures 4A-4B As shown, processor 102 can generate a first intensity relationship map 430 with intensity pixel values ​​of a first color (e.g., blue) and a second intensity relationship map 440 with intensity pixel values ​​of a second color (e.g., green). Subsequently, processor 102 can generate two sets of pixel projection data based on the first intensity relationship map 430 and the second intensity relationship map 440. In addition... Figure 2 In addition to process 200, or alternatively, processor 102 can generate relative pixel blocks of target features by identifying peak intensity clusters of two sets of pixel projection data in the columns and rows of the first pixel projection matrix 432 and the second pixel projection matrix 442 and determining the relative similarity between the identified peak intensity clusters. Figure 2 Similarly, for clarity, the intensity pixel data for each pixel in intensity relationship graphs 430 and 440 is represented in black and white in Figure 4. However, it can be understood that each pixel in intensity relationship graphs 430 and 440 can be represented in various grayscale levels, depending on the intensity detected by each pixel of the monochrome image sensor 150.

[0056] In one exemplary embodiment, the rows and columns of projection matrices 423 and 442 can be summed to determine similar pixel densities between matrices 423 and 442. For example, in Figure 4A In the first pixel projection matrix 432, summing all the black pixels in the rows of the first intensity relationship map 430 can represent the position of the pixel intensity distribution along the vertical axis. Furthermore, summing all the black pixels in the columns of the first intensity relationship map 430 can provide the position of the pixel intensity distribution along the horizontal axis. A similar process can be performed... Figure 4B The process is executed on the second pixel projection matrix 442. Then, the processor 102 can recognize... Figure 4A and Figure 4B The differences between intensity pixel blocks in the intensity relationship diagrams (e.g., the movement of pixel positions as shown in intensity relationship diagrams 430 and 440) are used to generate relative pixel blocks. Therefore, intensity clusters in intensity relationship diagrams 430 and 440 can be indirectly identified using the above-described methods and techniques to generate relative pixel blocks.

[0057] Generally, monochrome image sensors capture only monochrome images. Therefore, an image captured by a monochrome image sensor 150 can mimic a color image sensor by colorizing the captured image based on variable pixel intensities detected during illumination by colored light (e.g., red, green, and red light). Figure 5 A flowchart of an exemplary method 500 is shown, which generates a colorized image of a target site of a subject using a monochrome image sensor 150. In step 502, a light source 130 of the medical system 100 may sequentially illuminate a surface (e.g., the target site of a patient) with multiple colors, one color at a time. In some embodiments, the multiple colors may include at least one of red, green, or blue. In one embodiment, the surface may include the patient's gastrointestinal tissue. In step 504, the monochrome image sensor 150 may capture multiple image frames of the surface based on the multiple colors. In one embodiment, the multiple image frames may include at least one first image frame of a first color, a second image frame of a second color, and a third image frame of a third color. In one embodiment, the first color may be red, the second color may be green, and the third color may be blue. In step 506, the processor 102 may identify the target in the multiple image frames according to one or more of the imaging logic 104, target extraction logic 105, intensity normalization logic 106, and intensity correlation logic 107 in the memory 103. Furthermore, the position of the target in the first frame of the multiple image frames may differ from the position of the target in the second frame of the multiple image frames. Processor 102 may utilize one or more logics in memory 103 (e.g., imaging logic 104, target extraction logic 105, intensity normalization logic 106, and intensity correlation logic 107) to perform the steps described below. In step 508, processor 102 may highlight at least one feature (e.g., target features 203, 205, and / or 207) of a target in a plurality of image frames (e.g., frames 202, 204, and 206). In one embodiment, processor 102 may apply an edge filter to extract at least one edge from the at least one feature.

[0058] In step 510, processor 102 can normalize the color intensity of multiple image frames. In one embodiment, to normalize the color intensity of multiple image frames, light source 130 can illuminate the surface with a first color (e.g., red) from among multiple colors. Processor 102 can then determine the intensity of the illuminated first color and assign a normalized value to the intensity of the first color. Subsequently, light source 130 can illuminate the surface with a second color (e.g., green) from among multiple colors. Processor 102 can then determine the intensity of the illuminated second color and generate a normalized intensity value for the second color based on the normalized value.

[0059] In step 512, processor 102 may generate a color intensity map (e.g., intensity relationship maps 232, 234, or 236) of the target for each of a plurality of image frames (e.g., frames 202, 204, and 206). In one embodiment, the color intensity map may include a plurality of pixels, and each of the plurality of pixels may include a color intensity value. Further, the target in each of the plurality of image frames may be compared based on the color intensity maps of the plurality of image frames. In step 514, processor 102 may determine a correlation score by comparing each color intensity map in the plurality of image frames. In one embodiment, processor 102 may identify peak intensity clusters in the color intensity maps and estimate the motion of the target by comparing the peak intensity clusters in a first color intensity map and a second color intensity map.

[0060] In step 516, processor 102 may generate a color image based on the correlation score. For example, the correlation score may be determined in less than 150 milliseconds. In one embodiment, when the correlation score is higher than a predetermined threshold, processor 102 may determine matching pixels of the target between the first color intensity map and the second color intensity map. Additionally, or alternatively, processor 102 may downsample multiple frames and determine the correlation score after downsampling multiple frames. In one embodiment, processor 102 may downsample multiple frames by a factor of two, at least twice.

[0061] See again Figure 1 The figure illustrates a display 109 of a medical system 100 communicatively coupled to a processor 102 of an image processing device 101. The processor 102 is operable to transmit colorized images generated according to methods 200 and 500 to the display 109 for viewing by a user of the medical system 100. In some examples, the medical system 100 may be configured and operable to continuously perform the methods 200 and 500 shown and described herein, such that the display 109 may output multiple colorized images to provide continuous (e.g., live, real-time) imaging of one or more target objects.

[0062] Each of the aforementioned systems, devices, components, and methods can be used to detect the movement of target features in a target area of ​​a subject and generate a colorized image of the target area. By providing a medical device including an image processing system, a user can enhance the visualization of one or more features and / or properties of a target area within a subject's body during surgery using a monochrome image sensor. This medical device allows the user to accurately determine the location of the target area, thereby reducing overall surgical time, improving surgical efficiency, and avoiding unnecessary harm to the subject's body due to inaccurate positioning of the target object at the target treatment site.

[0063] It will be apparent to those skilled in the art that various modifications and variations can be made to the disclosed apparatus and methods without departing from the scope of this disclosure. It should be understood that the disclosed apparatus may include various suitable computer systems and / or computing units comprising multiple hardware components, such as processors and non-transitory computer-readable media, which allow the apparatus to perform one or more operations as described herein during a program. Other aspects of this disclosure will be apparent to those skilled in the art upon consideration of the description and practice of the features disclosed herein. The specification and examples are intended to be illustrative only.

[0064] It should be understood that Figure 1 The image processing device 101 can be any computing device. The image processing device 101 may also include input and output ports for connection to input and output devices such as a keyboard, mouse, touchscreen, monitor, display, etc. Of course, various system functions can be implemented in a distributed manner on similar platforms to distribute the processing load. Alternatively, these systems can be implemented through appropriate programming of a computer hardware platform.

[0065] In one embodiment, any of the disclosed systems, methods, and / or graphical user interfaces may be executed or implemented by a computing system conforming to or similar to that described herein. While not strictly required, aspects of this disclosure are described in the context of computer-executable instructions, such as routines executed by a data processing device (e.g., a server computer, wireless device, and / or personal computer). Those skilled in the art will understand that aspects of this disclosure can be practiced with other communication, data processing, or computer system configurations, including: internet devices, handheld devices (including personal digital assistants (“PDAs”), wearable computers, various cellular or mobile phones (including Voice over IP (“VoIP”) phones), dumb terminals, media players, gaming devices, virtual reality devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, set-top boxes, network PCs, minicomputers, mainframes, etc. In practice, the terms “computer,” “computing device,” etc., are generally used interchangeably herein and refer to any of the aforementioned devices and systems, as well as any data processor.

[0066] The aspects of this disclosure may be embodied in a dedicated computer and / or data processor specifically programmed, configured, and / or constructed to execute one or more computer-executable instructions as detailed herein. While aspects of this disclosure, such as certain functions, are described as performing only on a single device, this disclosure can also be practiced in a distributed environment where functions or modules are shared among different processing devices linked by a communication network, such as a local area network (“LAN”), a wide area network (“WAN”), and / or the Internet. Similarly, techniques presented herein involving multiple devices can be implemented in a single device. In a distributed computing environment, program modules may reside in local and / or remote memory storage devices.

[0067] Various aspects of this disclosure may be stored and / or distributed on non-transitory computer-readable media, including magnetically or optically readable computer optical discs, hardwired or pre-programmed chips (e.g., EEPROM semiconductor chips), nanotechnology memories, biological memories, or other data storage media. Alternatively, computer-implemented instructions, data structures, screen displays, and other data according to various aspects of this disclosure may be made available via the Internet and / or other networks (including wireless networks), on propagation media (e.g., electromagnetic waves, sound waves, etc. over a period of time), and / or they may be made available on any analog or digital network (packet-switched, circuit-switched, or other schemes).

[0068] The programmatic aspect of this technology can be considered a "product" or "manufactured item," typically appearing in the form of executable code and / or associated data, carried or embodied on some type of machine-readable medium. "Storage" type media includes any or all tangible memory of a computer, processor, or similar device, or related modules thereof, such as various semiconductor memories, tape drives, disk drives, etc., which provide non-transitory storage for software programming at any time. All or part of the software content can sometimes be communicated via the Internet or other various telecommunications networks. For example, such communication can enable software to be loaded from one computer or processor to another, such as from a management server or host of a mobile communication network to a server's computer platform and / or from a server to a mobile device. Therefore, another possible medium for carrying software elements includes light waves, radio waves, and electromagnetic waves (such as those used on physical interfaces between local devices) via wired and optical fixed-line telephone networks and via various air links. Physical elements carrying such waves, such as wired or wireless links, optical links, etc., can also be considered as media carrying software. As used herein, unless limited to non-transitory, tangible "storage" media, the term "readable medium" for a computer or machine refers to any medium that participates in providing instructions to a processor for execution.

[0069] Other embodiments of the invention will be apparent to those skilled in the art in light of the description and practice of the invention disclosed herein. The description and embodiments are intended to be considered exemplary only, and the true scope and spirit of the invention are indicated by the appended claims.

[0070] It should be understood that one or more aspects of any medical device described herein can be used in conjunction with any other medical device known in the art (such as a medical imaging system) or other endoscope (such as a colonoscope, bronchoscope, ureteroscope, duodenoscope, etc.) or other type of imager.

[0071] While the principles of this disclosure have been described herein with reference to illustrative examples of specific applications, it should be understood that this disclosure is not limited thereto. Those skilled in the art and who can appreciate the teachings provided herein will recognize that additional modifications, applications, and substitutions of equivalents are within the scope of the examples described herein. Therefore, this invention should not be considered as limited to the foregoing description.

Claims

1. A method for generating a color image using a monochrome image sensor, the method comprising: The surface is illuminated with multiple colors sequentially, one color at a time. Based on the multiple colors, multiple image frames of the surface are captured by a monochrome image sensor; Identify targets in the multiple image frames; Highlight at least one feature of the target in the plurality of image frames; Normalize the color intensity of the multiple image frames; Generate a color intensity map of the target for each of the plurality of image frames; Relevance scores are identified by comparing each color intensity map of the plurality of image frames; as well as The color image is generated based on the correlation score.

2. The method of claim 1, wherein the plurality of image frames includes at least a first image frame of a first color and a second image frame of a second color.

3. The method as described in claim 1 or 2, wherein normalizing the color intensity of the plurality of image frames comprises: The surface is illuminated with the first color of the plurality of colors; Determine the intensity of the first color that is being illuminated; as well as A normalized value is assigned to the intensity of the first color.

4. The method of claim 3, wherein normalizing the color intensity of the plurality of image frames further comprises: The surface is illuminated with the second color from the plurality of colors; Determine the intensity of the second color that is being illuminated; as well as The normalized intensity value of the second color is generated based on the normalized value.

5. The method of claim 1 or 2, wherein the plurality of colors includes at least one of red, green or blue.

6. The method of claim 1 or 2, wherein highlighting at least one feature of the target comprises: An edge filter is applied to extract at least one edge from the at least one feature.

7. The method of claim 1 or 2, wherein the color intensity map comprises a plurality of pixels; and Each of the plurality of pixels has a color intensity value.

8. The method of claim 1 or 2, wherein the target in each of the plurality of image frames is compared based on the color intensity map of the plurality of image frames.

9. The method of claim 1 or 2, further comprising: When the correlation score is higher than a predetermined threshold, the matching pixels of the target between the first color intensity map and the second color intensity map are determined.

10. The method of claim 1 or 2, further comprising: Downsampling is performed on the multiple image frames; as well as After downsampling the multiple image frames, the correlation score is determined.

11. The method of claim 1 or 2, further comprising: The multiple frames are downsampled at least twice with a factor of two.

12. The method of claim 1 or 2, further comprising: Identify the peak intensity clusters in the color intensity map; as well as The motion of the target is estimated by comparing the peak intensity clusters in the first color intensity map and the second color intensity map.

13. The method of claim 1 or 2, wherein the position of the target in the first frame of the plurality of frames is different from the position of the target in the second frame of the plurality of frames.

14. The method of claim 1 or 2, wherein the correlation score is determined within 150 milliseconds.

15. The method as described in claim 1 or 2, wherein the surface comprises gastrointestinal tissue.

Images