Endoscope imaging device and method, electronic equipment and storage medium

Abstract

The invention discloses an endoscope imaging device and method, electronic equipment and a storage medium, and relates to the technical field of image processing. The device comprises an illumination module which is used for providing an illumination light source and comprises a plurality of narrow-band light sources with different colors, and the plurality of narrow-band light sources comprise main spectral bands forming white light; the imaging module is used for imaging the reflected light after the illumination module irradiates the target part of the patient to obtain an RGB image; the processing module is used for decomposing the RGB image into a white light image and a plurality of non-white-light narrow-band images, performing enhancement processing on the plurality of non-white-light narrow-band images, and fusing the non-white-light narrow-band images with the white light image to obtain a narrow-band enhanced target image; wherein the plurality of non-white light narrow-band images respectively represent information formed by different tissue parts for absorbing narrow-band light with different wavelengths. Therefore, on the basis of a full-narrow-band illumination light source and an image decomposition-enhancement-fusion technology, the impression of white light hue is taken into consideration on the premise that focus details are highlighted through imaging.

Description

Technical Field

[0001] The present application relates to the field of image recognition, and in particular to an endoscopic imaging device, method, electronic device and storage medium. Background of the Invention

[0003] An endoscope is an optical instrument that consists of a cold light source lens, fiber optic wires, an image transmission system, a screen display system, etc. When in use, the doctor guides the endoscope into the patient's position to be examined, can directly detect changes in the relevant parts, and cooperate with surgical instruments to perform necessary diagnosis and treatment. In endoscope application scenarios, especially for gastrointestinal endoscopic examination, diagnosis and treatment, doctors usually use white light imaging mode that conforms to human vision for routine operations to obtain white light images, and when further exploring lesions, switch the white light imaging mode to narrow-band light imaging mode to obtain narrow-band light images.

[0004] Among them, white light images can well maintain colors that are consistent with the human eye's perception, but their wide spectrum characteristics have poor enhancement performance for specific tissues of organisms and cannot highlight detailed lesions; narrow-band light images can effectively highlight detailed lesions because the narrow-band band has strong absorption and reflection characteristics for specific tissues of organisms, but their lighting source only covers part of the visible spectrum band or changes the proportion of the white light spectrum, and the light band is incomplete, resulting in obvious differences in color tone from white light images.

[0005] Therefore, how to ensure that the imaging highlights the details of the lesion while taking into account the visual perception of the white light tone is an urgent problem to be solved. Summary of the invention

[0006] The embodiments of the present application provide an endoscopic imaging device, method, electronic device and storage medium, which are used to solve the problem that the prior art cannot ensure that the imaging highlights the details of the lesion while taking into account the visual effect of the white light tone.

[0007] On the one hand, an embodiment of the present application provides an endoscopic imaging device, comprising:

[0008] A lighting module, used for providing a lighting source, including a plurality of narrow-band light sources of different colors, wherein the plurality of narrow-band light sources contain a main spectral band constituting white light;

[0009] An imaging module, used to image the reflected light after the illumination module irradiates the target part of the patient, to obtain an RGB image;

[0010] The processing module is used to decompose the RGB image into a white light image and multiple non-white light narrow-band images, and after enhancing the multiple non-white light narrow-band images, fuse them with the white light image to obtain a narrow-band enhanced target image; wherein the multiple non-white light narrow-band images respectively represent information formed by different tissue parts that absorb narrow-band light of different wavelengths.

[0011] In some embodiments of the present application, the plurality of narrow-band light sources include a narrow-band blue-violet light source, a narrow-band blue light source, a narrow-band green light source, a narrow-band amber light source, and a narrow-band red light source.

[0012] In some embodiments of the present application, the processing module includes:

[0013] A decomposition unit, used for decomposing the RGB image into a white light image and a plurality of non-white light narrow band images, wherein main components of the plurality of non-white light narrow band images are respectively derived from a blue channel, a green channel, and a red channel of the RGB image;

[0014] An enhancement unit, used to perform image enhancement processing on the multiple non-white light narrow-band images respectively to obtain multiple non-white light narrow-band enhanced images;

[0015] The fusion unit is used to fuse the white light image with a plurality of non-white light narrow-band enhanced images to obtain a target image.

[0016] In some embodiments of the present application, the decomposition unit is used to: decompose an RGB image into a white light image and multiple non-white light narrow-band images, the multiple non-white light narrow-band images include a shallow image, a middle image, and a deep image; wherein the main component of the shallow image is derived from the blue channel, representing the information formed by the surface mucosa and capillary tissue with strong absorption characteristics for short-wavelength narrow-band light; the main component of the middle image is derived from the green channel, representing the information formed by the middle blood vessels and inflammatory sites with strong absorption characteristics for medium-wavelength narrow-band light; the main component of the deep image is derived from the red channel, representing the information formed by the deep blood vessels and bleeding points with strong absorption characteristics for long-wavelength narrow-band light.

[0017] In some embodiments of the present application, the enhancement unit is used to: perform image enhancement processing on multiple non-white light narrow-band images respectively to obtain multiple non-white light narrow-band enhanced images; wherein the image enhancement processing includes a global linear enhancement method, a global nonlinear enhancement method, and a local enhancement method, and the global nonlinear enhancement method is implemented based on a curve mapping function, including a gamma curve, an S-curve, a log curve, and a least squares fitting curve.

[0018] In some embodiments of the present application, the fusion unit is used to: fuse the detail components and brightness components of multiple non-white light narrow-band enhanced images, and the color components of the white light image into a target image; or, fuse the detail components of multiple non-white light narrow-band enhanced images, and the brightness components and color components of the white light image into a target image.

[0019] In some embodiments of the present application, the fusion unit is used to: assign multiple non-white light narrow-band enhanced images to the blue channel, green channel and red channel of the RGB image, respectively, to obtain a combined RGB image; convert the combined RGB image and the white light image to color spaces in which color and brightness are relatively independent, respectively, to obtain a first channel containing brightness and detail information, a second channel containing color information, and a third channel containing saturation information; after combining the first channel, the second channel and the third channel, convert them to the RGB space to obtain the target image.

[0020] In some embodiments of the present application, the fusion unit is used to: extract the detail components of each of multiple non-white light narrow-band enhanced images, and convert the white light image into a color space in which color and brightness are relatively independent; use the detail components to enhance the color channel or brightness channel of the color space to obtain an enhanced color space; and convert the enhanced color space to RGB space to obtain a target image.

[0021] In some embodiments of the present application, the fusion unit is used to: assign the non-white light narrow-band enhanced image to the blue channel, green channel and red channel of the RGB image respectively to obtain a combined RGB image; and fuse the combined RGB image with the white light image according to a preset ratio to obtain a target image.

[0022] On the one hand, an embodiment of the present application provides an endoscopic imaging method, which is applied to any of the above-mentioned endoscopic imaging devices, and the method includes:

[0023] Acquire an RGB image, the RGB image being obtained based on reflected light after the illumination module irradiates the target part of the patient; wherein the illumination module is used to provide an illumination light source, including a plurality of narrow-band light sources of different colors, the plurality of narrow-band light sources including a main spectral band constituting white light;

[0024] Decompose the RGB image into a white light image and multiple non-white light narrow band images;

[0025] After multiple non-white light narrow-band images are enhanced, they are fused with the white light image to obtain a narrow-band enhanced target image; wherein the multiple non-white light narrow-band images respectively represent information formed by different tissue parts that absorb narrow-band light of different wavelengths.

[0026] In some embodiments of the present application, the plurality of narrow-band light sources include a narrow-band blue-violet light source, a narrow-band blue light source, a narrow-band green light source, a narrow-band amber light source, and a narrow-band red light source.

[0027] In some embodiments of the present application, after a plurality of non-white light narrow-band images are enhanced, they are fused with a white light image to obtain a narrow-band enhanced target image, including: performing image enhancement processing on the plurality of non-white light narrow-band images respectively to obtain a plurality of non-white light narrow-band enhanced images; wherein main components of the plurality of non-white light narrow-band images are respectively derived from the blue channel, green channel, and red channel of the RGB image; and fusion processing of the white light image with the plurality of non-white light narrow-band enhanced images to obtain a target image.

[0028] In some embodiments of the present application, an RGB image is decomposed into a white light image and multiple non-white light narrow-band images, including: decomposing an RGB image into a white light image and multiple non-white light narrow-band images, the multiple non-white light narrow-band images include a shallow image, a middle image, and a deep image; wherein a main component of the shallow image is derived from the blue channel, representing information formed by surface mucosa and capillary tissue having strong absorption characteristics for short-wavelength narrow-band light; a main component of the middle image is derived from the green channel, representing information formed by middle-layer blood vessels and inflammatory sites having strong absorption characteristics for medium-wavelength narrow-band light; a main component of the deep image is derived from the red channel, representing information formed by deep blood vessels and bleeding sites having strong absorption characteristics for long-wavelength narrow-band light.

[0029] In some embodiments of the present application, image enhancement processing is performed on multiple non-white light narrow-band images respectively to obtain multiple non-white light narrow-band enhanced images, including: image enhancement processing is performed on multiple non-white light narrow-band images respectively to obtain multiple non-white light narrow-band enhanced images; wherein the image enhancement processing includes a global linear enhancement method, a global nonlinear enhancement method, and a local enhancement method, and the global nonlinear enhancement method is implemented based on a curve mapping function, including a gamma curve, an S-shaped curve, a log curve, and a least squares fitting curve.

[0030] In some embodiments of the present application, a white light image is fused with multiple non-white light narrow-band enhanced images to obtain a target image, including: fusing the detail components and brightness components of the multiple non-white light narrow-band enhanced images, and the color components of the white light image into the target image; or, fusing the detail components of the multiple non-white light narrow-band enhanced images, and the brightness components and color components of the white light image into the target image.

[0031] In some embodiments of the present application, detail components and brightness components of multiple non-white light narrow-band enhanced images and color components of a white light image are fused into a target image, including: assigning the multiple non-white light narrow-band enhanced images to the blue channel, green channel and red channel of the RGB image respectively to obtain a combined RGB image; converting the combined RGB image and the white light image to color spaces in which color and brightness are relatively independent, respectively, to obtain a first channel containing brightness and detail information, a second channel containing color information, and a third channel containing saturation information; after combining the first channel, the second channel and the third channel, converting them to the RGB space to obtain the target image.

[0032] In some embodiments of the present application, detail components of multiple non-white light narrow-band enhanced images, as well as brightness components and color components of white light images, are fused into a target image, including: extracting the detail components of each of the multiple non-white light narrow-band enhanced images, and converting the white light image to a color space where color and brightness are relatively independent; using the detail components, enhancing the color channel or brightness channel of the color space to obtain an enhanced color space; and converting the enhanced color space to RGB space to obtain the target image.

[0033] In some embodiments of the present application, a white light image is fused with a plurality of non-white light narrow-band enhanced images to obtain a target image, including: assigning the plurality of non-white light narrow-band enhanced images to the blue channel, green channel, and red channel of the RGB image, respectively, to obtain a combined RGB image; and fusing the combined RGB image with the white light image according to a preset ratio to obtain a target image.

[0034] On the one hand, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores program code, and when the program code is executed by the processor, the processor executes any one of the above-mentioned endoscopic imaging methods.

[0035] On the one hand, the present application provides a computer-readable storage medium, which includes a program code. When the storage medium is run on an electronic device, the program code is used to enable the electronic device to execute any of the above-mentioned endoscopic imaging methods.

[0036] The beneficial effects of this application are as follows:

[0037] An endoscopic imaging device, method, electronic device and storage medium provided in the embodiments of the present application adopt a full narrowband mode at the lighting end, without any broadband spectrum lighting, and the lighting combination covers the main spectral bands of visible light, thus avoiding the weakening effect of wide-band light on tissue vascular contrast. For multi-spectral fusion imaging, this solution proposes a multi-layer decomposition method based on the characteristics of the light absorption and reflection characteristics of the target object in the application scenario, decouples the image components, and makes each layer non-interfering with each other, so that the single-layer image can be differentiated and enhanced according to the actual scene and needs, while the white light layer can retain the original color tone information. In this way, it can not only ensure a look and feel close to the white light tone, but also provide better detail presentation at each level, thereby helping doctors to perform endoscopic exploration, diagnosis and treatment more efficiently and accurately.

[0038] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings.

[0039] BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0041] Figure 1 It is a schematic structural diagram of an endoscopic imaging device in an embodiment of the present application.

[0042] Figure 2 This is a flowchart of an implementation of an endoscopic imaging method in an embodiment of the present application.

[0043] Figure 3 A schematic diagram of the hardware structure of an electronic device in an embodiment of the present application.

[0044] Mode for Carrying Out the Invention

[0045] In order to make the purpose, technical scheme and advantages of the present application clearer, the technical scheme in the embodiment of the present application will be clearly and completely described below in conjunction with the drawings in the embodiment of the present application. Obviously, the described embodiment is only a part of the embodiment of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application. In the absence of conflict, the embodiments in the present application and the features in the embodiments can be arbitrarily combined with each other. In addition, although the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in an order different from that here.

[0046] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein.

[0047] The following is a brief introduction to the design concept of the embodiment of the present application:

[0048] In endoscopy application scenarios, especially for gastrointestinal endoscopy, diagnosis and treatment, doctors usually use the white light imaging mode that is consistent with human vision for routine operations to obtain white light images, and when further exploring lesions, switch the white light imaging mode to the narrow-band light imaging mode to obtain narrow-band light images. Among them, white light images can well maintain the color that is consistent with the perception of the human eye, but its wide spectrum characteristics have poor enhancement performance for specific tissues of organisms and cannot highlight detailed lesions; narrow-band light images, because the narrow-band band has strong absorption and reflection characteristics for specific tissues of organisms, can effectively highlight detailed lesions, but its illumination light source only covers part of the visible spectrum band or changes the white light spectrum ratio, and the light band is incomplete, resulting in a significant difference in its color tone from the white light image.

[0049] In view of this, the embodiment of the present application provides an endoscopic imaging device, method, electronic device and storage medium, wherein the device includes: an illumination module for providing an illumination light source, including multiple narrow-band light sources of different colors, wherein the multiple narrow-band light sources include the main spectral bands constituting white light; an imaging module for imaging the reflected light after the illumination module is irradiated on the target part of the patient to obtain an RGB image; a processing module for decomposing the RGB image into a white light image and multiple non-white light narrow-band images, and after enhancing the multiple non-white light narrow-band images, fusing them with the white light image to obtain a narrow-band enhanced target image; wherein the multiple non-white light narrow-band images respectively represent the information formed by different tissue parts that absorb narrow-band light of different wavelengths. In this way, a full narrow-band method is adopted at the illumination end, and no broadband spectrum illumination is adopted, and the illumination combination is spread over the main spectral bands of visible light, thus avoiding the weakening effect of broadband light on the contrast of tissue blood vessels. For multispectral fusion imaging, this solution proposes a multi-layer decomposition method based on the characteristics of light absorption and reflection characteristics of the target objects in the application scenario, decouples the image components so that each layer does not interfere with each other, and then can perform differentiated enhancement on the single-layer image according to the actual scenario and needs. At the same time, the white light layer can retain the original color tone information. In this case, it can not only ensure a visual perception close to the white light tone, but also provide better detail presentation at all levels, thereby helping doctors to perform endoscopic exploration, diagnosis and treatment more efficiently and accurately.

[0050] The preferred embodiments of the present application are described below in conjunction with the drawings in the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application. In addition, the embodiments and features in the embodiments of the present application may be combined with each other if there is no conflict.

[0051] refer to Figure 1 , is a schematic structural diagram of an endoscopic imaging device provided in an embodiment of the present application, comprising: an illumination module 101, an imaging module 102, and a processing module 103.

[0052] The lighting module 101 is used to provide a lighting source, including a plurality of narrow-band light sources of different colors, wherein the plurality of narrow-band light sources include the main spectral bands constituting white light and can be approximated as white light. The plurality of narrow-band light sources include a narrow-band blue-violet light source, a narrow-band blue light source, a narrow-band green light source, a narrow-band amber light source, and a narrow-band red light source. The specific light source device is a light emitting diode (LED). The specific narrow-band blue-violet light source may have a central wavelength range of [405, 425] nm, the narrow-band blue light source may have a central wavelength range of [450, 470] nm, the narrow-band green light source may have a central wavelength range of [525, 555] nm, the narrow-band amber light source may have a central wavelength range of [585, 615] nm, and the narrow-band red light source may have a central wavelength range of [620, 640] nm.

[0053] The imaging module 102 is used to image the reflected light after the illumination module 101 irradiates the target part of the patient to obtain an RGB image. The target part represents the tissue part of the patient that needs to be imaged. The imaging of the target part can reflect the patient's physical condition. After the emitted light of the illumination module 101 irradiates the patient's tissue part, it passes through the lens and the image sensor to form an original RGB image, which is composed of a blue channel, a green channel, and a red channel. The RGB image obtained by full narrow-band spectrum illumination can be regarded as an approximate white light image because the main spectral bands are not missing.

[0054] The processing module 103 is used to decompose the RGB image into a white light image and a plurality of non-white light narrow-band images, and after enhancing the plurality of non-white light narrow-band images, fuse them with the white light image to obtain a narrow-band enhanced target image; wherein the plurality of non-white light narrow-band images respectively represent information formed by different tissue parts that absorb narrow-band light of different wavelengths.

[0055] In some embodiments of the present application, the processing module 103 includes: a decomposition unit, an enhancement unit, and a fusion unit. Among them:

[0056] A decomposition unit, used for decomposing the RGB image into a white light image and a plurality of non-white light narrow band images, wherein main components of the plurality of non-white light narrow band images are respectively derived from a blue channel, a green channel, and a red channel of the RGB image;

[0057] An enhancement unit, used to perform image enhancement processing on the multiple non-white light narrow-band images respectively to obtain multiple non-white light narrow-band enhanced images;

[0058] The fusion unit is used to fuse the white light image with a plurality of non-white light narrow-band enhanced images to obtain a target image.

[0059] In some embodiments of the present application, the decomposition unit is used to:

[0060] The RGB image is decomposed into a white light image and multiple non-white light narrow-band images. The multiple non-white light narrow-band images include a shallow layer image, a middle layer image, and a deep layer image. Among them, the main component of the shallow layer image is derived from the blue channel, representing the information formed by the surface mucosa and capillary tissue with strong absorption characteristics for short-wavelength narrow-band light; the main component of the middle layer image is derived from the green channel, representing the information formed by the middle layer blood vessels and inflammation sites with strong absorption characteristics for medium-wavelength narrow-band light; the main component of the deep layer image is derived from the red channel, representing the information formed by the deep layer blood vessels and bleeding point sites with strong absorption characteristics for long-wavelength narrow-band light.

[0061] Among them, the surface mucosa and capillary tissue, middle layer blood vessels, inflammation sites, deep layer blood vessels, and bleeding point sites all belong to the aforementioned different tissue sites. The different tissue sites specifically include:

[0062] Tissue structure. For example, mucosa: The endoscope can clearly show the color, texture, and surface structure of the mucosa, helping doctors identify normal and abnormal conditions; blood vessels: They are the main detailed components in the human body. The distribution of blood vessels can be observed through endoscope imaging. Especially in areas such as the digestive tract, it can be used to detect blood vessel abnormalities or bleeding. Among them, blood vessels are divided into superficial mucosal capillaries, middle layer mucosal blood vessels, and deep layer blood vessels, and there are differences in the texture details of different blood vessels.

[0063] Abnormal lesion structure. For example, tumors: The shape, boundary, and its relationship with surrounding tissues of tumors all belong to detailed features; ulcers and inflammation: The areas affected by inflammation or ulcers will show characteristics such as redness, swelling, bleeding, or exudate; and, micro-tissue features: such as cysts or nodules.

[0064] Exemplarily, the specific implementation process of the decomposition unit is as follows:

[0065] The RGB image is decomposed into a white light image, a shallow layer image, a middle layer image, and a deep layer image.

[0066] Among them, the shallow layer image is denoted as Layer1. B2 is the mapped blue channel and can be approximated as Layer1; [b1 b2 b3] are decomposition parameters, and they satisfy the relationship b1 < b2 < b3; R1, G1, and B1 are the blue channel, green channel, and red channel of the original RGB image respectively.

[0067] Similarly, the middle layer image is denoted as Layer2. G2 is the mapped green channel and can be approximated as Layer2, [g1 g2 g3] are decomposition parameters, and they satisfy the relationships g1 < g2 and g3 < g2.

[0068] Similarly, denote the deep image as Layer3. R2 is the red channel after mapping, which can be approximated as Layer3. [r1 r2 r3] are the decomposition parameters, and they satisfy the relationship r3 < r2 < r1.

[0069] Similarly, denote the white image as Layer4. where is the decomposition parameter matrix.

[0070] By the above decomposition parameters, the advantage of decomposing the original RGB image into multiple layers of images is to decouple the image components. Each layer does not interfere with each other. Furthermore, each layer of the image can be enhanced differentially according to the actual scene and requirements. At the same time, the white light image can retain the original tone information.

[0071] In some embodiments of the present application, the enhancement unit is used for:

[0072] Performing image enhancement processing on multiple non-white-light narrowband images respectively to obtain multiple non-white-light narrowband enhanced images; wherein, the image enhancement processing includes a global linear enhancement method, a global non-linear enhancement method, and a local enhancement method. The global non-linear enhancement method is implemented based on a curve mapping function, including a gamma curve, an S curve, a log curve, and a least squares fitting curve.

[0073] Taking the enhancement of the shallow image LayerS1 as an example, the enhancement means include but are not limited to the global linear enhancement method: LayerS1 = α * Layer1, where α is the gain coefficient; the global non-linear enhancement method: LayerS1 = f(Layer1), where f() is the curve mapping function, and the representative curves can be a gamma curve, an S curve, a log curve, a least squares fitting curve, etc.; the local enhancement method (including but not limited to methods based on local histogram enhancement, multi-scale filtering enhancement, ROI region recognition enhancement, etc.).

[0074] In some embodiments of the present application, the fusion unit is used for: recombining the enhanced shallow image LayerS1, the enhanced middle image LayerS2, the enhanced deep image LayerS3, and the white light image LayerS4 to obtain a full narrowband enhanced white light image with a white light tone and distinct levels, that is, the target image, specifically including:

[0075] The detail components and brightness components of multiple non-white light narrow-band enhanced images and the color components of white light images are fused into the target image. The detail components are the detail information corresponding to the above-mentioned different tissue parts, including: edge: the area with significant grayscale or color changes in the image, which can be used to identify the boundary and shape of the object; corner point: the corner position with obvious changes in the image, which is used for feature matching and tracking; texture: the information describing the texture of the image surface, and the information about the surface structure of the object can be obtained by analyzing the texture; and contour: significant spatial structural features, such as protuberances, lesions, thick blood vessels, etc.

[0076] The above-mentioned process of fusing into a target image specifically includes: assigning multiple non-white light narrow-band enhanced images to the blue channel, green channel and red channel of the RGB image respectively to obtain a combined RGB image; converting the combined RGB image and the white light image to color spaces with relatively independent color and brightness respectively to obtain a first channel containing brightness and detail information, a second channel containing color information, and a third channel containing saturation information; after combining the first channel, the second channel and the third channel, converting them to the RGB space to obtain the target image.

[0077] For example, assign LayerS1 to the red channel, LayerS2 to the green channel, and LayerS3 to the blue channel to obtain the combined RGB image LayerS rgb , LayerS rgb , LayerS4 is converted to a color space where color and brightness are relatively independent, such as YUV, Lab, HSV space, etc. Taking HSV space as an example, convert LayerS rgb Convert from RGB space to HSV space to obtain its V channel containing brightness and detail information; convert LayerS4 from RGB space to HSV space to obtain its channel H4 containing color information and channel S4 containing saturation information; combine V, H4, and S4 into a new HSV image and convert back to RGB space to obtain the output fused image, which is the target image.

[0078] or,

[0079] The detail components of multiple non-white light narrow-band enhanced images, as well as the brightness components and color components of white light images, are fused into a target image. Specifically, the detail components of each of the multiple non-white light narrow-band enhanced images are extracted, and the white light image is converted to a color space with relatively independent color and brightness; the color channel or brightness channel of the color space is enhanced using the detail components to obtain an enhanced color space; after converting the enhanced color space to the RGB space, the target image is obtained. Among them, the means of extracting image detail components include but are not limited to edge detection (canny operator, sobel operator, etc.), high-pass filtering (Butterworth filtering, exponential high-pass filtering, etc.), inverse low-pass filtering (original image minus low-pass filtering), and image processing methods such as detail extraction models based on shallow neural networks.

[0080] For example, the detail components corresponding to LayerS1, LayerS2, and LayerS3 are extracted respectively and recorded as D1, D2, and D3 respectively, and D1, D2, and D3 are merged into D. Alternatively, LayerS1 is assigned to the red channel, LayerS2 is assigned to the green channel, and LayerS3 is assigned to the blue channel to obtain a combined RGB image, and the detail component is extracted from the combined RGB image, and the detail component is recorded as D. Further, LayerS4 is transferred to a color space in which color and brightness are relatively independent, such as YUV, Lab, HSV space, etc. Taking YUV space as an example, LayerS4 is transferred from RGB space to YUV space, and the Y channel, U channel, and V channel are extracted respectively. The Y channel is enhanced using the detail component D. One of the enhancement methods is: Y out =Y+β*D γ , β, γ are both enhancement coefficients, Y out To enhance the Y channel, a new YUV image is obtained. Further, the new YUV image is converted back to RGB space to obtain the target image.

[0081] Alternatively, multiple non-white light narrow-band enhanced images are respectively assigned to the blue channel, green channel and red channel of the RGB image to obtain a combined RGB image; the combined RGB image and the white light image are fused according to a preset ratio to obtain a target image.

[0082] For example, LayerS1 is assigned to the red channel of the original RGB image, LayerS2 is assigned to the green channel of the original RGB image, and LayerS3 is assigned to the blue channel of the original RGB image, thereby obtaining a combined RGB image, and the combined RGB image is recorded as LayerS rgb . Further, the image LayerS in the same RGB domain space rgb , LayerS4 is fused in proportion, the specific fusion formula is: I = k1 * LayerS rgb+k2*LayerS4, where k1 and k2 are LayerS rgb and the fusion ratio coefficient of LayerS4, I is the fused image, that is, the target image.

[0083] The above-mentioned endoscopic imaging device adopts a full narrowband method at the lighting end, does not use any broadband spectrum lighting, and the lighting combination covers the main spectral bands of visible light, thus avoiding the weakening effect of wide-band light on tissue vascular contrast. For multi-spectral fusion imaging, this solution proposes a multi-layer decomposition method based on the characteristics of the light absorption and reflection characteristics of the target object in the application scenario, decouples the image components, and makes each layer non-interfering with each other, so that the single-layer image can be differentiated and enhanced according to the actual scene and needs, while the white light layer can retain the original color tone information. In this case, it can not only ensure a look and feel close to the white light tone, but also provide better detail presentation at all levels, thereby helping doctors to perform endoscopic exploration, diagnosis and treatment more efficiently and accurately.

[0084] Based on the same inventive concept, the present application also provides an endoscopic imaging method. Figure 2 FIG. 1 is a flowchart of an implementation method of an endoscopic imaging method in an embodiment of the present application, comprising:

[0085] S201, acquiring an RGB image, the RGB image being obtained based on reflected light after the illumination module irradiates a target part of the patient; wherein the illumination module is used to provide an illumination light source, including a plurality of narrow-band light sources of different colors, the plurality of narrow-band light sources including a main spectral band constituting white light;

[0086] S202, decomposing the RGB image into a white light image and a plurality of non-white light narrow-band images;

[0087] S203, after enhancing the plurality of non-white light narrow-band images, fuse them with the white light image to obtain a narrow-band enhanced target image; wherein the plurality of non-white light narrow-band images respectively represent information formed by different tissue parts that absorb narrow-band light of different wavelengths.

[0088] In some embodiments of the present application, the plurality of narrow-band light sources include a narrow-band blue-violet light source, a narrow-band blue light source, a narrow-band green light source, a narrow-band amber light source, and a narrow-band red light source.

[0089] In some embodiments of the present application, after a plurality of non-white light narrow-band images are enhanced, they are fused with a white light image to obtain a narrow-band enhanced target image, including: performing image enhancement processing on the plurality of non-white light narrow-band images respectively to obtain a plurality of non-white light narrow-band enhanced images; wherein main components of the plurality of non-white light narrow-band images are respectively derived from the blue channel, green channel, and red channel of the RGB image; and fusion processing of the white light image with the plurality of non-white light narrow-band enhanced images to obtain a target image.

[0090] In some embodiments of the present application, an RGB image is decomposed into a white light image and multiple non-white light narrow-band images, including: decomposing an RGB image into a white light image and multiple non-white light narrow-band images, the multiple non-white light narrow-band images include a shallow image, a middle image, and a deep image; wherein a main component of the shallow image is derived from the blue channel, representing information formed by surface mucosa and capillary tissue having strong absorption characteristics for short-wavelength narrow-band light; a main component of the middle image is derived from the green channel, representing information formed by middle-layer blood vessels and inflammatory sites having strong absorption characteristics for medium-wavelength narrow-band light; a main component of the deep image is derived from the red channel, representing information formed by deep blood vessels and bleeding sites having strong absorption characteristics for long-wavelength narrow-band light.

[0091] In some embodiments of the present application, image enhancement processing is performed on multiple non-white light narrow-band images respectively to obtain multiple non-white light narrow-band enhanced images, including: image enhancement processing is performed on multiple non-white light narrow-band images respectively to obtain multiple non-white light narrow-band enhanced images; wherein the image enhancement processing includes a global linear enhancement method, a global nonlinear enhancement method, and a local enhancement method, and the global nonlinear enhancement method is implemented based on a curve mapping function, including a gamma curve, an S-shaped curve, a log curve, and a least squares fitting curve.

[0092] In some embodiments of the present application, a white light image is fused with multiple non-white light narrow-band enhanced images to obtain a target image, including: fusing the detail components and brightness components of the multiple non-white light narrow-band enhanced images, and the color components of the white light image into the target image; or, fusing the detail components of the multiple non-white light narrow-band enhanced images, and the brightness components and color components of the white light image into the target image.

[0093] In some embodiments of the present application, detail components and brightness components of multiple non-white light narrow-band enhanced images and color components of a white light image are fused into a target image, including: assigning the multiple non-white light narrow-band enhanced images to the blue channel, green channel and red channel of the RGB image respectively to obtain a combined RGB image; converting the combined RGB image and the white light image to color spaces in which color and brightness are relatively independent, respectively, to obtain a first channel containing brightness and detail information, a second channel containing color information, and a third channel containing saturation information; after combining the first channel, the second channel and the third channel, converting them to the RGB space to obtain the target image.

[0094] In some embodiments of the present application, detail components of multiple non-white light narrow-band enhanced images, as well as brightness components and color components of white light images, are fused into a target image, including: extracting the detail components of each of the multiple non-white light narrow-band enhanced images, and converting the white light image to a color space where color and brightness are relatively independent; using the detail components, enhancing the color channel or brightness channel of the color space to obtain an enhanced color space; and converting the enhanced color space to RGB space to obtain the target image.

[0095] In some embodiments of the present application, a white light image is fused with a plurality of non-white light narrow-band enhanced images to obtain a target image, including: assigning the plurality of non-white light narrow-band enhanced images to the blue channel, green channel, and red channel of the RGB image, respectively, to obtain a combined RGB image; and fusing the combined RGB image with the white light image according to a preset ratio to obtain a target image.

[0096] The above endoscopic imaging method and the technical solutions and corresponding technical effects of the endoscopic imaging method can be referenced to each other and will not be described in detail here.

[0097] In some possible embodiments, the endoscopic imaging device according to the present application may include at least a processor and a memory. The memory stores program code, and when the program code is executed by the processor, the processor executes the steps of the endoscopic imaging method according to various exemplary embodiments of the present application described in this specification. For example, the processor may execute the following steps: Figure 2 Follow the steps shown in .

[0098] Based on the same inventive concept, an electronic device is also provided in the embodiment of the present application, and the electronic device can realize the functions of the aforementioned endoscopic imaging method and device, referring to Figure 3 , the electronic device comprises:

[0099] At least one processor 301, and a memory 302 connected to the at least one processor 301. The specific connection medium between the processor 301 and the memory 302 is not limited in the embodiment of the present application. Figure 3 In the example, the processor 301 and the memory 302 are connected via the bus 300. The bus 300 is connected to the memory 302 via the bus 300. Figure 3 The bus 300 is represented by a bold line, and the connection between other components is only for schematic illustration and is not intended to be limiting. The bus 300 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. Alternatively, the processor 301 can also be called a controller, and there is no limitation on the name.

[0100] In the embodiment of the present application, the memory 302 stores instructions that can be executed by at least one processor 301. The at least one processor 301 can execute the endoscopic imaging method discussed above by executing the instructions stored in the memory 302. The processor 301 can implement Figure 2 The functions of each module in the device shown.

[0101] Among them, the processor 301 is the control center of the device, and can use various interfaces and lines to connect the various parts of the entire control device. By running or executing instructions stored in the memory 302 and calling data stored in the memory 302, the various functions of the device and process data, the device can be monitored as a whole.

[0102] In one possible design, the processor 301 may include one or more processing units, and the processor 301 may integrate an application processor and a modem processor, wherein the application processor mainly processes an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the modem processor may not be integrated into the processor 301. In some embodiments of the present application, the processor 301 and the memory 302 may be implemented on the same chip, and in some embodiments of the present application, they may also be implemented separately on separate chips.

[0103] The processor 301 may be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, and may implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the endoscopic imaging method disclosed in the embodiments of the present application may be directly embodied as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0104] The memory 302 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 302 may include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (Random Access Memory, RAM), a static random access memory (Static Random Access Memory, SRAM), a programmable read-only memory (Programmable Read Only Memory, PROM), a read-only memory (Read Only Memory, ROM), an electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), a magnetic memory, a disk, an optical disk, etc. The memory 302 is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 302 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.

[0105] By programming the processor 301, the code corresponding to the endoscopic imaging method described in the above embodiment can be fixed into the chip, so that the chip can execute the code when running. Figure 2 The steps of the endoscopic imaging method of the embodiment shown are as follows: How to design and program the processor 301 is a technique known to those skilled in the art and will not be described in detail here.

[0106] Based on the same inventive concept, an embodiment of the present application further provides a storage medium, which stores computer instructions. When the computer instructions are executed on a computer, the computer executes the endoscopic imaging method discussed above.

[0107] In some possible embodiments, various aspects of the endoscopic imaging method provided by the present application can also be implemented in the form of a program product, which includes a program code. When the program product is run on an apparatus, the program code is used to enable the control device to execute the steps of the endoscopic imaging method according to various exemplary embodiments of the present application described above in this specification.

[0108] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0109] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0110] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0111] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0112] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. An endoscopic imaging device, characterized in that: include: A lighting module, for providing a lighting light source, including a plurality of narrow-band light sources of different colors, wherein the plurality of narrow-band light sources contain a main spectral band constituting white light; An imaging module, used for imaging the reflected light after the illumination module irradiates the target part of the patient to obtain an RGB image; A processing module is used to decompose the RGB image into a white light image and multiple non-white light narrow-band images, and after enhancing the multiple non-white light narrow-band images, fuse them with the white light image to obtain a narrow-band enhanced target image; wherein the multiple non-white light narrow-band images respectively represent information formed by different tissue parts that absorb narrow-band light of different wavelengths.

2. The device according to claim 1, characterized in that: The plurality of narrow-band light sources include a narrow-band blue-violet light source, a narrow-band blue light source, a narrow-band green light source, a narrow-band amber light source, and a narrow-band red light source.

3. The device according to claim 1, characterized in that: The processing module comprises: a decomposition unit, configured to decompose the RGB image into the white light image and the plurality of non-white light narrow-band images, wherein main components of the plurality of non-white light narrow-band images are respectively derived from a blue channel, a green channel, and a red channel of the RGB image; an enhancement unit, configured to perform image enhancement processing on the plurality of non-white light narrow-band images respectively to obtain the plurality of non-white light narrow-band enhanced images; A fusion unit is used to fuse the white light image with the multiple non-white light narrow-band enhanced images to obtain the target image.

4. The device according to claim 3, characterized in that: The decomposition unit is used for: Decomposing the RGB image into the white light image and the plurality of non-white light narrow-band images, wherein the plurality of non-white light narrow-band images include a shallow layer image, a middle layer image, and a deep layer image; Among them, the main component of the shallow image comes from the blue channel, which represents the information formed by the surface mucosa and capillary tissue with strong absorption characteristics for short-wavelength narrow-band light; the main component of the middle-layer image comes from the green channel, which represents the information formed by the middle-layer blood vessels and inflammation sites with strong absorption characteristics for medium-wavelength narrow-band light; the main component of the deep image comes from the red channel, which represents the information formed by the deep blood vessels and bleeding points with strong absorption characteristics for long-wavelength narrow-band light.

5. The device according to claim 3, characterized in that: The enhancement unit is used for: Performing image enhancement processing on the multiple non-white light narrow-band images respectively to obtain the multiple non-white light narrow-band enhanced images; Among them, the image enhancement processing includes a global linear enhancement method, a global nonlinear enhancement method, and a local enhancement method. The global nonlinear enhancement method is implemented based on a curve mapping function, including a gamma curve, an S-curve, a log curve, and a least squares fitting curve.

6. The device according to claim 3, characterized in that: The fusion unit is used for: Merging the detail components and brightness components of the plurality of non-white light narrow-band enhanced images and the color components of the white light image into the target image; or, The detail components of the multiple non-white light narrow-band enhanced images, and the brightness component and color component of the white light image are fused into the target image.

7. The device according to claim 3, characterized in that: The fusion unit is used for: Assigning the non-white light narrow-band enhanced image to the blue channel, green channel and red channel of the RGB image respectively to obtain a combined RGB image; The combined RGB image and the white light image are fused according to a preset ratio to obtain the target image.

8. An endoscopic imaging method, applied to the endoscopic imaging device according to any one of claims 1 to 7, characterized in that: The method comprises: Acquire an RGB image, wherein the RGB image is obtained based on reflected light after the illumination module irradiates the target part of the patient; wherein the illumination module is used to provide an illumination light source, including a plurality of narrow-band light sources of different colors, and the plurality of narrow-band light sources contain a main spectral band constituting white light; Decomposing the RGB image into a white light image and a plurality of non-white light narrow band images; After the multiple non-white light narrow-band images are enhanced, they are fused with the white light image to obtain a narrow-band enhanced target image; wherein the multiple non-white light narrow-band images respectively represent information formed by different tissue parts that absorb narrow-band light of different wavelengths.

9. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a program code, and when the program code is executed by the processor, the processor executes the method as claimed in claim 8.

10. A computer-readable storage medium, characterized in that: The storage medium comprises a program code, and when the storage medium is run on an electronic device, the program code is used to enable the electronic device to execute the method according to claim 8.

Images