Imaging system, endoscope system, imaging method and computer storage medium

Abstract

The invention discloses an imaging system, an endoscope system, an imaging method and a computer storage medium, and relates to the technical field of medical instruments. The endoscope system includes a white light source portion, a guide portion, an imaging assembly, and a processor. The white light source part provides illumination light of a continuous visible spectrum to a detected area; the guide part is provided with a tip part configured to extend into a detected area; the imaging assembly comprises a common lens, a light splitting component, a gray level image sensor, a hyperspectral image sensor and an optical filter array; the common lens is used for emitting imaging light; the light splitting component is arranged on a light path of the imaging light; the grayscale image sensor converts the first imaging light into an electric signal from an optical signal to obtain a grayscale image; the optical filter array is used for decomposing the second imaging light into monochromatic light or narrow-band light with different wavelengths, and the hyperspectral image sensor is used for converting an optical signal decomposed by the optical filter array into an electric signal to obtain a hyperspectral image; and the processor performs imaging according to the grayscale image and / or the hyperspectral image.

Description

Technical Field

[0001] The present invention relates to the technical field of medical devices, and in particular to an imaging system, an endoscope system, an imaging method and a computer storage medium. Background Art

[0002] With the rapid development of miniaturized sensors and minimally invasive medical technologies, minimally invasive or non-invasive medical examinations and treatments based on endoscopes have become widely popular. Endoscopic technology enables imaging components to enter the interior of organisms (such as humans and animals) through natural orifices or small surgical incisions, thereby obtaining image information of the examined area, providing more intuitive and accurate diagnostic evidence. Electronic endoscopes integrate an imaging lens and image sensor at the front end of the probe, transmitting the resulting image to an image processing host computer, which, after processing, displays it on a screen for the user to observe.

[0003] During the development of endoscopes, special imaging as an important technology has been applied to a variety of endoscopic imaging devices, such as NBI (Narrow Band Imaging), BLI (Blue Light Imaging) and RDI (Red Dichromatic Imaging). These imaging technologies utilize the specificity of spectral absorption of specific tissues, use light of specific wavelengths to illuminate the imaging object to produce contrast, and use monochrome or color image sensors to receive images with specific contrast, which is convenient for users to observe.

[0004] This type of imaging technology often uses a multi-LED remote light-combining solution, where multiple LEDs or laser light sources are combined on the host computer. Light of the appropriate wavelength is then selected and transmitted to the body via a light guide to achieve specialized illumination. This method has a complex optical path and is relatively costly. If a monochrome image sensor is used for reception, the light source must switch synchronously with the sensor exposure, making it difficult to maintain a high frame rate. If a traditional color sensor is used for reception, the light source spectrum must be well matched to the sensor's received spectrum (color channel crosstalk in the color sensor), otherwise it will be difficult to achieve the desired narrowband imaging effect. Summary of the Invention

[0005] The main technical problem solved by the present invention is to provide an imaging system and an endoscope system with a simpler illumination structure in a special light imaging process.

[0006] According to a first aspect, an embodiment provides an endoscope system, comprising:

[0007] A white light source unit, used to provide continuous visible spectrum illumination light to the inspected area;

[0008] a guide portion having at least a tip portion configured to be inserted into a detected region;

[0009] an imaging assembly including a common lens, a light splitting component, a grayscale image sensor, a hyperspectral image sensor, and a filter array;

[0010] the common lens is disposed at the tip portion of the guide portion for collecting reflected light or scattered light after the illumination light irradiates the detected region to emit imaging light;

[0011] the light splitting component is disposed in an optical path of the imaging light for splitting the imaging light into first imaging light and second imaging light, the second imaging light having a greater intensity than the first imaging light;

[0012] the grayscale image sensor is disposed in an optical path of the first imaging light for converting the first imaging light from an optical signal to an electrical signal to obtain a grayscale image;

[0013] the filter array and the hyperspectral image sensor are sequentially disposed in an optical path of the second imaging light; the filter array is configured to split the second imaging light into monochromatic light or narrow-band light of different wavelengths, and the hyperspectral image sensor is configured to convert the monochromatic light or narrow-band light of different wavelengths obtained by splitting through the filter array from an optical signal to an electrical signal to obtain a hyperspectral image;

[0014] a processor including a plurality of imaging modes, the plurality of imaging modes including at least one of a first imaging mode, a second imaging mode, and a third imaging mode;

[0015] in the first imaging mode, the processor generates a display image of the detected region for output according to the grayscale image;

[0016] in the second imaging mode, the processor generates a display image of the detected region for output according to the hyperspectral image;

[0017] in the third imaging mode, the processor generates a display image of the detected region for output according to the grayscale image and the hyperspectral image.

[0018] According to a second aspect, an embodiment provides an endoscope system, comprising:

[0019] a white light source portion configured to provide illumination light of a continuous visible spectrum to a detected region;

[0020] a guide portion having at least a tip portion configured to be inserted into a detected region;

[0021] An imaging assembly comprising a common lens, a light splitting component, a grayscale image sensor, a hyperspectral image sensor and a filter array;

[0022] The common lens is arranged at the leading end of the guide portion, and is configured to collect reflected light or scattered light after the illumination light irradiates the detected region, so as to emit imaging light;

[0023] The light splitting component is arranged on the light path of the imaging light, and is configured to split the imaging light into first imaging light and second imaging light, the intensity of the second imaging light being greater than that of the first imaging light;

[0024] The grayscale image sensor is arranged on the light path of the first imaging light, and is configured to convert the first imaging light from an optical signal to an electrical signal to obtain a grayscale image;

[0025] The filter array and the hyperspectral image sensor are arranged on the light path of the second imaging light in sequence; the filter array is configured to split the second imaging light into monochromatic light or narrow-band light of different wavelengths, and the hyperspectral image sensor is configured to convert the monochromatic light or narrow-band light of different wavelengths obtained by splitting of the filter array from an optical signal to an electrical signal to obtain a hyperspectral image.

[0026] According to a third aspect, an embodiment provides an imaging system, comprising:

[0027] A white light source portion configured to provide illumination light of a continuous visible spectrum to a detected region;

[0028] An imaging assembly comprising a common lens, a light splitting component, a grayscale image sensor, a hyperspectral image sensor and a filter array;

[0029] The common lens is configured to collect reflected light or scattered light after the illumination light irradiates the detected region, so as to emit imaging light;

[0030] The light splitting component is arranged on the light path of the imaging light, and is configured to split the imaging light into first imaging light and second imaging light, the intensity of the second imaging light being greater than that of the first imaging light;

[0031] The grayscale image sensor is arranged on the light path of the first imaging light, and is configured to convert the first imaging light from an optical signal to an electrical signal to obtain a grayscale image;

[0032] The filter array and the hyperspectral image sensor are sequentially arranged on the light path of the second imaging light; the filter array is used to decompose the second imaging light into monochromatic light or narrow-band light of different wavelengths, and the hyperspectral image sensor is used to convert the monochromatic light or narrow-band light of different wavelengths decomposed by the filter array from an optical signal into an electrical signal to obtain a hyperspectral image.

[0033] According to a fourth aspect, an embodiment provides an imaging method, comprising:

[0034] Obtaining a checkerboard image of a first color channel corresponding to a filter color in a first filter unit and a checkerboard image of a second color channel corresponding to a filter color in a second filter unit in a hyperspectral image to be interpolated;

[0035] The first filter unit is a first filter unit that transmits a preset waveband, and the second filter unit is a second filter unit that transmits a waveband other than the preset waveband, and the first filter unit and the second filter unit are alternately and spacedly distributed in a horizontal direction and a vertical direction;

[0036] Interpolating pixel values of a first color corresponding to the first filter unit in a blank position in the checkerboard image of the first color channel to generate a complete image of the first color channel;

[0037] Interpolating pixel values of each pixel point in different color channels in the hyperspectral image according to the complete image of the first color channel and the checkerboard image of the second color channel to generate a display image of the detected region for output;

[0038] The blank position is a pixel point position where pixel values of the first color do not exist in the checkerboard image of the second color channel in the hyperspectral image.

[0039] According to a fifth aspect, an embodiment provides an imaging method, comprising:

[0040] Obtaining a hyperspectral image to be interpolated and a grayscale image corresponding to the hyperspectral image to be interpolated;

[0041] Obtaining image information of each pixel point in each color channel in the hyperspectral image, and obtaining pixel point positions with known pixel values and pixel point positions with pixel values to be interpolated in each color channel in the hyperspectral image according to the image information;

[0042] Obtaining a first pixel point corresponding to the pixel point position with the known pixel value and a second pixel point corresponding to the pixel point position with the pixel value to be interpolated in the grayscale image;

[0043] calculating gradients of the first pixel point and the second pixel point;

[0044] determining a pixel value of a pixel point position of the to-be-interpolated pixel value according to the gradient and a pixel value of a pixel point position of the known pixel value, to generate a display image of the detected region for output.

[0045] According to a sixth aspect, in an embodiment, a computer readable storage medium is provided, and the medium stores a computer program, which can be executed by a processor to implement the method described in any of the above embodiments.

[0046] According to the imaging system, the endoscope system, the imaging method and the computer storage medium of the above embodiments, the white light source is used in the endoscope system, which reduces the complexity of the light source, removes the synchronization requirement of the light source and the endoscope system, reduces the cost of the light source, and is more likely to be disposable. The grayscale image sensor and the hyperspectral image sensor are used in the endoscope system, the color channel and the texture channel are separated by the design of the double sensors, and high resolution and high spectral resolution can be realized at the same time. The endoscope system in the present application further comprises a filter array, and the filter array is used to realize imaging of multiple special lights, without the need for switching the light source and the like. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 FIG. 1 is a structural schematic diagram of an endoscope system in an embodiment;

[0048] Figure 2 FIG. 2 is a structural schematic diagram of an imaging system as an independent system in an embodiment;

[0049] Figure 3 FIG. 3 is a structural schematic diagram of an imaging system as a core component of an endoscope system in an embodiment;

[0050] Figure 4 FIG. 4 is a structural schematic diagram of an imaging assembly in an embodiment;

[0051] Figure 5 FIG. 5 is a component schematic diagram of a filter array in an embodiment;

[0052] Figure 6 FIG. 6 is a setting schematic diagram of a filter array in an embodiment;

[0053] Figure 7 FIG. 7 is a setting schematic diagram of a first sub-filter unit in a second case in an embodiment;

[0054] Figure 8 FIG. 8 is a setting schematic diagram of a second sub-filter unit in a second case in an embodiment;

[0055] Figure 9A flow chart of a method for generating a display image based on a grayscale image in one embodiment;

[0056] Figure 10 A method flow chart of a first method for generating a display image based on a hyperspectral image in one embodiment;

[0057] Figure 11 is a schematic diagram of the arrangement of the first filter unit in one embodiment;

[0058] Figure 12 is a method flow chart of a second method for generating a display image based on a hyperspectral image in one embodiment;

[0059] Figure 13 4 is a flowchart of step S240 in a second method for generating a display image based on a hyperspectral image in one embodiment;

[0060] Figure 14 4 is a flowchart of step S241 in a second method for generating a display image based on a hyperspectral image in one embodiment;

[0061] Figure 15 4 is a flowchart of step S260 in a second method for generating a display image based on a hyperspectral image in one embodiment;

[0062] Figure 16 is a schematic diagram of a 3×3 image extracted from a hyperspectral image in one embodiment;

[0063] Figure 17 4 is a flowchart of step S263 in a second method for generating a display image based on a hyperspectral image in one embodiment;

[0064] Figure 18 4 is a flowchart of step S265 in a second method for generating a display image based on a hyperspectral image in one embodiment;

[0065] Figure 19 4 is a flowchart of step S260 in a third method for generating a display image based on a hyperspectral image in one embodiment;

[0066] Figure 20 is a schematic diagram of a 5×5 image extracted from a hyperspectral image in one embodiment;

[0067] Figure 21 4 is a flowchart of step S264 in a third method for generating a display image based on a hyperspectral image in one embodiment;

[0068] Figure 22 4 is a flowchart of step S266 in a third method for generating a display image based on a hyperspectral image in one embodiment;

[0069] Figure 23 The flowchart is a method for generating a display image based on a grayscale image and a hyperspectral image in one embodiment. DETAILED DESCRIPTION

[0070] The present invention will be further described in detail below by means of specific embodiments in conjunction with the accompanying drawings. Similar elements in different embodiments are numbered with associated similar elements. In the following embodiments, many detailed descriptions are provided to enable the present application to be better understood. However, those skilled in the art will readily appreciate that some of the features may be omitted in different circumstances, or may be replaced by other elements, materials, or methods. In some cases, some operations related to the present application are not shown or described in the specification. This is to avoid the core portion of the present application being overwhelmed by excessive descriptions, and for those skilled in the art, it is not necessary to describe these related operations in detail. They will fully understand the related operations based on the description in the specification and the general technical knowledge in the art.

[0071] In addition, the features, operations, or characteristics described in the specification may be combined in any appropriate manner to form various embodiments. Furthermore, the steps or actions in the method description may be reordered or adjusted in a manner readily apparent to those skilled in the art. Therefore, the various sequences in the specification and drawings are provided solely for the purpose of clearly describing a particular embodiment and are not intended to be mandatory, unless otherwise specified.

[0072] The serial numbers assigned to components herein, such as "first," "second," etc., are used solely to distinguish the objects being described and do not convey any sequential or technical meaning. References to "connection" and "coupling" herein, unless otherwise specified, include both direct and indirect connections (couplings).

[0073] Please refer to Figure 1 In one embodiment, an endoscope system 1 is a complex optical and electronic imaging device primarily used for internal human or industrial inspections. The hardware structure includes an illumination system 11, an imaging system 12, a guide unit 13, a control system 14, an image transmission and display system 15, and a power supply system 16. The imaging system 12 can be a core component of the endoscope system 1 or used independently as a separate system.

[0074] Please refer to Figure 2 In one embodiment, when the imaging system 12 is used independently as a separate system, the imaging system 12 includes an illumination component 121, an imaging component 122, and a processor 123. The imaging system 12 uses the illumination component 121, the imaging component 122, and the processor 123 to image the inspected area.

[0075] Reference is made to Figure 3 In an embodiment, when the imaging system 12 is used as a core component in the endoscope system 1, the illumination component 121 is not required in the imaging system 12, and only the imaging component 122 and the processor 123 are included, and the illumination light is provided to the detected area by the illumination system 11 in the endoscope system 1.

[0076] In an embodiment, either the illumination system 11 in the endoscope system 1 or the illumination component 121 in the imaging system 12 can use one or more forms of light sources such as white light, red light, blue light, infrared light, ultraviolet light, and laser light, and the specific type of light source can be selected according to the application requirements. The application preferably uses ordinary white light as the illumination light, which can be generated by an LED or a xenon lamp light source, and is used to provide continuous visible spectrum illumination light to the detected area. Compared with other types of light sources, ordinary white light has good performance in color rendering, tissue recognition, and use safety, which helps to improve the clarity and authenticity of the image. In addition, the LED light source has the advantages of low power consumption and long service life, while the xenon lamp has high brightness and good spectral continuity, and the appropriate type of light source can be selected according to the specific application requirements.

[0077] In summary, in addition to the illumination system 11 and the illumination component 121 in the endoscope system 1 and the imaging system 12, the remaining parts of the imaging system 12 are the same, which are described in detail below.

[0078] Reference is made to Figure 4 In an embodiment, the imaging component 122 includes a shooting lens 1221, a light splitting component 1222, a grayscale image sensor 1223, a hyperspectral image sensor 1224, and a filter array 1225. The shooting lens 1221 is used to collect the reflected light or scattered light after the illumination light is irradiated to the detected area to emit the imaging light.

[0079] In an embodiment, when only the hyperspectral image sensor 1224 is arranged in the imaging component 122, the shooting lens 1221 is used to collect the optical image of the target area and guide the imaging light to the hyperspectral image sensor 1224. At this time, the imaging light path emitted by the shooting lens 1221 is only connected with the hyperspectral image sensor 1224, and does not involve the optical coupling relationship of other image sensors.

[0080] The filter array 1225 and the hyperspectral image sensor 1224 are sequentially arranged in the optical path of the imaging light emitted by the capturing lens 1221. The filter array 1225 is located between the capturing lens 1221 and the hyperspectral image sensor 1224 and is used to decompose the imaging light into monochromatic light or narrow-band light of different wavelengths, thereby performing wavelength-selective filtering on the imaging light. Specifically, the filter array 1225 can decompose the composite light from the capturing lens 1221 into monochromatic light or narrow-band light of different wavelengths, thereby achieving optical separation of the detected area in multiple spectral dimensions.

[0081] The hyperspectral image sensor 1224 is used to receive the optical signal processed by the optical filter array 1225 and convert the monochromatic light or narrow-band light corresponding to different wavelengths into electrical signals, thereby obtaining hyperspectral image data containing rich spectral information.

[0082] In one embodiment, when the imaging assembly 122 is provided with both a grayscale image sensor 1223 and a hyperspectral image sensor 1224, the shooting lens 1221 serves as a shared lens. The shared lens is also used to collect reflected light or scattered light after the illumination light hits the inspection area to emit imaging light.

[0083] Beam splitting component 1222 is positioned within the optical path of the imaging light emitted from the shared lens, and is used to split the imaging light from the shared lens into a first imaging light and a second imaging light according to a preset ratio or wavelength band characteristic, wherein the intensity of the second imaging light is greater than that of the first imaging light. This beam splitting component 1222 enables different types of image sensors to obtain their respective required optical signals based on the same field of view, thereby improving system compactness and image consistency.

[0084] The grayscale image sensor 1223 is disposed on the optical path of the first imaging light, and is configured to receive the portion of the imaging light and convert it into an electrical signal to generate a corresponding grayscale image.

[0085] Filter array 1225 and hyperspectral image sensor 1224 are sequentially positioned in the optical path of the second imaging light. Filter array 1225 is used to decompose the second imaging light into monochromatic light or narrowband light of different wavelengths, thereby performing wavelength-selective filtering on the second imaging light. Specifically, filter array 1225 decomposes the composite light from the imaging lens into monochromatic light or narrowband light of different wavelengths, thereby achieving optical separation of the detected area along multiple spectral dimensions.

[0086] The hyperspectral image sensor 1224 is used to convert the monochromatic light or narrow-band light decomposed by the filter array 1225 into an electrical signal from an optical signal to obtain a hyperspectral image.

[0087] It should be noted that the hyperspectral image sensor 1224 and grayscale image sensor 1223 used during the imaging process are used to collect spectral and texture information of the inspected area, respectively. The hyperspectral image sensor 1224 can capture spectral reflectance characteristics at different wavelengths, thereby reflecting differences in the composition, structure, or pathological status of the inspected area. The grayscale image sensor 1223 can generate detailed grayscale images, revealing topographical features such as the structural texture, edge contours, and light-dark variations of the inspected area. To ensure imaging consistency, the hyperspectral image sensor 1224 and the grayscale image sensor 1223 share the same spectroscopic component 1222, preferably a prism structure, located in the outgoing light path of the lens. This spectroscopic component 1222 allows both image sensors to simultaneously receive image information from the same optical perspective. The resulting hyperspectral image and grayscale image are highly spatially consistent, providing a good foundation for image registration.

[0088] In one embodiment, since hyperspectral image sensor 1224 and grayscale image sensor 1223 share a common lens and have minimal parallax, only affine or perspective transformation adjustments are required. Alternatively, to obtain a more accurate image, the images acquired by hyperspectral image sensor 1224 and grayscale image sensor 1223 can be spatially aligned to ensure image registration before image fusion.

[0089] It's important to note that the core goal of image registration is to align two images at the pixel level, ensuring that there is no misalignment or blur during subsequent fusion. If the two images share common edges or feature points, feature-based registration can be used. If the details of the two images are similar but slightly offset, optical flow-based registration can be used.

[0090] In summary, regardless of whether the imaging component 122 uses a single hyperspectral image sensor 1224 or a hyperspectral image sensor 1224 and a grayscale image sensor 1223 , the present application makes a special arrangement for the filter array 1225 in the hyperspectral image sensor 1224 , and generates different imaging methods based on this special arrangement, which will be described in detail below.

[0091] Please refer to Figure 5 In one embodiment, the filter array 1225 includes at least two types of filter units: a first filter unit 12251 that transmits a predetermined wavelength band, and a second filter unit 12252 that transmits wavelengths other than the predetermined wavelength band. The filters in the first filter unit 12251 and the filters in the second filter unit 12252 are alternately spaced horizontally and vertically to form a checkerboard pattern, thereby creating a uniform spatial coverage and facilitating subsequent image reconstruction and interpolation.

[0092] It should be noted that in the endoscope system 1, in order to improve the imaging contrast and functional information extraction capability of biological tissues, the spectral design of the filter array 1225 can be set based on the biological spectral response characteristics of the detected region. For example, human blood has obvious absorption peaks in the wavelength ranges of 415 nm±20 nm and 540 nm±20 nm, and therefore, the first filter unit 12251 in the filter array 1225 can be set to transmit the waveband, so that the imaging in the waveband can present a higher-contrast blood vessel image. For another example, in the waveband of 590 nm±10 nm, the gradient distribution characteristics of blood concentration are more significant, and by setting the first filter unit 12251 to transmit the waveband, an image reflecting the blood concentration distribution can be obtained.

[0093] In an embodiment, the present application selects a single color of filter with quantum efficiency greater than a set efficiency to form the first filter unit 12251 in the filter array 1225, or selects a single color of filter with imaging resolution greater than a set resolution to form the first filter unit 12251 in the filter array 1225, or selects a single color of filter with quantum efficiency greater than a set efficiency and imaging resolution greater than a set resolution to form the first filter unit 12251 in the filter array 1225. After the first filter unit 12251 is selected, the number of filters in the first filter unit 12251 needs to be greater than or equal to half of the total number of filters in the filter array 1225.

[0094] It should be noted that quantum efficiency refers to the ability of an image sensor or a certain filter waveband to respond to incident photons, specifically, how many of the photons falling on the image sensor pixels can be successfully converted into electrical signals. In the filter array 1225, if the quantum efficiency of a certain color of filter for the photons in the transmission waveband is high, it means that the quality of the color channel is good and the photosensitivity is high. The imaging resolution refers to the number of pixels covered by the color filter or its sampling density in the entire image. Selecting a filter with imaging resolution greater than a set resolution can make the imaging clearer while more details.

[0095] And, the number of filters in the first filter unit 12251 needs to be greater than or equal to half of the total number of filters in the filter array 1225, which can improve the sampling density of the image signal of the corresponding waveband of the first filter unit 12251, thereby enhancing the image resolution and edge detail performance of the corresponding waveband of the first filter unit 12251. At the same time, the information collected by the first filter unit 12251 corresponding to the waveband is used as the main reference color channel in the image reconstruction process, which assists the interpolation operation of other color channels, thereby improving the clarity and accuracy of the overall image. In addition, considering that the human eye has higher perceptual sensitivity to this waveband, increasing the proportion of the first filter unit 12251 can help obtain higher quality luminance information, and has higher signal-to-noise ratio and visual effect in image enhancement and subsequent analysis process.

[0096] Please refer to Figure 6 In the specific implementation process of the present application, the green filter is selected as the first filter unit 12251, that is, the green filter is distributed in a checkerboard pattern in the filter array 1225 as the first filter unit 12251, and the filters of the remaining colors except green are arranged as the second filter unit 12252 in the remaining positions except the distribution position of the first filter unit 12251. It can be observed that Figure 6 It can be seen that the second filter unit 12252 is also distributed in a checkerboard pattern in the filter array 1225.

[0097] In an embodiment, the second filter unit 12252 includes a first sub-filter unit 122521 and a second sub-filter unit 122522. The first sub-filter unit 122521 and the second sub-filter unit 122522 each include filters of at least two colors, and the colors of the filters in the first sub-filter unit 122521 and the second sub-filter unit 122522 are different. The spectral response difference of the filters of various colors in the first sub-filter unit 122521 is less than a first set value, and the spectral response difference of the filters of various colors in the second sub-filter unit 122522 is less than a second set value. The spectral response difference of the filters of various colors in the first sub-filter unit 122521 and the second sub-filter unit 122522 is greater than a third set value.

[0098] In one embodiment, to improve sampling continuity in the spectral transition region, filters of different colors in the first sub-filter unit 122521 are alternately arranged horizontally in the second filter unit 12252. Filters of different colors in the second sub-filter unit 122522 are also alternately arranged horizontally in the second filter unit 12252. In this case, the distribution of filters in each row of the first sub-filter units 122521 in the second filter unit 12252 may be the same as in the first case. For example, if the first sub-filter unit 122521 includes orange and red filters, then in the first case, filters in each row of the first sub-filter units 122521 are alternately arranged in the order of orange-red. Furthermore, the distribution of filters in each row of the first sub-filter units 122521 in the second filter unit 12252 may also be different as in the second case. For example, in the second case, the first sub-filter units 122521 in any row of the second filter units 12252 are alternately distributed in the order of orange-red, and the first sub-filter units 122521 in another adjacent row are alternately distributed in the order of red-orange.

[0099] In one embodiment, for special purposes, the present application further requires that the filters of different colors in the first sub-filter unit 122521 be alternately spaced in the vertical direction of the second filter unit 12252, and that the filters of different colors in the second sub-filter unit 122522 be alternately spaced in the vertical direction of the second filter unit 12252. That is, the first sub-filter array 1225 and the second sub-filter array 1225 in the present application satisfy the second condition described above. Please refer to Figure 7 and Figure 8 Similarly, taking the example that the first sub-filter unit 122521 includes orange and red filters and the second sub-filter unit 122522 includes blue and purple filters, the second case specifically defined in this application is as follows: the filters in the first sub-filter unit 122521 and the second sub-filter unit 122522 are as follows: Figure 7 and Figure 8 Set it up as shown.

[0100] In one embodiment, to avoid image interference or artifacts caused by the concentrated distribution of spectral contrast pixels and to improve the overall stability of the original image and spectral decoding efficiency, the present application further arranges the first sub-filter units 122521 and the second sub-filter units 122522 in alternating intervals. That is, when the first sub-filter units 122521 are arranged in the first row of the second filter units 12252, the second sub-filter units 122522 are arranged in the second row of the second filter units 12252; when the first sub-filter units 122521 are arranged in the first column of the second filter units 12252, the second sub-filter units 122522 are arranged in the second column of the second filter units 12252, and so on.

[0101] Based on the hardware architecture of the above-mentioned imaging component 122, whether it is an imaging system 12 used independently as a separate system or an imaging system 12 as a core component of the endoscope system 1, the processor 123 included therein can generate a display image for output of the detected area based on a grayscale image, or can generate a display image for output of the detected area based on a hyperspectral image, or can generate a display image for output of the detected area based on a grayscale image and a hyperspectral image.

[0102] Please refer to Figure 9 In one embodiment, the method in which the processor 123 generates a display image corresponding to the detected area for output based on the grayscale image includes the following steps.

[0103] Step S110: Obtain a preset color mapping table.

[0104] In one embodiment, the processor 123 calls a preset color mapping table to map each possible grayscale value to a specific color. The color mapping table can be a standard color scale or customized according to a specific application scenario, such as highlighting lesions with red or enhancing blood vessels with blue.

[0105] Step S120: determining the color value corresponding to the grayscale value of each pixel in the grayscale image according to the color mapping table to generate a display image for output in the detected area.

[0106] In one embodiment, the processor 123 traverses each pixel in the grayscale image, reads its grayscale value, searches for the color corresponding to each grayscale value in a preset color mapping table, and replaces each grayscale pixel with its corresponding color value, thereby generating a new color image.

[0107] In one embodiment, the processor 123 generates a display image corresponding to the detected area for output based on the hyperspectral image, including three methods. Among them, due to the different arrangements of the filter array, it is divided into the second method and the third method, which are described in detail below.

[0108] Please refer to Figure 10 ,The first method includes the following steps.

[0109] Step S210: Acquire a checkerboard image of a first color channel.

[0110] In one embodiment, the hyperspectral image includes a checkerboard image of a first color channel corresponding to the filter color in the first filter unit 12251 determined by the hyperspectral image sensor 1224 through the first filter unit 12251. If a green filter is selected as the first filter unit 12251 in this application, the checkerboard image of the first color channel is as follows: Figure 11 Shown is a green checkerboard image.

[0111] Step S230: Taking the point to be calculated as the center, select a pixel point corresponding to the first filter color in the checkerboard image of the first color channel.

[0112] In one embodiment, in order to calculate the pixel value of any point to be calculated, in the checkerboard image of the first color channel, with the point to be calculated as the center, a total of 16 pixel points corresponding to the first filter color are obtained in the four directions of the top, bottom, left, and right of the point to be calculated. If a green filter is selected as the first filter unit 12251 in this application, then the pixel points corresponding to the first filter color are as follows: Figure 11 Green pixels are shown.

[0113] Step S250: Calculate the pixel value of the point to be calculated using a bicubic interpolation method according to the pixel point corresponding to the selected first filter color.

[0114] In one embodiment, Figure 11 The selection in the above is taken as an example, and the 16 green pixels obtained in the four directions of the point to be calculated are described in the form of a coordinate system. Figure 11 The green pixels in . Here, the direction from top to bottom and from left to right is described. Figure 11 The coordinate point corresponding to each pixel in is (i, j), i and j are both positive integers, i∈[1, 7], j∈[1, 7].

[0115] Use the following formula to calculate the pixel value of the point to be calculated:

[0116]

[0117] Among them, G represents the pixel value of the point to be calculated, S1 and S0 represent the interpolation kernel coefficients, G11 represents the green pixel value corresponding to the filter color in the first filter unit 12251 corresponding to the coordinates (1, 4), G12 represents the green pixel value corresponding to the filter color in the first filter unit 12251 corresponding to the coordinates (2, 5), G13 represents the green pixel value corresponding to the filter color in the first filter unit 12251 corresponding to the coordinates (3, 6), and G14 represents the green pixel value corresponding to the filter color in the first filter unit 12251 corresponding to the coordinates (4, 7). G21 represents the green pixel value corresponding to the filter color in the first filter unit 12251 corresponding to the coordinates (2, 3), G22 represents the green pixel value corresponding to the filter color in the first filter unit 12251 corresponding to the coordinates (3, 4), G23 represents the green pixel value corresponding to the filter color in the first filter unit 12251 corresponding to the coordinates (4, 5), G24 represents the green pixel value corresponding to the filter color in the first filter unit 12251 corresponding to the coordinates (5, 6), The green pixel value corresponding to the filter color, G31 represents the green pixel value corresponding to the filter color in the first filter unit 12251 corresponding to the coordinates (3, 2), G32 represents the green pixel value corresponding to the filter color in the first filter unit 12251 corresponding to the coordinates (4, 3), G33 represents the green pixel value corresponding to the filter color in the first filter unit 12251 corresponding to the coordinates (5, 4), G34 represents the green pixel value corresponding to the filter color in the first filter unit 12251 corresponding to the coordinates (6, 5), G41 represents the green pixel value corresponding to the filter color in the first filter unit 12251 corresponding to the coordinates (4, 1), G42 represents the green pixel value corresponding to the filter color in the first filter unit 12251 corresponding to the coordinates (5, 2), G43 represents the green pixel value corresponding to the filter color in the first filter unit 12251 corresponding to the coordinates (6, 3), and G44 represents the green pixel value corresponding to the filter color in the first filter unit 12251 corresponding to the coordinates (7, 4).

[0118] The above method is used to calculate the pixel value of each point to be calculated, thereby generating a display image for output in the detected area.

[0119] Please refer to Figure 12 ,The second method includes the following steps.

[0120] Step S220: Acquire a checkerboard image of the first color channel and a checkerboard image of the second color channel.

[0121] In one embodiment, the hyperspectral image includes a checkerboard image of a first color channel corresponding to the filter color in the first filter unit 12251 determined by the hyperspectral image sensor 1224 through the first filter unit 12251, and a checkerboard image of a second color channel corresponding to the filter color in the second filter unit 12252 determined by the hyperspectral image sensor 1224 through the second filter unit 12252. If a green filter is selected as the first filter unit 12251 in this application, the checkerboard image of the first color channel is as follows: Figure 11 If the application selects an orange filter and a red filter as the first sub-filter unit 122521, and selects a purple filter and a blue filter as the second sub-filter unit 122522, please refer to Figure 6 The checkerboard image composed of orange, red, purple and blue pixels except the green pixels is the checkerboard image of the second color channel.

[0122] Step S240: interpolating the pixel values ​​of the first color corresponding to the first filter unit at the blank positions in the checkerboard image of the first color channel to generate a complete image of the first color channel.

[0123] Please refer to Figure 13 In one embodiment, when executing step S240 to interpolate the pixel values ​​of the first color corresponding to the first filter unit at the blank positions in the checkerboard image of the first color channel to generate a complete image of the first color channel, the following steps are also included.

[0124] Step S241: Acquire a first training image at full resolution, and generate a second training image corresponding to the checkerboard image of the first color channel based on the first training image.

[0125] Please refer to Figure 14 In one embodiment, when executing step S241 to obtain a first training image with full resolution and generating a second training image corresponding to the checkerboard image of the first color channel based on the first training image, the following steps are also included.

[0126] Step S2411: merging pixel points of the first color pixel value in the checkerboard image of the first color channel to generate a sampled image.

[0127] In one embodiment, since the checkerboard image of the first color channel is determined by the color correspondence of the filter in the first filter unit 12251, there are pixel points corresponding to the filters in the first color filter unit in the checkerboard image of the first color channel, and the first color pixel values ​​corresponding to the first color filter unit in these pixel points are all known, these pixel points are merged to generate a sampled image. Figure 11 ,Will Figure 11 All green pixels in are merged to generate the sampled image.

[0128] Step S2412: down-sample the sampled image according to a set specification to generate a first training image.

[0129] In one embodiment, an image of a predetermined size is cut out from the sampled image to perform downsampling, thereby generating a first training image. Specifically, an image of any size, such as 3×3, 4×4, or 5×5, is cut out from the sampled image to generate the first training image. Since the first training image is determined by the sampled image, and the pixel values ​​of all pixels in the sampled image are known, all pixels in the first training image are also known. Therefore, the first training image is the full-resolution first training image.

[0130] Step S2413: Generate a second training image according to the blank positions in the checkerboard image of the first color channel.

[0131] In one embodiment, the blank positions in the checkerboard image of the first color channel are used as a reference to extract the pixel positions where the corresponding pixel values ​​in the first training image do not exist, thereby generating a second training image. The blank positions are the pixel positions where the pixel values ​​of the first color do not exist in the checkerboard image of the second color channel in the hyperspectral image, that is, Figure 11 The white pixel position in .

[0132] Step S242: training the convolutional neural network model according to the first training image and the second training image.

[0133] In one embodiment, a first training image and a second training image are input into a convolutional neural network model, wherein the first training image is a fully-resolution image and the second training image is an image with missing pixel values. The convolutional neural network model is trained using the first training image and the second training image, so that the convolutional neural network model learns how to estimate the missing pixel values ​​in the image with missing pixel values ​​from the fully-resolution image. The present application uses a convolutional neural network with no less than 2 layers to extract local features in the first training image and the second training image and perform interpolation prediction. The mean square error is used as the loss function to measure the error between the inferred pixel value and the true pixel value. Finally, the Adam optimizer is used to optimize the parameters to ensure that the convolutional neural network model can effectively converge, thereby completing the training of the convolutional neural network model.

[0134] Step S243: Input the checkerboard image of the first color channel into the trained convolutional neural network model to generate a complete image of the first color channel.

[0135] In one embodiment, since the convolutional neural network model has been trained, the checkerboard image of the first color channel is input into the trained convolutional neural network model to obtain a complete image of the first color channel.

[0136] Step S260: interpolating the pixel values ​​of each pixel in the hyperspectral image in different color channels according to the complete image of the first color channel and the checkerboard image of the second color channel.

[0137] Please refer to Figure 15 In one embodiment, when executing step S260, interpolating the pixel values ​​of each pixel in the hyperspectral image in different color channels according to the complete image of the first color channel and the checkerboard image of the second color channel, the following steps are included.

[0138] Step S261: Determine a first native color pixel value, a first to-be-interpolated color pixel value, a second native color pixel value, an interpolated first color pixel value, and a second to-be-interpolated color pixel value.

[0139] In one embodiment, since the complete image of the first color channel is determined by the checkerboard image of the first color channel, and the checkerboard image of the first color channel is determined by the first filter unit 12251, the pixel value corresponding to the first color in each pixel point in the complete image of the first color channel is natively known, that is, the first native color pixel value, while the pixel values ​​of other color channels are unknown, that is, the first color pixel value to be interpolated.

[0140] In one embodiment, the checkerboard image of the second color channel is determined by the second filter unit 12252. Then, in the checkerboard image of the second color channel, the pixel value of each pixel corresponding to the second filter unit 12252 is natively known, that is, the second native color pixel value.

[0141] In one embodiment, a complete image of the first color channel is generated by interpolating the blank positions in the checkerboard image of the first color channel, and the blank positions in the checkerboard image of the first color channel are the pixel positions in the checkerboard image of the second color channel where the pixel values ​​of the first color do not exist. Therefore, the positions interpolated in the checkerboard image of the first color channel are the pixel positions in the checkerboard image of the second color channel where the pixel values ​​of the first color do not exist. This also indicates that the first color pixel values ​​have been interpolated at the pixel positions in the checkerboard image of the second color channel where the pixel values ​​of the first color do not exist, i.e., the interpolated first color pixel values.

[0142] In one embodiment, the second native color pixel values ​​and the interpolated first color pixel values ​​are removed from the checkerboard image of the second color channel, and the remaining values ​​are the color pixel values ​​that need to be interpolated in the checkerboard image of the second color channel, i.e., the second color pixel values ​​to be interpolated.

[0143] For example, Figure 16 As shown, the checkerboard image of the first color channel is a checkerboard image composed of green pixels. In the checkerboard image of the first color channel, the green pixel values ​​of all green pixels are the first native color pixel values. The pixel values ​​of all green pixels in the other color channels (such as Figure 16 The B and R to be calculated, that is, the blue pixel value and the red pixel value to be calculated) are the first color pixel values ​​to be interpolated.

[0144] The checkerboard image of the second color channel is a checkerboard image composed of blue pixels and red pixels. In the checkerboard image of the second color channel, the pixel values ​​of all blue pixels and the pixel values ​​of all red pixels are the second original color pixel values. The first color pixel value interpolated from the pixel position where the first color pixel value does not exist in the checkerboard image of the second color channel is the interpolated first color pixel value, that is, Figure 16 The green pixel value interpolated from the blue pixel and the red pixel (such as Figure 16 Among the blue and red pixels, G has been interpolated, that is, the green pixel value has been interpolated). The pixel values ​​of the remaining color channels in the pixel points with known pixel values ​​in the checkerboard image of the second color channel are the second color pixel values ​​to be interpolated. Figure 16In the embodiment, the to-be-calculated R (to-be-calculated red pixel value) in the blue pixel point is the second to-be-interpolated color pixel value in the checkerboard image of the second color channel. The to-be-calculated B (to-be-calculated blue pixel value) in the red pixel point is also the second to-be-interpolated color pixel value in the checkerboard image of the second color channel.

[0145] Step S263: determining the first to-be-interpolated color pixel value according to the first native color pixel value, the second native color pixel value and the interpolated first color pixel value.

[0146] Please refer to Figure 17 In an embodiment, the step S263 of determining the first to-be-interpolated color pixel value according to the first native color pixel value, the second native color pixel value and the interpolated first color pixel value includes the following steps.

[0147] Step S2631: obtaining the second native color pixel value and the interpolated first color pixel value of the pixel point closest to the first to-be-interpolated color pixel value in the horizontal direction or the vertical direction of the checkerboard image of the second color channel.

[0148] In an embodiment, after the first native color pixel value, the second native color pixel value and the interpolated first color pixel value are determined, the second native color pixel value and the interpolated first color pixel value of the pixel point closest to the first to-be-interpolated color pixel value in the horizontal direction or the vertical direction of the checkerboard image of the second color channel are obtained.

[0149] Since the checkerboard image of the first color channel and the checkerboard image of the second color channel are checkerboard images arranged in a cross manner, there must be the second native color pixel value and the interpolated first color pixel value required for calculating the first to-be-interpolated color pixel value on the adjacent pixel point in the horizontal direction or the vertical direction of the pixel point corresponding to the first to-be-interpolated color pixel value.

[0150] Step S2632: determining the first to-be-interpolated color pixel value according to the first native color pixel value, the second native color pixel value and the interpolated first color pixel value of the closest pixel point.

[0151] In an embodiment, also refer to Figure 16 The green pixel value of any green pixel point is the first native color pixel value, and the to-be-calculated B and / or the to-be-calculated R in the green pixel point are the first to-be-interpolated color pixel value.

[0152] In the horizontal direction of the green pixel, the two closest adjacent blue pixels are selected. The blue pixel values ​​(original B) of the two adjacent blue pixels are used as the second original color pixel value, and the interpolated green pixel values ​​(interpolated G) of the two adjacent blue pixels are used as the interpolated first color pixel value. The first color pixel value to be interpolated is calculated using the following formula:

[0153]

[0154] in, Figure 16 In the 3×3 image, from top to bottom and from left to right is the reference, B (1,2) G represents the blue pixel value of the pixel corresponding to the coordinate (1, 2), that is, the first color pixel value to be interpolated; (1,2) represents the green pixel value of the pixel corresponding to the coordinate (1, 2), that is, the first native color pixel value; B (1,1) G represents the blue pixel value of the pixel corresponding to the coordinate (1, 1), that is, the second native color pixel value; (1,1) represents the green pixel value of the pixel point corresponding to the coordinate (1, 1), that is, the interpolated first color pixel value; B (1,3) G represents the blue pixel value of the pixel corresponding to the coordinate (1, 3), that is, the second native color pixel value; (1,3) Represents the green pixel value of the pixel corresponding to the coordinates (1, 3), that is, the interpolated first color pixel value.

[0155] In the vertical direction of the green pixel, the two closest adjacent red pixels are selected. The red pixel values ​​(original R) of the two adjacent red pixels are used as the second original color pixel value, and the interpolated green pixel values ​​(interpolated G) of the two adjacent red pixels are used as the interpolated first color pixel value. The first color pixel value to be interpolated is calculated using the following formula:

[0156]

[0157] in, Figure 16 In the 3×3 image, from top to bottom and from left to right is the reference, R (1,2) The red pixel value of the pixel corresponding to the coordinate (1, 2), that is, the first color pixel value to be interpolated; G (1,2) represents the green pixel value of the pixel corresponding to the coordinate (1, 2), that is, the first native color pixel value; R (0,2) represents the red pixel value of the pixel corresponding to the coordinate (0, 2), that is, the second native color pixel value (not shown in the figure); G (0,2) represents the green pixel value of the pixel corresponding to the coordinate (0, 2), that is, the interpolated first color pixel value (not shown in the figure); R (2,2)G represents the red pixel value of the pixel corresponding to the coordinate (2, 2), that is, the second native color pixel value; (2,2) Represents the green pixel value of the pixel corresponding to the coordinate (2, 2), that is, the interpolated first color pixel value.

[0158] Step S265 : determining a second to-be-interpolated color pixel value according to the second native color pixel value and the interpolated first color pixel value.

[0159] Please refer to Figure 18 In one embodiment, when executing step S265 to determine the second color pixel value to be interpolated according to the second native color pixel value and the interpolated first color pixel value, the following steps are also included.

[0160] Step S2651: constructing an n×n interpolation unit with any pixel corresponding to the second to-be-interpolated color pixel value in the checkerboard image of the second color channel as the center point.

[0161] In one embodiment, in the checkerboard image of the second color channel, any pixel corresponding to the second color pixel value to be interpolated is selected as the center point to construct an n×n interpolation unit, where n is an odd number greater than 3.

[0162] Step S2653: Determine the second color pixel value to be interpolated.

[0163] In one embodiment, after determining the n×n interpolation units, the second color pixel value to be interpolated can be determined by using the second native color pixel value of the pixel point at the main diagonal vertex in the interpolation unit and the interpolated first color pixel value; the second color pixel value to be interpolated can also be determined by using the second native color pixel value of the pixel point at the sub-diagonal vertex in the interpolation unit and the interpolated first color pixel value; the second color pixel value to be interpolated can also be determined by using the second native color pixel value of the pixel point at the main diagonal vertex of the interpolation unit and the interpolated first color pixel value, as well as the second native color pixel value of the pixel point at the sub-diagonal vertex of the interpolation unit and the interpolated first color pixel value, which will be explained in detail below.

[0164] In one embodiment, since the distribution conditions of the first sub-filter units 122521 in each row of the second filter unit 12252 may be the same in a first situation, and the distribution conditions of the first sub-filter units 122521 in each row of the second filter unit 12252 may also be different in a second situation, in the interpolation unit, there may be a situation where the second native color pixel values ​​of the pixels at the vertices of the main diagonal or sub-diagonal are the same, and the second native color pixel values ​​of the pixels at the vertices of the main diagonal or sub-diagonal in the interpolation unit are different.

[0165] In one embodiment, when the second native color pixel values of the pixel points at the vertices of the main diagonal or the sub-diagonal in the interpolation unit are the same:

[0166] First, the gradient change of the main diagonal is calculated according to the second native color pixel values of the pixel points at the vertices of the main diagonal in the interpolation unit and the first color pixel values that have been interpolated, which is specifically calculated by using the following formula:

[0167] GradN = |B i-1,j-1 -B i+1,j+1 | + |2 x G i,j -(G i-1,j-1 + G i+1,j+1 )|

[0168] wherein GradN represents the gradient of the main diagonal, B i-1,j-1 represents the second native color pixel value of the pixel point at one vertex of the main diagonal, B i+1,j+1 represents the second native color pixel value of the pixel point at the other vertex of the main diagonal, G i,j represents the first color pixel value that has been interpolated in the pixel point corresponding to the second color pixel value to be interpolated, G i-1,j-1 represents the first color pixel value that has been interpolated in the pixel point at one vertex of the main diagonal, and G i+1,j+1 represents the first color pixel value that has been interpolated in the pixel point at the other vertex of the main diagonal.

[0169] Then, the gradient change of the sub-diagonal is calculated according to the second native color pixel values of the pixel points at the vertices of the sub-diagonal in the interpolation unit and the first color pixel values that have been interpolated, which is specifically calculated by using the following formula:

[0170] GradP = |B i-1,j+1 -B i+1,j-1 | + |2 x G i,j -(G i-1,j+1 + G i+1,j-1 )|

[0171] wherein GradP represents the gradient of the sub-diagonal, B i-1,j+1 represents the second native color pixel value of the pixel point at one vertex of the sub-diagonal, B i+1,j-1 represents the second native color pixel value of the pixel point at the other vertex of the sub-diagonal, G i,j represents the first color pixel value that has been interpolated in the pixel point corresponding to the second color pixel value to be interpolated, G i-1,j+1 represents the first color pixel value that has been interpolated in the pixel point at one vertex of the sub-diagonal, and G i+1,j-1 represents the first color pixel value that has been interpolated in the pixel point at the other vertex of the sub-diagonal.

[0172] If the gradient change of the main diagonal is smaller than the gradient change of the sub-diagonal, the second original color pixel value of the pixel point at the vertex of the main diagonal of the interpolation unit and the interpolated first color pixel value are used to determine the second color pixel value to be interpolated. Specifically, the calculation is performed using the following formula:

[0173]

[0174] Among them, B i,j Indicates the second color pixel value to be interpolated.

[0175] If the gradient change of the main diagonal is greater than the gradient change of the sub-diagonal, the second original color pixel value of the pixel point at the sub-diagonal vertex of the interpolation unit and the interpolated first color pixel value are used to determine the second color pixel value to be interpolated. Specifically, the calculation is performed using the following formula:

[0176]

[0177] Among them, B i,j Indicates the second color pixel value to be interpolated.

[0178] If the gradient change of the main diagonal is equal to the gradient change of the sub-diagonal, the second to-be-interpolated color pixel value is determined using the second native color pixel value and the interpolated first color pixel value of the pixel point at the vertex of the main diagonal of the interpolation unit, and the second native color pixel value and the interpolated first color pixel value of the pixel point at the vertex of the sub-diagonal of the interpolation unit. Specifically, the calculation is performed using the following formula:

[0179]

[0180] Among them, B i,j Indicates the second color pixel value to be interpolated.

[0181] In one embodiment, when the second native color pixel values ​​of the pixel points at the main diagonal or sub-diagonal vertices in the interpolation unit are different:

[0182] Calculate the pixel value corresponding to the second native color pixel value of the pixel point of one of the main diagonal or sub-diagonal vertices and the pixel value of the other vertex, and name the pixel value as the second native color pixel value. Specifically: update the interpolation unit with the pixel point of any vertex in the main diagonal or sub-diagonal as the center point. Update the second native color pixel value of any vertex in the main diagonal or sub-diagonal to the second color pixel value to be interpolated in the updated interpolation unit, and then use the above-mentioned interpolation unit to calculate the second color pixel value to be interpolated to calculate the second color pixel value to be interpolated in the updated interpolation unit. The second color pixel value to be interpolated calculated in this way is the pixel value corresponding to the second native color pixel value of the pixel point of one of the main diagonal or sub-diagonal vertices and the pixel value of the other vertex.

[0183] Specifically, refer to Figure 16 , the blue pixel value in the red pixel point corresponding to the coordinates (2, 2) is used as the second color pixel value to be interpolated, and the green pixel value in the red pixel point corresponding to the coordinates (2, 2) is used as the first color pixel value interpolated in the pixel point corresponding to the second color pixel value to be interpolated.

[0184] The blue pixel value of the pixel corresponding to the coordinate (1, 1) is the second native color pixel value of the pixel at one of the vertices of the main diagonal, and the green pixel value of the pixel corresponding to the coordinate (1, 1) is the interpolated first color pixel value of the pixel at one of the vertices of the main diagonal. The blue pixel value of the pixel corresponding to the coordinate (3, 3) is the second native color pixel value of the pixel at another vertex of the main diagonal, and the green pixel value of the pixel corresponding to the coordinate (3, 3) is the interpolated first color pixel value of the pixel at another vertex of the main diagonal. Then the gradient change of the main diagonal is:

[0185] GradN=|B 1,1 -B 3,3 |+|2×G 2,2 -(G 1,1 +G 3,3 )|

[0186] Among them, GradN represents the gradient of the main diagonal, B 1,1 Indicates the blue pixel value of the pixel corresponding to the coordinate (1, 1), B 3,3 Indicates the blue pixel value of the pixel corresponding to the coordinate (3, 3), G 2,2 Represents the green pixel value in the red pixel corresponding to the coordinate (2, 2), G 1,1 Indicates the green pixel value of the pixel corresponding to the coordinate (1, 1), G 3,3 Indicates the green pixel value of the pixel corresponding to the coordinate (3, 3).

[0187] The blue pixel value of the pixel point corresponding to the coordinate (1, 3) is the second primary color pixel value of the pixel point of one of the vertices of the sub-diagonal line, and the green pixel value of the pixel point corresponding to the coordinate (1, 3) is the interpolated first color pixel value of the pixel point of the other vertex of the sub-diagonal line. The blue pixel value of the pixel point corresponding to the coordinate (3, 1) is the second primary color pixel value of the pixel point of the other vertex of the sub-diagonal line, and the green pixel value of the pixel point corresponding to the coordinate (3, 1) is the interpolated first color pixel value of the pixel point of the other vertex of the sub-diagonal line. Then the gradient change of the sub-diagonal line is:

[0188] GradP = |B 1,3 -B 3,1 |+|2×G 2,2 -(G 1,3 +G 3,1 )|

[0189] wherein GradP represents the gradient of the sub-diagonal line, B 1,3 represents the blue pixel value of the pixel point corresponding to the coordinate (1, 3), B 3,1 represents the blue pixel value of the pixel point corresponding to the coordinate (3, 1), G 2,2 represents the green pixel value in the red pixel point corresponding to the coordinate (2, 2), G 1,3 represents the blue pixel value of the pixel point corresponding to the coordinate (1, 3), G 3,1 represents the green pixel value of the pixel point corresponding to the coordinate (3, 1).

[0190] If the gradient change of the main diagonal line is less than the gradient change of the sub-diagonal line, then:

[0191]

[0192] wherein B 2,2 represents the green pixel value in the red pixel point corresponding to the coordinate (2, 2).

[0193] If the gradient change of the main diagonal line is greater than the gradient change of the sub-diagonal line, then:

[0194]

[0195] wherein B 2,2 represents the green pixel value in the red pixel point corresponding to the coordinate (2, 2).

[0196] If the gradient change of the main diagonal line is equal to the gradient change of the sub-diagonal line, then:

[0197]

[0198] wherein B 2,2The green pixel value in the red pixel point corresponding to the coordinate (2, 2) is represented.

[0199] If the second native color pixel value of the pixel point corresponding to the coordinate (1, 1) is not a blue pixel value in the second native color pixel values of the pixel point corresponding to the coordinate (1, 1) and the pixel point corresponding to the coordinate (3, 3), the blue pixel value of the pixel point corresponding to the coordinate (1, 1) needs to be calculated to ensure that the second native color pixel values of the pixel point corresponding to the coordinate (1, 1) and the pixel point corresponding to the coordinate (3, 3) are both blue pixel values.

[0200] Taking the pixel point corresponding to the coordinate (1, 1) as the center point, an n x n interpolation unit is also constructed, and the blue pixel value of the pixel point corresponding to the coordinate (1, 1) is calculated in the same way as calculating B 2,2 .

[0201] It should be noted that when calculating B 2,2 , the pixel points at the vertices of the main diagonal and the auxiliary diagonal in the interpolation unit are used, and when calculating the blue pixel value of the pixel point corresponding to the coordinate (1, 1), the pixel points at the vertices of the main diagonal and the auxiliary diagonal in the interpolation unit are also used. Since the center point of the interpolation unit has been updated, the pixel points at the vertices of the main diagonal and the auxiliary diagonal in the interpolation unit are also updated.

[0202] In an embodiment, in the step S2652 of determining the second to-be-interpolated color pixel value, a 3 x 3 interpolation unit is constructed, and the interpolation unit can also be a 5 x 5 or 7 x 7 interpolation unit.

[0203] When a 5 x 5 interpolation unit is constructed, the pixel points at the vertices of the main diagonal or the auxiliary diagonal in the interpolation unit are also used to calculate the gradient change of the main diagonal or the auxiliary diagonal. Since the color change on the diagonal is approximately linear, the sum of the pixel values of the three pixel points on the intermediate diagonal can be determined according to the gradient change, and then the second to-be-interpolated color pixel value can be determined according to the sum of the pixel values of the three pixel points on the intermediate diagonal and the calculated gradient change. The specific formula used is:

[0204] T = B1 + B2 + B3

[0205] B1 = B2 - a

[0206] B3 = B2 + a

[0207] Then:

[0208] T = B2 - a + B2 + B2 + a = 3 x B2

[0209] Then:

[0210]

[0211] wherein, T represents the sum of the pixel values of the three pixel points of the middle diagonal line, B1 represents the pixel value of the pixel point of one vertex of the middle diagonal line, B2 represents the second color pixel value to be interpolated, B3 represents the pixel value of the pixel point of the other vertex of the middle diagonal line, and a represents the gradient change of the main diagonal line or the sub-diagonal line.

[0212] The pixel values of each pixel point in the hyperspectral image in different color channels are interpolated by using the above method, thereby generating a display image of the detected region for output.

[0213] In one embodiment, the third method comprises the following steps.

[0214] In one embodiment, the third method adopts the same method as in steps S220-S260 of the second method, and only a different method is adopted in step S260, which is specifically as follows.

[0215] Please refer to Figure 19 In one embodiment, when the pixel values of each pixel point in the hyperspectral image in different color channels are interpolated according to the complete image of the first color channel and the checkerboard image of the second color channel in step S260, the following steps are included.

[0216] Step S262: determining the first native color pixel value, the first color pixel value to be interpolated, the second native color pixel value, the first color pixel value that has been interpolated, and the second color pixel value to be interpolated.

[0217] Since the same method is adopted in step S262 as in step S261, no further description is given here, and only examples are given for further illustration. For example, as shown in Figure 20 The checkerboard image of the first color channel is a checkerboard image composed of green pixel points. In the checkerboard image of the first color channel, the green pixel values of all green pixel points are the first native color pixel value. The pixel values (such as R, B, P and A to be calculated in Figure 20 in the remaining color channels) of all green pixel points are the first color pixel value to be interpolated.

[0218] The checkerboard image of the second color channel is a checkerboard image composed of blue pixel points, purple pixel points, red pixel points and orange pixel points. In the checkerboard image of the second color channel, the pixel values of all blue pixel points, the pixel values of all purple pixel points, the pixel values of all red pixel points and the pixel values of all orange pixel points are the second native color pixel value. The first color pixel value interpolated in the pixel point position where the first color pixel value does not exist in the checkerboard image of the second color channel is the first color pixel value that has been interpolated, that is, in Figure 20 The green pixel value interpolated from the blue pixel, purple pixel, red pixel and orange pixel (such as Figure 20 Among the blue and purple pixels, G has been interpolated, that is, the green pixel value has been interpolated). The pixel values ​​of the remaining color channels in the pixel points with known pixel values ​​in the checkerboard image of the second color channel are the second color pixel values ​​to be interpolated. Figure 20 In the figure, the R, B, and A values ​​to be calculated for the purple pixel (the red pixel value to be calculated, the blue pixel value to be calculated, and the orange pixel value to be calculated) are the second color pixel values ​​to be interpolated in the checkerboard image of the second color channel. The R, P, and A values ​​to be calculated for the blue pixel and the B, P, and A values ​​to be calculated for the red pixel are also the second color pixel values ​​to be interpolated in the checkerboard image of the second color channel.

[0219] Step S264 : determining a first to-be-interpolated color pixel value according to the first native color pixel value, the second native color pixel value, and the interpolated first color pixel value.

[0220] Please refer to Figure 21 In one embodiment, determining the first color pixel value to be interpolated according to the first native color pixel value, the second native color pixel value and the interpolated first color pixel value in step S264 includes the following steps.

[0221] Step S2641: Obtain the second native color pixel value and the interpolated first color pixel value of the pixel point closest to the pixel point corresponding to the first color pixel value to be interpolated in the horizontal direction or vertical direction in the checkerboard image of the second color channel.

[0222] In one embodiment, after determining the first native color pixel value, the second native color pixel value, and the interpolated first color pixel value, the second native color pixel value and the interpolated first color pixel value of the pixel point in the first-order neighborhood closest to the pixel point corresponding to the first color pixel value to be interpolated are obtained in the horizontal direction or vertical direction of the checkerboard image of the second color channel.

[0223] Step S2642: Determine a first color pixel value to be interpolated according to the first native color pixel value, the second native color pixel value of the closest pixel point, and the interpolated first color pixel value.

[0224] In one embodiment, also referring to Figure 20 The green pixel value of any green pixel point is the first native color pixel value, and the R, B, P and / or A to be calculated in the green pixel value point are all the first color pixel values ​​to be interpolated.

[0225] In the horizontal direction of the green pixel, a purple pixel in the adjacent first-order neighborhood is selected. The purple pixel value (original P) of the purple pixel is used as the second original color pixel value, and the interpolated green pixel value (interpolated G) of the purple pixel is used as the interpolated first color pixel value. The first color pixel value to be interpolated is calculated using the following formula:

[0226] P (6,7) =G (6,7) +P (6,6) -G (6,6)

[0227] in, Figure 20 In the 5×5 image, from top to bottom and from left to right, the corresponding coordinates are (5, 5), (5, 6), (5, 7), (5, 8), (5, 9); (6, 5), (6, 6), (6, 7), (6, 8), (6, 9); (7, 5), (7, 6), (7, 7), (7, 8), (7, 9); (8, 5), (8, 6), (8, 7), (8, 8), (8, 9); (9, 5), (9, 6), (9, 7), (9, 8), (9, 9). (6,7) G represents the purple pixel value of the pixel corresponding to the coordinates (6, 7), that is, the first color pixel value to be interpolated; (6,7) represents the green pixel value of the pixel corresponding to the coordinates (6, 7), that is, the first native color pixel value; P (6,6) G represents the purple pixel value of the pixel corresponding to the coordinate (6, 6), that is, the second native color pixel value; (6,6) Represents the green pixel value of the pixel corresponding to the coordinate (6, 6), that is, the interpolated first color pixel value.

[0228] The same method is used to select a blue pixel point in the adjacent 1st-order neighborhood in the horizontal direction of the green pixel point, and the first color pixel value to be interpolated is calculated using the second native color pixel value of the blue pixel point and the interpolated first color pixel value. No further details are given here.

[0229] The same method is used to select an orange pixel in the adjacent first-order neighborhood in the vertical direction of the green pixel, and the first color pixel value to be interpolated is calculated using the second original color pixel value of the orange pixel and the interpolated first color pixel value. No further details are given here.

[0230] The same method is used to select a red pixel point in the adjacent 1st-order neighborhood in the horizontal direction of the green pixel point, and the first color pixel value to be interpolated is calculated using the second native color pixel value of the red pixel point and the interpolated first color pixel value. No further details are given here.

[0231] It should be noted that the "neighborhood" refers to a plurality of pixel points selected around the target pixel point with the target pixel point as the center, wherein 1 pixel point located in the up, down, left and right four directions of the target pixel point forms a four-direction neighborhood, which is a first-order neighborhood; 2 pixel points located in the up, down, left and right four directions of the target pixel point form a four-direction neighborhood, which is a second-order neighborhood, and so on; the pixel points located in the upper left, upper right, lower left and lower right directions of the target pixel point form an eight-direction neighborhood, which is a second-order neighborhood.

[0232] In this way, all first color pixel values to be interpolated in the checkerboard image of the first color channel can be calculated.

[0233] Step S266: determining the second color pixel value to be interpolated according to the second native color pixel value and the first color pixel value to be interpolated.

[0234] Please refer to Figure 22 In an embodiment, when the step S266 of determining the second color pixel value to be interpolated according to the second native color pixel value and the first color pixel value to be interpolated is performed, the following steps are further included.

[0235] Step S2661: calculating the second color pixel value to be interpolated according to the second native color pixel values of the two second-order neighborhood pixel points in the diagonal direction and the first color pixel value to be interpolated.

[0236] In an embodiment, taking the pixel point corresponding to any one second color pixel value to be interpolated in the checkerboard image of the second color channel as the center, the second native color pixel values of the two second-order neighborhood pixel points adjacent to the center pixel point and the first color pixel value to be interpolated are obtained, and the second color pixel value to be interpolated is calculated according to the second native color pixel values of the two second-order neighborhood pixel points in the diagonal direction and the first color pixel value to be interpolated.

[0237] It should be noted that the method adopted in this step is similar to the method of step S2653, the pixel point at the main diagonal vertex in the interpolation unit is the pixel point of the two second-order neighborhood pixel points adjacent to the center pixel point, and the pixel point at the auxiliary diagonal vertex is also the pixel point of the two second-order neighborhood pixel points adjacent to the center pixel point. However, in this step, the second native color pixel values of the pixel points at the main diagonal vertex and the auxiliary diagonal vertex are not the same, so the second color pixel value to be interpolated can be directly calculated using the second native color pixel value of the pixel point at the main diagonal vertex and the first color pixel value to be interpolated; and the second color pixel value to be interpolated can be calculated using the second native color pixel value of the pixel point at the auxiliary diagonal vertex and the first color pixel value to be interpolated.

[0238] Also refer to Figure 20In the 5x5 image of FIG. 6, the pixel point corresponding to the coordinate (7, 7) is taken as the pixel point corresponding to the second color pixel value to be interpolated, and B to be calculated is taken as the second color pixel value to be interpolated. The second color pixel value to be interpolated is calculated according to the second native color pixel values and the first color pixel values that have been interpolated of the pixel points of two 2-order neighborhoods adjacent to the pixel point in the diagonal direction, by using the same method as described above, and details are not described herein again.

[0239]

[0240] wherein P (7,7) represents the purple pixel value of the pixel point corresponding to the coordinate (7, 7), i.e., the second color pixel value to be interpolated; G (7,7) represents the green pixel value of the pixel point corresponding to the coordinate (7, 7), i.e., the first color pixel value that has been interpolated; P (6,6) represents the purple pixel value of the pixel point corresponding to the coordinate (6, 6), i.e., the second native color pixel value; G (6,6) represents the green pixel value of the pixel point corresponding to the coordinate (6, 6), i.e., the first color pixel value that has been interpolated; P (8,8) represents the purple pixel value of the pixel point corresponding to the coordinate (8, 8), i.e., the second native color pixel value; G (8,8) represents the green pixel value of the pixel point corresponding to the coordinate (8, 8), i.e., the first color pixel value that has been interpolated.

[0241] Similarly, referring to FIG. 6, Figure 20 In the 5x5 image of FIG. 6, the pixel point corresponding to the coordinate (7, 7) is taken as the pixel point corresponding to the second color pixel value to be interpolated, and B to be calculated is taken as the second color pixel value to be interpolated. The second color pixel value to be interpolated is calculated according to the second native color pixel values and the first color pixel values that have been interpolated of the pixel points of two 2-order neighborhoods adjacent to the pixel point in the diagonal direction, by using the same method as described above, and details are not described herein again.

[0242] In step S2662, the second color pixel value to be interpolated is calculated according to the second native color pixel values and the first color pixel values that have been interpolated of the pixel points of two 2-order neighborhoods in the vertical direction and / or the horizontal direction of the pixel point corresponding to the second color pixel value to be interpolated.

[0243] In one embodiment, in the vertical direction of the pixel point corresponding to the second color pixel value to be interpolated, the second native color pixel values and the first color pixel values that have been interpolated of the pixel points of two 2-order neighborhoods adjacent to the pixel point are obtained. The gradient change in the vertical direction is calculated according to the second native color pixel values and the first color pixel values that have been interpolated of the pixel points of the 2-order neighborhoods in the vertical direction, by using the following formula:

[0244] GradY = |A i-2,j -A i+2,j | + |2xG i,j-(G i-2,j +G i+2,j )|

[0245] wherein GradY represents the gradient in the vertical direction, A i-2,j represents the second native color pixel value of the pixel point of one of the 2nd-order neighborhoods in the vertical direction, A i+2,j represents the second native color pixel value of the pixel point of the other 2nd-order neighborhood in the vertical direction, G i,j represents the interpolated first color pixel value of the pixel point corresponding to the second color pixel value to be interpolated, G i-2,j represents the interpolated first color pixel value of the pixel point of one of the 2nd-order neighborhoods in the vertical direction, G i+2,j represents the interpolated first color pixel value of the pixel point of the other 2nd-order neighborhood in the vertical direction.

[0246] In one embodiment, in the horizontal direction of the pixel point corresponding to the second color pixel value to be interpolated, the second native color pixel value and the interpolated first color pixel value of the pixel points of the two adjacent 2nd-order neighborhoods are obtained. The gradient change in the horizontal direction is calculated according to the second native color pixel value and the interpolated first color pixel value of the pixel points of the 2nd-order neighborhoods in the horizontal direction, and the calculation is specifically performed by using the following formula:

[0247] GradX = |A i,j-2 -A i,j+2 | + |2 x G i,j -(G i,j-2 +G i,j+2 )|

[0248] wherein GradX represents the gradient in the horizontal direction, A i,j-2 represents the second native color pixel value of the pixel point of one of the 2nd-order neighborhoods in the horizontal direction, A i,j+2 represents the second native color pixel value of the pixel point of the other 2nd-order neighborhood in the horizontal direction, G i,j represents the interpolated first color pixel value of the pixel point corresponding to the second color pixel value to be interpolated, G i,j-2 represents the interpolated first color pixel value of the pixel point of one of the 2nd-order neighborhoods in the horizontal direction, G i,j+2 represents the interpolated first color pixel value of the pixel point of the other 2nd-order neighborhood in the horizontal direction.

[0249] If the gradient change in the vertical direction is less than the gradient change in the horizontal direction, the second color pixel value to be interpolated is determined by using the second native color pixel value and the interpolated first color pixel value of the pixel points of the 2nd-order neighborhoods in the vertical direction, and the calculation is specifically performed by using the following formula:

[0250]

[0251] Among them, A i,j Indicates the second color pixel value to be interpolated.

[0252] If the vertical gradient change is greater than the horizontal gradient change, the second original color pixel value of the pixel point in the second-order neighborhood in the horizontal direction and the interpolated first color pixel value are used to determine the second interpolated color pixel value. The calculation is specifically performed using the following formula:

[0253]

[0254] Among them, A i,j Indicates the second color pixel value to be interpolated.

[0255] If the gradient change in the vertical direction is equal to the gradient change in the horizontal direction, the second original color pixel value and the interpolated first color pixel value of the pixel point in the second-order neighborhood in the vertical direction, as well as the second original color pixel value and the interpolated first color pixel value of the pixel point in the second-order neighborhood in the horizontal direction, are used to determine the second color pixel value to be interpolated. Specifically, the calculation is performed using the following formula:

[0256]

[0257] Among them, A i,j Indicates the second color pixel value to be interpolated.

[0258] Specifically, refer to Figure 20 , the orange pixel value in the red pixel point corresponding to the coordinates (7, 7) is used as the second color pixel value to be interpolated, and the green pixel value in the red pixel point corresponding to the coordinates (7, 7) is used as the first color pixel value interpolated in the pixel point corresponding to the second color pixel value to be interpolated.

[0259] The orange pixel value of the pixel corresponding to the coordinate (5, 7) is the second native color pixel value of one of the second-order neighboring pixels in the vertical direction, and the green pixel value of the pixel corresponding to the coordinate (5, 7) is the interpolated first color pixel value of one of the second-order neighboring pixels in the vertical direction. The orange pixel value of the pixel corresponding to the coordinate (9, 7) is the second native color pixel value of another second-order neighboring pixel in the vertical direction, and the orange pixel value of the pixel corresponding to the coordinate (9, 7) is the interpolated first color pixel value of another second-order neighboring pixel in the vertical direction. Then the gradient change in the vertical direction is:

[0260] GradY=|A 5,7 -A 9,7 |+|2×G 7,7 -(G 5,7 +G9,7 )|

[0261] Among them, GradY represents the gradient in the vertical direction, A 5,7 Indicates the orange pixel value of the pixel corresponding to the coordinate (5, 7), A 9,7 Indicates the orange pixel value of the pixel corresponding to the coordinate (9, 7), G 7,7 Represents the green pixel value in the red pixel corresponding to the coordinate (7, 7), G 5,7 Indicates the green pixel value of the pixel corresponding to the coordinate (5, 7), G 9,7 Indicates the green pixel value of the pixel corresponding to the coordinates (9, 7).

[0262] The orange pixel value of the pixel corresponding to the coordinate (7, 5) is the second original color pixel value of one of the horizontal second-order neighboring pixels, and the green pixel value of the pixel corresponding to the coordinate (7, 5) is the interpolated first color pixel value of one of the horizontal second-order neighboring pixels. The orange pixel value of the pixel corresponding to the coordinate (9, 5) is the second original color pixel value of another horizontal second-order neighboring pixel, and the green pixel value of the pixel corresponding to the coordinate (9, 5) is the interpolated first color pixel value of another horizontal second-order neighboring pixel. Then the gradient change in the horizontal direction is:

[0263] GradX=|A 7,5 -A 7,9 |+|2×G 7,7 -(G 7,5 +G 7,9 )|

[0264] Among them, GradX represents the gradient in the horizontal direction, A 7,5 Indicates the orange pixel value of the pixel corresponding to the coordinate (7, 5), A 7,9 Indicates the orange pixel value of the pixel corresponding to the coordinate (7, 9), G 7,7 Represents the green pixel value in the red pixel corresponding to the coordinate (7, 7), G 7,5 Indicates the green pixel value of the pixel corresponding to the coordinate (7, 5), G 7,9 Indicates the green pixel value of the pixel corresponding to the coordinates (9, 7).

[0265] If the vertical gradient change is smaller than the horizontal gradient change, then:

[0266]

[0267] Among them, A 7,7 Represents the orange pixel value in the red pixel corresponding to the coordinate (7, 7).

[0268] If the gradient change in the vertical direction is greater than the gradient change in the horizontal direction, then:

[0269]

[0270] wherein A 7,7 represents the orange pixel value in the red pixel point corresponding to the coordinate (7, 7).

[0271] If the gradient change in the vertical direction is equal to the gradient change in the horizontal direction, then:

[0272]

[0273] wherein A 7,7 represents the orange pixel value in the red pixel point corresponding to the coordinate (7, 7).

[0274] Reference is made to Figure 23 In an embodiment, the method for generating a display image corresponding to the detected region for output by the processor 123 based on the grayscale image and the hyperspectral image includes the following steps.

[0275] Step S310: Obtain image information of each pixel point in each color channel of the hyperspectral image.

[0276] In an embodiment, after the grayscale image and the hyperspectral image are registered, the present application adopts the color difference constant criterion to utilize the correlation between the color channels to ensure that the color relationship is preserved during the interpolation process, thereby improving the accuracy of the interpolation.

[0277] In the color difference constant criterion, it is assumed that the changes in different color channels at the same spatial position are similar, that is, the spectral characteristics between the color channels change smoothly and do not suddenly mutate. If a certain channel (such as the red channel) has a specific brightness change between two pixel points, other channels (such as the green channel) should have a similar brightness change trend. In this way, when interpolating at the unknown point, the color difference information of the known point can be utilized to ensure that the completed pixel point conforms to the spectral consistency. Based on the above criterion, the image information of each pixel point in each color channel of the hyperspectral image is obtained.

[0278] Step S320: Obtain the pixel point positions of the known pixel values and the pixel point positions of the pixel values to be interpolated in each color channel of the hyperspectral image according to the image information.

[0279] In one embodiment, the first filter unit 12251 and the second filter unit 12252 in the hyperspectral image sensor 1224 are arranged in a predetermined manner. With this arrangement, the hyperspectral image sensor 1224 can only collect information about the wavelengths covered by the filters, while information about wavelengths not covered must be interpolated. Each pixel in the hyperspectral image can only sense light transmitted by the filters, while light of other wavelengths is blocked. This means that only some pixels have directly measured spectral values, while the remaining pixels need to be inferred through interpolation. The image information of the hyperspectral image will mark which pixels have directly measured values ​​(i.e., known points) and which require interpolation (i.e., unknown points).

[0280] Since the arrangement of the first filter unit 12251 and the second filter unit 12252 is already determined, the arrangement information of the filters will be stored in the image information of the hyperspectral image. By reading this image information, the pixel position of the known pixel value and the pixel position of the pixel value to be interpolated can be directly obtained.

[0281] Step S330: obtaining a first pixel point corresponding to a pixel point position of a known pixel value and a second pixel point corresponding to a pixel point position of a pixel value to be interpolated in the grayscale image.

[0282] In one embodiment, a grayscale image reflects brightness variations in a scene, and its gradient indicates the brightness variation trend between pixels. Because grayscale images and multiple spectral channels of a hyperspectral image share a similar spatial structure, a first pixel corresponding to a pixel position with a known pixel value in the grayscale image and a second pixel corresponding to a pixel position with a pixel value to be interpolated can serve as a reliable reference.

[0283] Step S340: Calculate the gradient of the first pixel and the second pixel.

[0284] In one embodiment, in the hyperspectral image, only some first pixels have known values. The values ​​of adjacent pixels can be inferred by combining the values ​​of these known pixels with the gradient of the grayscale image.

[0285] In one embodiment, the gradient in each direction of the grayscale image is calculated using the following formula:

[0286]

[0287] in, Represents the brightness change along the x-axis in the grayscale image, G(x,y) represents the brightness value corresponding to the second pixel in the grayscale image, and G(x-1,y) represents the brightness value corresponding to the first pixel in the grayscale image.

[0288] Step S350: determining the pixel value of the pixel point position of the pixel value to be interpolated according to the gradient and the pixel value of the pixel point position of the known pixel value.

[0289] In an embodiment, since the color channels in the hyperspectral image are related to the gradient of the grayscale image, the pixel value of the pixel point position of the pixel value to be interpolated is calculated by using the following formula:

[0290]

[0291] wherein F(x, y) represents the pixel value of the pixel point position of the pixel value to be interpolated, and F(x-1, y) represents the pixel value of the pixel point position of the known pixel value.

[0292] In an embodiment, in the interpolation process provided in the present application, the color channel pixel value corresponding to each filter needs to be interpolated, and the core idea of this method is to introduce the correlation between the color channels into the interpolation calculation by using the gradient information of the grayscale image, so as to improve the accuracy of the interpolation.

[0293] Suppose that in the filter array 1225, the arrangement of the first row of filters is a purple filter, a blue filter, a green filter, a purple filter, a blue filter, and a green filter, wherein the known pixel values are the first purple pixel value, the first blue pixel value, and the first green pixel value, then according to the interpolation of the known points of the same color:

[0294] The second purple pixel value is calculated by using the first purple pixel value and the gradient of the grayscale image:

[0295]

[0296] wherein F(x, y) represents the pixel value of the pixel point position of the pixel value to be interpolated, and F(x-1, y) represents the pixel value of the pixel point position of the known pixel value. 紫(4) F(x, y) represents the second purple pixel value, F(x-1, y) represents the first purple pixel value, 紫(1) represents the gradient of purple.

[0297] The second blue pixel value is calculated by using the first blue pixel value and the gradient of the grayscale image:

[0298]

[0299] wherein F(x, y) represents the pixel value of the pixel point position of the pixel value to be interpolated, and F(x-1, y) represents the pixel value of the pixel point position of the known pixel value. 蓝(5) F(x, y) represents the second blue pixel value, F(x-1, y) represents the first blue pixel value, 蓝(2) represents the gradient of blue.

[0300] The second green pixel value is calculated by using the first green pixel value and the gradient of the grayscale image:

[0301] ​​

[0302] Among them, F 绿(5) Represents the second green pixel value, F 绿(2) represents the first green pixel value, Represents a gradient of green.

[0303] Because filters determine the transmittance of a pixel for a specific portion of the spectrum, filters of the same color represent the same spectral channel. The purpose of interpolation is to complete the spectral information of the same channel, so when calculating the second purple pixel, the first purple pixel must be used, and known blue or green points cannot be used.

[0304] The above method is used to calculate the pixel value of each pixel point position of the pixel value to be interpolated, thereby generating a display image for output in the detected area.

[0305] In one embodiment, if the imaging system 12 is used independently as a separate system, the above content has described the entire workflow of the complete imaging system 12.

[0306] In one embodiment, if the imaging system 12 is a core component of the endoscope system 1, the guide portion 13 of the endoscope system 1 includes at least a distal end configured to penetrate the inspection area, allowing it to enter the human body or the interior of the device, flexibly extend and retract, and transmit optical and imaging signals. The control system 14 is used by the doctor or operator to control the operation and function of the endoscope. The image transmission and display system 15 transmits the displayed image to an external display device for the doctor to observe. The power supply system 16 provides the power required for the operation of the endoscope system 1.

[0307] Those skilled in the art will appreciate that all or part of the functions of the various methods in the above embodiments can be implemented by hardware or by computer program. When all or part of the functions in the above embodiments are implemented by computer program, the program can be stored in a computer-readable storage medium, and the storage medium can include: read-only memory, random access memory, disk, optical disk, hard disk, etc., and the program is executed by a computer to implement the above functions. For example, the program is stored in the memory of the device, and when the program in the memory is executed by the processor, all or part of the above functions can be implemented. In addition, when all or part of the functions in the above embodiments are implemented by computer program, the program can also be stored in a storage medium such as a server, another computer, disk, optical disk, flash disk or mobile hard disk, and saved in the memory of the local device by downloading or copying, or the system of the local device is updated. When the program in the memory is executed by the processor, all or part of the functions in the above embodiments can be implemented.

[0308] The above examples are used to illustrate the present invention, which are only used to help understand the present invention and are not intended to limit the present invention. Those skilled in the art can make several simple deductions, modifications or substitutions based on the concept of the present invention.

Claims

1. An endoscope system, characterized in that: include: A white light source unit, used to provide continuous visible spectrum illumination light to the inspected area; a guide portion having at least a tip portion configured to probe into the detected area; An imaging assembly comprising a common lens, a light-splitting component, a grayscale image sensor, a hyperspectral image sensor, and a filter array; The common lens is arranged at the front end of the guide portion, and is used to collect reflected light or scattered light after the illumination light irradiates the inspection area, so as to emit imaging light; The light splitting component is arranged on the optical path of the imaging light, and is used to split the imaging light into a first imaging light and a second imaging light, wherein the intensity of the second imaging light is greater than the intensity of the first imaging light; The grayscale image sensor is arranged on the optical path of the first imaging light, and is used to convert the first imaging light from an optical signal to an electrical signal to obtain a grayscale image; The filter array and the hyperspectral image sensor are sequentially arranged on the optical path of the second imaging light; the filter array is used to decompose the second imaging light into monochromatic light or narrow-band light of different wavelengths, and the hyperspectral image sensor is used to convert the monochromatic light or narrow-band light of different wavelengths decomposed by the filter array from optical signals to electrical signals to obtain a hyperspectral image; a processor comprising a plurality of imaging modes, the plurality of imaging modes comprising at least one of a first imaging mode, a second imaging mode, and a third imaging mode; In the first imaging mode, the processor generates a display image of the detected area for output according to the grayscale image; In the second imaging mode, the processor generates a display image of the detected area for output according to the hyperspectral image; In the third imaging mode, the processor generates a display image of the detected area for output according to the grayscale image and the hyperspectral image.

2. The endoscope system according to claim 1, wherein: The filter array includes a first filter unit that transmits a preset wavelength band and a second filter unit that transmits wavelength bands other than the preset wavelength band. The filters in the first filter unit and the filters in the second filter unit are alternately spaced in the horizontal and vertical directions.

3. The endoscope system according to claim 2, wherein: The second filter unit includes a first sub-filter unit and a second sub-filter unit, the first sub-filter unit includes filters of at least two colors, and the second sub-filter unit includes filters of at least two colors, wherein the filters in the first sub-filter unit and the second sub-filter unit have different colors; The spectral response difference between the filters of each color in the first sub-filter unit is less than a first set value, the spectral response difference between the filters of each color in the second sub-filter unit is less than a second set value, and the spectral response difference between the filters of any color in the first sub-filter unit and the filters of any color in the second sub-filter unit is greater than a third set value.

4. The endoscope system according to claim 3, wherein: The filters of different colors in the first sub-filter unit are alternately distributed in the horizontal and vertical directions of the second filter unit, and the filters of different colors in the second sub-filter unit are alternately distributed in the horizontal and vertical directions of the second filter unit.

5. The endoscope system according to claim 3 or 4, wherein: The first sub-filter units and the second sub-filter units are alternately distributed.

6. The endoscope system according to claim 2, wherein: The hyperspectral image includes a checkerboard image of a first color channel corresponding to a filter color in the first filter unit determined by the hyperspectral image sensor through the first filter unit, and a checkerboard image of a second color channel corresponding to a filter color in the second filter unit determined by the hyperspectral image sensor through the second filter unit; The processor obtains a checkerboard image of the first color channel, interpolates pixel values ​​of the first color corresponding to the first filter unit at blank positions in the checkerboard image of the first color channel to generate a complete image of the first color channel; and interpolates pixel values ​​of each pixel point in the hyperspectral image in different color channels based on the complete image of the first color channel and the checkerboard image of the second color channel to generate a display image of the detected area for output; The blank position is a pixel position where the pixel value of the first color does not exist in the checkerboard image of the second color channel in the hyperspectral image.

7. The endoscope system according to claim 6, wherein: The processor interpolates pixel values ​​of the first color corresponding to the first filter unit at blank positions in the checkerboard image of the first color channel to generate a complete image of the first color channel, including: Acquire a first training image at full resolution, and generate a second training image corresponding to the checkerboard image of the first color channel based on the first training image; The convolutional neural network model is trained according to the first training image and the second training image, and the checkerboard image of the first color channel is input into the trained convolutional neural network model to generate a complete image of the first color channel.

8. The endoscope system according to claim 6, wherein: The processor interpolates pixel values ​​of each pixel in the hyperspectral image in different color channels according to the complete image of the first color channel and the checkerboard image of the second color channel to generate a display image for output of the detected area, including: Each pixel point in the checkerboard image of the second color channel includes a second native color pixel value corresponding to the filter color in the second filter unit, a first color pixel value interpolated at the blank position, and a second color pixel value to be interpolated; The second to-be-interpolated color pixel value is determined according to the second native color pixel value and the interpolated first color pixel value, so as to complete the interpolation of the pixel values ​​of each pixel point in the hyperspectral image in different color channels.

9. The endoscope system according to claim 8, wherein: The processor determines the second to-be-interpolated color pixel value according to the second native color pixel value and the interpolated first color pixel value, including: In the hyperspectral image, constructing an n×n interpolation unit with any pixel point in the checkerboard image of the second color channel as a center point, where n is an odd number; Determine the second color pixel value to be interpolated by using the second native color pixel value of the pixel point at the main diagonal vertex in the interpolation unit and the interpolated first color pixel value; or, Determine the second color pixel value to be interpolated by using the second native color pixel value of the pixel point at the sub-diagonal vertex in the interpolation unit and the interpolated first color pixel value; or, The second color pixel value to be interpolated is determined by using the second native color pixel value and the interpolated first color pixel value of the pixel point at the main diagonal vertex of the interpolation unit, as well as the second native color pixel value and the interpolated first color pixel value of the pixel point at the sub-diagonal vertex of the interpolation unit.

10. The endoscope system according to claim 9, wherein: When the second native color pixel values ​​of the pixel points at the main diagonal or sub-diagonal vertices in the interpolation unit are the same, Calculating a gradient change of the main diagonal line according to the second native color pixel value of the pixel point at the main diagonal vertex in the interpolation unit and the interpolated first color pixel value; Calculating a gradient change of the sub-diagonal line according to the second native color pixel value of the pixel point at the sub-diagonal line vertex in the interpolation unit and the interpolated first color pixel value; When the gradient change of the main diagonal is less than the gradient change of the sub-diagonal, determining the second color pixel value to be interpolated by using the second native color pixel value of the pixel point at the vertex of the main diagonal of the interpolation unit and the interpolated first color pixel value; When the gradient change of the main diagonal is greater than the gradient change of the sub-diagonal, the second color pixel value to be interpolated is determined by using the second native color pixel value of the pixel point at the sub-diagonal vertex of the interpolation unit and the interpolated first color pixel value; When the gradient change of the main diagonal is equal to the gradient change of the secondary diagonal, the second color pixel value to be interpolated is determined by using the second native color pixel value and the interpolated first color pixel value of the pixel point at the vertex of the main diagonal of the interpolation unit, as well as the second native color pixel value and the interpolated first color pixel value of the pixel point at the vertex of the secondary diagonal of the interpolation unit.

11. The endoscope system according to claim 1, wherein: The processor generates a display image for output of the detected area according to the grayscale image and the hyperspectral image, including: Obtaining image information of each pixel in each color channel in the hyperspectral image; Acquire pixel locations of known pixel values ​​and pixel locations of pixel values ​​to be interpolated in each color channel of the hyperspectral image according to the image information; Acquire a first pixel point corresponding to the position of the pixel point of the known pixel value and a second pixel point corresponding to the position of the pixel point of the pixel value to be interpolated in the grayscale image; Calculate the gradient of the first pixel and the second pixel; The pixel value of the pixel point position of the pixel value to be interpolated is determined according to the gradient and the pixel value of the pixel point position of the known pixel value, so as to generate a display image for output in the detected area.

12. An endoscope system, characterized in that: include: A white light source unit, used to provide continuous visible spectrum illumination light to the inspected area; a guide portion having at least a tip portion configured to probe into the detected area; An imaging assembly comprising a common lens, a light-splitting component, a grayscale image sensor, a hyperspectral image sensor, and a filter array; The common lens is arranged at the front end of the guide portion, and is used to collect reflected light or scattered light after the illumination light irradiates the inspection area, so as to emit imaging light; The light splitting component is arranged on the optical path of the imaging light, and is used to split the imaging light into a first imaging light and a second imaging light, wherein the intensity of the second imaging light is greater than the intensity of the first imaging light; The grayscale image sensor is arranged on the optical path of the first imaging light, and is used to convert the first imaging light from an optical signal to an electrical signal to obtain a grayscale image; The filter array and the hyperspectral image sensor are sequentially arranged on the optical path of the second imaging light; the filter array is used to decompose the second imaging light into monochromatic light or narrow-band light of different wavelengths, and the hyperspectral image sensor is used to convert the monochromatic light or narrow-band light of different wavelengths decomposed by the filter array from optical signals to electrical signals to obtain a hyperspectral image.

13. The endoscope system according to claim 12, wherein: The filter array includes a first filter unit that transmits a preset wavelength band and a second filter unit that transmits wavelength bands other than the preset wavelength band. The filters in the first filter unit and the filters in the second filter unit are alternately spaced in the horizontal and vertical directions.

14. The endoscope system according to claim 13, wherein: The second filter unit includes a first sub-filter unit and a second sub-filter unit, the first sub-filter unit includes filters of at least two colors, and the second sub-filter unit includes filters of at least two colors, wherein the filters in the first sub-filter unit and the second sub-filter unit have different colors; The spectral response difference between the filters of each color in the first sub-filter unit is less than a first set value, the spectral response difference between the filters of each color in the second sub-filter unit is less than a second set value, and the spectral response difference between the filters of any color in the first sub-filter unit and the filters of any color in the second sub-filter unit is greater than a third set value.

15. The endoscope system according to claim 14, wherein: The filters of different colors in the first sub-filter unit are alternately spaced in the horizontal direction of the second filter unit, and the filters of different colors in the second sub-filter unit are alternately spaced in the horizontal direction of the second filter unit.

16. The endoscope system according to claim 15, wherein: The filters of different colors in the first sub-filter unit are alternately spaced in the vertical direction of the second filter unit, and the filters of different colors in the second sub-filter unit are alternately spaced in the vertical direction of the second filter unit.

17. The endoscope system according to claim 13 or 14, wherein: The first sub-filter units and the second sub-filter units are alternately distributed.

18. The endoscope system according to claim 13, wherein: The number of filters in the first filter unit is greater than or equal to half of the total number of all filters in the filter array.

19. The endoscope system according to claim 13, wherein: The endoscope system also includes a processor; The hyperspectral image includes a checkerboard image of a first color channel corresponding to a filter color in the first filter unit determined by the hyperspectral image sensor through the first filter unit, and a checkerboard image of a second color channel corresponding to a filter color in the second filter unit determined by the hyperspectral image sensor through the second filter unit; The processor obtains a checkerboard image of the first color channel, interpolates pixel values ​​of the first color corresponding to the first filter unit at blank positions in the checkerboard image of the first color channel to generate a complete image of the first color channel; and interpolates pixel values ​​of each pixel point in the hyperspectral image in different color channels based on the complete image of the first color channel and the checkerboard image of the second color channel to generate a display image of the detected area for output; The blank position is a pixel position where the pixel value of the first color does not exist in the checkerboard image of the second color channel in the hyperspectral image.

20. The endoscope system according to claim 19, wherein: The processor interpolates pixel values ​​of the first color corresponding to the first filter unit at blank positions in the checkerboard image of the first color channel to generate a complete image of the first color channel, including: Acquire a first training image at full resolution, and generate a second training image corresponding to the checkerboard image of the first color channel based on the first training image; The convolutional neural network model is trained according to the first training image and the second training image, and the checkerboard image of the first color channel is input into the trained convolutional neural network model to generate a complete image of the first color channel.

21. The endoscope system according to claim 19, wherein: The processor interpolates pixel values ​​of each pixel in the hyperspectral image in different color channels according to the complete image of the first color channel and the checkerboard image of the second color channel to generate a display image for output of the detected area, including: Each pixel point in the checkerboard image of the second color channel includes a second native color pixel value corresponding to the filter color in the second filter unit, a first color pixel value interpolated at the blank position, and a second color pixel value to be interpolated; The second to-be-interpolated color pixel value is determined according to the second native color pixel value and the interpolated first color pixel value, so as to complete the interpolation of the pixel values ​​of each pixel point in the hyperspectral image in different color channels.

22. The endoscope system according to claim 21, wherein: The processor determines the second to-be-interpolated color pixel value according to the second native color pixel value and the interpolated first color pixel value, including: In the hyperspectral image, an n×n interpolation unit is constructed with any pixel corresponding to the second to-be-interpolated color pixel value in the checkerboard image of the second color channel as a center point, where n is an odd number; Determine the second color pixel value to be interpolated by using the second native color pixel value of the pixel point at the main diagonal vertex in the interpolation unit and the interpolated first color pixel value; or, Determine the second color pixel value to be interpolated by using the second native color pixel value of the pixel point at the sub-diagonal vertex in the interpolation unit and the interpolated first color pixel value; or, The second color pixel value to be interpolated is determined by using the second native color pixel value and the interpolated first color pixel value of the pixel point at the main diagonal vertex of the interpolation unit, as well as the second native color pixel value and the interpolated first color pixel value of the pixel point at the sub-diagonal vertex of the interpolation unit.

23. The endoscope system according to claim 22, wherein: When the second native color pixel values ​​of the pixel points at the main diagonal or sub-diagonal vertices in the interpolation unit are the same, Calculating a gradient change of the main diagonal line according to the second native color pixel value of the pixel point at the main diagonal vertex in the interpolation unit and the interpolated first color pixel value; Calculating a gradient change of the sub-diagonal line according to the second native color pixel value of the pixel point at the sub-diagonal line vertex in the interpolation unit and the interpolated first color pixel value; When the gradient change of the main diagonal is less than the gradient change of the sub-diagonal, determining the second color pixel value to be interpolated by using the second native color pixel value of the pixel point at the vertex of the main diagonal of the interpolation unit and the interpolated first color pixel value; When the gradient change of the main diagonal is greater than the gradient change of the sub-diagonal, the second color pixel value to be interpolated is determined by using the second native color pixel value of the pixel point at the sub-diagonal vertex of the interpolation unit and the interpolated first color pixel value; When the gradient change of the main diagonal is equal to the gradient change of the secondary diagonal, the second color pixel value to be interpolated is determined by using the second native color pixel value and the interpolated first color pixel value of the pixel point at the vertex of the main diagonal of the interpolation unit, as well as the second native color pixel value and the interpolated first color pixel value of the pixel point at the vertex of the secondary diagonal of the interpolation unit.

24. The endoscope system according to claim 23, wherein: When the second native color pixel values ​​of the pixel points at the main diagonal or sub-diagonal vertices in the interpolation unit are different, The interpolation unit is updated with a pixel point at any vertex in the main diagonal or the secondary diagonal as the center point, and the second native color pixel value of any vertex in the main diagonal or the secondary diagonal is updated to the second color pixel value to be interpolated in the updated interpolation unit; The updated interpolation unit is used to determine a second color pixel value to be interpolated, so as to determine a second native color pixel value of any vertex in the main diagonal or the sub-diagonal.

25. The endoscope system according to claim 21, wherein: The processor determines the second to-be-interpolated color pixel value according to the second native color pixel value and the interpolated first color pixel value, including: Taking any pixel point corresponding to the second color pixel value to be interpolated in the checkerboard image of the second color channel as the center; Obtaining the second native color pixel values ​​and the interpolated first color pixel values ​​of two adjacent second-order neighboring pixels in a diagonal direction of the pixel corresponding to the second color pixel value to be interpolated, and calculating the second color pixel value to be interpolated based on the second native color pixel values ​​and the interpolated first color pixel values ​​of the two second-order neighboring pixels in the diagonal direction; and / or, Obtaining, in a vertical direction of the pixel corresponding to the second to-be-interpolated color pixel value, the second native color pixel values ​​and the interpolated first color pixel values ​​of two adjacent second-order neighboring pixels, and calculating a vertical gradient change based on the second native color pixel values ​​and the interpolated first color pixel values ​​of the two second-order neighboring pixels in the vertical direction; Obtaining, in a horizontal direction of the pixel corresponding to the second to-be-interpolated color pixel value, the second native color pixel values ​​and the interpolated first color pixel values ​​of two adjacent second-order neighboring pixels, and calculating a horizontal gradient change based on the second native color pixel values ​​and the interpolated first color pixel values ​​of the two second-order neighboring pixels in the horizontal direction; When the gradient change in the vertical direction is smaller than the gradient change in the horizontal direction, determining the second color pixel value to be interpolated by using the second native color pixel value of the pixel point in the second-order neighborhood in the vertical direction and the interpolated first color pixel value; When the gradient change in the vertical direction is greater than the gradient change in the horizontal direction, determining the second color pixel value to be interpolated by using the second native color pixel value of the pixel point in the second-order neighborhood in the horizontal direction and the interpolated first color pixel value; When the gradient change in the vertical direction is equal to the gradient change in the horizontal direction, the second native color pixel value and the interpolated first color pixel value of the pixel point in the second-order neighborhood in the vertical direction, as well as the second native color pixel value and the interpolated first color pixel value of the pixel point in the second-order neighborhood in the horizontal direction, are used to determine the second color pixel value to be interpolated.

26. The endoscope system according to claim 12, wherein: The endoscope system also includes a processor; The processor obtains image information of each pixel in each color channel in the hyperspectral image; Acquire pixel locations of known pixel values ​​and pixel locations of pixel values ​​to be interpolated in each color channel of the hyperspectral image according to the image information; Acquire a first pixel point corresponding to the position of the pixel point of the known pixel value and a second pixel point corresponding to the position of the pixel point of the pixel value to be interpolated in the grayscale image; Calculate the gradient of the first pixel and the second pixel; The pixel value of the pixel point position of the pixel value to be interpolated is determined according to the gradient and the pixel value of the pixel point position of the known pixel value, so as to generate a display image for output in the detected area.

27. An imaging system, characterized in that include: A white light source unit, used to provide continuous visible spectrum illumination light to the inspected area; An imaging assembly comprising a common lens, a light-splitting component, a grayscale image sensor, a hyperspectral image sensor, and a filter array; The shared lens is used to collect reflected light or scattered light after the illumination light irradiates the inspection area to emit imaging light; The light splitting component is arranged on the optical path of the imaging light, and is used to split the imaging light into a first imaging light and a second imaging light, wherein the intensity of the second imaging light is greater than the intensity of the first imaging light; The grayscale image sensor is arranged on the optical path of the first imaging light, and is used to convert the first imaging light from an optical signal to an electrical signal to obtain a grayscale image; The filter array and the hyperspectral image sensor are sequentially arranged on the optical path of the second imaging light; the filter array is used to decompose the second imaging light into monochromatic light or narrow-band light of different wavelengths, and the hyperspectral image sensor is used to convert the monochromatic light or narrow-band light of different wavelengths decomposed by the filter array from optical signals to electrical signals to obtain a hyperspectral image.

28. An imaging method, characterized in that: include: Acquire a checkerboard image of a first color channel corresponding to the filter color in the first filter unit and a checkerboard image of a second color channel corresponding to the filter color in the second filter unit in the hyperspectral image to be interpolated; The first filter unit is a first filter unit that transmits a preset wavelength band, and the second filter unit is a second filter unit that transmits wavelength bands other than the preset wavelength band. The first filter unit and the second filter unit are alternately spaced in the horizontal direction and the vertical direction. interpolating pixel values ​​of the first color corresponding to the first filter unit at blank positions in the checkerboard image of the first color channel to generate a complete image of the first color channel; interpolating pixel values ​​of each pixel point in the hyperspectral image in different color channels according to the complete image of the first color channel and the checkerboard image of the second color channel to generate a display image of the detected area for output; The blank position is a pixel point position where the pixel value of the first color does not exist in the checkerboard image of the second color channel in the hyperspectral image.

29. The imaging method according to claim 28, wherein Interpolating pixel values ​​of the first color corresponding to the first filter unit at blank positions in the checkerboard image of the first color channel to generate a complete image of the first color channel includes: Acquire a first training image at full resolution, and generate a second training image corresponding to the checkerboard image of the first color channel based on the first training image; The convolutional neural network model is trained according to the first training image and the second training image, and the checkerboard image of the first color channel is input into the trained convolutional neural network model to generate a complete image of the first color channel.

30. The imaging method according to claim 28, wherein The interpolating pixel values ​​of each pixel point in the hyperspectral image in different color channels according to the complete image of the first color channel and the checkerboard image of the second color channel to generate a display image for output of the detected area includes: Each pixel point in the checkerboard image of the second color channel includes a second native color pixel value corresponding to the filter color in the second filter unit, a first color pixel value interpolated at the blank position, and a second color pixel value to be interpolated; The second to-be-interpolated color pixel value is determined according to the second native color pixel value and the interpolated first color pixel value, so as to complete the interpolation of the pixel values ​​of each pixel point in the hyperspectral image in different color channels.

31. The imaging method according to claim 30, wherein The determining the second to-be-interpolated color pixel value according to the second native color pixel value and the interpolated first color pixel value includes: In the hyperspectral image, constructing an n×n interpolation unit with any pixel point in the checkerboard image of the second color channel as a center point, where n is an odd number; Determine the second color pixel value to be interpolated by using the second native color pixel value of the pixel point at the main diagonal vertex in the interpolation unit and the interpolated first color pixel value; or, Determine the second color pixel value to be interpolated by using the second native color pixel value of the pixel point at the sub-diagonal vertex in the interpolation unit and the interpolated first color pixel value; or, The second color pixel value to be interpolated is determined by using the second native color pixel value and the interpolated first color pixel value of the pixel point at the main diagonal vertex of the interpolation unit, as well as the second native color pixel value and the interpolated first color pixel value of the pixel point at the sub-diagonal vertex of the interpolation unit.

32. The imaging method according to claim 31, wherein When the second native color pixel values ​​of the pixel points at the main diagonal or sub-diagonal vertices in the interpolation unit are the same, Calculating a gradient change of the main diagonal line according to the second native color pixel value of the pixel point at the main diagonal vertex in the interpolation unit and the interpolated first color pixel value; Calculating a gradient change of the sub-diagonal line according to the second native color pixel value of the pixel point at the sub-diagonal line vertex in the interpolation unit and the interpolated first color pixel value; When the gradient change of the main diagonal is less than the gradient change of the sub-diagonal, determining the second color pixel value to be interpolated by using the second native color pixel value of the pixel point at the vertex of the main diagonal of the interpolation unit and the interpolated first color pixel value; When the gradient change of the main diagonal is greater than the gradient change of the sub-diagonal, the second color pixel value to be interpolated is determined by using the second native color pixel value of the pixel point at the sub-diagonal vertex of the interpolation unit and the interpolated first color pixel value; When the gradient change of the main diagonal is equal to the gradient change of the secondary diagonal, the second color pixel value to be interpolated is determined by using the second native color pixel value and the interpolated first color pixel value of the pixel point at the vertex of the main diagonal of the interpolation unit, as well as the second native color pixel value and the interpolated first color pixel value of the pixel point at the vertex of the secondary diagonal of the interpolation unit.

33. The imaging method according to claim 32, wherein When the second native color pixel values ​​of the pixel points at the main diagonal or sub-diagonal vertices in the interpolation unit are different, Updating the center point with a pixel point of any vertex in the main diagonal or the secondary diagonal to update the interpolation unit, and updating the second native color pixel value of any vertex in the main diagonal or the secondary diagonal to the second to-be-interpolated color pixel value in the updated interpolation unit; The updated interpolation unit is used to determine a second color pixel value to be interpolated, so as to determine a second native color pixel value of any vertex in the main diagonal or the sub-diagonal.

34. The imaging method according to claim 30, wherein The determining the second to-be-interpolated color pixel value according to the second native color pixel value and the interpolated first color pixel value includes: Taking any pixel point corresponding to the second color pixel value to be interpolated in the checkerboard image of the second color channel as the center; Obtaining the second native color pixel values ​​and the interpolated first color pixel values ​​of two adjacent second-order neighboring pixels in a diagonal direction of the pixel corresponding to the second color pixel value to be interpolated, and calculating the second color pixel value to be interpolated based on the second native color pixel values ​​and the interpolated first color pixel values ​​of the two second-order neighboring pixels in the diagonal direction; and / or, Obtaining, in a vertical direction of the pixel corresponding to the second to-be-interpolated color pixel value, the second native color pixel values ​​and the interpolated first color pixel values ​​of two adjacent second-order neighboring pixels, and calculating a vertical gradient change based on the second native color pixel values ​​and the interpolated first color pixel values ​​of the two second-order neighboring pixels in the vertical direction; Obtaining, in a horizontal direction of the pixel corresponding to the second to-be-interpolated color pixel value, the second native color pixel values ​​and the interpolated first color pixel values ​​of two adjacent second-order neighboring pixels, and calculating a horizontal gradient change based on the second native color pixel values ​​and the interpolated first color pixel values ​​of the two second-order neighboring pixels in the horizontal direction; When the gradient change in the vertical direction is smaller than the gradient change in the horizontal direction, determining the second color pixel value to be interpolated by using the second native color pixel value of the pixel point in the second-order neighborhood in the vertical direction and the interpolated first color pixel value; When the gradient change in the vertical direction is greater than the gradient change in the horizontal direction, determining the second color pixel value to be interpolated by using the second native color pixel value of the pixel point in the second-order neighborhood in the horizontal direction and the interpolated first color pixel value; When the gradient change in the vertical direction is equal to the gradient change in the horizontal direction, the second native color pixel value and the interpolated first color pixel value of the pixel point in the second-order neighborhood in the vertical direction, as well as the second native color pixel value and the interpolated first color pixel value of the pixel point in the second-order neighborhood in the horizontal direction, are used to determine the second color pixel value to be interpolated.

35. An imaging method, characterized in that include: Acquire a hyperspectral image to be interpolated and a grayscale image corresponding to the hyperspectral image to be interpolated; Obtaining image information of each pixel point in each color channel of the hyperspectral image, and obtaining pixel point positions of known pixel values ​​and pixel point positions of pixel values ​​to be interpolated in each color channel of the hyperspectral image based on the image information; Acquire a first pixel point corresponding to the position of the pixel point of the known pixel value and a second pixel point corresponding to the position of the pixel point of the pixel value to be interpolated in the grayscale image; Calculate the gradient of the first pixel and the second pixel; The pixel value of the pixel point position of the pixel value to be interpolated is determined according to the gradient and the pixel value of the pixel point position of the known pixel value, so as to generate a display image for output in the detected area.

36. A computer-readable storage medium, characterized in that The medium stores a computer program, which can be executed by a processor to implement the method according to any one of claims 28 to 35.

Images