Imaging system, endoscope system, imaging method and computer storage medium

Abstract

The invention discloses an imaging system, an endoscope system, an imaging method and a computer storage medium, and relates to the technical field of medical instruments. The endoscope system includes a white light source portion, a guide portion, and an imaging assembly. The white light source part is used for providing illumination light with a continuous visible spectrum for a detected area; the guide part is at least provided with a tip part configured to extend into the detected area; the imaging assembly comprises a shooting lens, a hyperspectral image sensor and an optical filter array; the shooting lens is arranged at the tip part of the guide part and is used for collecting reflected light or scattered light after the illumination light irradiates the detected area so as to emit imaging light; the optical filter array and the hyperspectral image sensor are sequentially arranged on a light path of the imaging light; the optical filter array is used for decomposing the imaging light into monochromatic light or narrow-band light with different wavelengths, and the hyperspectral image sensor is used for converting the monochromatic light or narrow-band light with different wavelengths decomposed by the optical filter array into electric signals from optical signals so as to obtain a hyperspectral image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical devices, in particular to an imaging system, an endoscope system, an imaging method and a computer storage medium. BACKGROUND

[0002] With the rapid development of miniaturized sensors and minimally invasive medical technology, minimally invasive or non-invasive medical examination and treatment methods based on endoscopes have been widely used in clinical practice. Endoscope technology enables components with imaging functions to enter the body through natural cavities of the human or animal body or through small channels formed by surgery, thereby obtaining image information of internal target areas and providing intuitive and accurate visual evidence for disease diagnosis and treatment.

[0003] In modern endoscope systems, electronic endoscopes integrate image sensors and imaging lenses at the front end of the probe, and the collected images are transmitted to the image processing host through signals and are presented in real time on display devices for observation and analysis by physicians.

[0004] With the continuous evolution of endoscope technology, special light imaging technology as a key innovation means has been widely integrated into various endoscope imaging devices. This technology enhances image contrast by illuminating tissues with specific wavelengths of light based on the absorption characteristics of tissues to different wavelengths of light spectrum, thereby improving lesion visualization. Image acquisition can be done by monochrome or color image sensors.

[0005] Traditional special light imaging usually uses a multi-band LED or laser light illumination scheme, that is, multiple light sources are integrated at the end of the endoscope host, and different wavelengths of light are guided into the body through a light guide beam to achieve specific band illumination. This optical device structure is complex and has high manufacturing cost. If a monochrome image sensor is used, the switching of the light source and the exposure timing of the image sensor need to be accurately controlled to achieve multi-spectral image acquisition, which is difficult to achieve in high frame rate imaging. If a traditional color image sensor is used, the emission band of the light source and the receiving channel of the sensor must have good matching, otherwise the imaging quality will be affected by channel crosstalk, especially in narrowband light imaging.

[0006] To improve spectral resolution and imaging quality, some endoscope systems have introduced hyperspectral imaging sensors, which can obtain high-dimensional data of control and spectral information. However, most current hyperspectral endoscopes use a dual-sensor design, which requires high-precision pre-registration before shipment, increasing production costs and limiting system miniaturization and production efficiency. SUMMARY

[0007] The technical problem solved by the present application is to provide an imaging system, an imaging method using a single hyperspectral imaging sensor for imaging, and a corresponding endoscope system.

[0008] According to a first aspect, an endoscope system is provided in an embodiment, comprising:

[0009] a white light source unit configured to provide a continuous visible spectrum of illumination light to a detected region;

[0010] a guide unit having at least a tip portion configured to be inserted into the detected region;

[0011] an imaging assembly comprising a taking lens, a hyperspectral image sensor, and a filter array;

[0012] the taking lens is disposed at the tip portion of the guide unit, and is configured to collect reflected light or scattered light after the illumination light irradiates the detected region, to emit imaging light;

[0013] the filter array and the hyperspectral image sensor are sequentially disposed in an optical path of the imaging light; the filter array is configured to decompose the imaging light into monochromatic light or narrow-band light of different wavelengths, and the hyperspectral image sensor is configured to convert the monochromatic light or narrow-band light of different wavelengths decomposed by the filter array from optical signals to electrical signals, to obtain a hyperspectral image.

[0014] According to a second aspect, an imaging system is provided in an embodiment, comprising:

[0015] a white light source unit configured to provide a continuous visible spectrum of illumination light to a detected region;

[0016] an imaging assembly comprising a taking lens, a hyperspectral image sensor, and a filter array;

[0017] the taking lens is configured to collect reflected light or scattered light after the illumination light irradiates the detected region, to emit imaging light;

[0018] the filter array and the hyperspectral image sensor are sequentially disposed in an optical path of the imaging light; the filter array is configured to decompose the imaging light into monochromatic light or narrow-band light of different wavelengths, and the hyperspectral image sensor is configured to convert the monochromatic light or narrow-band light of different wavelengths decomposed by the filter array from optical signals to electrical signals, to obtain a hyperspectral image.

[0019] According to a third aspect, an imaging method is provided in an embodiment, comprising:

[0020] obtaining a checkerboard image of a first color channel corresponding to a filter color in a first filter unit, and a checkerboard image of a second color channel corresponding to a filter color in a second filter unit, in a hyperspectral image to be interpolated;

[0021] The first filter unit is a first filter unit that transmits a preset wavelength band, and the second filter unit is a second filter unit that transmits wavelength bands other than the preset wavelength band. The first filter unit and the second filter unit are alternately spaced in the horizontal direction and the vertical direction.

[0022] interpolating pixel values ​​of the first color corresponding to the first filter unit at blank positions in the checkerboard image of the first color channel to generate a complete image of the first color channel;

[0023] interpolating pixel values ​​of each pixel point in the hyperspectral image in different color channels according to the complete image of the first color channel and the checkerboard image of the second color channel to generate a display image of the detected area for output;

[0024] The blank position is a pixel point position where the pixel value of the first color does not exist in the checkerboard image of the second color channel in the hyperspectral image.

[0025] According to a fourth aspect, an embodiment provides a computer-readable storage medium, on which a computer program is stored. The computer program can be executed by a processor to implement the method described in any one of the above embodiments.

[0026] According to the imaging system, endoscope system, imaging method, and computer storage medium of the above-mentioned embodiments, the use of a white light source in the endoscope system reduces the complexity of the light source, eliminates the synchronization requirement between the light source and the endoscope system, reduces the cost of the light source, and makes it easier to achieve disposable. The endoscope system only uses a hyperspectral image sensor, which greatly reduces the complexity of the image sensing part while ensuring the image quality requirements of special light imaging and white light imaging. The endoscope system in this application also includes a filter array, which uses the filter array to achieve imaging of multiple special lights without the need for switching light sources and other operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 A schematic structural diagram of an endoscope system in one embodiment;

[0028] Figure 2 A schematic diagram of the structure of an imaging system in an embodiment used as a separate system independently;

[0029] Figure 3 A schematic diagram of the structure of an imaging system as a core component of an endoscope system in an embodiment;

[0030] Figure 4 is a schematic structural diagram of an imaging assembly in one embodiment;

[0031] Figure 5 A schematic diagram of a filter array in an embodiment;

[0032] Figure 6 A schematic diagram of a filter array in an embodiment;

[0033] Figure 7 A schematic diagram of a first sub-filter unit in a second case in an embodiment;

[0034] Figure 8 A schematic diagram of a second sub-filter unit in a second case in an embodiment;

[0035] Figure 9 A method flowchart of a first method of generating a display image based on a hyperspectral image in an embodiment;

[0036] Figure 10 A schematic diagram of a first filter unit in an embodiment;

[0037] Figure 11 A method flowchart of a second method of generating a display image based on a hyperspectral image in an embodiment;

[0038] Figure 12 A method flowchart of step S240 in the second method of generating a display image based on a hyperspectral image in an embodiment;

[0039] Figure 13 A method flowchart of step S241 in the second method of generating a display image based on a hyperspectral image in an embodiment;

[0040] Figure 14 A method flowchart of step S260 in the second method of generating a display image based on a hyperspectral image in an embodiment;

[0041] Figure 15 A schematic diagram of a 3x3 image extracted in a hyperspectral image in an embodiment;

[0042] Figure 16 A method flowchart of step S263 in the second method of generating a display image based on a hyperspectral image in an embodiment;

[0043] Figure 17 A method flowchart of step S265 in the second method of generating a display image based on a hyperspectral image in an embodiment;

[0044] Figure 18 A method flowchart of step S260 in a third method of generating a display image based on a hyperspectral image in an embodiment;

[0045] Figure 19 FIG. 1 is a schematic diagram of a 5x5 image extracted from a hyperspectral image in one embodiment;

[0046] Figure 20 FIG. 4 is a flow chart of step S264 in the third method of generating a display image based on a hyperspectral image in one embodiment;

[0047] Figure 21 FIG. 5 is a flow chart of step S266 in the third method of generating a display image based on a hyperspectral image in one embodiment. DETAILED DESCRIPTION

[0048] The application will be further described below in connection with the drawings. Like numbers in different figures represent similar elements. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the application. Flowever, it will be apparent to one skilled in the art that the application can be practiced without these specific details. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the application.

[0049] In addition, features, operations, or steps described in the specification can be combined in any suitable manner without departing from the scope of the application. Similarly, the various steps or actions in a method can be combined, reordered, or split into further steps or actions without departing from the scope of the application. Accordingly, the description and drawings are to be regarded as illustrative in nature and not as restrictive.

[0050] The serial numbers of components in this application, such as "first", "second", etc., are used only to distinguish the described objects, and do not have any sequential or technical meaning. Unless otherwise specified, "connected" or "coupled" in this application includes direct and indirect connections (couplings).

[0051] Reference should be made to Figure 1 In one embodiment, the endoscope system 1 is a complex optical and electronic imaging device, mainly used for internal human body or industrial detection, and includes an illumination system 11, an imaging system 12, a guide portion 13, a control system 14, an image transmission and display system 15, and a power supply system 16 in hardware structure. The imaging system 12 can be used as a core component in the endoscope system 1, or can be used as a separate system independently.

[0052] Referring to Figure 2 In an embodiment, when the imaging system 12 is applied independently as a separate system, the imaging system 12 comprises the illumination assembly 121, the imaging assembly 122, and the processor 123. The imaging system 12 images the detected region by using the illumination assembly 121, the imaging assembly 122, and the processor 123.

[0053] Referring to Figure 3 In an embodiment, when the imaging system 12 is applied as a core component in the endoscope system 1, the imaging system 12 does not need the illumination assembly 121, and only comprises the imaging assembly 122 and the processor 123, which uses the illumination system 11 in the endoscope system 1 to provide the illumination light to the detected region.

[0054] In an embodiment, the illumination system 11 in the endoscope system 1 or the illumination assembly 121 in the imaging system 12 can adopt one or more forms of light sources such as white light, red light, blue light, infrared light, ultraviolet light, and laser light, and the specific type of light source can be selected according to the application requirements. The application preferably adopts ordinary white light as the illumination light, which can be generated by an LED or a xenon lamp light source, and is used to provide the detected region with continuous visible spectrum illumination light. Compared with other types of light sources, ordinary white light has good performance in color rendering, tissue recognition, and use safety, which helps to improve the clarity and authenticity of the image. In addition, the LED light source has the advantages of low power consumption and long service life, and the xenon lamp has high brightness and good spectral continuity, so the appropriate type of light source can be selected according to the specific application requirements.

[0055] In summary, in addition to the illumination system 11 and the illumination assembly 121 in the endoscope system 1 and the imaging system 12, the remaining parts of the imaging system 12 are the same, which are described in detail below.

[0056] Referring to Figure 4 In an embodiment, the imaging assembly 122 comprises a shooting lens 1221, a light splitting component 1222, a filter array 1223, and a hyperspectral image sensor 1224. The shooting lens 1221 is used to collect the reflected light or scattered light after the illumination light irradiates the detected region to emit the imaging light.

[0057] In an embodiment, when only the hyperspectral image sensor 1224 is arranged in the imaging assembly 122, the shooting lens 1221 is used to collect the optical image of the target region and guide the imaging light to the hyperspectral image sensor 1224. At this time, the imaging light path emitted by the shooting lens 1221 is only connected with the hyperspectral image sensor 1224, and does not involve the optical coupling relationship of other image sensors.

[0058] The filter array 1223 and the hyperspectral image sensor 1224 are arranged in sequence on the light path of the imaging light emitted by the photographing lens 1221, and the filter array 1223 is located between the photographing lens 1221 and the hyperspectral image sensor 1224, and is used for decomposing the imaging light into monochromatic light or narrow-band light of different wavelengths to perform wavelength-selective filtering on the imaging light. Specifically, the filter array 1223 can decompose the composite light from the photographing lens 1221 into monochromatic light or narrow-band light of different wavelengths, thereby realizing optical separation of the detected region in multiple spectral dimensions.

[0059] The hyperspectral image sensor 1224 is used for receiving the light signal processed by the filter array 1223, and converting the monochromatic light or narrow-band light corresponding to different wavelengths into an electrical signal, thereby obtaining hyperspectral image data containing rich spectral information.

[0060] The present application specially sets the filter array 1223 in the hyperspectral image sensor 1224, and different imaging methods are generated based on this special setting, which will be described in detail below.

[0061] Please refer to Figure 5 In an embodiment, the filter array 1223 includes at least two types of filter units, one type being a first filter unit 12231 that transmits a preset wavelength band, and the other type being a second filter unit 12232 that transmits the remaining wavelength bands except the preset wavelength band. The filters in the first filter unit 12231 and the filters in the second filter unit 12232 are alternately and spacedly distributed in the horizontal direction and the vertical direction to form a checkerboard distribution, thereby forming an arrangement structure that is uniformly covered in space and is beneficial to subsequent image reconstruction and interpolation.

[0062] It should be noted that in the endoscope system 1, in order to improve the imaging contrast of biological tissues and the ability to extract functional information, the spectral design of the filter array 1223 can be set based on the biological spectral response characteristics of the detected region. For example, human blood has obvious absorption peaks in the wavelength ranges of 415 nm±20 nm and 540 nm±20 nm, and therefore the first filter unit 12231 that transmits this wavelength band can be arranged in the filter array 1223, so that the imaging in this wavelength band can present a blood vessel image with higher contrast. For example, in the wavelength band of 590 nm±10 nm, the gradient distribution characteristics of blood concentration are more obvious, and by arranging the first filter unit 12231 that transmits this wavelength band, an image reflecting the distribution of blood concentration can be obtained.

[0063] In one embodiment, the present application selects a single color of filter in the filter array 1223 whose quantum efficiency is greater than a set efficiency to form the first filter unit 12231, or selects a single color of filter in the filter array 1223 whose imaging resolution is greater than a set resolution to form the first filter unit 12231, or selects a single color of filter in the filter array 1223 whose quantum efficiency is greater than a set efficiency and whose imaging resolution is greater than a set resolution to form the first filter unit 12231. After the first filter unit 12231 is selected, the number of filters in the first filter unit 12231 needs to be greater than or equal to half of the total number of filters in the filter array 1223.

[0064] It should be noted that quantum efficiency refers to the ability of an image sensor or a certain filter waveband to respond to incident photons, specifically how many of the photons that fall on the image sensor pixels can be successfully converted into an electrical signal. In the filter array 1223, if the quantum efficiency of a certain color of filter for the photons in the waveband is high, it means that the quality of the color channel is good and the light sensitivity is high. Imaging resolution refers to the number of pixels covered by the color filter or its sampling density in the entire image. Selecting a filter with an imaging resolution greater than a set resolution can make the image clearer while retaining more details.

[0065] In addition, the number of filters in the first filter unit 12231 needs to be greater than or equal to half of the total number of filters in the filter array 1223, which can improve the sampling density of the image signal in the waveband corresponding to the first filter unit 12231, thereby enhancing the image resolution and edge detail performance of the waveband corresponding to the first filter unit 12231. At the same time, the information collected by the first filter unit 12231 in the waveband is used as the main reference channel in the image reconstruction process to assist the interpolation operation of other wavebands, thereby improving the clarity and accuracy of the overall image. In addition, considering that the human eye has higher perception sensitivity to this waveband, increasing the proportion of the first filter unit 12231 helps to obtain higher quality luminance information, which has higher signal-to-noise ratio and visual effect in the image enhancement and subsequent analysis process.

[0066] Please refer to Figure 6 In the specific implementation process of the present application, the green filter is selected as the first filter unit 12231, that is, the green filter as the first filter unit 12231 is distributed in a checkerboard pattern in the filter array 1223, and the filters of the remaining colors except green are set as the second filter unit 12232 in the remaining positions other than the distribution position of the first filter unit 12231. It can be observed that Figure 6 It can be seen that the second filter unit 12232 is also distributed in a checkerboard pattern in the filter array 1223.

[0067] In one embodiment, the second filter unit 12232 includes a first sub-filter unit 122321 and a second sub-filter unit 122322. At least two colors of filters are included in each of the first sub-filter unit 122321 and the second sub-filter unit 122322, and the colors of the filters in the first sub-filter unit 122321 and the second sub-filter unit 122322 are different. The spectral response difference of the filters of each color in the first sub-filter unit 122321 is less than a first set value, and the spectral response difference of the filters of each color in the second sub-filter unit 122322 is less than a second set value. The spectral response difference of the filters of each color in the first sub-filter unit 122321 and the second sub-filter unit 122322 is greater than a third set value.

[0068] In one embodiment, in order to improve the sampling continuity of the spectral transition region, the filters of different colors in the first sub-filter unit 122321 are alternately and spacedly distributed in the horizontal direction of the second filter unit 12232. The filters of different colors in the second sub-filter unit 122322 are also alternately and spacedly distributed in the horizontal direction of the second filter unit 12232. In this case, the distribution in each row of the first sub-filter unit 122321 in the second filter unit 12232 can be the same as the first case. For example, the first sub-filter unit 122321 includes orange and red filters, and in the first case, the first sub-filter unit 122321 in each row is alternately and spacedly distributed in the order of orange-red. In addition, the distribution in each row of the first sub-filter unit 122321 in the second filter unit 12232 can also be the second case. For example, in the second case, the first sub-filter unit 122321 in any row of the second filter unit 12232 is alternately and spacedly distributed in the order of orange-red, and the first sub-filter unit 122321 in the adjacent row is alternately and spacedly distributed in the order of red-orange.

[0069] In one embodiment, in order to meet the special requirements, the filters of different colors in the first sub-filter unit 122321 are alternately and spacedly distributed in the vertical direction of the second filter unit 12232, and the filters of different colors in the second sub-filter unit 122322 are also alternately and spacedly distributed in the vertical direction of the second filter unit 12232. That is, the first sub-filter array 1223 and the second sub-filter array 1223 in the present application meet the second case described above. Please refer to Figure 7 and Figure 8Similarly, taking the example that the first sub-filter unit 122521 includes orange and red filters and the second sub-filter unit 122522 includes blue and purple filters, the second case specifically defined in this application is as follows: the filters in the first sub-filter unit 122521 and the second sub-filter unit 122522 are as follows: Figure 7 and Figure 8 Set it up as shown.

[0070] In one embodiment, to avoid image interference or artifacts caused by the concentrated distribution of spectral contrast pixels and to improve the overall stability of the original image and spectral decoding efficiency, the present application further arranges the first sub-filter units 122321 and the second sub-filter units 122322 in an alternating pattern in the vertical direction of the second filter units 12232. That is, when the first sub-filter units 122321 are arranged in the first row of the second filter units 12232, the second sub-filter units 122322 are arranged in the second row of the second filter units 12232, and so on.

[0071] Based on the hardware architecture of the above-mentioned imaging component 122, whether it is an imaging system 12 used independently as a separate system or an imaging system 12 as a core component of the endoscope system 1, the processor 123 included therein can generate a display image for output of the detected area based on a grayscale image, or can generate a display image for output of the detected area based on a hyperspectral image, or can generate a display image for output of the detected area based on a grayscale image and a hyperspectral image.

[0072] In one embodiment, the processor 123 generates a display image corresponding to the detected area for output based on the hyperspectral image, including three methods. Among them, due to the different arrangements of the filter array, it is divided into the second method and the third method, which are described in detail below.

[0073] Please refer to Figure 9 ,The first method includes the following steps.

[0074] Step S210: Acquire a checkerboard image of a first color channel.

[0075] In one embodiment, the hyperspectral image includes a checkerboard image of a first color channel corresponding to the filter color in the first filter unit 12231 determined by the hyperspectral image sensor 1224 through the first filter unit 12231. If a green filter is selected as the first filter unit 12231 in this application, the checkerboard image of the first color channel is as follows: Figure 10 Shown is a green checkerboard image.

[0076] Step S230: Taking the point to be calculated as the center, select a pixel point corresponding to the first filter color in the checkerboard image of the first color channel.

[0077] In one embodiment, in order to calculate the pixel value of any point to be calculated, in the checkerboard image of the first color channel, with the point to be calculated as the center, a total of 16 pixel points corresponding to the first filter color are obtained in the four directions of the top, bottom, left, and right of the point to be calculated. If a green filter is selected as the first filter unit 12231 in this application, then the pixel points corresponding to the first filter color are as follows: Figure 10 Green pixels are shown.

[0078] Step S250: Calculate the pixel value of the point to be calculated using a bicubic interpolation method according to the pixel point corresponding to the selected first filter color.

[0079] In one embodiment, Figure 10 The selection in the above is taken as an example, and the 16 green pixels obtained in the four directions of the point to be calculated are described in the form of a coordinate system. Figure 10 The green pixels in . Here, the direction from top to bottom and from left to right is described. Figure 10 The coordinate point corresponding to each pixel in is (i, j), i and j are both positive integers, i∈[1, 7], j∈[1, 7].

[0080] Use the following formula to calculate the pixel value of the point to be calculated:

[0081]

[0082] Among them, G represents the pixel value of the point to be calculated, S1 and S0 represent the interpolation kernel coefficients, G11 represents the green pixel value corresponding to the filter color in the first filter unit 12231 corresponding to the coordinates (1, 4), G12 represents the green pixel value corresponding to the filter color in the first filter unit 12231 corresponding to the coordinates (2, 5), G13 represents the green pixel value corresponding to the filter color in the first filter unit 12231 corresponding to the coordinates (3, 6), and G14 represents the green pixel value corresponding to the filter color in the first filter unit 12231 corresponding to the coordinates (4, 7). G21 represents the green pixel value corresponding to the filter color in the first filter unit 12231 corresponding to the coordinates (2, 3), G22 represents the green pixel value corresponding to the filter color in the first filter unit 12231 corresponding to the coordinates (3, 4), G23 represents the green pixel value corresponding to the filter color in the first filter unit 12231 corresponding to the coordinates (4, 5), G24 represents the green pixel value corresponding to the filter color in the first filter unit 12231 corresponding to the coordinates (5, 6), The green pixel value corresponding to the filter color, G31 represents the green pixel value corresponding to the filter color in the first filter unit 12231 corresponding to the coordinates (3, 2), G32 represents the green pixel value corresponding to the filter color in the first filter unit 12231 corresponding to the coordinates (4, 3), G33 represents the green pixel value corresponding to the filter color in the first filter unit 12231 corresponding to the coordinates (5, 4), G34 represents the green pixel value corresponding to the filter color in the first filter unit 12231 corresponding to the coordinates (6, 5), G41 represents the green pixel value corresponding to the filter color in the first filter unit 12231 corresponding to the coordinates (4, 1), G42 represents the green pixel value corresponding to the filter color in the first filter unit 12231 corresponding to the coordinates (5, 2), G43 represents the green pixel value corresponding to the filter color in the first filter unit 12231 corresponding to the coordinates (6, 3), and G44 represents the green pixel value corresponding to the filter color in the first filter unit 12231 corresponding to the coordinates (7, 4).

[0083] The above method is used to calculate the pixel value of each point to be calculated, thereby generating a display image for output in the detected area.

[0084] Please refer to Figure 11 ,The second method includes the following steps.

[0085] Step S220: Acquire a checkerboard image of the first color channel and a checkerboard image of the second color channel.

[0086] In one embodiment, the hyperspectral image includes a checkerboard image of a first color channel corresponding to the filter colors in the first filter unit 12231 determined by the hyperspectral image sensor 1224 through the first filter unit 12231, and a checkerboard image of a second color channel corresponding to the filter colors in the second filter unit 12232 determined by the hyperspectral image sensor 1224 through the second filter unit 12232. If the green filter is selected in this application as the first filter unit 12231, the checkerboard image of the first color channel is a green checkerboard image as shown in FIG. 3A. If the orange filter and the red filter are selected in this application as the first sub-filter unit 122321, and the purple filter and the blue filter are selected as the second sub-filter unit 122322, please refer to FIG. 3B. The checkerboard image composed of orange, red, purple and blue except the green pixel points in FIG. 3B is the checkerboard image of the second color channel. Figure 10 Figure 6

[0087] Step S240: Interpolating the pixel values of the first color corresponding to the first filter unit in the blank positions of the checkerboard image of the first color channel to generate a complete image of the first color channel.

[0088] Please refer to FIG. 3A and FIG. 3B. Figure 12 In one embodiment, when performing step S240 of interpolating the pixel values of the first color corresponding to the first filter unit in the blank positions of the checkerboard image of the first color channel to generate a complete image of the first color channel, the following steps are further included.

[0089] Step S241: Obtaining a full-resolution first training image, and generating a second training image corresponding to the checkerboard image of the first color channel according to the first training image.

[0090] Please refer to FIG. 3A and FIG. 3B. Figure 13 In one embodiment, when performing step S241 of obtaining a full-resolution first training image, and generating a second training image corresponding to the checkerboard image of the first color channel according to the first training image, the following steps are further included.

[0091] Step S2411: Merging the pixel values of the first color in the checkerboard image of the first color channel to generate a sampling image.

[0092] In one embodiment, since the checkerboard image of the first color channel is determined by the filter colors in the first filter unit 12231, in the checkerboard image of the first color channel, there are pixel points corresponding to the filters in the first color filter unit. Merging these pixel points generates a sampling image. Please refer to FIG. 3A and FIG. 3B. Figure 10 Figure 10 ​​​All green pixel points in the sampling image are merged to generate a sampling image.

[0093] Step S2412: The sampling image is down-sampled according to a set specification to generate a first training image.

[0094] In an embodiment, an image of a set specification is cropped from the sampling image to complete the down-sampling, thereby generating the first training image. That is, an image of any specification such as 3x3, 4x4, or 5x5 is cropped from the sampling image to generate the first training image. Since the first training image is determined from the sampling image, and the pixel values of all pixel points in the sampling image are known, all pixel points in the first training image are also known, and thus the first training image is a full-resolution first training image.

[0095] Step S2413: A second training image is generated according to the blank positions in the checkerboard image of the first color channel.

[0096] In an embodiment, the second training image is generated by extracting, as a contrast, the pixel point positions in the first training image in which the corresponding pixel values do not exist, from the blank positions in the checkerboard image of the first color channel. The blank positions are the pixel point positions in which the pixel values of the first color in the checkerboard image of the second color channel in the hyperspectral image do not exist, that is, the white pixel point positions in the following formula. Figure 10

[0097] Step S242: The convolutional neural network model is trained according to the first training image and the second training image.

[0098] In an embodiment, the first training image and the second training image are input into the convolutional neural network model, where the first training image is a full-resolution image, and the second training image is an image in which the pixel values are missing. The first training image and the second training image are used to train the convolutional neural network model, so that the convolutional neural network model learns how to estimate the missing pixel values in the image in which the pixel values are missing from the full-resolution image. The present application uses a convolutional neural network with no less than 2 layers to extract local features in the first training image and the second training image and perform interpolation prediction. The mean square error is used as a loss function to measure the error between the inferred pixel values and the true pixel values. Finally, the Adam optimizer is used for parameter optimization to ensure that the convolutional neural network model can effectively converge, thereby completing the training of the convolutional neural network model.

[0099] Step S243: The checkerboard image of the first color channel is input into the trained convolutional neural network model to generate a complete image of the first color channel.

[0100] ​In one embodiment, since the convolutional neural network model has been trained, the checkerboard image of the first color channel is input into the trained convolutional neural network model, and the complete image of the first color channel can be obtained.

[0101] Step S260: Interpolating the pixel values of each pixel point in different color channels in the hyperspectral image according to the complete image of the first color channel and the checkerboard image of the second color channel.

[0102] Please refer to Figure 14 In one embodiment, when performing step S260 of interpolating the pixel values of each pixel point in different color channels in the hyperspectral image according to the complete image of the first color channel and the checkerboard image of the second color channel, the following steps are included.

[0103] Step S261: Determining the first native color pixel value, the first color pixel value to be interpolated, the second native color pixel value, the interpolated first color pixel value, and the second color pixel value to be interpolated.

[0104] In one embodiment, since the complete image of the first color channel is determined by the checkerboard image of the first color channel, and the checkerboard image of the first color channel is determined by the first filter unit 12231, the pixel value corresponding to the first color in each pixel point in the complete image of the first color channel is originally known, i.e., the first native color pixel value, and the pixel values of other color channels are unknown, i.e., the first color pixel value to be interpolated.

[0105] In one embodiment, the checkerboard image of the second color channel is determined by the second filter unit 12232, and in the checkerboard image of the second color channel, the pixel values of each pixel point corresponding to the second filter unit 12232 are originally known, i.e., the second native color pixel value.

[0106] In one embodiment, interpolating the blank positions in the checkerboard image of the first color channel generates the complete image of the first color channel, and the blank positions in the checkerboard image of the first color channel are the pixel point positions where the pixel values of the first color in the checkerboard image of the second color channel do not exist. Therefore, the pixel point positions where the pixel values of the first color in the checkerboard image of the second color channel do not exist are interpolated in the checkerboard image of the first color channel, and it can be concluded that the first color pixel values have been interpolated in the pixel point positions where the pixel values of the first color in the checkerboard image of the second color channel do not exist, i.e., the interpolated first color pixel value.

[0107] In one embodiment, the second native color pixel values ​​and the interpolated first color pixel values ​​are removed from the checkerboard image of the second color channel, and the remaining values ​​are the color pixel values ​​that need to be interpolated in the checkerboard image of the second color channel, i.e., the second color pixel values ​​to be interpolated.

[0108] For example, Figure 15 As shown, the checkerboard image of the first color channel is a checkerboard image composed of green pixels. In the checkerboard image of the first color channel, the pixel values ​​of all green pixels are the first native color pixel values. The pixel values ​​of all green pixels in the other color channels (such as Figure 15 The B and R to be calculated, that is, the blue pixel value and the red pixel value to be calculated) are the first color pixel values ​​to be interpolated.

[0109] The checkerboard image of the second color channel is a checkerboard image composed of blue pixels and red pixels. In the checkerboard image of the second color channel, the pixel values ​​of all blue pixels and the pixel values ​​of all red pixels are the second original color pixel values. The first color pixel value interpolated from the pixel position where the first color pixel value does not exist in the checkerboard image of the second color channel is the interpolated first color pixel value, that is, Figure 15 The green pixel value is interpolated from the blue pixel and the red pixel (such as Figure 15 Among the blue and red pixels, G has been interpolated, that is, the green pixel value has been interpolated). The pixel values ​​of the remaining color channels in the pixel points with known pixel values ​​in the checkerboard image of the second color channel are the second color pixel values ​​to be interpolated. Figure 15 In the example, R (the red pixel value to be calculated) for the blue pixel is the second color pixel value to be interpolated in the checkerboard image of the second color channel. B (the blue pixel value to be calculated) for the red pixel is also the second color pixel value to be interpolated in the checkerboard image of the second color channel.

[0110] Step S263 : determining a first to-be-interpolated color pixel value according to the first native color pixel value, the second native color pixel value, and the interpolated first color pixel value.

[0111] Please refer to Figure 16 In one embodiment, determining the first color pixel value to be interpolated according to the first native color pixel value, the second native color pixel value and the interpolated first color pixel value in step S263 includes the following steps.

[0112] Step S2631: Obtain the second native color pixel value and the interpolated first color pixel value of the pixel point closest to the pixel point corresponding to the first color pixel value to be interpolated in the horizontal direction or vertical direction in the checkerboard image of the second color channel.

[0113] In one embodiment, after determining the first native color pixel value, the second native color pixel value, and the interpolated first color pixel value, the second native color pixel value and the interpolated first color pixel value of the pixel point closest to the pixel point corresponding to the first color pixel value to be interpolated are obtained in the horizontal direction or vertical direction of the checkerboard image of the second color channel.

[0114] Since the checkerboard image of the first color channel and the checkerboard image of the second color channel are checkerboard images arranged crosswise with each other, the second native color pixel value required for calculating the first color pixel value to be interpolated and the interpolated first color pixel value must be present at the adjacent pixel points in the horizontal or vertical direction of the pixel point corresponding to the first color pixel value to be interpolated.

[0115] Step S2632: determining a first color pixel value to be interpolated according to the first original color pixel value, the second original color pixel value of the closest pixel point, and the interpolated first color pixel value.

[0116] In one embodiment, also referring to Figure 15 The green pixel value of any green pixel point is the first native pixel value, and the to-be-calculated B and / or to-be-calculated R in the green pixel point is the first to-be-interpolated color pixel value.

[0117] In the horizontal direction of the green pixel, the two closest adjacent blue pixels are selected. The blue pixel values ​​(original B) of the two adjacent blue pixels are used as the second original color pixel value, and the interpolated green pixel values ​​(interpolated G) of the two adjacent blue pixels are used as the interpolated first color pixel value. The first color pixel value to be interpolated is calculated using the following formula:

[0118]

[0119] in, Figure 15 In the 3×3 image, from top to bottom and from left to right is the reference, B (1,2) The blue pixel value of the pixel corresponding to the coordinate (1, 2), that is, the first color pixel value to be interpolated; G (1,2) represents the green pixel value of the pixel corresponding to the coordinate (1, 2), that is, the first native pixel value; B (1,1) G represents the blue pixel value of the pixel corresponding to the coordinate (1, 1), that is, the second native color pixel value; (1,1) represents the green pixel value of the pixel point corresponding to the coordinate (1, 1), that is, the interpolated first color pixel value; B (1,3) G represents the blue pixel value of the pixel corresponding to the coordinate (1, 3), that is, the second native color pixel value; (1,3)G represents the green pixel value of the pixel point corresponding to the coordinate (1, 3), i.e. the first color pixel value after interpolation.

[0120] In the vertical direction of the green pixel point, the nearest two adjacent red pixel points are selected, the red pixel value (native R) of the two adjacent red pixel points is the second native color pixel value, the green pixel value (interpolated G) of the two adjacent red pixel points is the first color pixel value after interpolation, and the first color pixel value to be interpolated is calculated by using the following formula:

[0121]

[0122] Wherein, Figure 14 In the 3x3 image, taking the direction from top to bottom and from left to right as the reference, R (1,2) G represents the green pixel value of the pixel point corresponding to the coordinate (1, 2), i.e. the first color pixel value after interpolation. (1,2) G represents the green pixel value of the pixel point corresponding to the coordinate (1, 2), i.e. the first color pixel value after interpolation. (0,2) G represents the green pixel value of the pixel point corresponding to the coordinate (1, 2), i.e. the first color pixel value after interpolation. (0,2) G represents the green pixel value of the pixel point corresponding to the coordinate (1, 2), i.e. the first color pixel value after interpolation. (2,2) G represents the green pixel value of the pixel point corresponding to the coordinate (1, 2), i.e. the first color pixel value after interpolation. (2,2) G represents the green pixel value of the pixel point corresponding to the coordinate (1, 2), i.e. the first color pixel value after interpolation.

[0123] Step S265: determining the second color pixel value to be interpolated according to the second native color pixel value and the first color pixel value after interpolation.

[0124] Please refer to Figure 17 In an embodiment, when performing step S263 of determining the second color pixel value to be interpolated according to the second native color pixel value and the first color pixel value after interpolation, the following steps are further included.

[0125] Step S2651: constructing an n x n interpolation unit with the pixel point corresponding to any second color pixel value to be interpolated in the checkerboard image of the second color channel as the center point.

[0126] In an embodiment, in the checkerboard image of the second color channel, any pixel point corresponding to a second color pixel value to be interpolated is selected as the center point to construct an n x n interpolation unit, where n is an odd number greater than or equal to 3.

[0127] Step S2652: determining the second color pixel value to be interpolated.

[0128] In one embodiment, after determining the n x n interpolation unit, the second native color pixel value of the pixel point at the vertex of the main diagonal line in the interpolation unit can be determined by using the second native color pixel value and the first color pixel value of the pixel point at the vertex of the main diagonal line in the interpolation unit; the second native color pixel value of the pixel point at the vertex of the sub-diagonal line in the interpolation unit can be determined by using the second native color pixel value and the first color pixel value of the pixel point at the vertex of the sub-diagonal line in the interpolation unit; and the second native color pixel value of the pixel point at the vertex of the main diagonal line in the interpolation unit can be determined by using the second native color pixel value and the first color pixel value of the pixel point at the vertex of the main diagonal line in the interpolation unit, and the second native color pixel value and the first color pixel value of the pixel point at the vertex of the sub-diagonal line in the interpolation unit. The following will be described in detail.

[0129] In one embodiment, the distribution of the first sub-filter units 122321 in each row of the second filter unit 12232 can have the same first case, and the distribution of the first sub-filter units 122321 in each row of the second filter unit 12232 can also have the second different case. Therefore, the second native color pixel value of the pixel point at the vertex of the main diagonal line or the sub-diagonal line in the interpolation unit can be the same, and the second native color pixel value of the pixel point at the vertex of the main diagonal line or the sub-diagonal line in the interpolation unit can be different.

[0130] In one embodiment, when the second native color pixel value of the pixel point at the vertex of the main diagonal line or the sub-diagonal line in the interpolation unit is the same:

[0131] First, the gradient change of the main diagonal line is calculated according to the second native color pixel value of the pixel point at the vertex of the main diagonal line in the interpolation unit and the first color pixel value of the pixel point at the vertex of the main diagonal line in the interpolation unit, and the calculation is performed by using the following formula:

[0132] GradN = |B i-1,j-1 -B i+1,j+1 | + |2 x G i,j -(G i-1,j-1 + G i+1,j+1 )|

[0133] Wherein, GradN represents the gradient of the main diagonal line, B i-1,j-1 represents the second native color pixel value of the pixel point at one vertex of the main diagonal line, B i+1,j+1 represents the second native color pixel value of the pixel point at the other vertex of the main diagonal line, G i,j represents the first color pixel value of the pixel point in the interpolation unit, G i-1,j-1 represents the first color pixel value of the pixel point at one vertex of the main diagonal line in the interpolation unit, and G i+1,j+1the interpolated first color pixel value of the pixel point of the other vertex of the main diagonal.

[0134] The gradient change of the sub-diagonal is calculated according to the second original color pixel value of the pixel point of the vertex of the sub-diagonal in the interpolation unit and the interpolated first color pixel value, and the calculation is performed by using the following formula:

[0135] GradP = |B i-1,j+1 -B i+1,j-1 | + |2 x G i,j -(G i-1,j+1 + G i+1,j-1 )|

[0136] Wherein, GradP represents the gradient of the sub-diagonal, B i-1,j+1 represents the second original color pixel value of the pixel point of one vertex of the sub-diagonal, B i+1,j-1 represents the second original color pixel value of the pixel point of the other vertex of the sub-diagonal, G i,j represents the interpolated first color pixel value of the pixel point corresponding to the second color pixel value to be interpolated, G i-1,j+1 represents the interpolated first color pixel value of the pixel point of one vertex of the sub-diagonal, G i+1,j-1 represents the interpolated first color pixel value of the pixel point of the other vertex of the sub-diagonal.

[0137] If the gradient change of the main diagonal is less than the gradient change of the sub-diagonal, the second color pixel value to be interpolated is determined by using the second original color pixel value of the pixel point of the vertex of the main diagonal in the interpolation unit and the interpolated first color pixel value, and the calculation is performed by using the following formula:

[0138]

[0139] Wherein, B i,j represents the second color pixel value to be interpolated.

[0140] If the gradient change of the main diagonal is greater than the gradient change of the sub-diagonal, the second color pixel value to be interpolated is determined by using the second original color pixel value of the pixel point of the vertex of the sub-diagonal in the interpolation unit and the interpolated first color pixel value, and the calculation is performed by using the following formula:

[0141]

[0142] Wherein, B i,j represents the second color pixel value to be interpolated.

[0143] If the gradient variation of the main diagonal is equal to the gradient variation of the sub-diagonal, the second to-be-interpolated color pixel value is determined by using the second native color pixel value of the pixel point of the vertex of the main diagonal of the interpolation unit and the first color pixel value which has been interpolated, the second native color pixel value of the pixel point of the vertex of the sub-diagonal of the interpolation unit and the first color pixel value which has been interpolated, and the following formula is used for calculation:

[0144]

[0145] wherein B i,j represents the second to-be-interpolated color pixel value.

[0146] In an embodiment, when the second native color pixel values of the pixel points of the vertices of the main diagonal or the sub-diagonal in the interpolation unit are different:

[0147] The pixel value corresponding to the second native color pixel value of the pixel point of the vertex of the other diagonal in the pixel point of one of the vertices of the main diagonal or the sub-diagonal is calculated, and the pixel value is also named as the second native color pixel value. Specifically, the pixel point of any one of the vertices of the main diagonal or the sub-diagonal is taken as the center point, and an updated interpolation unit is taken. The second native color pixel value of any one of the vertices of the main diagonal or the sub-diagonal is updated to the second to-be-interpolated color pixel value in the updated interpolation unit, and the second to-be-interpolated color pixel value in the updated interpolation unit is calculated by using the above-mentioned way of calculating the second to-be-interpolated color pixel value in the interpolation unit, and the second to-be-interpolated color pixel value calculated in this way is the pixel value corresponding to the second native color pixel value of the pixel point of the vertex of the other diagonal in the pixel point of one of the vertices of the main diagonal or the sub-diagonal.

[0148] Specifically, also referring to Figure 15 the green pixel value in the red pixel point corresponding to the coordinate (2, 2) is taken as the second to-be-interpolated color pixel value, and the green pixel value in the red pixel point corresponding to the coordinate (2, 2) is the first color pixel value which has been interpolated in the pixel point corresponding to the second to-be-interpolated color pixel value.

[0149] the blue pixel value of the pixel point corresponding to the coordinate (1, 1) is the second native color pixel value of the pixel point of one of the vertices of the main diagonal, and the green pixel value of the pixel point corresponding to the coordinate (1, 1) is the first color pixel value which has been interpolated in the pixel point of one of the vertices of the main diagonal. The blue pixel value of the pixel point corresponding to the coordinate (3, 3) is the second native color pixel value of the pixel point of the other vertex of the main diagonal, and the green pixel value of the pixel point corresponding to the coordinate (3, 3) is the first color pixel value which has been interpolated in the pixel point of the other vertex of the main diagonal. Then the gradient variation of the main diagonal is:

[0150] GradN=|B 1,1-B 3,3 |+|2×G 2,2 -(G 1,1 +G 3,3 )|

[0151] Among them, GradN represents the gradient of the main diagonal, B 1,1 Indicates the blue pixel value of the pixel corresponding to the coordinate (1, 1), B 3,3 Indicates the blue pixel value of the pixel corresponding to the coordinate (3, 3), G 2,2 Represents the green pixel value in the red pixel corresponding to the coordinate (2, 2), G 1,1 Indicates the blue pixel value of the pixel corresponding to the coordinate (1, 1), G 3,3 Indicates the green pixel value of the pixel corresponding to the coordinate (3, 3).

[0152] The blue pixel value of the pixel corresponding to the coordinate (1, 3) is the second native color pixel value of the pixel at one of the vertices of the sub-diagonal line, and the green pixel value of the pixel corresponding to the coordinate (1, 3) is the interpolated first color pixel value of the pixel at one of the vertices of the sub-diagonal line. The blue pixel value of the pixel corresponding to the coordinate (3, 1) is the second native color pixel value of the pixel at the other vertex of the sub-diagonal line, and the green pixel value of the pixel corresponding to the coordinate (3, 1) is the interpolated first color pixel value of the pixel at the other vertex of the sub-diagonal line. Then the gradient change of the sub-diagonal line is:

[0153] GradP=|B 1,3 -B 3,1 |+|2×G 2,2 -(G 1,3 +G 3,1 )|

[0154] Among them, GradP represents the gradient of the secondary diagonal, B 1,3 Indicates the blue pixel value of the pixel corresponding to the coordinate (1, 3), B 3,1 Indicates the blue pixel value of the pixel corresponding to the coordinate (3, 1), G 2,2 Represents the green pixel value in the red pixel corresponding to the coordinate (2, 2), G 1,3 Indicates the blue pixel value of the pixel corresponding to the coordinate (1, 3), G 3,1 Indicates the green pixel value of the pixel corresponding to the coordinate (3, 1).

[0155] If the gradient change of the main diagonal is less than the gradient change of the secondary diagonal, then:

[0156]

[0157] Among them, B 2,2Represents the green pixel value in the red pixel corresponding to the coordinate (2, 2).

[0158] If the gradient change of the main diagonal is greater than the gradient change of the secondary diagonal, then:

[0159]

[0160] Among them, B 2,2 Represents the green pixel value in the red pixel corresponding to the coordinate (2, 2).

[0161] If the gradient change of the main diagonal is equal to the gradient change of the secondary diagonal, then:

[0162]

[0163] Among them, B 2,2 Represents the green pixel value in the red pixel corresponding to the coordinate (2, 2).

[0164] If, among the second native color pixel values ​​of the pixel point corresponding to the coordinates (1, 1) and the pixel point corresponding to the coordinates (3, 3), the second native color pixel value of the pixel point corresponding to the coordinates (1, 1) is not a blue pixel value, then it is necessary to calculate the blue pixel value of the pixel point corresponding to the coordinates (1, 1) to ensure that the second native color pixel values ​​of the pixel point corresponding to the coordinates (1, 1) and the pixel point corresponding to the coordinates (3, 3) are both blue pixel values.

[0165] With the pixel corresponding to the coordinate (1, 1) as the center point, we also construct an n×n interpolation unit and sample the above calculation B. 2,2 Calculate the blue pixel value of the pixel corresponding to the coordinate (1, 1) in this way.

[0166] It should be noted that when calculating B 2,2 When calculating the blue pixel value of the pixel at the coordinate (1, 1), the pixels at the main and sub-diagonal vertices of the interpolation unit are used. Since the center point of the interpolation unit has been updated, the pixels at the main and sub-diagonal vertices of the interpolation unit are also updated.

[0167] In one embodiment, in step S2652 , when determining the second color pixel value to be interpolated, a 3×3 interpolation unit is constructed. The interpolation unit may also be a 5×5 or 7×7 interpolation unit.

[0168] When the interpolation unit is 5x5, the gradient change of the main diagonal or the sub-diagonal is calculated by using the pixel points of the vertexes of the main diagonal or the sub-diagonal of the interpolation unit. Since the color change on the diagonal is approximately linear, the pixel value of the three pixel points of the middle diagonal can be determined according to the gradient change, and the second color pixel value to be interpolated is determined according to the pixel value of the three pixel points of the middle diagonal and the calculated gradient change. The specific formula is as follows:

[0169] T = B1 + B2 + B3

[0170] B1 = B2 - a

[0171] B3 = B2 + a

[0172] Then:

[0173] T = B2 - a + B2 + B2 + a = 3 x B2

[0174] Then:

[0175]

[0176] Wherein, T represents the sum of the pixel values of the three pixel points of the middle diagonal, B1 represents the pixel value of the pixel point of one vertex of the middle diagonal, B2 represents the second color pixel value to be interpolated, B3 represents the pixel value of the pixel point of another vertex of the middle diagonal, and a represents the gradient change of the main diagonal or the sub-diagonal.

[0177] The pixel values of the pixel points in different color channels of the hyperspectral image are interpolated by using the above method, so as to generate a display image of the detected area for output.

[0178] In one embodiment, the third method comprises the following steps.

[0179] In one embodiment, the third method uses the same method as in steps S220-S260 of the second method, and only uses a different method in step S260, which is as follows.

[0180] Please refer to Figure 18 In one embodiment, when the pixel values of the pixel points in different color channels of the hyperspectral image are interpolated according to the complete image of the first color channel and the checkerboard image of the second color channel in step S260, the following steps are included.

[0181] Step S262: determining the first native color pixel value, the first color pixel value to be interpolated, the second native color pixel value, the interpolated first color pixel value, and the second color pixel value to be interpolated.

[0182] In one embodiment, since the complete image of the first color channel is determined by the checkerboard image of the first color channel, and the checkerboard image of the first color channel is determined by the first filter unit 12251, the pixel value corresponding to the first color in each pixel point in the complete image of the first color channel is natively known, that is, the first native color pixel value, while the pixel values ​​of other color channels are unknown, that is, the first color pixel value to be interpolated.

[0183] In one embodiment, the checkerboard image of the second color channel is determined by the second filter unit 12252. Then, in the checkerboard image of the second color channel, the pixel value of each pixel corresponding to the second filter unit 12252 is natively known, that is, the second native color pixel value.

[0184] In one embodiment, a complete image of the first color channel is generated by interpolating the blank positions in the checkerboard image of the first color channel, and the blank positions in the checkerboard image of the first color channel are the pixel positions in the checkerboard image of the second color channel where the pixel values ​​of the first color do not exist. Therefore, the positions interpolated in the checkerboard image of the first color channel are the pixel positions in the checkerboard image of the second color channel where the pixel values ​​of the first color do not exist. This also indicates that the first color pixel values ​​have been interpolated at the pixel positions in the checkerboard image of the second color channel where the pixel values ​​of the first color do not exist, i.e., the interpolated first color pixel values.

[0185] In one embodiment, the second native color pixel values ​​and the interpolated first color pixel values ​​are removed from the checkerboard image of the second color channel, and the remaining values ​​are the color pixel values ​​that need to be interpolated in the checkerboard image of the second color channel, i.e., the second color pixel values ​​to be interpolated.

[0186] For example, Figure 19 As shown, the checkerboard image of the first color channel is a checkerboard image composed of green pixels. In the checkerboard image of the first color channel, the green pixel values ​​of all green pixels are the first native color pixel values. The pixel values ​​of all green pixels in the other color channels (such as Figure 19 The R, B, P and A to be calculated in , that is, the red pixel value, blue pixel value, purple pixel value and orange pixel value to be calculated) are the first color pixel values ​​to be interpolated.

[0187] The checkerboard image of the second color channel is a checkerboard image composed of blue pixel points, purple pixel points, red pixel points and orange pixel points. In the checkerboard image of the second color channel, the pixel value of all the blue pixel points, the pixel value of all the purple pixel points, the pixel value of all the red pixel points and the pixel value of all the orange pixel points are the second primary color pixel value. The first color pixel value interpolated in the pixel point position where the first color pixel value does not exist in the checkerboard image of the second color channel is the first color pixel value that has been interpolated, that is, the green pixel value that has been interpolated in the blue pixel point and the purple pixel point in Figure 19 The first color pixel value that has been interpolated is the green pixel value that has been interpolated in the blue pixel point and the purple pixel point in Figure 19 The first color pixel value that has been interpolated is the green pixel value that has been interpolated in the blue pixel point and the purple pixel point in Figure 19 The first color pixel value that has been interpolated is the green pixel value that has been interpolated in the blue pixel point and the purple pixel point in

[0188] Step S264: determining the first color pixel value to be interpolated according to the first primary color pixel value, the second primary color pixel value and the first color pixel value that has been interpolated.

[0189] Please refer to Figure 20 In an embodiment, the step S264 of determining the first color pixel value to be interpolated according to the first primary color pixel value, the second primary color pixel value and the first color pixel value that has been interpolated includes the following steps.

[0190] Step S2641: obtaining the second primary color pixel value and the first color pixel value that has been interpolated of the pixel point closest to the pixel point corresponding to the first color pixel value to be interpolated in the horizontal direction or the vertical direction of the checkerboard image of the second color channel.

[0191] In an embodiment, after the first primary color pixel value, the second primary color pixel value and the first color pixel value that has been interpolated are determined, the second primary color pixel value and the first color pixel value that has been interpolated of the pixel point in the 1st-order neighborhood closest to the pixel point corresponding to the first color pixel value to be interpolated in the horizontal direction or the vertical direction of the checkerboard image of the second color channel are obtained.

[0192] Step S2642: determining the first to-be-interpolated color pixel value according to the first primary color pixel value, the second primary color pixel value of the closest pixel point, and the interpolated first color pixel value.

[0193] In one embodiment, also referring to Figure 19 , the green pixel value of any one of the green pixel points is the first primary color pixel value, and the R, B, P and / or A to be calculated in the green pixel value point are all the first to-be-interpolated color pixel value.

[0194] In the horizontal direction of the green pixel point, one purple pixel point in the 1st-order neighborhood is selected, the purple pixel value (primary P) of the purple pixel point is the second primary color pixel value, the interpolated green pixel value (interpolated G) of the purple pixel point is the interpolated first color pixel value, and the first to-be-interpolated color pixel value is calculated by using the following formula:

[0195] P (6,7) =G (6,7) +P (6,6) -G (6,6)

[0196] Wherein, Figure 20 In the 5x5 image, the coordinates corresponding to the direction from top to bottom and from left to right are (5, 5), (5, 6), (5, 7), (5, 8), (5, 9); (6, 5), (6, 6), (6, 7), (6, 8), (6, 9); (7, 5), (7, 6), (7, 7), (7, 8), (7, 9); (8, 5), (8, 6), (8, 7), (8, 8), (8, 9); (9, 5), (9, 6), (9, 7), (9, 8), (9, 9).P (6,7) represents the purple pixel value of the pixel point corresponding to the coordinate (6, 7), i.e. the first to-be-interpolated color pixel value; G (6,7) represents the green pixel value of the pixel point corresponding to the coordinate (6, 7), i.e. the first primary color pixel value; P (6,6) represents the purple pixel value of the pixel point corresponding to the coordinate (6, 6), i.e. the second primary color pixel value; G (6,6) represents the green pixel value of the pixel point corresponding to the coordinate (6, 6), i.e. the interpolated first color pixel value.

[0197] In the horizontal direction of the green pixel point, one purple pixel point in the 1st-order neighborhood is selected, the purple pixel value (primary P) of the purple pixel point is the second primary color pixel value, the interpolated green pixel value (interpolated G) of the purple pixel point is the interpolated first color pixel value, and the first to-be-interpolated color pixel value is calculated by using the following formula:

[0198] The same method is used in the vertical direction of the green pixel point, and one orange pixel point in the adjacent first-order neighborhood is selected. The first color pixel value is calculated by using the second original color pixel value of the orange pixel point and the first color pixel value that has been interpolated. Details are not repeated here.

[0199] The same method is used in the horizontal direction of the green pixel point, and one red pixel point in the adjacent first-order neighborhood is selected. The first color pixel value is calculated by using the second original color pixel value of the red pixel point and the first color pixel value that has been interpolated. Details are not repeated here.

[0200] It should be noted that the "neighborhood" refers to a plurality of pixel points around the target pixel point with the target pixel point as the center. One pixel point in the four directions of the target pixel point, i.e., up, down, left, and right, constitutes a four-direction neighborhood, which is a first-order neighborhood. Two pixel points in the four directions of the target pixel point, i.e., up, down, left, and right, constitute a four-direction neighborhood, which is a second-order neighborhood. Pixel points in the left-up, right-up, left-down, and right-down directions of the target pixel point constitute an eight-direction neighborhood, which is a second-order neighborhood.

[0201] In this way, all the first color pixel values in the checkerboard image of the first color channel can be calculated.

[0202] Step S266: determining the second color pixel value to be interpolated according to the second original color pixel value and the first color pixel value that has been interpolated.

[0203] Please refer to Figure 21 In one embodiment, when step S266 is performed to determine the second color pixel value to be interpolated according to the second original color pixel value and the first color pixel value that has been interpolated, the following steps are further included.

[0204] Step S2661: in the diagonal direction of the pixel point corresponding to the second color pixel value to be interpolated, the second color pixel value to be interpolated is calculated according to the second original color pixel values of the two second-order neighborhood pixel points in the diagonal direction and the first color pixel value that has been interpolated.

[0205] In one embodiment, with the pixel point corresponding to any second color pixel value to be interpolated in the checkerboard image of the second color channel as the center, the second original color pixel values of the two second-order neighborhood pixel points and the first color pixel value that has been interpolated are obtained. The second color pixel value to be interpolated is calculated according to the second original color pixel values of the two second-order neighborhood pixel points in the diagonal direction and the first color pixel value that has been interpolated.

[0206] It should be noted that the method adopted in this step is similar to the method of step S2653, the pixel point of the main diagonal line vertex in the interpolation unit is the pixel point of the two adjacent 2-order neighborhoods, and the pixel point of the auxiliary diagonal line vertex is also the pixel point of the two adjacent 2-order neighborhoods. However, in this step, the second native color pixel value of the pixel points of the main diagonal line vertex and the auxiliary diagonal line vertex is not the same, so the second native color pixel value of the pixel point of the main diagonal line vertex and the first color pixel value which has been interpolated can be directly used to calculate the second to-be-interpolated color pixel value; and the second native color pixel value of the pixel point of the auxiliary diagonal line vertex and the first color pixel value which has been interpolated can be directly used to calculate the second to-be-interpolated color pixel value.

[0207] Also refer to Figure 19 In the 5*5 image of FIG. 5, the pixel point corresponding to the coordinate (7, 7) is taken as the pixel point corresponding to the second to-be-interpolated color pixel value, and B is taken as the second to-be-interpolated color pixel value. The 3*3 interpolation unit is constructed with the pixel point as the center, the second to-be-interpolated color pixel value is calculated according to the second native color pixel value of the pixel point of the main diagonal line vertex of the interpolation unit and the first color pixel value which has been interpolated, and the second to-be-interpolated color pixel value is calculated by using the following formula:

[0208]

[0209] Wherein, P (7,7) represents the purple pixel value of the pixel point corresponding to the coordinate (7, 7), that is, the second to-be-interpolated color pixel value; G (7,7) represents the green pixel value of the pixel point corresponding to the coordinate (7, 7), that is, the first color pixel value which has been interpolated; P (6,6) represents the purple pixel value of the pixel point corresponding to the coordinate (6, 6), that is, the second native color pixel value; G (6,6) represents the green pixel value of the pixel point corresponding to the coordinate (6, 6), that is, the first color pixel value which has been interpolated.

[0210] Also refer to Figure 19 In the 5*5 image of FIG. 5, the pixel point corresponding to the coordinate (7, 7) is taken as the pixel point corresponding to the second to-be-interpolated color pixel value, and B is taken as the second to-be-interpolated color pixel value. The 3*3 interpolation unit is constructed with the pixel point as the center, the second to-be-interpolated color pixel value is calculated according to the second native color pixel value of the pixel point of the main diagonal line vertex of the interpolation unit and the first color pixel value which has been interpolated, and the second to-be-interpolated color pixel value is calculated by using the following formula:

[0211] Step S2662: In the vertical direction and / or the horizontal direction of the pixel point corresponding to the second to-be-interpolated color pixel value, the second to-be-interpolated color pixel value is calculated according to the second native color pixel value and the first color pixel value which has been interpolated of the pixel points of the two 2-order neighborhoods in the vertical direction and / or the horizontal direction.

[0212] In one embodiment, in the vertical direction of the pixel point corresponding to the second color pixel value to be interpolated, the second native color pixel values and the first color pixel values interpolated of the pixel points of the two adjacent 2-order neighborhoods are obtained. The gradient change in the vertical direction is calculated according to the second native color pixel values and the first color pixel values interpolated of the pixel points of the 2-order neighborhoods in the vertical direction, and the calculation is specifically performed by using the following formula.

[0213] GradY = |A i-2,j -A i+2,j | + |2 x G i,j -(G i-2,j + G i+2,j )|

[0214] wherein GradY represents the gradient in the vertical direction, A i-2,j represents the second native color pixel value of the pixel point of one of the 2-order neighborhoods in the vertical direction, A i+2,j represents the second native color pixel value of the pixel point of the other 2-order neighborhood in the vertical direction, G i,j represents the first color pixel value interpolated in the pixel point corresponding to the second color pixel value to be interpolated, G i-2,j represents the first color pixel value interpolated of the pixel point of one of the 2-order neighborhoods in the vertical direction, G i+2,j represents the first color pixel value interpolated of the pixel point of the other 2-order neighborhood in the vertical direction.

[0215] In one embodiment, in the horizontal direction of the pixel point corresponding to the second color pixel value to be interpolated, the second native color pixel values and the first color pixel values interpolated of the pixel points of the two adjacent 2-order neighborhoods are obtained. The gradient change in the horizontal direction is calculated according to the second native color pixel values and the first color pixel values interpolated of the pixel points of the 2-order neighborhoods in the horizontal direction, and the calculation is specifically performed by using the following formula.

[0216] GradX = |A i,j-2 -A i,j+2 | + |2 x G i,j -(G i,j-2 + G i,j+2 )|

[0217] wherein GradX represents the gradient in the horizontal direction, A i,j-2 represents the second native color pixel value of the pixel point of one of the 2-order neighborhoods in the horizontal direction, A i,j+2 represents the second native color pixel value of the pixel point of the other 2-order neighborhood in the horizontal direction, G i,j represents the first color pixel value interpolated in the pixel point corresponding to the second color pixel value to be interpolated, G i,j-2G represents the interpolated first color pixel value of the pixel point in one of the 2nd order neighborhoods in the horizontal direction i,j+2 G represents the interpolated first color pixel value of the pixel point in the other of the 2nd order neighborhoods in the horizontal direction. If the gradient change in the vertical direction is less than the gradient change in the horizontal direction, the second native color pixel value and the interpolated first color pixel value of the pixel point in the 2nd order neighborhood in the vertical direction determine the second color pixel value to be interpolated, which is calculated by using the following formula.

[0218]

[0219] wherein A i,j G represents the second color pixel value to be interpolated.

[0220] If the gradient change in the vertical direction is greater than the gradient change in the horizontal direction, the second native color pixel value and the interpolated first color pixel value of the pixel point in the 2nd order neighborhood in the horizontal direction determine the second color pixel value to be interpolated, which is calculated by using the following formula.

[0221]

[0222] wherein A i,j G represents the second color pixel value to be interpolated.

[0223] If the gradient change in the vertical direction is equal to the gradient change in the horizontal direction, the second native color pixel value and the interpolated first color pixel value of the pixel point in the 2nd order neighborhood in the vertical direction, and the second native color pixel value and the interpolated first color pixel value of the pixel point in the 2nd order neighborhood in the horizontal direction determine the second color pixel value to be interpolated, which is calculated by using the following formula.

[0224]

[0225] wherein A i,j G represents the second color pixel value to be interpolated.

[0226] Specifically, also referring to Figure 20 the orange pixel value in the red pixel point corresponding to the coordinate (7, 7) as the second color pixel value to be interpolated, and the green pixel value in the red pixel point corresponding to the coordinate (7, 7) as the interpolated first color pixel value corresponding to the second color pixel value to be interpolated.

[0227] The orange pixel value of the pixel point corresponding to the coordinate (5, 7) is the second original color pixel value of the pixel point of one of the 2-order neighborhoods in the vertical direction, and the green pixel value of the pixel point corresponding to the coordinate (5, 7) is the interpolated first color pixel value of the pixel point of one of the 2-order neighborhoods in the vertical direction. The orange pixel value of the pixel point corresponding to the coordinate (9, 7) is the second original color pixel value of the pixel point of another 2-order neighborhood in the vertical direction, and the green pixel value of the pixel point corresponding to the coordinate (9, 7) is the interpolated first color pixel value of the pixel point of another 2-order neighborhood in the vertical direction. Then the gradient change in the vertical direction is:

[0228] GradY = |A 5,7 -A 9,7 | + |2 x G 7,7 -(G 5,7 + G 9,7 )|

[0229] wherein GradY represents the gradient in the vertical direction, A 5,7 represents the orange pixel value of the pixel point corresponding to the coordinate (5, 7), A 9,7 represents the orange pixel value of the pixel point corresponding to the coordinate (9, 7), G 7,7 represents the green pixel value in the red pixel point corresponding to the coordinate (7, 7), G 5,7 represents the green pixel value of the pixel point corresponding to the coordinate (5, 7), and G 9,7 represents the green pixel value of the pixel point corresponding to the coordinate (9, 7).

[0230] The orange pixel value of the pixel point corresponding to the coordinate (7, 5) is the second original color pixel value of the pixel point of one of the 2-order neighborhoods in the horizontal direction, and the green pixel value of the pixel point corresponding to the coordinate (7, 5) is the interpolated first color pixel value of the pixel point of one of the 2-order neighborhoods in the horizontal direction. The orange pixel value of the pixel point corresponding to the coordinate (9, 5) is the second original color pixel value of the pixel point of another 2-order neighborhood in the horizontal direction, and the green pixel value of the pixel point corresponding to the coordinate (9, 5) is the interpolated first color pixel value of the pixel point of another 2-order neighborhood in the horizontal direction. Then the gradient change in the horizontal direction is:

[0231] GradX = |A 7,5 -A 7,9 | + |2 x G 7,7 -(G 7,5 + G 7,9 )|

[0232] wherein GradX represents the gradient in the horizontal direction, A 7,5 represents the orange pixel value of the pixel point corresponding to the coordinate (7, 5), A 7,9G represents the green pixel value of the pixel point corresponding to the coordinate (7, 9). 7,7 G represents the green pixel value of the pixel point corresponding to the coordinate (7, 9). 7,5 G represents the green pixel value of the pixel point corresponding to the coordinate (7, 9). 7,9 G represents the green pixel value of the pixel point corresponding to the coordinate (7, 9).

[0233] If the gradient change in the vertical direction is less than the gradient change in the horizontal direction, then:

[0234]

[0235] A represents the orange pixel value of the pixel point corresponding to the coordinate (7, 9). 7,7 G represents the green pixel value of the pixel point corresponding to the coordinate (7, 9).

[0236] If the gradient change in the vertical direction is greater than the gradient change in the horizontal direction, then:

[0237]

[0238] A represents the orange pixel value of the pixel point corresponding to the coordinate (7, 9). 7,7 G represents the green pixel value of the pixel point corresponding to the coordinate (7, 9).

[0239] If the gradient change in the vertical direction is equal to the gradient change in the horizontal direction, then:

[0240]

[0241] A represents the orange pixel value of the pixel point corresponding to the coordinate (7, 9). 7,7 G represents the green pixel value of the pixel point corresponding to the coordinate (7, 9).

[0242] In one embodiment, if the imaging system 12 is applied independently as a separate system, the above description has set forth the entire workflow of the complete imaging system 12.

[0243] In one embodiment, if the imaging system 12 is applied as a core component in the endoscope system 1, the guide part 13 in the endoscope system 1 at least has a tip part configured to be inserted into the detected area, for entering the human body or the internal part of the equipment, flexible stretching and transmission of optical signal and image signal. The control system 14 is used for the doctor or operator to control the operation and function of the endoscope. The image transmission and display system 15 transmits the display image to the external display device for the doctor to observe. The power supply system 16 provides the power required for the operation of the endoscope system 1.

[0244] Those skilled in the art can understand that all or part of the functions of various methods in the above embodiments can be realized by hardware or by a computer program. When all or part of the functions in the above embodiments are realized by a computer program, the program can be stored in a computer readable storage medium, which can include a read-only memory, a random access memory, a magnetic disk, an optical disk, a hard disk, and the like. The above functions are realized by executing the program by a computer. For example, the program is stored in a memory of a device, and the above functions are realized by executing the program in the memory by a processor. In addition, when all or part of the functions in the above embodiments are realized by a computer program, the program can also be stored in a storage medium such as a server, another computer, a disk, an optical disk, a flash disk, or a mobile hard disk, and is saved in a memory of a local device by downloading or copying, or the system of the local device is updated, and the above functions are realized by executing the program in the memory by a processor.

[0245] The above application of specific examples to the present application is described, which is only used to help understand the present application and does not limit the present application. According to the idea of the present application, those skilled in the art can make several simple deductions, deformations, or substitutions.

Claims

1. An endoscope system, characterized in that: include: A white light source unit, used to provide continuous visible spectrum illumination light to the inspected area; a guide portion having at least a tip portion configured to probe into the detected area; An imaging component, comprising a shooting lens, a hyperspectral image sensor, and a filter array; The shooting lens is arranged at the front end of the guide portion, and is used to collect the reflected light or scattered light after the illumination light irradiates the inspection area, so as to emit imaging light; The filter array and the hyperspectral image sensor are sequentially arranged on the optical path of the imaging light; the filter array is used to decompose the imaging light into monochromatic light or narrow-band light of different wavelengths, and the hyperspectral image sensor is used to convert the monochromatic light or narrow-band light of different wavelengths decomposed by the filter array from optical signals to electrical signals to obtain a hyperspectral image.

2. The endoscope system according to claim 1, wherein: The filter array includes a first filter unit that transmits a preset wavelength band and a second filter unit that transmits wavelength bands other than the preset wavelength band. The filters in the first filter unit and the filters in the second filter unit are alternately spaced in the horizontal and vertical directions.

3. The endoscope system according to claim 2, wherein: The second filter unit includes a first sub-filter unit and a second sub-filter unit, wherein the first sub-filter unit includes filters of at least two colors, and the second sub-filter unit includes filters of at least two colors, wherein the filters in the first sub-filter unit and the second sub-filter unit have different colors; The spectral response difference between the filters of each color in the first sub-filter unit is less than a first set value, the spectral response difference between the filters of each color in the second sub-filter unit is less than a second set value, and the spectral response difference between the filters of any color in the first sub-filter unit and the filters of any color in the second sub-filter unit is greater than a third set value.

4. The endoscope system according to claim 3, wherein: The filters of different colors in the first sub-filter unit are alternately spaced in the horizontal direction of the second filter unit, and the filters of different colors in the second sub-filter unit are alternately spaced in the horizontal direction of the second filter unit.

5. The endoscope system according to claim 4, wherein: The filters of different colors in the first sub-filter unit are alternately spaced in the vertical direction of the second filter unit, and the filters of different colors in the second sub-filter unit are alternately spaced in the vertical direction of the second filter unit.

6. The endoscope system according to claim 3 or 4, wherein: The first sub-filter units and the second sub-filter units are alternately distributed.

7. The endoscope system according to claim 2, wherein: The number of filters in the first filter unit is greater than or equal to half of the total number of all filters in the filter array.

8. The endoscope system according to claim 2, wherein: The endoscope system also includes a processor; The hyperspectral image includes a checkerboard image of a first color channel corresponding to a filter color in the first filter unit determined by the hyperspectral image sensor through the first filter unit, and a checkerboard image of a second color channel corresponding to a filter color in the second filter unit determined by the hyperspectral image sensor through the second filter unit; The processor obtains a checkerboard image of the first color channel, interpolates pixel values ​​of the first color corresponding to the first filter unit at blank positions in the checkerboard image of the first color channel to generate a complete image of the first color channel; and interpolates pixel values ​​of each pixel point in the hyperspectral image in different color channels based on the complete image of the first color channel and the checkerboard image of the second color channel to generate a display image of the detected area for output; The blank position is a pixel point position where the pixel value of the first color does not exist in the checkerboard image of the second color channel in the hyperspectral image.

9. The endoscope system according to claim 8, wherein: The processor interpolates pixel values ​​of the first color corresponding to the first filter unit at blank positions in the checkerboard image of the first color channel to generate a complete image of the first color channel, including: Acquire a first training image at full resolution, and generate a second training image corresponding to the checkerboard image of the first color channel based on the first training image; The convolutional neural network model is trained according to the first training image and the second training image, and the checkerboard image of the first color channel is input into the trained convolutional neural network model to generate a complete image of the first color channel.

10. The endoscope system according to claim 9, wherein: The processor interpolates pixel values ​​of each pixel in the hyperspectral image in different color channels according to the complete image of the first color channel and the checkerboard image of the second color channel to generate a display image for output of the detected area, including: Each pixel point in the checkerboard image of the second color channel includes a second native color pixel value corresponding to the filter color in the second filter unit, a first color pixel value interpolated at the blank position, and a second color pixel value to be interpolated; The second to-be-interpolated color pixel value is determined according to the second native color pixel value and the interpolated first color pixel value, so as to generate a display image for output in the detected area.

11. The endoscope system according to claim 10, wherein: The processor determines the second to-be-interpolated color pixel value according to the second native color pixel value and the interpolated first color pixel value, including: In the hyperspectral image, constructing an n×n interpolation unit with any pixel point in the checkerboard image of the second color channel as a center point, where n is an odd number; Determine the second color pixel value to be interpolated by using the second native color pixel value of the vertex pixel point of the main diagonal line in the interpolation unit and the interpolated first color pixel value; or, Determine the second color pixel value to be interpolated by using the second native color pixel value of the vertex pixel point of the sub-diagonal line in the interpolation unit and the interpolated first color pixel value; or, The second color pixel value to be interpolated is determined by using the second native color pixel value and the interpolated first color pixel value of the vertex pixel point of the main diagonal line of the interpolation unit, as well as the second native color pixel value and the interpolated first color pixel value of the vertex pixel point of the sub-diagonal line of the interpolation unit.

12. The endoscope system according to claim 11, wherein: When the second native color pixel values ​​of the vertex pixels of the main diagonal or the sub-diagonal in the interpolation unit are the same, Calculating a gradient change of the main diagonal line according to the second native color pixel value of the vertex pixel point of the main diagonal line in the interpolation unit and the interpolated first color pixel value; Calculating a gradient change of the sub-diagonal line according to the second native color pixel value of the vertex pixel point of the sub-diagonal line in the interpolation unit and the interpolated first color pixel value; When the gradient change of the main diagonal line is smaller than the gradient change of the sub-diagonal line, determining the second color pixel value to be interpolated by using the second native color pixel value of the vertex pixel point of the main diagonal line of the interpolation unit and the interpolated first color pixel value; When the gradient change of the main diagonal is greater than the gradient change of the sub-diagonal, determining the second color pixel value to be interpolated by using the second native color pixel value of the vertex pixel point of the sub-diagonal of the interpolation unit and the interpolated first color pixel value; When the gradient change of the main diagonal is equal to the gradient change of the secondary diagonal, the second color pixel value to be interpolated is determined using the second native color pixel value and the interpolated first color pixel value of the vertex pixel point of the main diagonal of the interpolation unit, as well as the second native color pixel value and the interpolated first color pixel value of the vertex pixel point of the secondary diagonal of the interpolation unit.

13. The endoscope system according to claim 12, wherein: When the second native color pixel values ​​of the vertex pixels of the main diagonal or sub-diagonal in the interpolation unit are different, Updating the center point with a pixel point of any vertex in the main diagonal or the secondary diagonal to update the interpolation unit, and updating the second native color pixel value of any vertex in the main diagonal or the secondary diagonal to the second to-be-interpolated color pixel value in the updated interpolation unit; The updated interpolation unit is used to determine a second color pixel value to be interpolated, so as to determine a second native color pixel value of any vertex in the main diagonal or the sub-diagonal.

14. The endoscope system according to claim 10, wherein: The processor determines the second to-be-interpolated color pixel value according to the second native color pixel value and the interpolated first color pixel value, including: Taking any pixel point corresponding to the second color pixel value to be interpolated in the checkerboard image of the second color channel as the center; Obtaining the second native color pixel values ​​and the interpolated first color pixel values ​​of two adjacent second-order neighboring pixels in a diagonal direction of the pixel corresponding to the second color pixel value to be interpolated, and calculating the second color pixel value to be interpolated based on the second native color pixel values ​​and the interpolated first color pixel values ​​of the two second-order neighboring pixels in the diagonal direction; and / or, Obtaining, in a vertical direction of the pixel corresponding to the second to-be-interpolated color pixel value, the second native color pixel values ​​and the interpolated first color pixel values ​​of two adjacent second-order neighboring pixels, and calculating a vertical gradient change based on the second native color pixel values ​​and the interpolated first color pixel values ​​of the two second-order neighboring pixels in the vertical direction; Obtaining, in a horizontal direction of the pixel corresponding to the second to-be-interpolated color pixel value, the second native color pixel values ​​and the interpolated first color pixel values ​​of two adjacent second-order neighboring pixels, and calculating a horizontal gradient change based on the second native color pixel values ​​and the interpolated first color pixel values ​​of the two second-order neighboring pixels in the horizontal direction; When the gradient change in the vertical direction is smaller than the gradient change in the horizontal direction, determining the second color pixel value to be interpolated by using the second native color pixel value of the pixel point in the second-order neighborhood in the vertical direction and the interpolated first color pixel value; When the gradient change in the vertical direction is greater than the gradient change in the horizontal direction, determining the second color pixel value to be interpolated by using the second native color pixel value of the pixel point in the second-order neighborhood in the horizontal direction and the interpolated first color pixel value; When the gradient change in the vertical direction is equal to the gradient change in the horizontal direction, the second native color pixel value and the interpolated first color pixel value of the pixel point in the second-order neighborhood in the vertical direction, as well as the second native color pixel value and the interpolated first color pixel value of the pixel point in the second-order neighborhood in the horizontal direction, are used to determine the second color pixel value to be interpolated.

15. An imaging system, characterized in that: include: A white light source unit, used to provide continuous visible spectrum illumination light to the inspected area; An imaging component, comprising a shooting lens, a hyperspectral image sensor, and a filter array; The shooting lens is used to collect the reflected light or scattered light after the illumination light illuminates the inspection area to emit imaging light; The filter array and the hyperspectral image sensor are sequentially arranged on the optical path of the imaging light; the filter array is used to decompose the imaging light into monochromatic light or narrow-band light of different wavelengths, and the hyperspectral image sensor is used to convert the monochromatic light or narrow-band light of different wavelengths decomposed by the filter array from optical signals to electrical signals to obtain a hyperspectral image.

16. An imaging method, characterized in that: include: Acquire a checkerboard image of a first color channel corresponding to the filter color in the first filter unit and a checkerboard image of a second color channel corresponding to the filter color in the second filter unit in the hyperspectral image to be interpolated; The first filter unit is a first filter unit that transmits a preset wavelength band, and the second filter unit is a second filter unit that transmits wavelength bands other than the preset wavelength band. The first filter unit and the second filter unit are alternately spaced in the horizontal direction and the vertical direction. interpolating pixel values ​​of the first color corresponding to the first filter unit at blank positions in the checkerboard image of the first color channel to generate a complete image of the first color channel; interpolating pixel values ​​of each pixel point in the hyperspectral image in different color channels according to the complete image of the first color channel and the checkerboard image of the second color channel to generate a display image of the detected area for output; The blank position is a pixel point position where the pixel value of the first color does not exist in the checkerboard image of the second color channel in the hyperspectral image.

17. The imaging method according to claim 16, wherein: Interpolating pixel values ​​of the first color corresponding to the first filter unit at blank positions in the checkerboard image of the first color channel to generate a complete image of the first color channel includes: Acquire a first training image at full resolution, and generate a second training image corresponding to the checkerboard image of the first color channel based on the first training image; The convolutional neural network model is trained according to the first training image and the second training image, and the checkerboard image of the first color channel is input into the trained convolutional neural network model to generate a complete image of the first color channel.

18. The imaging method according to claim 17, wherein: The interpolating pixel values ​​of each pixel point in the hyperspectral image in different color channels according to the complete image of the first color channel and the checkerboard image of the second color channel to generate a display image for output of the detected area includes: Each pixel point in the checkerboard image of the second color channel includes a second native color pixel value corresponding to the filter color in the second filter unit, a first color pixel value interpolated at the blank position, and a second color pixel value to be interpolated; The second to-be-interpolated color pixel value is determined according to the second native color pixel value and the interpolated first color pixel value, so as to generate a display image for output in the detected area.

19. The imaging method according to claim 18, wherein The determining the second to-be-interpolated color pixel value according to the second native color pixel value and the interpolated first color pixel value includes: In the hyperspectral image, constructing an n×n interpolation unit with any pixel point in the checkerboard image of the second color channel as a center point, where n is an odd number; Determine the second color pixel value to be interpolated by using the second native color pixel value of the vertex pixel point of the main diagonal line in the interpolation unit and the interpolated first color pixel value; or, Determine the second color pixel value to be interpolated by using the second native color pixel value of the vertex pixel point of the sub-diagonal line in the interpolation unit and the interpolated first color pixel value; or, The second color pixel value to be interpolated is determined by using the second native color pixel value and the interpolated first color pixel value of the vertex pixel point of the main diagonal line of the interpolation unit, as well as the second native color pixel value and the interpolated first color pixel value of the vertex pixel point of the sub-diagonal line of the interpolation unit.

20. The imaging method according to claim 19, wherein When the second native color pixel values ​​of the vertex pixels of the main diagonal or the sub-diagonal in the interpolation unit are the same, Calculating a gradient change of the main diagonal line according to the second native color pixel value of the vertex pixel point of the main diagonal line in the interpolation unit and the interpolated first color pixel value; Calculating a gradient change of the sub-diagonal line according to the second native color pixel value of the vertex pixel point of the sub-diagonal line in the interpolation unit and the interpolated first color pixel value; When the gradient change of the main diagonal line is smaller than the gradient change of the sub-diagonal line, determining the second color pixel value to be interpolated by using the second native color pixel value of the vertex pixel point of the main diagonal line of the interpolation unit and the interpolated first color pixel value; When the gradient change of the main diagonal is greater than the gradient change of the sub-diagonal, determining the second color pixel value to be interpolated by using the second native color pixel value of the vertex pixel point of the sub-diagonal of the interpolation unit and the interpolated first color pixel value; When the gradient change of the main diagonal is equal to the gradient change of the secondary diagonal, the second color pixel value to be interpolated is determined using the second native color pixel value and the interpolated first color pixel value of the vertex pixel point of the main diagonal of the interpolation unit, as well as the second native color pixel value and the interpolated first color pixel value of the vertex pixel point of the secondary diagonal of the interpolation unit.

21. The imaging method according to claim 19, wherein When the second native color pixel values ​​of the vertex pixels of the main diagonal or sub-diagonal in the interpolation unit are different, Updating the center point with a pixel point of any vertex in the main diagonal or the secondary diagonal to update the interpolation unit, and updating the second native color pixel value of any vertex in the main diagonal or the secondary diagonal to the second to-be-interpolated color pixel value in the updated interpolation unit; The updated interpolation unit is used to determine a second color pixel value to be interpolated, so as to determine a second native color pixel value of any vertex in the main diagonal or the sub-diagonal.

22. The imaging method according to claim 18, wherein The determining the second to-be-interpolated color pixel value according to the second native color pixel value and the interpolated first color pixel value includes: Taking any pixel point corresponding to the second color pixel value to be interpolated in the checkerboard image of the second color channel as the center; Obtaining the second native color pixel values ​​and the interpolated first color pixel values ​​of two adjacent second-order neighboring pixels in a diagonal direction of the pixel corresponding to the second color pixel value to be interpolated, and calculating the second color pixel value to be interpolated based on the second native color pixel values ​​and the interpolated first color pixel values ​​of the two second-order neighboring pixels in the diagonal direction; and / or, Obtaining, in a vertical direction of the pixel corresponding to the second to-be-interpolated color pixel value, the second native color pixel values ​​and the interpolated first color pixel values ​​of two adjacent second-order neighboring pixels, and calculating a vertical gradient change based on the second native color pixel values ​​and the interpolated first color pixel values ​​of the two second-order neighboring pixels in the vertical direction; Obtaining, in a horizontal direction of the pixel corresponding to the second to-be-interpolated color pixel value, the second native color pixel values ​​and the interpolated first color pixel values ​​of two adjacent second-order neighboring pixels, and calculating a horizontal gradient change based on the second native color pixel values ​​and the interpolated first color pixel values ​​of the two second-order neighboring pixels in the horizontal direction; When the gradient change in the vertical direction is smaller than the gradient change in the horizontal direction, determining the second color pixel value to be interpolated by using the second native color pixel value of the pixel point in the second-order neighborhood in the vertical direction and the interpolated first color pixel value; When the gradient change in the vertical direction is greater than the gradient change in the horizontal direction, determining the second color pixel value to be interpolated by using the second native color pixel value of the pixel point in the second-order neighborhood in the horizontal direction and the interpolated first color pixel value; When the gradient change in the vertical direction is equal to the gradient change in the horizontal direction, the second native color pixel value and the interpolated first color pixel value of the pixel point in the second-order neighborhood in the vertical direction, as well as the second native color pixel value and the interpolated first color pixel value of the pixel point in the second-order neighborhood in the horizontal direction, are used to determine the second color pixel value to be interpolated.

23. A computer-readable storage medium, characterized in that A computer program is stored on the medium, and the computer program can be executed by a processor to implement the method according to any one of claims 16 to 22.

Images