Image processing method, endoscope system, endoscope camera device and storage medium

Abstract

An image processing method, comprising: acquiring a visible-light scene image and a near-infrared light scene image, the near-infrared light scene image being obtained by collecting near-infrared light in a preset waveband range; and processing the visible-light scene image and the near-infrared light scene image to obtain a processed visible-light scene image and a processed near-infrared light scene image; and fusing the processed visible-light scene image and the processed near-infrared light scene image to obtain a fused image.

Description

Image processing method, endoscope system, endoscope camera device and storage medium

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on August 31, 2023, with application number 2023111219897, and invention name “Image processing method, endoscope system, endoscope camera equipment and storage medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the technical field of medical endoscopes, and in particular to an image processing method, an endoscope system, an endoscope camera device, and a storage medium. Background Art

[0003] With the continuous advancement of medical technology, endoscopes have become an essential medical aid in hospitals for examining, diagnosing, and treating patients. During the examination and diagnosis of patients, endoscopes allow for clear visualization of diseased tissue, playing a crucial supporting role in the doctor's diagnosis. Furthermore, during surgical procedures, the use of real-time endoscope images enables doctors to treat patients more efficiently.

[0004] Although some 4K endoscopes or fluorescent endoscopes already exist in the relevant technology, when these endoscopes encounter some special situations, such as tissue bleeding blocking the lens field of view, water mist on the lens blocking the field of view, insufficient brightness making it difficult to see dark details or unclear blood vessel patterns, their performance is unsatisfactory, which interferes with doctors' surgery, reduces the efficiency of the surgery, and increases the risk of surgery.

[0005] The above content is only used to assist in understanding the technical solution of this application and does not constitute an admission that the above content is prior art.

[0006] Summary of the Invention

[0007] According to various embodiments of the present application, an image processing method, an endoscope system, an endoscope camera device, and a storage medium are provided.

[0008] The present invention provides an image processing method for an endoscope camera system, comprising:

[0009] Acquire a visible light scene image and a near-infrared light scene image, wherein the near-infrared light scene image is obtained by collecting near-infrared light within a preset wavelength band;

[0010] Processing the visible light scene image and the near-infrared light scene image to obtain processed visible light scene image and near-infrared light scene image;

[0011] The processed visible light scene image and near-infrared light scene image are fused to obtain a fused image.

[0012] In one embodiment, the endoscope camera system includes a camera, the camera includes an image sensor and a beam splitter prism, and the step of acquiring the visible light scene image and the near-infrared light scene image includes:

[0013] Reflected light from an object in a target scene after being irradiated by a visible light source and a near-infrared light source is received through a beam splitter prism, and the optical paths of the reflected light from the object irradiated by the visible light and the reflected light from the object irradiated by the near-infrared light are changed through the beam splitter prism to form a visible light path and a near-infrared light path;

[0014] The visible light of the visible light path and the near-infrared light of the near-infrared light path are received by an image sensor to generate a visible light scene image and a near-infrared light scene image.

[0015] In one embodiment, the camera further includes a filter, and the step of receiving the visible light of the visible light path and the near-infrared light of the near-infrared light path through the image sensor to generate the visible light scene image and the near-infrared light scene image includes:

[0016] Filtering the near-infrared light in the near-infrared light path through a filter to obtain reflected light of an object irradiated by the near-infrared light within a preset wavelength range;

[0017] The image sensor receives the visible light in the visible light path and the near-infrared light filtered by the filter to generate a visible light scene image and a near-infrared light scene image.

[0018] In one embodiment, the preset wavelength range is between about 820 nm and about 1100 nm.

[0019] In one embodiment, the endoscope camera system further includes a camera host, the camera host includes a first FPGA chip, and the step of processing the visible light scene image and the near-infrared light scene image to obtain the processed visible light scene image and the near-infrared light scene image includes:

[0020] The visible light scene image and the near-infrared light scene image are decoded and processed by the first FPGA chip to obtain processed visible light scene image and near-infrared light scene image; the processing includes at least one of wide dynamic processing and detail enhancement processing.

[0021] In one embodiment, the processing of the visible light scene image further includes at least one of the following: defect correction, photometry, noise filtering, white balance, color restoration, color correction, saturation correction, gamma correction, color space conversion, contrast correction, and sharpness correction;

[0022] The processing of the near-infrared light scene image further includes at least one of the following: defect correction and noise filtering.

[0023] In one embodiment, the step of processing the visible light scene image and the near-infrared light scene image to obtain processed visible light scene image and near-infrared light scene image further includes:

[0024] The visible light scene image and the near-infrared light scene image are corrected and processed by the first FPGA chip to obtain processed visible light scene image and near-infrared light scene image.

[0025] In one embodiment, the camera host further includes: a second FPGA chip, and the method further includes:

[0026] receiving, through the second FPGA chip, a target image transmitted by the first FPGA chip; the target image comprising at least one of a processed visible light scene image, a near-infrared light scene image, and the fused image;

[0027] The target image is transmitted to a peripheral device through the second FPGA chip.

[0028] In one embodiment, the method further comprises:

[0029] Receiving, through the second FPGA chip, a configuration command sent by the peripheral device;

[0030] The configuration command is sent to the first FPGA chip through the second FPGA chip.

[0031] In one embodiment, the fusion is achieved by any one of the following methods: weighted average method, maximum method, minimum method, wavelet transform fusion, principal component analysis fusion, feature extraction and fusion, machine learning fusion, pyramid fusion, and fluorescent labeling fusion.

[0032] The embodiment of the present application further provides an endoscope system, the endoscope system including an endoscope camera system, the endoscope camera system including a camera head and a camera host;

[0033] The camera is used to obtain visible light scene images and near-infrared light scene images, and the near-infrared light scene images are obtained by collecting near-infrared light in a preset band range;

[0034] The camera host is used to process the visible light scene image and the near-infrared light scene image to obtain processed visible light scene image and near-infrared light scene image; and fuse the processed visible light scene image and near-infrared light scene image to obtain a fused image.

[0035] In one embodiment, the endoscope system further includes a light source and a lens, the light source includes a visible light source and a near-infrared light source, and the camera includes a beam splitter prism visible light image sensor and a near-infrared light image sensor arranged along the light path transmission direction;

[0036] The visible light source and the near-infrared light source are used to illuminate objects in the target scene through the lens, forming reflected light from the objects illuminated by the visible light and reflected light from the objects illuminated by the near-infrared light respectively;

[0037] The beam splitter prism is used to receive the reflected light after the visible light source and the near-infrared light source illuminate the object in the target scene through the lens, and change the optical path of the reflected light of the visible light irradiating the object and the reflected light of the near-infrared light irradiating the object to form a visible light path and a near-infrared light path;

[0038] The visible light image sensor is configured to receive visible light from the visible light path and generate a visible light scene image;

[0039] The near-infrared light image sensor is used to receive the near-infrared light in the near-infrared light path and generate a near-infrared light scene image.

[0040] In one embodiment, the camera further includes a filter, and the filter is disposed between the beam splitter prism and the near-infrared image sensor;

[0041] The optical filter is used to filter the near-infrared light in the near-infrared light path to obtain reflected light of the object irradiated by the near-infrared light within a preset wavelength range;

[0042] The near-infrared light image sensor is further configured to receive the near-infrared light filtered by the filter and generate a near-infrared light scene image.

[0043] In one embodiment, the preset wavelength range is between about 820 nm and about 1100 nm.

[0044] In one embodiment, the camera host includes a dual FPGA component module, and the dual FPGA component module includes a first FPGA chip and a second FPGA chip;

[0045] The first FPGA chip is used to decode and process the visible light scene image and the near-infrared light scene image, and to perform correction processing on the visible light scene image and the near-infrared light scene image to obtain processed visible light scene image and near-infrared light scene image; the processing includes at least one of wide dynamic processing and detail enhancement processing.

[0046] In one embodiment, the first FPGA chip is further configured to fuse the processed visible light scene image and near-infrared light scene image to obtain a fused image.

[0047] In one embodiment, the first FPGA chip is further configured to transmit a target image to the second FPGA chip via a high-speed transmission interface; the target image includes at least one of the processed visible light scene image, the near-infrared light scene image, and the fused image;

[0048] The second FPGA chip is used to receive the target image transmitted by the first chip and transmit the target image to the peripheral device; it is also used to receive the configuration command sent by the peripheral device and send the configuration command to the first FPGA chip.

[0049] In one embodiment, the peripheral device includes: a display and a mobile terminal, and the host further includes: a power drive module;

[0050] The display is used to display the processed visible light scene image, near-infrared light scene image and fused image transmitted by the second FPGA chip;

[0051] The mobile terminal is configured to interact with the host and store data, receive images transmitted by the second FPGA chip, and send configuration commands to the second FPGA chip to adjust image-related parameters;

[0052] The power drive module is used to supply power to the host.

[0053] In one embodiment, the present application also proposes an endoscopic camera device, which includes: a memory, a processor, and an image processing program of the endoscopic camera system stored on the memory and runnable on the processor, wherein the image processing program of the endoscopic camera system is configured to implement the steps of the image processing method of the endoscopic camera system as described above.

[0054] In one embodiment, the present application also proposes a storage medium on which an image processing program of an endoscopic camera system is stored. When the image processing program of the endoscopic camera system is executed by a processor, the steps of the image processing method of the endoscopic camera system as described above are implemented.

[0055] The details of one or more embodiments of the present application are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the present application will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] To better describe and illustrate the embodiments and / or examples of the inventions disclosed herein, reference may be made to one or more of the accompanying drawings. The additional details or examples used to describe the accompanying drawings should not be considered to limit the scope of any of the disclosed embodiments, the presently described embodiments and / or examples, and the best mode of these embodiments currently understood.

[0057] FIG1 is a schematic structural diagram of an endoscope system according to one or more embodiments.

[0058] FIG2 is a schematic diagram of components of a light source assembly module in an endoscope system embodiment in one or more embodiments.

[0059] FIG3 is a schematic diagram of components of a camera assembly module in an endoscope system embodiment in one or more embodiments.

[0060] FIG4 is a schematic diagram of components of a camera host assembly module in an endoscope system embodiment in one or more embodiments.

[0061] FIG5 is a schematic diagram of the structure of an endoscopic imaging device in a hardware operating environment according to one or more embodiments.

[0062] FIG6 is a flow chart of an image processing method of an endoscope camera system in one or more embodiments.

[0063] FIG. 7 is another flowchart of an image processing method of an endoscope camera system in one or more embodiments.

[0064] FIG8 is another flowchart of an image processing method of an endoscope camera system in one or more embodiments.

[0065] FIG9 is another flowchart of an image processing method of an endoscope camera system in one or more embodiments.

[0066] FIG. 10 is another flowchart of an image processing method of an endoscope camera system in one or more embodiments.

[0067] FIG. 11 is a schematic diagram of functional modules of an endoscope camera system according to one or more embodiments.

[0068] The realization of the objectives, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0069] It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application.

[0070] The main solutions of the embodiments of the present application include: obtaining a visible light scene image and a near-infrared light scene image, wherein the near-infrared light scene image is obtained by collecting near-infrared light within a preset wavelength range, processing the visible light scene image and the near-infrared light scene image to obtain a processed visible light scene image and a near-infrared light scene image, and fusing the processed visible light scene image and the near-infrared light scene image to obtain a fused image. Since the near-infrared light scene image is formed by using near-infrared light within a preset wavelength range, the present application improves the clarity and recognizability of imaging when the endoscope is in a complex scene and / or poor imaging conditions during surgery. The present application not only helps medical staff diagnose and treat patients more quickly, but also can promptly and effectively deal with emergencies during surgery, greatly saving operation time and reducing the pain of patients.

[0071] 1 , an exemplary embodiment of an endoscope system of the present application is provided. The endoscope system includes an endoscope camera system, which includes a camera head and a camera host. The camera head is connected to the camera host, wherein:

[0072] The camera is used to obtain visible light scene images and near-infrared light scene images, and the near-infrared light scene images are obtained by collecting near-infrared light in a preset band range;

[0073] The camera host is used to process the visible light scene image and the near-infrared light scene image to obtain processed visible light scene image and near-infrared light scene image, and fuse the processed visible light scene image and near-infrared light scene image to obtain a fused image.

[0074] In this embodiment, the reflected visible light and reflected near-infrared light generated by the object being illuminated are collected by a camera and converted, thereby obtaining a visible light scene image and a near-infrared light scene image.

[0075] Before obtaining the near-infrared light scene image, the reflected near-infrared light can be processed to obtain reflected near-infrared light within a preset band range, and then the near-infrared light scene image is obtained based on the reflected near-infrared light within the band range.

[0076] In one embodiment, the preset wavelength range may be between about 820 nm and about 1100 nm, for example, between 820 nm and 1100 nm.

[0077] Compared to fluorescent endoscopes, which have a wavelength range of 760-805nm, the near-infrared light in this embodiment, which has a wavelength range of about 820nm to about 1100nm, can effectively penetrate blood and water mist that visible light and fluorescence cannot penetrate, and obtain the required clear image. The principle is mainly to utilize some physical properties of near-infrared light in this wavelength range. First, the absorption rate of near-infrared light in blood and water mist is low, and the interference is weak; second, the wavelength of near-infrared light in this wavelength range is long, the energy is large, and it has strong penetrating ability. In addition, fluorescent endoscopes require the injection of contrast agents into human tissues in advance before they can be developed in the fluorescent band. However, this application does not require the injection of contrast agents, and its purpose is not for development, but to penetrate blood and water mist to obtain clear images.

[0078] Therefore, in this embodiment, the near-infrared light within this wavelength range can effectively penetrate the blood and water mist that visible light cannot penetrate, thereby obtaining the required clear image.

[0079] In order to obtain images with clearer details and more prominent effective information, in some embodiments, the obtained visible light scene image and near-infrared light scene image can be processed and fused by the camera host. Fusion of the two images processed by the camera host can improve the clarity and recognizability of the final image, helping users better cope with complex and changing surgical environments. In this embodiment, specific methods for fusing visible light scene images and near-infrared light scene images include but are not limited to the following two:

[0080] Method 1: Near-infrared scene images are mainly non-full-color images (such as black and white images). Under special circumstances, near-infrared scene images have the characteristics of high clarity and high recognizability, while visible light scene images are full-color images (RGB images). After the visible light scene image and the near-infrared scene image are fused, the resulting fused image is a full-color image with clarity and recognizability under special circumstances.

[0081] Method 2 is to independently display the visible light scene image and the near-infrared light scene image in a fused screen. For example, the two images can be displayed correspondingly in two separate display windows, or they can be displayed correspondingly through a picture-in-picture display screen, or they can be displayed correspondingly by switching the display to call out two separate display screens. The examples are not exhaustive here.

[0082] In one embodiment, the endoscope system further includes a light source and a lens, and the lens is connected to the light source and the camera.

[0083] Exemplarily, the lens can be composed of a lens barrel, an objective lens and an eyepiece, wherein the lens barrel is a cylindrical structure for accommodating optical elements, the objective lens is a lens located at the front end of the lens barrel, used to focus light and magnify the observed object, and the eyepiece is a lens located at the rear end of the lens barrel, used to observe and magnify the object imaged by the objective lens.

[0084] As shown in FIG2 , the light source includes a visible light source and a near-infrared light source.

[0085] As shown in FIG3 , as an exemplary embodiment, the camera includes: a beam splitter prism visible light image sensor and a near-infrared light image sensor arranged along the light transmission direction; wherein:

[0086] The visible light source and the near-infrared light source are used to illuminate objects in the target scene through the lens of the endoscope, forming reflected light of the objects illuminated by visible light and reflected light of the objects illuminated by near-infrared light, respectively, and transmit them to the camera of the endoscope through the lens to collect visible light scene images and near-infrared light scene images.

[0087] The beam splitter prism is used to receive the reflected light after the visible light source and the near-infrared light source illuminate the object in the target scene through the lens, and change the optical path of the reflected light of the visible light irradiating the object and the reflected light of the near-infrared light irradiating the object to form a visible light path and a near-infrared light path;

[0088] The visible light image sensor is configured to receive visible light from the visible light path and generate a visible light scene image;

[0089] The near-infrared light image sensor is used to receive the near-infrared light in the near-infrared light path and generate a near-infrared light scene image.

[0090] In one embodiment, the camera further includes a filter, and the filter is disposed between the beam splitter prism and the near-infrared image sensor;

[0091] The optical filter is used to filter the near-infrared light in the near-infrared light path to obtain reflected light of the object irradiated by the near-infrared light within a preset wavelength range;

[0092] The near-infrared light image sensor is further configured to receive the near-infrared light filtered by the filter and generate a near-infrared light scene image.

[0093] Exemplarily, as shown in FIG2 , the light source further includes components such as an LED lamp, a power supply, a light source controller, and a light guide. These components work together to ensure that the light source system provides the light source required for a specific application.

[0094] Among them, the light source controller is used to control various parameters of the light source, such as light source type, light source intensity, and color temperature; the LED lamp is used to provide light; the power supply is used to provide power to the light source system; and the light guide is used to transmit light to the objective lens in the lens.

[0095] In this embodiment, the light source integrates both visible and near-infrared light sources. Specifically, a light source controller controls the light source to provide visible and near-infrared light. This light is transmitted through a light guide in the lens to the objective lens, where it is emitted and illuminates objects within the target scene. The reflected light from the illuminated objects then passes through the lens and reaches the camera. In another embodiment, two separate devices, one for visible light and one for near-infrared light, are used together as the light source.

[0096] As shown in FIG3 , after the reflected light from the object is illuminated and enters the camera, it first passes through a dichroic prism, which changes the optical paths of the reflected light from the visible light irradiating the object and the reflected light from the near-infrared light irradiating the object, forming a visible light path and a near-infrared light path.

[0097] Then, the near-infrared light transmitted through the near-infrared light path is filtered by a filter to obtain reflected light of the object irradiated by the near-infrared light in a preset wavelength range.

[0098] Subsequently, the image sensor receives the reflected visible light and the reflected near-infrared light and generates a visible light scene image and a near-infrared scene image. In one embodiment, the visible light image sensor receives visible light from the visible light path to generate the visible light scene image, and the near-infrared light image sensor receives near-infrared light from the near-infrared light path and filtered by a light filter to generate the near-infrared scene image.

[0099] In one embodiment, the camera may further include: a control module connected to the visible light image sensor and the near-infrared light image sensor, and configured to provide the driving voltage required for the two image sensors to transmit image data and to issue related control commands.

[0100] As an embodiment, the camera host includes a dual FPGA component module, the dual FPGA component module includes a first FPGA chip (i.e., chip A shown in FIG4 ) and a second FPGA chip (i.e., chip B shown in FIG4 ), the first FPGA chip and the second FPGA chip are connected via a high-speed transmission interface; wherein:

[0101] The first FPGA chip is used to process and fuse the visible light scene image and the near-infrared light scene image to obtain the processed visible light scene image, the near-infrared light scene image and the fused image, and transmit the target image to the second FPGA chip through a high-speed transmission interface. The processing includes at least one of wide dynamic processing and detail enhancement processing; the target image includes the processed visible light scene image and the near-infrared light scene image, and at least one of the fused images.

[0102] In this embodiment, after the visible light scene image and the near-infrared light scene image are acquired by the camera, the system transmits the images to the first FPGA chip of the camera host. The first FPGA chip performs image processing on the visible light scene image and the near-infrared light scene image. The specific processing may be as follows:

[0103] First, the scene image data is decoded, and the decoded image is used for subsequent image processing, wherein the image includes a visible light scene image and a near-infrared light scene image.

[0104] Then, when the depth of field is insufficient and the dark portion of the image cannot be distinguished, the wide dynamic range module in the first FPGA chip can be used to perform wide dynamic range processing on the scene image to enhance the brightness of the scene image and make the dark portion of the image clearer.

[0105] Among them, wide dynamic processing is used to display the details of the high-brightness and low-brightness areas of the image. It can retain details of a large range of brightness differences in an image, thereby obtaining a more information-rich image even under complex lighting conditions.

[0106] At the same time, when the details of the image are not clear enough, such as the lines of blood vessels, the detail enhancement module in the first FPGA chip can also be used to perform detail enhancement processing on the scene image, which can effectively enhance the detail lines. When an overexposed scene occurs, the metering module can be used to solve the overexposure phenomenon.

[0107] Finally, after processing and fusing the visible light scene image and the near-infrared light scene image, the first FPGA chip transmits the resulting two processed images, and / or the fused image, to the second FPGA chip via the high-speed transmission interface. In other words, in some embodiments, the first FPGA chip can transmit the processed visible light scene image and near-infrared light scene image separately to the second FPGA chip via the high-speed transmission interface, or can transmit the fused image separately to the second FPGA chip via the high-speed transmission interface. Of course, the first FPGA chip can also transmit the visible light scene image, the near-infrared light scene image, and the fused image together to the second FPGA chip.

[0108] The second FPGA chip is used to transmit the processed visible light scene image and near-infrared light scene image, and / or fused image to a peripheral device.

[0109] In one embodiment, first, the processed visible light image, processed near-infrared light image and fused image transmitted from the first FPGA chip via the high-speed transmission interface are received by the second FPGA chip.

[0110] Then, the target image can be determined. The second FPGA chip can use the above three images or part of the three images as the target image. In some embodiments, the visible light scene image, the near-infrared light scene image, and at least one of the fused image can be used as the target image, for example, the visible light scene image alone can be used as the target image, the near-infrared light scene image alone can be used as the target image, or the fused image alone can be used as the target image. Of course, the fused image and the visible light scene image can be used as the target image together, or the fused image and the near-infrared light scene image can be used as the target image together. For the determined target image, the second FPGA chip can transmit the target image to a peripheral device (such as a display) for display via the DP interface or the HDMI interface. In addition, the second FPGA chip can also be used for image compression storage, menu synthesis, human-computer interaction, configuration command issuance, etc.

[0111] In one embodiment, processing of visible light scene images and near-infrared light scene images may also include: one or more of decoding, defect correction, photometry, noise filtering, white balance, color restoration, color correction, saturation correction, gamma correction, color space conversion, contrast correction, and sharpness correction of the visible light scene images, and one or more of decoding, defect correction, and noise filtering of the near-infrared light scene images.

[0112] In one embodiment, as shown in FIG4 , the camera host further includes: a power drive module for supplying power to the entire host system.

[0113] In one embodiment, the camera host further includes: one or more of a FLASH flash memory module, an EMMC module, an SD module, a USB transmission module, a DDR4 storage module, a SATA module, a DP module, an HDMI module, a WIFI module, and a communication module; wherein:

[0114] FLASH module, EMMC module and SD module are used for program storage and booting;

[0115] USB module for image capture, video recording, and program storage and upgrade;

[0116] DDR4 memory module for image access and freezing;

[0117] SATA module for image capture and video recording;

[0118] DP module, used for DP1.2 standard video transmission and display, including SN75DP130 interface chip and DP1.2 standard interface, to achieve DP1.2 standard video transmission and display;

[0119] HDMI module, used for HDMI2.0 standard video transmission and display, including TMDS181 interface chip and HDMI2.0 standard interface, to achieve HDMI2.0 standard video transmission and display;

[0120] WIFI module, used to transmit image data to the mobile terminal in real time for storage and display and interaction with the host system;

[0121] Communication module, used for communication with peripheral devices, including SPI communication, UART communication, I2C communication and network port communication.

[0122] Furthermore, as an embodiment, the peripheral device includes a display and a mobile terminal, the display is connected to the second FPGA chip via a DP interface or an HDMI interface, and the mobile terminal is wirelessly connected to the second FPGA chip, wherein:

[0123] The display can adopt a 4K ultra-high-definition display to display the images transmitted by the host system in real time.

[0124] The second FPGA chip is further configured to send an image to a display for display via a DP interface or an HDMI interface, and to interact with the mobile terminal, receive a configuration command sent by the mobile terminal, and send the configuration command to the first FPGA chip;

[0125] The first FPGA chip is further configured to adjust image parameters according to the configuration instructions.

[0126] In one embodiment, the second FPGA chip receives the configuration command from the mobile terminal via a wireless connection, and sends the configuration command to the first FPGA chip. The first FPGA chip makes corresponding adjustments to the specific processing function of the image according to the command. In this embodiment, the configuration command can be the specific processing of the image selected by the user in the mobile terminal, such as the above-mentioned image wide dynamic range, image detail enhancement, image fusion and other processing. The user can select one or more of them, enable or disable them. After the user selects, the mobile terminal will generate the corresponding configuration command in real time and send it to the host system via a wireless connection. The host system makes corresponding adjustments to the image based on the configuration command.

[0127] Through the above scheme, this embodiment proposes an endoscope camera system, which obtains visible light scene images through a camera, and collects near-infrared light in a preset band range to obtain near-infrared light scene images, processes the visible light scene image and the near-infrared light scene image through a host to obtain processed visible light scene images and near-infrared light scene images, and fuses the processed visible light scene image and near-infrared light scene image to obtain a fused image. Since the near-infrared light scene image is formed by near-infrared light in a preset band range (for example, a band between 820nm-1100nm), the present application scheme improves the clarity and recognizability of endoscope imaging in complex scenes and / or poor imaging conditions, so that the system can adapt to more scenes. In this embodiment, specific methods for fusing visible light scene images and near-infrared light scene images include but are not limited to the following two: Method 1, the infrared light scene image is mainly a non-full-color image (such as a black and white image, etc.), and under special circumstances, the infrared light scene image has the characteristics of high clarity and high recognizability, while the visible light scene image is a full-color image (RGB image). After the fusion process, the obtained fused image is a full-color image with clarity and recognizability under special circumstances; Method 2, the visible light scene image and the near-infrared light scene image are independently displayed in a fused screen. For example, the two images can be displayed correspondingly in two separate display windows, or the two images can be displayed correspondingly through a picture-in-picture display screen, or the two images can be displayed correspondingly by switching the display to call out two separate display screens. The examples are not exhaustive here.

[0128] It should be noted that the camera, host, and lens in the endoscope camera system of the above embodiment constitute the endoscope of the embodiment of the present application. For its specific structure and principle, please refer to the above embodiment and will not be repeated here.

[0129] Refer to Figure 5, which is a schematic diagram of the structure of an endoscope camera device in the hardware operating environment involved in an embodiment of the present application.

[0130] As shown in FIG5 , the endoscopic imaging device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to implement connection and communication between these components. The user interface 1003 may include a display screen and an input unit such as a keyboard. Optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a wireless fidelity (WI-FI) interface). The memory 1005 may be a high-speed random access memory (RAM) memory or a stable non-volatile memory (NVM), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0131] Those skilled in the art will appreciate that the structure shown in FIG5 does not constitute a limitation on the endoscopic imaging device, and may include more or fewer components than shown, or a combination of certain components, or a different arrangement of components.

[0132] As shown in FIG5 , the memory 1005 as a storage medium may include an operating system, a data storage module, a network communication module, a user interface module, and an image processing program of an endoscope camera system.

[0133] In the endoscopic camera device shown in Figure 5, the network interface 1004 can be used to communicate data with other devices; the user interface 1003 can be used to interact with the user; the processor 1001 and the memory 1005 in the endoscopic camera device of the present application can be set in the endoscopic camera device, and the endoscopic camera device calls the image processing program of the endoscopic camera system stored in the memory 1005 through the processor 1001, and executes the image processing method of the endoscopic camera system provided in the embodiment of the present application.

[0134] Based on the above system architecture and device structure, an embodiment of the method of this application is proposed.

[0135] 6 , which is a flow chart of an image processing method for an endoscope camera system according to an embodiment of the present application, in this embodiment, the image processing method for an endoscope camera system includes:

[0136] Step S1000: Acquire a visible light scene image and a near-infrared light scene image, wherein the near-infrared light scene image is obtained by collecting near-infrared light within a preset wavelength band;

[0137] Visible light imaging technology can be utilized in endoscopy. When visible light strikes a tissue surface, some of the light is reflected, while some is transmitted through the tissue. The imaging device collects the reflected light and converts it into a visible light image. However, visible light in the 400nm-700nm wavelength range has limited penetrating power. Small particles generated during surgery can almost completely block visible light, affecting visible light image quality and leading to loss of surgical information.

[0138] Furthermore, while many 4K endoscopes or fluorescent endoscopes are already on the market, their performance is generally good in common scenarios. However, they often fail to meet demand in special circumstances, such as when bleeding obstructs the lens' field of view, when fog obstructs the lens' field of view, when insufficient light prevents details in dark areas, or when vascular patterns are unclear. These special circumstances can interfere with the surgeon's surgical procedures, reducing surgical efficiency and increasing the risk to the patient.

[0139] To address the aforementioned issues, improve the clarity and legibility of endoscopic imaging, and adapt endoscopic imaging to a wider range of surgical scenarios, this embodiment proposes an image processing method for an endoscopic camera system. This image processing method for an endoscopic camera system is based on the endoscopic system described in the aforementioned embodiment. The method utilizes near-infrared light in a specific wavelength band to effectively penetrate blood and water mist, which are impenetrable to visible light, to produce the desired clear image.

[0140] As shown in Figure 1, the endoscope system includes a light source, a camera host, a camera head and a lens. The light source includes: a visible light source and a near-infrared light source.

[0141] In this embodiment, visible light and near-infrared light are provided by a light source, emitted through a lens and illuminate objects in a target scene, and then the reflected visible light and reflected near-infrared light generated by the illuminated objects from the lens are collected and converted by a camera to obtain visible light scene images and near-infrared light scene images.

[0142] In one embodiment, before converting the reflected near-infrared light and obtaining the near-infrared light scene image, the reflected near-infrared light can be processed to obtain reflected near-infrared light in a special band range. Specifically, the reflected near-infrared light is filtered by a filter in the camera to obtain reflected near-infrared light in a band range of 820nm-1100nm, and then based on the reflected near-infrared light in this band range, the near-infrared light scene image is converted and obtained.

[0143] Compared to fluorescent endoscopes, which have a wavelength range of 760-805nm, this embodiment uses near-infrared light in the wavelength range of about 820nm to about 1100nm, which can effectively penetrate blood and water mist that visible light cannot penetrate, and obtain the required clear image. Its principle is mainly to utilize some physical properties of near-infrared light in this wavelength range. First, the absorption rate of near-infrared light in blood and water mist is low, and the interference is weak; second, the wavelength of near-infrared light in this wavelength range is long, the energy is large, and it has strong penetrating ability. In addition, fluorescent endoscopes require the injection of contrast agents into human tissues in advance before they can be developed in the fluorescent band. However, this application does not require the injection of contrast agents, and its purpose is not for development, but to penetrate blood and water mist to obtain clear images.

[0144] Therefore, this embodiment uses near-infrared light in the wavelength range of about 820nm to about 1100nm to effectively penetrate blood and water mist that visible light cannot penetrate, thereby obtaining the required clear image.

[0145] Step S2000: Processing the visible light scene image and the near-infrared light scene image to obtain processed visible light scene image and near-infrared light scene image;

[0146] In one embodiment, in order to obtain images with clearer details and more significant effective information, the acquired visible light scene images and near-infrared light scene images can be further processed. The processed images can help users better cope with complex and changing surgical environments.

[0147] In one embodiment, the system processes the visible light scene images and near-infrared light scene images acquired by the camera, and the processing may include decoding, defect correction, light metering, noise filtering, white balance, color restoration, color correction, saturation correction, gamma correction, wide dynamic, color space conversion, detail enhancement, contrast correction, sharpness correction, etc. Among them, wide dynamic processing of the image can effectively improve the brightness of the scene and obtain an image that can clearly display the dark part; detail enhancement processing of the image can effectively enhance the detail texture, so that the image can better meet the various needs of users.

[0148] In the above-mentioned image processing, processing modules may be set in the system corresponding to different functions, such as a light metering module, a wide dynamic module, and a detail enhancement module.

[0149] By processing images, it is possible to cope with complex and changeable surgical environments. For example, when an overexposed scene occurs, the efficient light metering module can effectively solve the overexposure phenomenon; when the depth of field is insufficient and the dark image cannot be distinguished, the wide dynamic module can effectively enhance the brightness of the scene, so that the dark image can be clearly seen; when the details of the image are not clear enough, such as the lines of blood vessels, the detail enhancement module can effectively enhance the details; when encountering water mist or blood scenes and the image cannot be seen clearly, the fusion of near-infrared images through water mist and blood with visible light images can effectively solve this problem.

[0150] Step S3000: fusing the processed visible light scene image and near-infrared light scene image to obtain a fused image.

[0151] Although visible light imaging has a limited penetration depth, allowing only surface tissue or shallow structures to be observed, limiting observation of deeper tissues or organs, it also offers advantages such as being more consistent with human observation and more intuitively understandable and observable. Therefore, to make endoscopic imaging clearer, more recognizable, and adaptable to a wider range of surgical scenarios, the imaging advantages of visible light and near-infrared light within the aforementioned special wavelength range can be combined to achieve even better imaging results.

[0152] The combining method may be to fuse the two images, that is, to fuse the processed visible light scene image and the near-infrared light scene image to obtain a fused image.

[0153] In this embodiment, when the endoscope encounters water mist or blood scenes during surgery, resulting in inability to form clear images, the near-infrared light scene image that passes through the water mist and blood is fused with the visible light scene image to effectively solve this problem.

[0154] Fusion of the two images processed by the camera host can improve the clarity and recognizability of the final image, helping users better cope with complex and changing surgical environments. In this embodiment, the specific methods for fusing the visible light scene image and the near-infrared light scene image include but are not limited to the following two:

[0155] Method 1: Near-infrared scene images are mainly non-full-color images (such as black and white images). In special environments, infrared scene images have high clarity and recognizability, while visible light scene images are full-color images (RGB images). After the visible light scene image and the near-infrared scene image are fused, the resulting fused image is a full-color image with clarity and recognizability in special environments.

[0156] Method 2 is to independently display the visible light scene image and the near-infrared light scene image in a fused screen. For example, the two images can be displayed correspondingly in two separate display windows, or they can be displayed correspondingly through a picture-in-picture display screen, or they can be displayed correspondingly by switching the display to call out two separate display screens. The examples are not exhaustive here.

[0157] For the first approach above, the fusion processing method may include:

[0158] Weighted averaging: This method performs a weighted average of corresponding pixels in the visible light scene image and the near-infrared scene image to produce a new fused image. The weights can be adjusted to highlight different features based on application requirements.

[0159] Maximum or minimum method: Select the maximum or minimum pixel in the visible light scene image and the near-infrared light scene image respectively as the corresponding pixel of the fused image. This can emphasize the different characteristics of the two bands.

[0160] Wavelet transform fusion: The visible light scene image and the near-infrared light scene image are subjected to wavelet transform respectively, the coefficients are fused in the transform domain, and finally an inverse wavelet transform is performed to obtain a fused image. This can preserve the low-frequency and high-frequency information of the image.

[0161] Principal component analysis (PCA) fusion: The visible light scene image and the near-infrared light scene image are projected onto their principal components respectively, and then the principal components are weighted averaged or certain principal components are selected for fusion.

[0162] Feature extraction and fusion: Extract meaningful features such as texture and edges from visible light scene images and near-infrared light scene images, and then fuse these features to produce a fused image.

[0163] Machine learning fusion: Use machine learning methods, such as support vector machines and neural networks, to input visible light scene images and near-infrared light scene images into the model for training, and then use the model to predict the fusion results.

[0164] Pyramid fusion: The visible light scene image and the near-infrared light scene image are decomposed into pyramids, then fused at different scales, and finally the fused image is restored by merging.

[0165] Fluorescent marker fusion: Superimpose visible light scene images and near-infrared light scene images, thereby directly mapping the information of the near-infrared light scene image to the visible light image, providing richer biological information.

[0166] In this embodiment, by acquiring a visible light scene image and a near-infrared light scene image, wherein the near-infrared light scene image is obtained by collecting near-infrared light within a preset wavelength band, processing the visible light scene image and the near-infrared light scene image to obtain processed visible light scene image and near-infrared light scene image, and fusing the processed visible light scene image and the near-infrared light scene image to obtain a fused image, the clarity and recognizability of imaging during surgery with an endoscope in complex scenes and / or poor imaging conditions are improved.

[0167] In one embodiment, referring to FIG7 , FIG7 is another flow chart of an endoscopic imaging method according to an embodiment of the present application. As shown in FIG3 , the endoscopic system includes an endoscopic imaging system, the endoscopic imaging system includes a camera, and the camera includes: a visible light image sensor, a near-infrared light image sensor, and a beam splitter prism;

[0168] As an implementation manner, based on the embodiment shown in FIG6 , step S1000 may include:

[0169] Reflected light from an object in a target scene after being irradiated by a visible light source and a near-infrared light source is received through a beam splitter prism, and the optical paths of the reflected light from the object irradiated by the visible light and the reflected light from the object irradiated by the near-infrared light are changed through the beam splitter prism to form a visible light path and a near-infrared light path;

[0170] The visible light of the visible light path and the near-infrared light of the near-infrared light path are received by an image sensor to generate a visible light scene image and a near-infrared light scene image.

[0171] As another implementation, based on the embodiment shown in FIG. 6 , step S1000 may include:

[0172] Step S1010: Receiving reflected light from an object in a target scene after being illuminated by a visible light source and a near-infrared light source through a beam splitter prism, and changing the optical paths of the reflected light from the object illuminated by the visible light and the reflected light from the object illuminated by the near-infrared light through the beam splitter prism to form a visible light path and a near-infrared light path;

[0173] Step S1020: filtering the near-infrared light in the near-infrared light path through a filter to obtain reflected light of the object irradiated by the near-infrared light within a preset wavelength range;

[0174] Step S1030: receiving the visible light in the visible light path and the near-infrared light filtered by the filter through an image sensor to generate a visible light scene image and a near-infrared light scene image.

[0175] In one embodiment, as shown in FIG2 , the light source further includes components such as a light source controller and a light guide.

[0176] Among them, the light source controller is used to control various parameters of the light source, such as light source type, light source intensity, and color temperature, and the light guide is used to transmit light to the objective lens in the lens.

[0177] In this embodiment, the light source controller controls the light source to provide visible light and near-infrared light, which is transmitted to the objective lens through the light guide in the lens and emitted to illuminate objects in the target scene. The reflected light after the object is illuminated then passes through the lens to reach the camera.

[0178] After the reflected light from the object is illuminated and enters the camera, it first passes through the dichroic prism, which changes the optical paths of the reflected light from the visible light irradiating the object and the reflected light from the near-infrared light irradiating the object, making the two reflected lights non-coplanar, forming a visible light path and a near-infrared light path.

[0179] Then, the near-infrared light transmitted through the near-infrared light path is filtered by a filter to obtain the reflected light of the object irradiated by near-infrared light in the band range of about 820nm to about 1100nm. The near-infrared light in this band can effectively penetrate blood, water and water mist that visible light cannot penetrate, and obtain the required clear image. The principle is mainly to utilize some physical properties of the near-infrared light in this band. First, the near-infrared light in this band has a low absorption rate in blood, water and water mist, and is subject to weak interference; second, the near-infrared light in this band has a longer wavelength, greater energy, and stronger penetrating ability.

[0180] Afterwards, visible light and near-infrared light are received by an image sensor and a visible light scene image and a near-infrared light scene image are generated. In one embodiment, visible light from the visible light path is received by a visible light image sensor to generate a visible light scene image, and near-infrared light from the near-infrared light path and filtered by a light filter is received by a near-infrared light image sensor to generate a near-infrared light scene image.

[0181] In this embodiment, a beam splitter receives reflected light from objects within a target scene illuminated by a visible light source and a near-infrared light source, and changes the optical paths of the reflected light from the visible light-illuminated object and the reflected light from the near-infrared light-illuminated object by the beam splitter, forming a visible light path and a near-infrared light path. A filter filters the near-infrared light from the near-infrared light path to obtain reflected light from the object illuminated by the near-infrared light within a preset wavelength range. An image sensor receives the visible light from the visible light path and the near-infrared light filtered by the filter to generate a visible light scene image and a near-infrared light scene image. This generates the desired target image, providing a foundation for subsequent image processing to improve image clarity and recognizability to meet imaging requirements in different scenarios.

[0182] In one embodiment, referring to FIG8 , FIG8 is another flow chart of an endoscopic imaging method according to an embodiment of the present application. Based on the embodiment shown in FIG6 , the endoscopic imaging system further includes a camera host, which includes a first FPGA chip. Step S2000 may include:

[0183] Step S2010: decoding the visible light scene image and the near-infrared light scene image, and performing wide dynamic range processing and / or detail enhancement processing on the visible light scene image and the near-infrared light scene image through the first FPGA chip to obtain processed visible light scene image and near-infrared light scene image.

[0184] Step S2020: Correcting the visible light scene image and the near-infrared light scene image by using the first FPGA chip to obtain processed visible light scene image and near-infrared light scene image.

[0185] In this embodiment, after obtaining the visible light scene image and the near-infrared light scene image in step S1000, the system transmits the images to the first FPGA chip, which performs image processing on the images. In one embodiment, the image processing may include the following operations:

[0186] First, the scene image data is decoded, and the decoded image is used for subsequent image processing, wherein the image includes a visible light scene image and a near-infrared light scene image.

[0187] When insufficient depth of field makes dark areas difficult to distinguish, the wide dynamic range module applies wide dynamic range processing to the scene image, boosting its brightness and making dark areas clearer. Wide dynamic range processing is used to reveal details in both high- and low-brightness areas of an image. Wide dynamic range processing preserves details across a wide range of brightness differences within a single image, resulting in a more informative image even in complex lighting conditions.

[0188] At the same time, when the details of the image are not clear enough, such as the lines of blood vessels, the detail enhancement module can be used to perform detail enhancement processing on the scene image to effectively enhance the detail lines. When overexposed scenes appear, the metering module can be used to solve the overexposure phenomenon. The detail enhancement processing can include the following methods:

[0189] The Laplacian filter enhances the high-frequency components of an image, namely edges and details. It does this by calculating the difference between each pixel and its surrounding pixels, thereby enhancing the details of the image.

[0190] Unsharp masking is a process that first creates a blurred version of the original image, typically using a Gaussian blur, then calculates the difference between the original and blurred images to get the high-frequency details. Finally, the high-frequency details are added back to the original image to enhance the image's details.

[0191] High-pass filters are used to enhance high-frequency components in an image, such as edges and details. Common high-pass filters such as Sobel and Prewitt can be used, which achieve this by enhancing high-frequency components in the frequency domain.

[0192] Reduce noise and enhance image details through the Wiener filter. It smoothes the image by averaging the noise and signal, thereby highlighting the details.

[0193] Through the histogram equalization method, the histogram of the image is stretched to increase the distribution range of pixel intensity, thereby improving the contrast of the image.

[0194] Through the adaptive histogram equalization method, histogram equalization is applied to small patches of the image to preserve the details in the image and avoid excessive enhancement of the global contrast.

[0195] And CLAHE (Contrast Limited Adaptive Histogram Equalization) applies histogram equalization to small areas of the image to enhance details and avoid over-enhancement. It is particularly useful when processing images with a large dynamic range.

[0196] Through multi-scale Retinex, the image is weighted averaged at different scales to improve the dynamic range and contrast of the image while enhancing details.

[0197] Adaptive Retinex is used to enhance the local contrast of the image, with different degrees of enhancement depending on the needs of different areas.

[0198] In addition, the correction processing of the visible light scene image and the near-infrared light scene image may include:

[0199] The visible light scene image can be processed by defect correction, noise filtering, white balance, color restoration, color correction, saturation correction, gamma correction, color space conversion, contrast correction, sharpness correction, etc.

[0200] Perform defect correction, noise filtering and other processing on near-infrared light scene images.

[0201] The images include visible light scene images and near-infrared light scene images.

[0202] In this embodiment, by decoding the visible light scene image and the near-infrared light scene image, and performing wide dynamic processing and / or detail enhancement and correction processing, the processed visible light image and the near-infrared light image are obtained, which improves the clarity and recognizability of the image, making the endoscopic imaging applicable to more scenarios.

[0203] In one embodiment, referring to FIG. 9 , FIG. 9 is another flow chart of an endoscopic imaging method according to an embodiment of the present application, wherein the imaging host further includes a second FPGA chip, and after step S3000 , further includes:

[0204] Step S4000: receiving, through the second FPGA chip, the processed visible light scene image, the near-infrared light scene image, and the fused image transmitted by the first FPGA chip;

[0205] Step S5000: transmitting the processed visible light scene image, near-infrared light scene image and fused image to a peripheral device through the second FPGA chip.

[0206] In this embodiment, the first FPGA chip is mainly responsible for image processing and transmission, and the second FPGA chip is mainly responsible for interaction with peripheral device modules. After the first FPGA chip processes and fuses the visible light scene image and the near-infrared light scene image, the two processed images and the fused image are transmitted to the second FPGA chip through a high-speed transmission interface.

[0207] In this embodiment, first, the second FPGA chip receives the processed visible light image, the processed near-infrared light image, and the fused image transmitted by the first FPGA chip through the high-speed transmission interface.

[0208] Then, the second FPGA chip can transmit the above three images to a peripheral device for display through the DP interface or the HDMI interface, and the peripheral device can be a display.

[0209] In addition, the peripheral device may also include a mobile terminal, and the endoscope may also establish a connection with the mobile terminal via a WiFi module. Through the WiFi connection, the second FPGA may also transmit the above-mentioned image to the mobile terminal in real time for display or storage by the terminal.

[0210] In one embodiment, referring to FIG. 10 , FIG. 10 is another flow chart of an endoscopic imaging method according to an embodiment of the present application. In this embodiment, after step S5000, the following steps are further included:

[0211] Step S6000: receiving a configuration command sent by the peripheral device through the second FPGA chip;

[0212] Step S7000: Send the configuration command to the first FPGA chip through the second FPGA chip.

[0213] In this embodiment, the second FPGA chip can also receive configuration commands from the mobile terminal through a WiFi connection, and send the configuration commands to the first FPGA chip. The first FPGA chip makes corresponding adjustments to the specific processing functions of the image according to the commands. In this embodiment, the configuration command can be the specific processing of the image selected by the user on the mobile terminal, such as the above-mentioned image wide dynamic range, image detail enhancement, image fusion and other processing. The user can select one or several of them, enable or not enable. After the user selects, the mobile terminal will generate the corresponding configuration command in real time and send it to the endoscope camera system through a WiFi connection. The endoscope camera system makes corresponding adjustments to the system based on the received configuration command.

[0214] In this embodiment, the second FPGA chip receives the processed visible light scene image, near-infrared light scene image, and fused image transmitted by the first FPGA chip, transmits the processed visible light scene image, near-infrared light scene image, and fused image to a peripheral device, and receives configuration commands sent by the peripheral device and sends the configuration commands to the first FPGA chip. This makes human-computer interaction more convenient, enables easy configuration of the endoscopic camera system to cope with various situations that arise during surgery, and can record the surgical progress in real time.

[0215] In one embodiment, referring to FIG. 11 , an embodiment of the present application further provides an endoscopic camera system 1100 , the endoscopic camera system comprising:

[0216] An image acquisition module 1110 is configured to acquire a visible light scene image and a near-infrared light scene image, wherein the near-infrared light scene image is obtained by acquiring near-infrared light within a preset wavelength band;

[0217] An image processing module 1120 is configured to process the visible light scene image and the near-infrared light scene image to obtain processed visible light scene images and near-infrared light scene images;

[0218] The image fusion module 1130 is configured to fuse the processed visible light scene image and near-infrared light scene image to obtain a fused image.

[0219] In one embodiment, the endoscope camera system 1100 includes a camera, which includes an image sensor and a beam splitter prism, and the image acquisition module 1110 is used to:

[0220] Reflected light from an object in a target scene after being irradiated by a visible light source and a near-infrared light source is received through a beam splitter prism, and the optical paths of the reflected light from the object irradiated by the visible light and the reflected light from the object irradiated by the near-infrared light are changed through the beam splitter prism to form a visible light path and a near-infrared light path;

[0221] The visible light of the visible light path and the near-infrared light of the near-infrared light path are received by an image sensor to generate a visible light scene image and a near-infrared light scene image.

[0222] In one embodiment, the camera further includes a filter, and the image acquisition module 1110 is configured to:

[0223] Filtering the near-infrared light in the near-infrared light path through a filter to obtain reflected light of an object irradiated by the near-infrared light within a preset wavelength range;

[0224] The image sensor receives the visible light in the visible light path and the near-infrared light filtered by the filter to generate a visible light scene image and a near-infrared light scene image.

[0225] In one embodiment, the preset wavelength range is between about 820 nm and about 1100 nm.

[0226] In one embodiment, the endoscope camera system 1100 further includes a camera host, the camera host includes a first FPGA chip, and the image processing module 1120 is configured to:

[0227] The visible light scene image and the near-infrared light scene image are decoded and processed by the first FPGA chip to obtain processed visible light scene image and near-infrared light scene image; the processing includes at least one of wide dynamic processing and detail enhancement processing.

[0228] In one embodiment, the processing of the visible light scene image by the image processing module 1120 further includes at least one of the following: defect correction, photometry, noise filtering, white balance, color restoration, color correction, saturation correction, gamma correction, color space conversion, contrast correction, and sharpness correction;

[0229] The processing of the near-infrared light scene image by the image processing module 1120 further includes at least one of the following: defect correction and noise filtering.

[0230] In one embodiment, the image processing module 1120 is further configured to:

[0231] The visible light scene image and the near-infrared light scene image are corrected and processed by the first FPGA chip to obtain processed visible light scene image and near-infrared light scene image.

[0232] In one embodiment, the camera host further includes a second FPGA chip, and the endoscope camera system 1100 is further configured to:

[0233] receiving, through the second FPGA chip, a target image transmitted by the first FPGA chip; the target image comprising at least one of a processed visible light scene image, a near-infrared light scene image, and the fused image;

[0234] The target image is transmitted to a peripheral device through the second FPGA chip.

[0235] In one embodiment, the endoscopic camera system 1100 is further configured to:

[0236] Receiving, through the second FPGA chip, a configuration command sent by the peripheral device;

[0237] The configuration command is sent to the first FPGA chip through the second FPGA chip.

[0238] In one embodiment, the fusion is achieved by any one of the following methods: weighted average method, maximum method, minimum method, wavelet transform fusion, principal component analysis fusion, feature extraction and fusion, machine learning fusion, pyramid fusion, and fluorescent labeling fusion.

[0239] For the implementation principle of the endoscope camera system of this embodiment, please refer to the above embodiments and will not be repeated here.

[0240] In one embodiment, an embodiment of the present application also provides an endoscopic camera device, which includes: a memory, a processor, and an image processing program of the endoscopic camera system stored on the memory and runnable on the processor, wherein the image processing program of the endoscopic camera system is configured to implement the steps of each embodiment of the image processing method of the endoscopic camera system as described above.

[0241] For the implementation principle of this embodiment, please refer to the above embodiments and will not be repeated here.

[0242] In one embodiment, an embodiment of the present application also provides a computer-readable storage medium, on which an image processing program of an endoscopic camera system is stored. When the image processing program of the endoscopic camera system is executed by a processor, the steps of each embodiment of the image processing method of the endoscopic camera system as described above are implemented.

[0243] For the implementation principle of this embodiment, please refer to the above embodiments and will not be repeated here.

[0244] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or system comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or system. In the absence of further limitations, an element defined by the phrase "comprising an endoscopic camera" does not exclude the presence of other identical elements in the process, method, article, or system comprising the element.

[0245] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0246] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0247] Each module in the above-mentioned endoscopic camera system can be implemented in whole or in part through software, hardware, or a combination thereof. The network interface can be an Ethernet card or a wireless network card, for example. Each of the above-mentioned modules can be embedded in or independent of the server's processor in hardware form, or can be stored in the server's memory in software form, so that the processor can call and execute the corresponding operations of each module.

[0248] As used in this application, the terms "component," "module," and "system" are intended to refer to a computer-related entity, which can be hardware, a combination of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable code, a thread of execution, a program, and / or a computer. As an illustration, both an application running on a server and a server can be a component. One or more components can reside in a process and / or a thread of execution, and a component can be located within a computer and / or distributed between two or more computers.

[0249] The above are merely exemplary embodiments of the present application and are not intended to limit the patent scope of the present application. Any equivalent structural or equivalent process transformations made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. An image processing method for an endoscope camera system, comprising the following steps: Acquire a visible light scene image and a near-infrared light scene image, wherein the near-infrared light scene image is obtained by collecting near-infrared light within a preset wavelength band; Processing the visible light scene image and the near-infrared light scene image to obtain processed visible light scene image and near-infrared light scene image; The processed visible light scene image and near infrared light scene image are fused to obtain a fused image.

2. The image processing method of the endoscope camera system according to claim 1, wherein the endoscope camera system comprises a camera, the camera comprises an image sensor and a beam splitter prism, and the step of acquiring the visible light scene image and the near-infrared light scene image comprises: Reflected light after the visible light source and the near-infrared light source illuminate an object in a target scene is received through a beam splitter prism, and the optical paths of the reflected light after the visible light illuminates the object and the reflected light after the near-infrared light illuminates the object are changed through the beam splitter prism to form a visible light path and a near-infrared light path; Visible light in the visible light path and near-infrared light in the near-infrared light path are received by an image sensor to generate a visible light scene image and a near-infrared light scene image.

3. The image processing method of the endoscope camera system according to claim 2, wherein the camera further comprises a filter, and the step of receiving the visible light of the visible light path and the near-infrared light of the near-infrared light path through the image sensor to generate the visible light scene image and the near-infrared light scene image comprises: The near-infrared light of the near-infrared light path is filtered by a filter to obtain reflected light of an object irradiated by the near-infrared light within a preset wavelength range; The visible light in the visible light path and the near infrared light filtered by the filter are received by the image sensor to generate a visible light scene image and a near infrared light scene image.

4. The endoscopic imaging method according to any one of claims 1 to 3, wherein the preset wavelength range is between about 820 nm and about 1100 nm.

5. The endoscopic imaging method according to any one of claims 1 to 4, wherein the endoscopic imaging system further comprises an imaging host, the imaging host comprises a first FPGA chip, and the step of processing the visible light scene image and the near-infrared light scene image to obtain the processed visible light scene image and the near-infrared light scene image comprises: The visible light scene image and the near-infrared light scene image are decoded and processed by the first FPGA chip to obtain processed visible light scene image and near-infrared light scene image; the processing includes at least one of wide dynamic processing and detail enhancement processing.

6. The endoscope camera method according to claim 5, wherein the processing of the visible light scene image further comprises at least one of the following: defect correction, photometry, noise filtering, white balance, color restoration, color correction, saturation correction, gamma correction, color space conversion, contrast correction, and sharpness correction; The processing of the near-infrared light scene image also includes at least one of the following: defect correction and noise filtering.

7. The endoscopic imaging method according to claim 5 or 6, wherein the step of processing the visible light scene image and the near-infrared light scene image to obtain the processed visible light scene image and the near-infrared light scene image further comprises: The visible light scene image and the near-infrared light scene image are corrected and processed by the first FPGA chip to obtain processed visible light scene image and near-infrared light scene image.

8. The endoscope imaging method according to any one of claims 5 to 7, wherein the imaging host further comprises a second FPGA chip, and the method further comprises: Receiving, by the second FPGA chip, a target image transmitted by the first FPGA chip; The target image includes at least one of a processed visible light scene image, a near infrared light scene image and the fused image; The target image is transmitted to a peripheral device through the second FPGA chip.

9. The endoscopic imaging method according to claim 8, further comprising: Receiving a configuration command sent by the peripheral device through the second FPGA chip; The configuration command is sent to the first FPGA chip through the second FPGA chip.

10. The endoscopic imaging method according to any one of claims 1 to 9, wherein the fusion is achieved by any one of the following methods: weighted average method, maximum method, minimum method, wavelet transform fusion, principal component analysis fusion, feature extraction and fusion, machine learning fusion, pyramid fusion, and fluorescent marker fusion.

11. An endoscope system, the endoscope system comprising an endoscope camera system, the endoscope camera system comprising a camera head and a camera host; The camera is used to obtain visible light scene images and near-infrared light scene images, and the near-infrared light scene images are obtained by collecting near-infrared light conversion within a preset band range; The camera host is used to process the visible light scene image and the near-infrared light scene image to obtain processed visible light scene image and near-infrared light scene image; and fuse the processed visible light scene image and near-infrared light scene image to obtain a fused image.

12. The endoscope system according to claim 11, further comprising a light source and a lens, wherein the light source comprises a visible light source and a near-infrared light source, and the camera comprises a beam splitter prism, a visible light image sensor and a near-infrared light image sensor arranged along the transmission direction of the light path; The visible light source and the near-infrared light source are used to illuminate objects in the target scene through the lens, forming reflected light of the objects illuminated by the visible light and reflected light of the objects illuminated by the near-infrared light, respectively; The beam splitter prism is used to receive the reflected light after the visible light source and the near-infrared light source illuminate the object in the target scene through the lens, and change the optical path of the reflected light of the object illuminated by the visible light and the reflected light of the object illuminated by the near-infrared light to form a visible light path and a near-infrared light path; The visible light image sensor is used to receive the visible light of the visible light path and generate a visible light scene image; The near-infrared light image sensor is used to receive the near-infrared light in the near-infrared light path and generate a near-infrared light scene image.

13. The endoscope system according to claim 12, wherein the camera head further comprises a filter, wherein the filter is disposed between the beam splitter prism and the near-infrared light image sensor; The optical filter is used to filter the near-infrared light in the near-infrared light path to obtain reflected light of an object irradiated by the near-infrared light within a preset wavelength range; The near-infrared light image sensor is further used to receive the near-infrared light filtered by the filter to generate a near-infrared light scene image.

14. The endoscope system according to any one of claims 11 to 13, wherein the preset wavelength range is between about 820 nm and about 1100 nm.

15. The endoscope system according to any one of claims 11 to 14, wherein the camera host comprises a dual FPGA component module, and the dual FPGA component module comprises a first FPGA chip and a second FPGA chip; The first FPGA chip is used to decode and process the visible light scene image and the near-infrared light scene image, and to perform correction processing on the visible light scene image and the near-infrared light scene image to obtain processed visible light scene image and near-infrared light scene image; the processing includes at least one of wide dynamic processing and detail enhancement processing.

16. The endoscope system as described in claim 15, wherein the first FPGA chip is also used to fuse the processed visible light scene image and near-infrared light scene image to obtain the fused image.

17. The endoscope system according to any one of claims 15 to 16, wherein the first FPGA chip is further used to transmit a target image to a second FPGA chip through a high-speed transmission interface; the target image comprises at least one of the processed visible light scene image, the near-infrared light scene image and the fused image; The second FPGA chip is used to receive the target image transmitted by the first chip, and transmit the target image to a peripheral device.

18. The endoscope system according to any one of claims 15 to 17, wherein the peripheral device comprises: Display and / or mobile terminal; The display is used to display the processed visible light scene image and near infrared scene image transmitted by the second FPGA chip. Light scene image and fused image; The mobile terminal is used to interact with the host and store data, and is also used to receive images transmitted by the second FPGA chip.

19. An endoscopic imaging device, comprising: A memory, a processor, and an image processing program of an endoscopic camera system stored in the memory and executable on the processor, wherein the image processing program of the endoscopic camera system is configured to implement the steps of the endoscopic camera method according to any one of claims 1 to 10.

20. A storage medium storing an image processing program of an endoscopic camera system, wherein the image processing program of the endoscopic camera system, when executed by a processor, implements the steps of the endoscopic camera method according to any one of claims 1 to 10.

Images