Image processing method, device and equipment

Abstract

The present application provides an image processing method, apparatus, and device. The method comprises: determining a target development mode corresponding to a fluorescent image; wherein the target development mode is the same as a configured initial development mode, or the target development mode is different from the initial development mode; and, under the target development mode, acquiring a target fluorescent image corresponding to a specified location within a target object. Through the technical solution of the present application, the development mode can be automatically adjusted to match the feature information corresponding to the target region of interest, thereby effectively switching between positive and negative development modes.

Description

Technical Field

[0001] The present application relates to the field of medical technology, and in particular to an image processing method, apparatus and device. Background Art

[0002] Endoscopes are commonly used medical devices consisting of a light-guiding structure and a set of lenses. Once inside a target object, the endoscope can capture visible light and fluorescence images of a specific location within the target object, generating a fused image based on these two images. This fused image clearly displays normal and diseased tissue at a specific location within the target object. This fused image allows for the distinction between normal and diseased tissue at a specific location within the target object, allowing for examination and treatment of the target object based on the fused image and accurate determination of which tissues require resection.

[0003] When acquiring fluorescence images of a specific location within a target object, multiple development modes are available, and the fluorescence images captured under different development modes differ in form. Because of the multiple development modes, medical personnel must manually configure the development mode to use based on their experience. Once a development mode is determined, it cannot be adjusted. This configuration may result in poor fluorescence image quality, which in turn may lead to poor fusion image quality, failing to clearly display normal and diseased tissue at the specific location within the target object, resulting in a poor user experience. Summary of the Invention

[0004] The present application provides an image processing method, the method comprising:

[0005] Determining a target development mode corresponding to the fluorescent image; wherein the target development mode is the same as the configured initial development mode, or the target development mode is different from the initial development mode;

[0006] In the target development mode, a target fluorescence image corresponding to a specified position inside the target object is acquired.

[0007] Exemplarily, determining the target development mode corresponding to the fluorescent image includes:

[0008] Determining a target region of interest corresponding to the specified position inside the target object based on an initial visible light image and an initial fluorescence image corresponding to the specified position inside the target object; wherein the initial visible light image includes a normal tissue subregion, the initial fluorescence image includes a fluorescence target region, and the target region of interest is a remaining subregion of the fluorescence target region excluding the normal tissue subregion; wherein the initial fluorescence image is a fluorescence image acquired under the initial development mode; if the initial development mode is a positive development mode, the fluorescence target region is a development region, and if the initial development mode is a negative development mode, the fluorescence target region is a non-development region;

[0009] A target development mode is determined based on feature information corresponding to the target region of interest.

[0010] Exemplarily, determining the target region of interest corresponding to the designated position inside the target object based on the initial visible light image and the initial fluorescence image corresponding to the designated position inside the target object includes:

[0011] determining the normal tissue sub-region from the initial visible light image;

[0012] determining the fluorescent target area from the initial fluorescent image;

[0013] An intersection region between the fluorescent target region and the normal tissue sub-region is determined, and the remaining sub-regions in the fluorescent target region except the intersection region are determined as the target region of interest.

[0014] Exemplarily, determining the fluorescent target area from the initial fluorescent image includes:

[0015] Based on the pixel value corresponding to each pixel in the initial fluorescent image, selecting a target pixel that matches the fluorescent target area from all the pixels in the initial fluorescent image;

[0016] A connected domain consisting of all target pixels is obtained, and the fluorescent target area is determined from the initial fluorescent image based on the connected domain.

[0017] Exemplarily, selecting a target pixel that matches the fluorescent target area from all pixels of the initial fluorescent image based on a pixel value corresponding to each pixel in the initial fluorescent image includes: if the initial fluorescent image is a fluorescent image in a positive development mode, determining that the pixel in the initial fluorescent image is a target pixel when the pixel value corresponding to the pixel is greater than a first threshold; or

[0018] If the initial fluorescent image is a fluorescent image in a negative development mode, when a pixel value corresponding to a pixel point in the initial fluorescent image is less than a second threshold, the pixel point is determined to be a target pixel point.

[0019] Exemplarily, if the characteristic information is a first number of all pixels in the target region of interest, determining the target development mode based on the characteristic information corresponding to the target region of interest includes:

[0020] If the first number is less than a number threshold, determining that the target development mode is a positive development mode; if the first number is not less than the number threshold, determining that the target development mode is a negative development mode;

[0021] Alternatively, if the ratio of the first number to the second number of all pixels in the initial fluorescent image is less than a ratio threshold, it is determined that the target development mode is a positive development mode; if the ratio of the first number to the second number is not less than the ratio threshold, it is determined that the target development mode is a negative development mode.

[0022] Exemplarily, in the target development mode, obtaining a target fluorescence image corresponding to a specified position inside the target object includes: if the target development mode is different from the configured initial development mode, switching the initial development mode to the target development mode, and obtaining a target fluorescence image corresponding to the specified position inside the target object in the target development mode.

[0023] Exemplarily, in the target development mode, after acquiring a target fluorescence image corresponding to a specified position inside the target object, the method further includes:

[0024] determining a normal tissue subregion from a target visible light image corresponding to a specified position inside the target object;

[0025] determining a fluorescent target area from the target fluorescent image;

[0026] determining an intersection region between the fluorescent target region and the normal tissue subregion, and removing the intersection region from the target fluorescent image to obtain a new fluorescent image;

[0027] The new fluorescence image and the target visible light image are fused to obtain a fused image.

[0028] Exemplarily, the removing process of the intersection area in the target fluorescence image to obtain a new fluorescence image includes: performing brightness attenuation processing on the intersection area to obtain the new fluorescence image; or performing brightness remapping processing on the intersection area to obtain the new fluorescence image; or performing brightness attenuation processing and brightness remapping processing on the intersection area to obtain a new fluorescence image.

[0029] Exemplarily, the fusing of the new fluorescence image and the target visible light image to obtain a fused image includes: contrast enhancing the new fluorescence image to obtain an enhanced fluorescence image; color mapping the enhanced fluorescence image to obtain a mapped fluorescence image; and fusing the mapped fluorescence image and the target visible light image to obtain a fused image.

[0030] The present application provides an image processing device, comprising: a determination module for determining a target development mode corresponding to a fluorescent image; wherein the target development mode is the same as a configured initial development mode, or the target development mode is different from the initial development mode; and an acquisition module for acquiring, under the target development mode, a target fluorescent image corresponding to a specified position inside the target object.

[0031] The present application proposes an image processing device, comprising: a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor; the processor is used to execute the machine-executable instructions to implement the image processing method disclosed in the above example of the present application.

[0032] It can be seen from the above technical solution that in the embodiment of the present application, the target development mode can be determined, and the fluorescent image corresponding to the specified position inside the target object can be obtained under the target development mode. The target development mode can be the same as or different from the configured initial development mode, that is, the development mode can be automatically adjusted. After setting the development mode, the development mode can be automatically adjusted according to the visible light image and the fluorescent image of the actual scene, and the development mode can be switched in real time, thereby effectively switching the positive development mode and the negative development mode (that is, switching the positive development mode to the negative development mode, or switching the negative development mode to the positive development mode), so that the fluorescence image corresponding to the development mode has a better effect and can highlight the diseased tissue information. After generating a fused image based on the fluorescence image, the fused image has a better effect and can clearly display the normal tissue and diseased tissue at the specified position inside the target object, and can accurately determine which tissues need to be removed, so that the user experience is better. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments of the present application or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in this application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings of the embodiments of the present application.

[0034] Figure 1 is a schematic structural diagram of an endoscope system in one embodiment of the present application;

[0035] Figure 2 is a functional structural diagram of an endoscope system in one embodiment of the present application;

[0036] Figure 3A and Figure 3B is a schematic diagram of a visible light image and a fluorescent image in one embodiment of the present application;

[0037] Figure 3C and Figure 3D is a schematic diagram of a fused image in one embodiment of the present application;

[0038] Figure 4 is a flowchart of an image processing method in one embodiment of the present application;

[0039] Figure 5A and Figure 5B is a schematic diagram of determining a target region of interest in one embodiment of the present application;

[0040] Figure 6 is a flowchart of an image processing method in one embodiment of the present application;

[0041] Figure 7 is a flowchart of an image processing method in one embodiment of the present application;

[0042] Figure 8 It is a structural diagram of an image processing device in one embodiment of the present application. DETAILED DESCRIPTION

[0043] The terms used in the embodiments of this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The singular forms "a," "the," and "the" used in this application and claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to any or all possible combinations of one or more associated listed items.

[0044] It should be understood that although the terms first, second, third, etc. may be used to describe various information in the embodiments of the present application, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" used may also be interpreted as "at the time of" or "when" or "in response to determining".

[0045] See also Figure 1The figure shows the structure of the endoscope system, which may include: an endoscope, a light source, a camera system host, a display device and a storage device, wherein the display device and the storage device are external devices. Figure 1 This is just an example of an endoscope system and is not intended to be limiting.

[0046] For example, an endoscope can be inserted into a designated location (i.e., a location to be inspected, which is an area within the patient that needs to be inspected, and there is no restriction on this designated location) within a target object (e.g., a patient or other subject), capture an image of the designated location within the target object, and output the image of the designated location within the target object to a display device and a storage device. A user (e.g., a medical professional, etc.) can observe the image displayed on the display device to inspect bleeding sites, tumor sites, abnormal sites, etc. at the designated location within the target object. The user can access the images stored in the storage device for postoperative review and surgical training.

[0047] The endoscope can capture images of a specified location inside a target object and input the images to the camera system host. The light source can provide light for the endoscope, that is, emit illumination light from the front end of the endoscope, so that the endoscope can capture relatively clear images of the inside of the target object. After receiving the image, the camera system host can input the image to a storage device, which stores the image. In subsequent processes, the user can access the image in the storage device, or access the video (a video composed of a large number of images) in the storage device. After receiving the image, the camera system host can also input the image to a display device, which displays the image. The user can observe the image displayed by the display device in real time.

[0048] See also Figure 2 The figure shows a functional structure diagram of an endoscope system. The endoscope may include a camera optical system, an imaging unit, a processing unit and an operating unit. The camera optical system is used to focus the light from the observation site, and the camera optical system is composed of one or more lenses. The imaging unit is used to perform photoelectric conversion on the light received by each pixel to generate image data. The imaging unit is composed of sensors such as CMOS (Complementary Metal Oxide Semiconductor) or CCD (Charge Coupled Device). The processing unit is used to convert the image data into a digital signal and send the converted digital signal (such as the pixel value of each pixel) to the camera system host. The operating unit may include but is not limited to switches, buttons and touch panels, etc., which are used to receive indication signals for the switching action of the endoscope, indication signals for the switching action of the light source, etc., and output the indication signals to the camera system host.

[0049] The light source may include an illumination control unit and an illumination unit. The illumination control unit is configured to receive an instruction signal from the camera system host and control the illumination unit to provide illumination light to the endoscope based on the instruction signal.

[0050] The camera system host is used to process the image data received from the endoscope and transmit it to the display device and storage device. The display device and storage device are external devices of the camera system host.

[0051] The camera system host may include an image input unit, an image processing unit, an intelligent processing unit, a video encoding unit, a control unit, and an operating unit. The image input unit is used to receive signals from the endoscope and transmit the received signals to the image processing unit. The image processing unit is used to perform ISP (Image Signal Processing) operations on the images input by the image input unit, including but not limited to brightness conversion, sharpening, fluorescence staining, scaling, etc. The processed images are transmitted to the intelligent processing unit, the video encoding unit, or a display device. The intelligent processing unit performs intelligent analysis on the images, including but not limited to scene classification based on deep learning, instrument head detection, gauze detection, and fog classification. The processed images are transmitted to the image processing unit or the video encoding unit. The image processing unit processes the processed images by the intelligent processing unit in ways that include but are not limited to brightness conversion, frame stacking, and scaling. The video encoding unit is used to encode and compress the images and transmit them to a storage device. The control unit is used to control the various modules of the endoscope, including but not limited to the lighting mode, image processing mode, intelligent processing mode, and video encoding mode. The input unit may include but is not limited to a switch, a button, and a touch panel, and is used to receive an external instruction signal and output the received instruction signal to the control unit.

[0052] In actual use, it was found that visible light images are not able to distinguish diseased tissue (such as cancerous tissue, etc.) sufficiently, and the doctor's experience is often required to judge and remove the diseased tissue, which can easily lead to problems such as excessive removal of normal tissue or incomplete removal of diseased tissue. In order to distinguish normal tissue from diseased tissue, in one possible embodiment, a white light endoscope and a fluorescent endoscope can be inserted into a specified position inside the target object. The white light endoscope collects images of the specified position inside the target object, and the fluorescent endoscope also collects images of the specified position inside the target object. For the convenience of distinction, the images collected by the white light endoscope are called visible light images (also called white light images), and the images collected by the fluorescent endoscope are called fluorescent images.

[0053] Among them, when a fluorescent contrast agent, such as ICG (Indocyanine Green), is injected into a designated location inside the target object, the fluorescent contrast agent is more absorbed by the diseased tissue. When the excitation light is irradiated on the diseased tissue, the fluorescent contrast agent will produce fluorescence, allowing the fluorescent endoscope to capture fluorescent images. Based on the fluorescent images, normal tissue and diseased tissue can be distinguished, helping medical staff to accurately distinguish diseased tissue.

[0054] The visible light image can be an image generated based on visible light, see Figure 3A The figure is a schematic diagram of a visible light image of a specified position inside a target object. The fluorescence image can be an image generated based on fluorescence, see Figure 3B , which is a schematic diagram of a fluorescent image of a specified location inside a target object.

[0055] In summary, it can be seen that during the actual use of the endoscope system, visible light images and fluorescence images of a specified location within the target object can be collected. That is, the camera system host can obtain visible light images and fluorescence images. After obtaining the visible light image and fluorescence image, a fused image can be generated based on the visible light image and the fluorescence image. The fused image can clearly display normal tissue and diseased tissue at the specified location within the target object, and based on the fused image, normal tissue and diseased tissue at the specified location within the target object can be distinguished.

[0056] When collecting fluorescent images of specified locations inside the target object, there are multiple development modes, and the forms of fluorescent images collected in different development modes are different. For example, during use, a fluorescence endoscope is divided into a positive development mode (positive development mode can also be called positive development mode) and a negative development mode (negative development mode can also be called negative development mode). In the positive development mode, the location of the diseased tissue shows fluorescence, that is, the developed area of ​​the fluorescent image corresponds to the diseased tissue. In the negative development mode, the location outside the diseased tissue shows fluorescence, and the location of the diseased tissue does not show fluorescence, that is, the non-developed area of ​​the fluorescent image corresponds to the diseased tissue.

[0057] In the positive development mode, a fluorescent contrast agent injection method that matches the positive development mode needs to be used. Based on the injection method of the fluorescent contrast agent, it is necessary to ensure that the diseased tissue is visualized while the normal tissue is not. In the negative development mode, a fluorescent contrast agent injection method that matches the negative development mode needs to be used. Based on the injection method of the fluorescent contrast agent, it is necessary to ensure that the diseased tissue is not visualized while the normal tissue is visualized.

[0058] See also Figure 3CAs shown, the fluorescence image on the upper side is a fluorescence image in positive development mode. The developed area of ​​the fluorescence image corresponds to the diseased tissue, and the undeveloped area of ​​the fluorescence image corresponds to normal tissue. By fusing the visible light image and the fluorescence image, a fused image is generated. In the fused image, the area corresponding to the diseased tissue is the developed area. This area can be stained, making it easier for medical staff to observe the corresponding area of ​​the diseased tissue by dyeing it with a corresponding color.

[0059] The fluorescence image below is a negative development mode fluorescence image. The developed areas of the fluorescence image correspond to normal tissue, while the undeveloped areas correspond to diseased tissue. By fusing the visible light image and the fluorescence image, a fused image is generated. In the fused image, the areas corresponding to the diseased tissue are undeveloped, while the areas corresponding to normal tissue are developed. The areas corresponding to the normal tissue can be stained, thereby highlighting the diseased tissue.

[0060] Because there are multiple development modes, such as positive and negative, medical personnel must manually configure the development mode to use based on their experience to capture fluorescence images. Once a development mode is determined, it cannot be adjusted. This configuration can result in poor fluorescence image quality, which in turn can lead to poor fusion image quality, inability to clearly display normal and diseased tissue at specific locations within the target object, and a poor user experience.

[0061] For example, if the imaging mode configured by the medical staff is the negative imaging mode, and in the actual scenario, the area of ​​the lesion tissue at the specified location inside the target object is small, then when the negative imaging mode is used, refer to Figure 3D As shown, a large area of ​​background will be stained, making it difficult to observe the diseased tissue.

[0062] In response to the above findings, an image processing method is proposed in an embodiment of the present application, which can automatically adjust the development mode. That is, after the development mode is set, the development mode can be automatically adjusted according to the visible light image and fluorescent image of the actual scene, and the development mode can be switched in real time, thereby effectively switching between the positive development mode and the negative development mode (that is, switching the positive development mode to the negative development mode, or switching the negative development mode to the positive development mode). For example, assuming that the development mode configured by the medical staff is the negative development mode, and the area of ​​the diseased tissue at the specified location inside the target object is small, then the negative development mode can be switched to the positive development mode, and the fluorescent image can be collected in the positive development mode. In this way, the area of ​​the diseased tissue is stained, and the large area of ​​the background is not stained, so that the diseased tissue can be easily observed.

[0063] The technical solutions of the embodiments of the present application are described below in conjunction with specific embodiments.

[0064] In an embodiment of the present application, an image processing method is provided. This method can be applied to a camera system host, such as by an image processing unit or intelligent processing unit of the camera system host, without limitation. In the image processing method, a target development mode corresponding to a fluorescent image can be determined. The target development mode can be the same as a configured initial development mode, or the target development mode can be different from the initial development mode. In the target development mode, a target fluorescent image corresponding to a specified location within the target object is acquired.

[0065] For example, the imaging mode initially configured by a medical professional is called the initial imaging mode. Based on the initial imaging mode, the medical professional can manually switch the imaging mode to the target imaging mode. The switched imaging mode is called the target imaging mode. In other words, the medical professional manually switches the initial imaging mode to the target imaging mode. In this case, the target imaging mode and the initial imaging mode can be different. For example, the initial imaging mode can be a positive imaging mode, while the target imaging mode can be a negative imaging mode, or the initial imaging mode can be a negative imaging mode, while the target imaging mode can be a positive imaging mode.

[0066] For another example, a target region of interest corresponding to a specified location within the target object can be determined based on an initial visible light image and an initial fluorescence image corresponding to the specified location within the target object, and a target development mode can be determined based on the characteristic information corresponding to the target region of interest. In other words, the target development mode matches the characteristic information corresponding to the target region of interest, automatically adjusting the development mode based on the visible light image and the fluorescence image of the actual scene, and switching the development mode in real time, thereby effectively switching between a positive development mode and a negative development mode. In this case, the target development mode and the initial development mode can be different or the same. For example, the initial development mode is a positive development mode, while the target development mode is a negative development mode or a positive development mode, or the initial development mode is a negative development mode, while the target development mode is a positive development mode or a negative development mode.

[0067] The following describes the image processing method proposed in the embodiment of the present application in conjunction with specific application scenarios. Figure 4 FIG. 1 is a flow chart of the image processing method, which may include:

[0068] Step 401: Acquire an initial visible light image and an initial fluorescence image corresponding to a specified position inside a target object. The acquisition time of the initial visible light image and the acquisition time of the initial fluorescence image may be the same.

[0069] For example, when it is necessary to collect an image of a specified position inside a target object (such as a patient or other subject) (i.e., the position to be inspected, such as the area inside the patient's body that needs to be inspected), an endoscope (such as a white light endoscope and a fluorescence endoscope) can be inserted into the specified position inside the target object, and the white light endoscope collects a visible light image of the specified position inside the target object (for the convenience of distinction, recorded as an initial visible light image), and the fluorescence endoscope collects a fluorescence image of the specified position inside the target object (for the convenience of distinction, recorded as an initial fluorescence image), and the camera system host obtains the initial visible light image and the initial fluorescence image from the white light endoscope and the fluorescence endoscope. Figure 3A The figure shows the initial visible light image of the specified position inside the target object. Figure 3B , which is a schematic diagram of an initial fluorescence image of a specified location inside a target object.

[0070] For example, since the fluorescent endoscope is divided into a positive development mode and a negative development mode during use, it is necessary to pre-configure the development mode. For example, medical staff configure the development mode based on experience. For the convenience of distinction, the pre-configured development mode is called the initial development mode. The initial development mode can be a positive development mode or a negative development mode, and there is no restriction on this.

[0071] In step 401, an initial fluorescence image corresponding to a specified location within the target object is acquired in an initial development mode. Specifically, the initial fluorescence image is a fluorescence image corresponding to the initial development mode. For example, if the initial development mode is a positive development mode, the developed area of ​​the initial fluorescence image corresponds to diseased tissue, and the undeveloped area of ​​the initial fluorescence image corresponds to normal tissue. If the initial development mode is a negative development mode, the developed area of ​​the initial fluorescence image corresponds to normal tissue, and the undeveloped area of ​​the initial fluorescence image corresponds to diseased tissue.

[0072] Step 402: Determine a target region of interest corresponding to the designated location within the target object based on an initial visible light image and an initial fluorescence image corresponding to the designated location within the target object. Exemplarily, the initial visible light image includes a normal tissue subregion, the initial fluorescence image includes a fluorescence target region, and the target region of interest is the remaining subregion of the fluorescence target region excluding the normal tissue subregion. Exemplarily, the initial fluorescence image is a fluorescence image acquired in an initial development mode. If the initial development mode is a positive development mode, the fluorescence target region is a developed region in the initial fluorescence image (i.e., the developed region corresponds to the diseased tissue). If the initial development mode is a negative development mode, the fluorescence target region is a non-developed region in the initial fluorescence image (i.e., the non-developed region corresponds to the diseased tissue).

[0073] Exemplarily, normal tissue and diseased tissue may exist at a designated location within the target object. When acquiring an initial visible light image of the designated location within the target object using a white light endoscope, the initial visible light image may include a sub-image corresponding to normal tissue, and this sub-image is referred to as a normal tissue sub-region in the initial visible light image. The initial visible light image may include a sub-image corresponding to diseased tissue, and this sub-image is referred to as a diseased tissue sub-region in the initial visible light image. This embodiment does not involve the processing method of the diseased tissue sub-region in the initial visible light image. In addition, when acquiring an initial fluorescent image of the designated location within the target object using a fluorescent endoscope, the initial fluorescent image may include a sub-image corresponding to normal tissue, and this sub-image is referred to as a normal tissue sub-region in the initial fluorescent image. This embodiment does not involve the processing method of the normal tissue sub-region in the initial fluorescent image. The initial fluorescent image may include a sub-image corresponding to diseased tissue, and this sub-image is referred to as a fluorescent target region in the initial fluorescent image. It should be noted that the fluorescent target area may include sub-areas corresponding to the diseased tissue. In actual applications, considering possible errors, the fluorescent target area may include sub-areas corresponding to normal tissue in addition to sub-areas corresponding to the diseased tissue, that is, the fluorescent target area includes sub-areas corresponding to the diseased tissue and sub-areas corresponding to the normal tissue.

[0074] In summary, it can be seen that the initial visible light image may include a normal tissue subregion (i.e., the normal tissue subregion is located in the initial visible light image), and the initial fluorescence image may include a fluorescence target region (i.e., the fluorescence target region is located in the initial fluorescence image). Based on this, the remaining subregion in the fluorescence target region, excluding the normal tissue subregion, can be determined as the target region of interest (the target region of interest is located in the initial fluorescence image). That is, a subregion matching the normal tissue subregion is determined from the fluorescence target region, and this subregion is removed from the fluorescence target region. The remaining subregion is the target region of interest.

[0075] In a possible implementation, the target region of interest may be determined using the following steps:

[0076] Step 4021: Determine a normal tissue sub-region from the initial visible light image.

[0077] For example, the initial visible light image includes a normal tissue sub-region, and the normal tissue sub-region can be determined from the initial visible light image. There is no limitation on the determination method, as long as the normal tissue sub-region can be determined from the initial visible light image. Figure 5A As shown, the image on the left is the initial visible light image, and S01 is the normal tissue sub-region in the initial visible light image, which is recorded as the first region of interest.

[0078] For example, the normal tissue subregion can be determined by inputting an initial visible light image into a trained target network model to obtain a normal tissue subregion in the initial visible light image. The target network model training process may include, but is not limited to, obtaining a sample image and calibration information corresponding to the sample image, the calibration information including a normal tissue subregion in the sample image and a calibration label value corresponding to the normal tissue subregion; and training the initial network model based on the sample image and the calibration information corresponding to the sample image to obtain a trained target network model.

[0079] The above process of the embodiment of the present application is described below in conjunction with specific application scenarios.

[0080] During the actual surgical process, due to the injection problem of fluorescent contrast agent, normal tissues such as blood vessels will also be visualized, and the visualization of normal tissues will interfere with medical staff. Therefore, in this embodiment, it is necessary to identify the visualization interference of normal tissue, that is, it is necessary to identify the normal tissue sub-region from the fluorescent target region. In order to identify the normal tissue sub-region from the fluorescent target region, a target network model can be trained, and the normal tissue sub-region can be determined from the initial visible light image based on the target network model.

[0081] First, an initial network model can be obtained. The initial network model can be a machine learning model, such as a deep learning model or a neural network model, and there is no restriction on the type of the initial network model. The initial network model can be a detection model used to determine a region of interest (a normal tissue subregion) from a visible light image, that is, to provide a segmented region in the visible light image, and the segmented region serves as the region of interest. There is no restriction on the structure of the initial network model, as long as it can achieve the above-mentioned functions.

[0082] Secondly, a large number of sample images can be obtained (each sample image is a visible light image inside the target object), which include positive sample images and negative sample images. For each positive sample image, the user can manually calibrate the normal tissue sub-region, that is, calibrate the calibration frame where the normal tissue sub-region is located in the positive sample image, and give the calibration label value corresponding to the calibration frame, indicating that this calibration frame is the region of interest. If the calibration label value is 0, it means that the calibration frame is the region of interest. In addition, for each negative sample image, the user can manually calibrate the abnormal tissue sub-region, that is, calibrate the calibration frame where the abnormal tissue sub-region is located in the negative sample image, and give the calibration label value corresponding to the calibration frame, indicating that this calibration frame is not the region of interest. If the calibration label value is 1, it means that the calibration frame is not the region of interest.

[0083] To summarize, a large number of sample images can be acquired, and calibration information corresponding to each sample image can be obtained. The calibration information may include the normal tissue sub-region in the sample image (i.e., the normal tissue sub-region is represented by a calibration box) and the calibration label value corresponding to the normal tissue sub-region (such as 0, 1, etc.).

[0084] In practical applications, the target object may contain multiple types of normal tissue. Some types of normal tissue may have imaging interference, such as blood vessels, while other types of normal tissue do not. Therefore, when calibrating the calibration frame of the normal tissue subregion in the positive sample image, the calibration frame of the normal tissue with imaging interference can be calibrated, without calibrating the calibration frame of the normal tissue without imaging interference. Similarly, when calibrating the calibration frame of the abnormal tissue subregion in the negative sample image, the calibration frame of the normal tissue without imaging interference and the calibration frame of the diseased tissue subregion can be calibrated.

[0085] Again, the sample images and the calibration information corresponding to the sample images can be used to train the initial network model, and the trained network model is recorded as the target network model. There is no restriction on this training process.

[0086] In summary, a trained target network model can be obtained. After obtaining the target network model, the target network model can be deployed and used to identify normal tissue sub-regions, such as blood vessel regions, in visible light images. For example, after obtaining an initial visible light image, the initial visible light image can be input into the target network model, which processes the initial visible light image to obtain normal tissue sub-regions in the initial visible light image. There are no restrictions on this process.

[0087] Step 4022: Determine the fluorescent target area from the initial fluorescent image.

[0088] For example, the initial fluorescence image may include a fluorescence target area, and the fluorescence target area can be determined from the initial fluorescence image. There is no limitation on the determination method, as long as the fluorescence target area can be determined from the initial fluorescence image. Figure 5A As shown, the image on the right is the initial fluorescence image, and S02 is the fluorescence target area in the initial fluorescence image, which is recorded as the second region of interest.

[0089] For example, the following steps may be used to determine the fluorescent target area. Of course, the following steps are only examples of determining the fluorescent target area, and there is no limitation on the method of determining the fluorescent target area.

[0090] Step 40221: Based on the pixel value corresponding to each pixel in the initial fluorescent image, select a target pixel that matches the fluorescent target area from all the pixels in the initial fluorescent image.

[0091] For example, for the initial fluorescent image, based on the characteristics of the initial fluorescent image, the brightness of the developed area of ​​the initial fluorescent image is relatively large, and the brightness of the non-developed area of ​​the initial fluorescent image is relatively small. Therefore, based on the pixel value (such as brightness value) corresponding to each pixel point in the initial fluorescent image, the target pixel point that matches the fluorescent target area can be selected by setting an adaptive threshold.

[0092] For example, in the positive development mode, the fluorescent target area corresponds to the development area, and the normal tissue sub-area corresponds to the non-development area, that is, the brightness value of the fluorescent target area is relatively large, and the brightness value of the normal tissue sub-area is relatively small. Therefore, the sub-area with a large brightness value can be used as the fluorescent target area. In the negative development mode, the fluorescent target area corresponds to the non-development area, and the normal tissue sub-area corresponds to the development area. That is, the brightness value of the fluorescent target area is relatively small, and the brightness value of the normal tissue sub-area is relatively large. Therefore, the sub-area with a small brightness value can be used as the fluorescent target area. Based on the above principle, in order to select the target pixel point that matches the fluorescent target area, the following processing can be used:

[0093] Case 1: If the initial fluorescent image is a fluorescent image in a positive development mode, that is, the initial fluorescent image is obtained in the positive development mode, and the positive development mode is used to indicate that the developed area of ​​the initial fluorescent image is a fluorescent target area, then, for each pixel point in the initial fluorescent image, if the pixel value (such as brightness value) corresponding to the pixel point is greater than a first threshold, it is determined that the pixel point is a target pixel point; if the pixel value corresponding to the pixel point is not greater than the first threshold, it is determined that the pixel point is not a target pixel point.

[0094] Obviously, after performing the above processing on all pixels in the initial fluorescence image, target pixels can be selected from all pixels in the initial fluorescence image. The number of target pixels is multiple. In summary, it can be seen that the target pixel is a pixel in the initial fluorescence image whose pixel value is greater than the first threshold.

[0095] For example, a binarization method can be used to determine the target pixel in the initial fluorescent image. Binarization is to set the pixel value of the pixel on the image to 0 or 255. For example, if the pixel value ranges from 0 to 255, an appropriate binarization threshold can be set. If the pixel value of the pixel is not less than the binarization threshold, the pixel value of the pixel is set to 255. If the pixel value of the pixel is less than the binarization threshold, the pixel value of the pixel is set to 0, thereby obtaining a binarized image. The pixel value of the pixel in the binarized image is 0 or 255.

[0096] Based on the above principles, a threshold can be pre-configured as a first threshold. The first threshold can be configured based on experience and is not limited to this. The first threshold can indicate the boundary between the developed and non-developed areas, and can distinguish the developed and non-developed areas in the initial fluorescence image. Based on this, since the brightness value of the fluorescent target area is relatively large, while the brightness value of the normal tissue sub-area is relatively small, the sub-area with the large brightness value can be regarded as the fluorescent target area. Therefore, for each pixel in the initial fluorescence image, if the pixel value corresponding to the pixel is greater than the first threshold, the pixel is determined to be the target pixel. If the pixel value corresponding to the pixel is not greater than the first threshold, the pixel is determined not to be the target pixel.

[0097] Case 2: If the initial fluorescent image is a fluorescent image obtained in a negative development mode, i.e., the initial fluorescent image is obtained in the negative development mode, and the negative development mode is used to indicate that the non-developed area of ​​the initial fluorescent image is a fluorescent target area, then, for each pixel in the initial fluorescent image, if the pixel value (e.g., brightness value) corresponding to the pixel is less than a second threshold, the pixel is determined to be a target pixel; if the pixel value corresponding to the pixel is not less than the second threshold, the pixel is determined not to be a target pixel. After performing the above processing on all pixels in the initial fluorescent image, target pixels can be selected from all pixels in the initial fluorescent image, i.e., pixels in the initial fluorescent image whose pixel value is less than the second threshold.

[0098] For example, a threshold value can be pre-configured as the second threshold value. This second threshold value can be configured based on experience and is not limited to this. The second threshold value can represent the boundary between the developed and non-developed areas, and can distinguish the developed and non-developed areas in the initial fluorescence image. Based on this, since the brightness value of the fluorescent target area is relatively low, while the brightness value of the normal tissue sub-area is relatively high, the sub-area with a low brightness value can be regarded as the fluorescent target area. Therefore, for each pixel in the initial fluorescence image, if the pixel value corresponding to the pixel is less than the second threshold value, the pixel is determined to be the target pixel. If the pixel value corresponding to the pixel is not less than the second threshold value, the pixel is determined not to be the target pixel.

[0099] Step 40222: Obtain the connected domain consisting of all target pixels.

[0100] For example, after the target pixels are determined from the initial fluorescence image, a connected region detection algorithm can be used to determine the connected domain composed of these target pixels (i.e., the connected region of the target pixels). The number of connected domains can be at least one, and there is no restriction on the process of determining the connected domain.

[0101] Among them, the area composed of adjacent target pixels is called a connected domain. The connected domain detection algorithm refers to finding and marking each connected domain in the initial fluorescence image. In other words, based on the connected domain detection algorithm, each connected domain in the initial fluorescence image can be obtained, and there is no restriction on this.

[0102] Step 40223: Determine the fluorescent target area from the initial fluorescent image based on the connected domain.

[0103] For example, the number of connected domains may be at least one. If the number of connected domains is one, the connected domain is used as the fluorescent target region in the initial fluorescent image. If the number of connected domains is at least two, the connected domain with the largest area is used as the fluorescent target region in the initial fluorescent image. Alternatively, all connected domains are used as the fluorescent target region in the initial fluorescent image. Alternatively, all connected domains are sorted in descending order of area, and the top K connected domains are selected as the fluorescent target regions in the initial fluorescent image, where K is a positive integer, such as 1, 2, 3, etc., without limitation.

[0104] In summary, based on steps 40221 to 40223, the fluorescent target area can be determined from the initial fluorescent image. Figure 5A As shown, S02 is the fluorescent target area in the initial fluorescent image.

[0105] Step 4023: Determine the intersection area of ​​the fluorescent target area and the normal tissue sub-area.

[0106] For example, after determining the fluorescent target region from the initial fluorescent image and the normal tissue subregion from the initial visible light image, the intersection region of the fluorescent target region and the normal tissue subregion can be determined, for example, see Figure 5B As shown, S01 is the normal tissue subregion in the initial visible light image, S02 is the fluorescent target region in the initial fluorescence image, and S03 is the intersection of the fluorescent target region and the normal tissue subregion. For example, a normal tissue subregion can be first determined from the initial fluorescence image. The position of the normal tissue subregion is the same as the position of S01 in the initial visible light image. Then, the intersection of the normal tissue subregion and the fluorescent target region can be determined. The intersection region S03 of the normal tissue subregion and the fluorescent target region can be recorded as the third region of interest.

[0107] Step 4024: Determine the remaining sub-regions in the fluorescent target region in the initial fluorescent image except the intersection region as the target region of interest, that is, the target region of interest in the initial fluorescent image.

[0108] For example, after the fluorescence target region S02 and the intersection region S03 are determined from the initial fluorescence image, the remaining subregions of the fluorescence target region S02 except the intersection region S03 can be determined as the target region of interest. Figure 5B As shown, S04 is the target region of interest in the initial fluorescence image, and the target region of interest in the initial fluorescence image can be recorded as a fourth region of interest.

[0109] In summary, based on steps 4021 to 4024 , the target region of interest, ie, the remaining sub-regions in the fluorescent target region except the normal tissue sub-region, can be determined from the initial fluorescence image.

[0110] Step 403: Determine a target development mode based on the feature information corresponding to the target region of interest.

[0111] Exemplarily, the target development mode is related to characteristic information corresponding to the target region of interest, i.e., the target development mode can be determined based on this characteristic information, and the method for determining this is not limited. For example, the characteristic information corresponding to the target region of interest may include, but is not limited to: the area value of all pixels in the target region of interest, the average value of the pixel values ​​of all pixels in the target region of interest (such as the average brightness value), the total number of all pixels in the target region of interest (referred to as the first number), etc. There is no limitation on this characteristic information, as long as the target development mode can be determined based on the characteristic information corresponding to the target region of interest. For example, the characteristic information may also include the distribution and change rate of the target region of interest.

[0112] For example, if the characteristic information is the area value of all pixels within the target region of interest, then if the area value is less than a preset area threshold, it can be determined that the target development mode is a positive development mode; if the area value is not less than the preset area threshold, it can be determined that the target development mode is a negative development mode. Alternatively, if the area value is less than the preset area threshold, it can be determined that the target development mode is a negative development mode; if the area value is not less than the preset area threshold, it can be determined that the target development mode is a positive development mode.

[0113] For another example, if the characteristic information is the average brightness value of all pixels within the target region of interest, then if the average brightness value is less than a preset brightness threshold, the target development mode is determined to be a positive development mode; if the average brightness value is not less than the preset brightness threshold, the target development mode is determined to be a negative development mode. Alternatively, if the average brightness value is less than the preset brightness threshold, the target development mode is determined to be a negative development mode; if the average brightness value is not less than the preset brightness threshold, the target development mode is determined to be a positive development mode.

[0114] For another example, if the characteristic information is a first number of all pixels within a target region of interest, then if the first number is less than a preset number threshold, it can be determined that the target development mode is a positive development mode; if the first number is not less than the preset number threshold, it can be determined that the target development mode is a negative development mode. Alternatively, if the first number is less than the preset number threshold, it can be determined that the target development mode is a negative development mode; if the first number is not less than the preset number threshold, it can be determined that the target development mode is a positive development mode.

[0115] Of course, the above methods are just a few examples of determining the target development mode based on feature information, and there is no limitation to this. For the sake of convenience of description, the feature information is taken as an example of the first number of all pixels in the target area of ​​interest, that is, the target development mode is determined based on the first number of all pixels in the target area of ​​interest.

[0116] Exemplarily, determining the target development mode based on the first number of all pixels in the target region of interest may include, but is not limited to: Method 1: If the first number is less than a number threshold (which can be configured based on experience), the target development mode is determined to be a positive development mode. If the first number is not less than the number threshold, the target development mode is determined to be a negative development mode. Method 2: A second number of all pixels in the initial fluorescent image (i.e., the total number of all pixels in the initial fluorescent image) may be determined, and the target development mode is determined based on the first number and the second number. For example, if the ratio of the first number to the second number is less than a ratio threshold (which can be configured based on experience), the target development mode is determined to be a positive development mode. If the ratio of the first number to the second number is not less than the ratio threshold, the target development mode is determined to be a negative development mode.

[0117] In approach 1, if the first number is less than the number threshold, it indicates that the total number of all pixels within the target ROI is relatively small, i.e., the fluorescent target area is relatively small. Therefore, the target development mode can be determined as a positive development mode, i.e., the relatively small fluorescent target area is developed, making it easier for medical personnel to observe the fluorescent target area. Alternatively, if the first number is not less than the number threshold, it indicates that the total number of all pixels within the target ROI is relatively large, i.e., the fluorescent target area is relatively large, while the normal tissue sub-area is relatively small. Therefore, the target development mode can be determined as a negative development mode, i.e., the relatively small normal tissue sub-area is developed, thereby staining the normal tissue sub-area and highlighting the diseased tissue.

[0118] Similarly, in Method 2, if the ratio of the first number to the second number is less than the ratio threshold, it indicates that the fluorescent target area is relatively small, and the target development mode is determined to be a positive development mode, and the relatively small fluorescent target area is developed. Alternatively, if the ratio of the first number to the second number is not less than the ratio threshold, it indicates that the fluorescent target area is relatively large, while the normal tissue sub-area is relatively small, and the target development mode is determined to be a negative development mode, that is, the relatively small normal tissue sub-area is developed.

[0119] Step 404: In target development mode, a target fluorescence image corresponding to a specified location inside the target object is acquired. For example, a target visible light image and a target fluorescence image corresponding to the specified location inside the target object are acquired. The target visible light image and the target fluorescence image can be acquired at the same time.

[0120] Exemplarily, a target visible light image of a specified position inside the target object can be collected by a white light endoscope, and a target fluorescent image of a specified position inside the target object can be collected by a fluorescent endoscope, and the camera system host can obtain the target visible light image and the target fluorescent image from the white light endoscope and the fluorescent endoscope.

[0121] In step 404, a target fluorescence image corresponding to a specified location within the target object is acquired in the target development mode. Specifically, the target fluorescence image is a fluorescence image corresponding to the target development mode. For example, if the target development mode is a positive development mode, the developed area of ​​the target fluorescence image corresponds to diseased tissue, and the undeveloped area of ​​the target fluorescence image corresponds to normal tissue. If the target development mode is a negative development mode, the developed area of ​​the target fluorescence image corresponds to normal tissue, and the undeveloped area of ​​the target fluorescence image corresponds to diseased tissue.

[0122] In one possible embodiment, acquiring a target fluorescence image corresponding to a specified location within a target object in a target development mode may include, but is not limited to: if the target development mode is different from a currently used initial development mode, switching the initial development mode to the target development mode, and acquiring a target fluorescence image corresponding to the specified location within the target object in the target development mode. If the target development mode is the same as the initial development mode, maintaining the initial development mode unchanged, i.e., directly using the initial development mode as the target development mode, and acquiring a target fluorescence image corresponding to the specified location within the target object in the target development mode.

[0123] For example, if the initial development mode is the positive development mode and the target development mode is the positive development mode, the initial development mode can be maintained as the positive development mode, and in the positive development mode, a target fluorescence image corresponding to a specified position inside the target object is acquired. For another example, if the initial development mode is the positive development mode and the target development mode is the negative development mode, the positive development mode can be switched to the negative development mode, and in the negative development mode, a target fluorescence image corresponding to a specified position inside the target object is acquired. For another example, if the initial development mode is the negative development mode and the target development mode is the negative development mode, the initial development mode can be maintained as the negative development mode, and in the negative development mode, a target fluorescence image corresponding to a specified position inside the target object is acquired. For another example, if the initial development mode is the negative development mode and the target development mode is the positive development mode, the negative development mode can be switched to the positive development mode, and in the positive development mode, a target fluorescence image corresponding to a specified position inside the target object is acquired.

[0124] After obtaining the target visible light image and target fluorescence image corresponding to the specified position inside the target object, the target visible light image and the target fluorescence image can be fused to obtain a fused image and output the fused image. The fusion process will not be described in detail. Figure 3C shown.

[0125] Another image processing method is proposed in the embodiment of this application. Figure 6 As shown, the method includes:

[0126] Step 601 : Acquire an initial visible light image and an initial fluorescence image corresponding to a specified position inside a target object. The initial fluorescence image is a fluorescence image corresponding to an initial development mode.

[0127] Step 602: Determine a target region of interest corresponding to the designated position inside the target object based on the initial visible light image and the initial fluorescence image corresponding to the designated position inside the target object.

[0128] Step 603: Determine a target development mode based on the feature information corresponding to the target region of interest.

[0129] Step 604 : In the target development mode, a target fluorescent image corresponding to a designated position inside the target object is acquired, and a target visible light image corresponding to the designated position inside the target object is acquired.

[0130] Illustratively, steps 601 to 604 may refer to steps 401 to 404 .

[0131] Step 605: Generate a new fluorescence image based on the target fluorescence image. For example, based on the target visible light image and the target fluorescence image, the following steps can be used to generate a new fluorescence image:

[0132] Step 6051: Determine a normal tissue sub-region from the target visible light image.

[0133] Step 6052: Determine the fluorescent target area from the target fluorescent image.

[0134] Step 6053: Determine the intersection area between the fluorescent target area and the normal tissue sub-area.

[0135] Exemplarily, steps 6051 to 6053 may refer to steps 4021 to 4023, and the initial visible light image may be replaced by the target visible light image, and the initial fluorescence image may be replaced by the target fluorescence image.

[0136] Step 6054: remove the intersection region in the target fluorescence image to obtain a new fluorescence image, that is, use the target fluorescence image after the intersection region is removed as the new fluorescence image.

[0137] Exemplarily, the removal process may include, but is not limited to, brightness attenuation and / or brightness remapping. For example, brightness attenuation may be performed on the intersection region in the target fluorescence image to obtain a new fluorescence image; or brightness remapping may be performed on the intersection region in the target fluorescence image to obtain a new fluorescence image; or brightness attenuation and brightness remapping may be performed on the intersection region in the target fluorescence image to obtain a new fluorescence image. Of course, the above are only a few examples of removal processes and are not intended to be limiting.

[0138] Step 606: Fuse the new fluorescence image and the target visible light image to obtain a fused image. For example, based on the target visible light image and the new fluorescence image, the fused image can be generated using the following steps:

[0139] Step 6061: Contrast enhancement is performed on the new fluorescence image to obtain an enhanced fluorescence image (i.e., a contrast-enhanced fluorescence image). For example, after obtaining the new fluorescence image, contrast enhancement can be performed on the new fluorescence image to enhance the contrast of the fluorescence image. Contrast enhancement methods may include, but are not limited to, histogram averaging, local contrast enhancement, and the like. There is no limitation on the contrast enhancement method.

[0140] Step 6062: Color mapping is performed on the enhanced fluorescence image to obtain a mapped fluorescence image (i.e., a color-mapped fluorescence image). For example, after obtaining the contrast-enhanced fluorescence image, color mapping can be performed on the fluorescence image, i.e., the fluorescence image can be dyed to obtain a color-mapped fluorescence image. This embodiment does not limit the color mapping method. For example, when dyeing the fluorescence image, different fluorescence brightness levels in the fluorescence image can correspond to different hues and saturations.

[0141] Step 6063: Fuse the mapped fluorescence image and the target visible light image to obtain a fused image. The fused image is the final output image, which is an image obtained by staining the target visible light image.

[0142] For example, the mapped fluorescence image can be superimposed on the target visible light image for display, thereby obtaining the fused image, thereby dyeing the target visible light image to achieve the effect of image development. The obtained fused image is the output image, that is, the visible light image after dyeing.

[0143] Exemplarily, the target visible light image may be a target visible light image in an RGB color space, and the mapped fluorescence image may be superimposed on the target visible light image in the RGB color space to obtain a fused image. Alternatively, the target visible light image in the RGB color space may be converted into a target visible light image in a YUV color space, and the mapped fluorescence image may be superimposed on the target visible light image in the YUV color space to obtain a fused image. Alternatively, the target visible light image in the RGB color space may be converted into a target visible light image in a LAB color space, and the mapped fluorescence image may be superimposed on the target visible light image in the LAB color space to obtain a fused image. Alternatively, the target visible light image in the RGB color space may be converted into a target visible light image in an XYZ color space, and the mapped fluorescence image may be superimposed on the target visible light image in the XYZ color space to obtain a fused image.

[0144] It can be seen from the above technical solution that in the embodiment of the present application, the development mode can be automatically adjusted according to the characteristic information corresponding to the target area of ​​interest, so that the development mode matches the characteristic information corresponding to the target area of ​​interest. After setting the development mode, the development mode can be automatically adjusted according to the visible light image and the fluorescent image of the actual scene, and the development mode can be switched in real time, thereby effectively switching the positive development mode and the negative development mode (that is, switching the positive development mode to the negative development mode, or switching the negative development mode to the positive development mode), so that the development mode matches the actual scene, and the fluorescent image has a better effect and can highlight the diseased tissue information. After generating a fused image based on the fluorescent image, the fused image has a better effect and can clearly display the normal tissue and diseased tissue at the specified position inside the target object, and can accurately determine which tissues need to be removed, so that the user experience is better. The above method switches the development mode in real time when the medical staff selects the development mode, and supports automatic switching of the development mode.

[0145] Another image processing method is proposed in the embodiment of this application. Figure 7 As shown, the method includes:

[0146] Step 701: Determine whether the smart development mode is enabled, that is, whether the smart development mode is allowed to be enabled. If not, step 702 may be executed; if yes, step 703 may be executed.

[0147] Step 702: When the intelligent development mode is off, the user is prompted to manually select a target development mode, and the user manually selects the target development mode. For example, when the intelligent development mode is off, a manual switching mode can be activated, and the user can manually switch the development mode in real time. For example, when the initial development mode is the positive development mode, the user manually switches the positive development mode to the negative development mode (i.e., the negative development mode is the target development mode). When the initial development mode is the negative development mode, the user manually switches the negative development mode to the positive development mode (i.e., the positive development mode is the target development mode).

[0148] Step 703: When the intelligent development mode is turned on, the target development mode is determined based on the initial visible light image and the initial fluorescence image corresponding to the specified position inside the target object. For example, the initial visible light image and the initial fluorescence image are obtained, and the target region of interest is determined based on the initial visible light image and the initial fluorescence image, and the target development mode is determined based on the feature information corresponding to the target region of interest. For example, the target development mode is determined based on the initial visible light image and the initial fluorescence image corresponding to the specified position inside the target object. The implementation method can be seen in Figure 4 or Figure 6 As shown, no further details will be given here.

[0149] Step 704 : In the target development mode, a target fluorescent image corresponding to a designated position inside the target object is acquired, and a target visible light image corresponding to the designated position inside the target object is acquired.

[0150] Based on the same application concept as the above method, an image processing device is proposed in the embodiment of the present application, see Figure 8 FIG. 1 is a schematic structural diagram of the device, which includes:

[0151] The determination module 81 is used to determine a target development mode corresponding to the fluorescent image; wherein the target development mode is the same as the configured initial development mode, or the target development mode is different from the initial development mode; the acquisition module 82 is used to acquire a target fluorescent image corresponding to a specified position inside the target object under the target development mode.

[0152] Exemplarily, when determining the target development mode corresponding to the fluorescence image, the determination module 81 is specifically used to: determine the target region of interest corresponding to the specified position inside the target object based on the initial visible light image and the initial fluorescence image corresponding to the specified position inside the target object; wherein the initial visible light image includes a normal tissue sub-region, the initial fluorescence image includes a fluorescence target region, and the target region of interest is the remaining sub-region of the fluorescence target region except the normal tissue sub-region; wherein the initial fluorescence image is a fluorescence image acquired under the initial development mode; if the initial development mode is a positive development mode, the fluorescence target region is a development region, and if the initial development mode is a negative development mode, the fluorescence target region is a non-development region; and determine the target development mode based on the characteristic information corresponding to the target region of interest.

[0153] Exemplarily, when determining the target region of interest corresponding to the specified position inside the target object based on the initial visible light image and the initial fluorescence image corresponding to the specified position inside the target object, the determination module 81 is specifically used to: determine the normal tissue sub-region from the initial visible light image; determine the fluorescence target region from the initial fluorescence image; determine the intersection region of the fluorescence target region and the normal tissue sub-region, and determine the remaining sub-regions of the fluorescence target region except the intersection region as the target region of interest.

[0154] Exemplarily, when determining the fluorescent target area from the initial fluorescent image, the determination module 81 is specifically used to: select target pixel points that match the fluorescent target area from all pixel points of the initial fluorescent image based on the pixel value corresponding to each pixel point in the initial fluorescent image; obtain a connected domain composed of all target pixel points, and determine the fluorescent target area from the initial fluorescent image based on the connected domain.

[0155] Exemplarily, the determination module 81 is specifically used to select a target pixel point that matches the fluorescent target area from all the pixels of the initial fluorescent image based on the pixel value corresponding to each pixel point in the initial fluorescent image: if the initial fluorescent image is a fluorescent image under a positive development mode, when the pixel value corresponding to the pixel point in the initial fluorescent image is greater than a first threshold, determine that the pixel point is a target pixel point; or, if the initial fluorescent image is a fluorescent image under a negative development mode, when the pixel value corresponding to the pixel point in the initial fluorescent image is less than a second threshold, determine that the pixel point is a target pixel point.

[0156] Exemplarily, if the characteristic information is a first number of all pixels in the target region of interest, the determination module 81 is specifically used to determine the target development mode based on the characteristic information corresponding to the target region of interest: if the first number is less than a number threshold, determine that the target development mode is a positive development mode; if the first number is not less than the number threshold, determine that the target development mode is a negative development mode; or, if the ratio of the first number to the second number of all pixels in the initial fluorescent image is less than a ratio threshold, determine that the target development mode is a positive development mode; if the ratio of the first number to the second number is not less than the ratio threshold, determine that the target development mode is a negative development mode.

[0157] Exemplarily, when the acquisition module 82 acquires the target fluorescence image corresponding to the specified position inside the target object in the target development mode, it is specifically used to: if the target development mode is different from the configured initial development mode, switch the initial development mode to the target development mode, and acquire the target fluorescence image corresponding to the specified position inside the target object in the target development mode.

[0158] Exemplarily, the acquisition module 82, in the target development mode, is further used, after acquiring the target fluorescence image corresponding to the specified position inside the target object, to: determine a normal tissue sub-region from the target visible light image corresponding to the specified position inside the target object; determine a fluorescent target region from the target fluorescence image; determine an intersection region between the fluorescent target region and the normal tissue sub-region, and remove the intersection region in the target fluorescence image to obtain a new fluorescence image; and fuse the new fluorescence image and the target visible light image to obtain a fused image.

[0159] Exemplarily, the acquisition module 82 removes the intersection area in the target fluorescence image to obtain a new fluorescence image by: performing brightness attenuation processing on the intersection area to obtain the new fluorescence image; or performing brightness remapping processing on the intersection area to obtain the new fluorescence image; or performing brightness attenuation processing and brightness remapping processing on the intersection area to obtain a new fluorescence image.

[0160] Exemplarily, the acquisition module 82 fuses the new fluorescence image and the target visible light image to obtain a fused image, which is specifically used to: perform contrast enhancement on the new fluorescence image to obtain an enhanced fluorescence image; perform color mapping on the enhanced fluorescence image to obtain a mapped fluorescence image; and fuse the mapped fluorescence image and the target visible light image to obtain a fused image.

[0161] Based on the same application concept as the above method, an image processing device (i.e., a camera system host) is also proposed in an embodiment of the present application. The image processing device may include: a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor; the processor is used to execute the machine-executable instructions to implement the image processing method disclosed in the above example of the present application.

[0162] Based on the same application concept as the above method, an embodiment of the present application also provides a machine-readable storage medium, on which a number of computer instructions are stored. When the computer instructions are executed by a processor, the image processing method disclosed in the above example of the present application can be implemented.

[0163] The machine-readable storage medium may be any electronic, magnetic, optical, or other physical storage device that may contain or store information, such as executable instructions, data, and the like. For example, the machine-readable storage medium may be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, a storage drive (such as a hard disk drive), a solid-state drive, any type of storage disk (such as a CD, DVD, etc.), or similar storage media, or a combination thereof.

[0164] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email transceiver, game console, tablet computer, wearable device, or any combination of these devices.

[0165] For the convenience of description, the above devices are described as being divided into various units according to their functions. Of course, when implementing this application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0166] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the embodiments of the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0167] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0168] Furthermore, these computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0169] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable device to implement the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0170] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Determining a target development mode corresponding to the fluorescent image; wherein the target development mode is the same as the configured initial development mode, or the target development mode is different from the initial development mode; In the target development mode, obtaining a target fluorescence image corresponding to a specified position inside the target object; Wherein, determining the target development mode corresponding to the fluorescent image includes: Determining a target region of interest corresponding to the specified position inside the target object based on an initial visible light image and an initial fluorescence image corresponding to the specified position inside the target object; wherein the initial visible light image includes a normal tissue subregion, the initial fluorescence image includes a fluorescence target region, and the target region of interest is a remaining subregion of the fluorescence target region excluding the normal tissue subregion; wherein the initial fluorescence image is a fluorescence image acquired under the initial development mode; if the initial development mode is a positive development mode, the fluorescence target region is a development region, and if the initial development mode is a negative development mode, the fluorescence target region is a non-development region; A target development mode is determined based on feature information corresponding to the target region of interest.

2. The method according to claim 1, characterized in that The determining of a target region of interest corresponding to a designated position inside the target object based on an initial visible light image and an initial fluorescence image corresponding to the designated position inside the target object includes: determining the normal tissue sub-region from the initial visible light image; determining the fluorescent target area from the initial fluorescent image; An intersection region between the fluorescent target region and the normal tissue sub-region is determined, and the remaining sub-regions in the fluorescent target region except the intersection region are determined as the target region of interest.

3. The method according to claim 2, characterized in that Determining the fluorescent target area from the initial fluorescent image includes: Based on the pixel value corresponding to each pixel in the initial fluorescent image, selecting a target pixel that matches the fluorescent target area from all the pixels in the initial fluorescent image; A connected domain consisting of all target pixels is obtained, and the fluorescent target area is determined from the initial fluorescent image based on the connected domain.

4. The method according to claim 3, characterized in that The step of selecting a target pixel point that matches the fluorescent target area from all pixels of the initial fluorescent image based on a pixel value corresponding to each pixel point in the initial fluorescent image includes: If the initial fluorescent image is a fluorescent image in a positive development mode, when a pixel value corresponding to a pixel point in the initial fluorescent image is greater than a first threshold, the pixel point is determined to be a target pixel point; or If the initial fluorescent image is a fluorescent image in a negative development mode, when a pixel value corresponding to a pixel point in the initial fluorescent image is less than a second threshold, the pixel point is determined to be a target pixel point.

5. The method according to claim 1, wherein If the characteristic information is a first number of all pixels in the target region of interest, determining the target development mode based on the characteristic information corresponding to the target region of interest includes: If the first number is less than a number threshold, determining that the target development mode is a positive development mode; if the first number is not less than the number threshold, determining that the target development mode is a negative development mode; Alternatively, if the ratio of the first number to the second number of all pixels in the initial fluorescent image is less than a ratio threshold, it is determined that the target development mode is a positive development mode; if the ratio of the first number to the second number is not less than the ratio threshold, it is determined that the target development mode is a negative development mode.

6. The method according to claim 1, characterized in that Acquiring a target fluorescent image corresponding to a specified position inside a target object in the target development mode includes: If the target development mode is different from the configured initial development mode, the initial development mode is switched to the target development mode, and a target fluorescence image corresponding to a designated position inside the target object is acquired in the target development mode.

7. The method according to claim 1, characterized in that After acquiring the target fluorescence image corresponding to the designated position inside the target object in the target development mode, the method further includes: determining a normal tissue subregion from a target visible light image corresponding to a specified position inside the target object; determining a fluorescent target area from the target fluorescent image; determining an intersection region between the fluorescent target region and the normal tissue subregion, and removing the intersection region from the target fluorescent image to obtain a new fluorescent image; The new fluorescence image and the target visible light image are fused to obtain a fused image.

8. The method according to claim 7, characterized in that The removing the intersection area in the target fluorescence image to obtain a new fluorescence image includes: Performing brightness attenuation processing on the intersection area to obtain the new fluorescence image; or, Performing brightness remapping processing on the intersection area to obtain the new fluorescence image; or, The intersection area is subjected to brightness attenuation processing and brightness remapping processing to obtain a new fluorescence image.

9. The method according to claim 7, characterized in that The fusing the new fluorescence image and the target visible light image to obtain a fused image includes: performing contrast enhancement on the new fluorescence image to obtain an enhanced fluorescence image; performing color mapping on the enhanced fluorescence image to obtain a mapped fluorescence image; The mapped fluorescence image and the target visible light image are fused to obtain a fused image.

10. An image processing device, characterized in that: The device comprises: a determination module, configured to determine a target development mode corresponding to a fluorescence image; wherein the target development mode is the same as a configured initial development mode, or the target development mode is different from the initial development mode; wherein, when determining the target development mode corresponding to the fluorescence image, the determination module is specifically configured to: determine a target region of interest corresponding to a specified position inside the target object based on an initial visible light image and an initial fluorescence image corresponding to the specified position inside the target object; wherein the initial visible light image includes a normal tissue subregion, the initial fluorescence image includes a fluorescence target region, and the target region of interest is a subregion remaining in the fluorescence target region except the normal tissue subregion; wherein the initial fluorescence image is a fluorescence image acquired under the initial development mode; if the initial development mode is a positive development mode, the fluorescence target region is a development region, and if the initial development mode is a negative development mode, the fluorescence target region is a non-development region; and determine the target development mode based on feature information corresponding to the target region of interest; The acquisition module is used to acquire a target fluorescent image corresponding to a specified position inside the target object in the target development mode.

11. The device according to claim 10, It is characterized by: in, The determining module determines the target region of interest corresponding to the designated position inside the target object based on the initial visible light image and the initial fluorescence image corresponding to the designated position inside the target object, specifically for: determining the normal tissue subregion from the initial visible light image; determining the fluorescence target region from the initial fluorescence image; determining an intersection region between the fluorescence target region and the normal tissue subregion, and determining the remaining subregions of the fluorescence target region except the intersection region as the target region of interest; The determining module is specifically configured to determine the fluorescent target area from the initial fluorescent image by: selecting target pixels matching the fluorescent target area from all pixels of the initial fluorescent image based on the pixel value corresponding to each pixel in the initial fluorescent image; obtaining a connected domain consisting of all target pixels, and determining the fluorescent target area from the initial fluorescent image based on the connected domain; The determining module is configured to select a target pixel that matches the fluorescent target area from all pixels of the initial fluorescent image based on the pixel value corresponding to each pixel in the initial fluorescent image: if the initial fluorescent image is a fluorescent image in a positive development mode, when the pixel value corresponding to the pixel in the initial fluorescent image is greater than a first threshold, determine that the pixel is a target pixel; or if the initial fluorescent image is a fluorescent image in a negative development mode, when the pixel value corresponding to the pixel in the initial fluorescent image is less than a second threshold, determine that the pixel is a target pixel; Wherein, if the characteristic information is a first number of all pixels in the target region of interest, the determination module is specifically configured to determine the target development mode based on the characteristic information corresponding to the target region of interest: if the first number is less than a number threshold, determine that the target development mode is a positive development mode; if the first number is not less than the number threshold, determine that the target development mode is a negative development mode; or, if a ratio of the first number to a second number of all pixels in the initial fluorescent image is less than a ratio threshold, determine that the target development mode is a positive development mode; if the ratio of the first number to the second number is not less than the ratio threshold, determine that the target development mode is a negative development mode; The acquisition module is configured to acquire, when in the target development mode, a target fluorescence image corresponding to a specified position inside the target object: if the target development mode is different from the configured initial development mode, switch the initial development mode to the target development mode, and acquire, in the target development mode, a target fluorescence image corresponding to the specified position inside the target object; Wherein, the acquisition module, after acquiring the target fluorescence image corresponding to the designated position inside the target object in the target development mode, is further configured to: determine a normal tissue subregion from the target visible light image corresponding to the designated position inside the target object; determine a fluorescent target region from the target fluorescence image; determine an intersection region between the fluorescent target region and the normal tissue subregion, and remove the intersection region from the target fluorescence image to obtain a new fluorescent image; and fuse the new fluorescent image with the target visible light image to obtain a fused image; The acquisition module removes the intersection area from the target fluorescence image to obtain a new fluorescence image by performing brightness attenuation on the intersection area to obtain the new fluorescence image; or performs brightness remapping on the intersection area to obtain the new fluorescence image; or performs brightness attenuation and brightness remapping on the intersection area to obtain the new fluorescence image; Among them, the acquisition module fuses the new fluorescence image and the target visible light image to obtain a fused image, which is specifically used to: perform contrast enhancement on the new fluorescence image to obtain an enhanced fluorescence image; perform color mapping on the enhanced fluorescence image to obtain a mapped fluorescence image; and fuse the mapped fluorescence image and the target visible light image to obtain a fused image.

12. An image processing device, characterized in that: include: a processor and a machine-readable storage medium storing machine-executable instructions capable of being executed by the processor; The processor is configured to execute machine-executable instructions to implement the method steps described in any one of claims 1-9.

Images