Endoscopic Image Processing Method, Device, Storage Medium and Electronic Device

Abstract

The present application provides an endoscopic image processing method, device, storage medium, and electronic device. First, an effective endoscopic image in the endoscopic image to be processed is obtained, then a white light endoscopic image and a fluorescence endoscopic image are obtained based on the effective endoscopic image. After that, information on the target object region in the white light endoscopic image is obtained based on the white light endoscopic image, and finally, the corresponding target object region in the fluorescence endoscopic image is determined based on the target object region information. Among them, the fluorescence image uses fluorescence to outline and display each region in the image, and its display effect is relatively clear and intuitive. Compared with observing the microvessels and glandular structures on the surface of the target object, the method of identifying the target object type by observing and comparing the fluorescence intensity of the target object region and the non-target object region in the fluorescence image is simpler, effectively reducing the difficulty of identifying the target object type, and thus effectively alleviating the technical problem of the large difficulty in identifying the target object type at present.

Description

Technical Field

[0001] The present application relates to the field of medical technology assistance, and particularly to an endoscopic image processing method, device, storage medium and electronic device. Background Art

[0002] Using WLE (White-light endoscopy) to identify abnormal conditions of the body is a commonly used medical method at present.

[0003] However, when the area covered by the target object (i.e., the abnormal area) in the body is small, it is difficult to accurately identify the target object in the body only by WLE. Therefore, currently, blue laser imaging magnifying endoscopy and narrow-band imaging magnifying endoscopy are usually used to observe the microvessels and glandular structures in the body to identify the target object. However, due to the uneven distribution of glands and microvessels, the difficulty of identification is relatively large. Summary of the Invention

[0004] The present application provides an endoscopic image processing method, device, storage medium and electronic device, which are used to alleviate the technical problem of large difficulty in identifying the type of target object.

[0005] In order to solve the above technical problems, the present application provides the following technical solutions:

[0006] The present application provides an endoscopic image processing method, including:

[0007] Obtain an effective endoscopic image in the endoscopic image to be processed;

[0008] Perform classification processing on the effective endoscopic image to obtain a white-light endoscopic image and a fluorescence endoscopic image;

[0009] Perform information extraction processing on the white-light endoscopic image to obtain the target object area information in the white-light endoscopic image;

[0010] Determine the corresponding target object area in the fluorescence endoscopic image according to the target object area information.

[0011] Among them, the step of obtaining an effective endoscopic image in the endoscopic image to be processed includes:

[0012] Input the endoscopic image to be processed into a trained endoscopic image processing network to obtain the features of the endoscopic image to be processed;

[0013] When the features of the endoscopic image to be processed are the same as the features of the effective endoscopic image, determine the endoscopic image to be processed with the features of the endoscopic image to be processed as the effective endoscopic image.

[0014] Among them, the step of inputting the endoscopic image to be processed into the trained endoscopic image processing network to obtain the features of the endoscopic image to be processed includes:

[0015] Input the endoscopic image to be processed into the feature extraction module of the endoscopic image to be processed in the trained endoscopic image processing network, so as to extract the features of the endoscopic image to be processed through the feature extraction module of the endoscopic image to be processed, and obtain the features of the endoscopic image to be processed.

[0016] Among them, the step of classifying the effective endoscopic image to obtain a white light endoscopic image and a fluorescence endoscopic image includes:

[0017] Input the effective endoscopic image into the effective endoscopic image feature extraction module in the trained endoscopic image processing network, so as to extract the features of the effective endoscopic image through the effective endoscopic image feature extraction module, and obtain the features of the effective endoscopic image;

[0018] When the features of the effective endoscopic image are the same as the features of the white light image, determine that the effective endoscopic image with the features of the effective endoscopic image is a white light endoscopic image;

[0019] When the features of the effective endoscopic image are the same as the features of the fluorescence image, determine that the effective endoscopic image with the features of the effective endoscopic image is a fluorescence endoscopic image.

[0020] Among them, the step of extracting information from the white light endoscopic image to obtain the target object region information in the white light endoscopic image includes:

[0021] Input the white light endoscopic image into the white light endoscopic image feature extraction module in the trained endoscopic image processing network, so as to extract the features of the white light endoscopic image through the white light endoscopic image feature extraction module, and obtain the features of the white light endoscopic image;

[0022] When the features of the white light endoscopic image are the same as the features of the target object image, determine that the white light endoscopic image with the features of the white light endoscopic image is the target object image;

[0023] Input the target object image into the image segmentation module in the trained endoscopic image processing network, so as to perform feature segmentation on the target object image through the image segmentation module, and obtain a target object region with target object region features and the position information of the target object region;

[0024] Use the position information of the target object region as the target object region information.

[0025] Among them, the step of determining the corresponding target object area in the fluorescence endoscopy image according to the target object area information includes:

[0026] Input the target object images with all marked target object areas and the fluorescence endoscopy image into the image matching module in the trained endoscopy image processing network, so as to determine the corresponding relationship between each fluorescence endoscopy image and the target object images of each marked target object area through the image matching module;

[0027] Determine the corresponding target object areas in each fluorescence endoscopy image according to the corresponding relationship and the target object area information.

[0028] Among them, after the step of determining the corresponding target object area in the fluorescence endoscopy image according to the target object area information, it further includes:

[0029] Determine the target object type according to the fluorescence brightness parameters of the corresponding target object areas in each fluorescence endoscopy image and the fluorescence brightness parameters of the non-target object areas in each fluorescence endoscopy image.

[0030] An embodiment of the present application further provides an endoscopy image processing device, including:

[0031] An acquisition module, configured to acquire a valid endoscopy image in the to-be-processed endoscopy image;

[0032] A classification module, configured to perform classification processing on the valid endoscopy image to obtain a white light endoscopy image and a fluorescence endoscopy image;

[0033] An information extraction module, configured to perform information extraction processing on the white light endoscopy image to obtain the target object area information in the white light endoscopy image;

[0034] A determination module, configured to determine the corresponding target object area in the fluorescence endoscopy image according to the target object area information.

[0035] An embodiment of the present application further provides a computer-readable storage medium, in which multiple instructions are stored, and the instructions are suitable for being loaded by a processor to execute the steps in any one of the above endoscopy image processing methods.

[0036] An embodiment of the present application further provides an electronic device, including a processor and a memory, the processor is electrically connected to the memory, the memory is used to store instructions and data, and the processor is used to execute the steps in any one of the above endoscopy image processing methods.

[0037] An endoscopic image processing method, apparatus, storage medium, and electronic device are provided in an embodiment of the present application. First, an effective endoscopic image in a to-be-processed endoscopic image is obtained. Then, the effective endoscopic image is classified to obtain a white-light endoscopic image and a fluorescence endoscopic image. After that, information extraction processing is performed on the white-light endoscopic image to obtain target object region information in the white-light endoscopic image. Finally, a corresponding target object region in the fluorescence endoscopic image is determined according to the target object region information. Among them, the fluorescence image uses fluorescence to outline and display each region in the image, and its display effect is relatively clear and intuitive. Moreover, compared with determining the type of the target object by observing the microvessels and glandular structures on the surface of the target object, the method of identifying the type of the target object by observing and comparing the fluorescence intensity of the target object region and the fluorescence intensity of the non-target object region in the fluorescence image is simpler, effectively reducing the difficulty of identifying the type of the target object, thereby effectively alleviating the technical problem of the large difficulty in identifying the type of the current target object. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The following will clearly show the technical solutions and other beneficial effects of the present application by describing the specific embodiments of the present application in detail with reference to the accompanying drawings.

[0039] Figure 1 It is a schematic flowchart of the endoscopic image processing method provided in an embodiment of the present application.

[0040] Figure 2 It is a schematic scene diagram of the endoscopic image processing method provided in an embodiment of the present application.

[0041] Figure 3 It is another schematic flowchart of the endoscopic image processing method provided in an embodiment of the present application.

[0042] Figure 4 It is a schematic structural diagram of the endoscopic image processing apparatus provided in an embodiment of the present application.

[0043] Figure 5 It is a schematic structural diagram of the electronic device provided in an embodiment of the present application.

[0044] Figure 6 It is another schematic structural diagram of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0045] The following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of protection of the present application.

[0046] An endoscopic image processing method, apparatus, storage medium, and electronic device are provided in an embodiment of the present application.

[0047] As Figure 1 shown, Figure 1 is a schematic flowchart of the endoscopic image processing method provided in an embodiment of the present application. The specific process can be as follows:

[0048] 101. Obtain a valid endoscopic image from the endoscopic image to be processed.

[0049] Among them, the endoscopic image to be processed is an endoscopic image for which the type of target object needs to be recognized, and the valid endoscopic image is a clear endoscopic image. Specifically, the endoscopic image is an image taken during an endoscopic examination (i.e., an examination of a hollow organ or tissue). In actual application, due to the variable states of organs and tissues in the human body, it is easy for the endoscopic images taken during the endoscopic examination to be blurred. For example, since there is inflammation in the patient's stomach, a large amount of gastric mucus will be secreted. When performing a gastroscopy on the patient's stomach, since the gastric mucus covers the gastroscope lens, the image taken by the gastroscope is turbid and blurred, and the endoscopic doctor cannot accurately obtain the information of the target object based on the blurred image. Therefore, in order to avoid the above situation, it is necessary to first screen the images taken by the endoscope to obtain clear endoscopic images, so as to facilitate subsequent recognition of the target object based on the clear endoscopic images.

[0050] In this embodiment, the endoscopic image to be processed is input into a trained endoscopic image processing network for processing to obtain the characteristics of the endoscopic image to be processed. When the characteristics of the endoscopic image to be processed are the same as the characteristics of the valid endoscopic image, the endoscopic image to be processed with the characteristics of the endoscopic image to be processed is determined as the valid endoscopic image. Optionally, the trained endoscopic image processing network includes a trained ResNet network, and the characteristics of the endoscopic image to be processed include the resolution characteristics in the endoscopic image to be processed, which are used to reflect the resolution size of the image.

[0051] Specifically, the endoscopic image to be processed is input into the feature extraction module of the trained ResNet network for the endoscopic image to be processed, so as to extract the features of the endoscopic image to be processed through the feature extraction module of the endoscopic image to be processed to obtain the resolution characteristics. When the resolution characteristics are the same as the characteristics of the valid endoscopic image, it means that the resolution of the endoscopic image to be processed meets the resolution size of the clear image. Therefore, the endoscopic image to be processed is determined as the valid endoscopic image.

[0052] For example, the image resolution size for characterizing effective endoscopic image features is greater than or equal to 1024×768. The feature extraction module for the endoscopic image to be processed extracts features from the endoscopic image A to be processed, obtaining resolution features characterized by a resolution of 1024×768. That is, the resolution features reflected by the endoscopic image A to be processed are the same as the effective endoscopic image features. Therefore, the endoscopic image A to be processed is determined as an effective endoscopic image.

[0053] 102. Classify the effective endoscopic images to obtain white light endoscopic images and fluorescence endoscopic images.

[0054] Among them, the white light endoscopic image is an image obtained by shooting with a white light endoscope. The white light endoscope is an endoscope that uses white light as the shooting light source. The fluorescence endoscopic image is a fluorescence angiography image. Specifically, an endoscopic physician can usually identify the position of the target object by observing the color and shape of each region of the organ in the white light endoscopic image. However, when the target object area is small, it is difficult to identify the type of the target object by the naked eye. The fluorescence angiography image uses fluorescence to label the blood flow in the blood vessels. The endoscopic physician can identify the type of the target object by observing the change of the fluorescence brightness in the target object area over time. Even if the target object area is small, the type of the target object can be accurately identified by observing the change of the fluorescence brightness over time. However, the position of the target object cannot be accurately observed only from the fluorescence image. Therefore, in order to more intuitively observe the type of the target object, it is necessary to first determine the position of the target object in the fluorescence endoscopic image.

[0055] In the actual application process, a fluorescent agent (for example, indocyanine green injection) can be intravenously injected into the patient first, and an interference filter that can transmit fluorescence is placed at the tip of the endoscope. Since the interference filter is provided at the tip of the endoscope, the endoscopic physician can synchronously obtain the white light endoscopic image and the fluorescence endoscopic image.

[0056] In this embodiment, in order to ensure the accuracy of subsequent target object type recognition, after determining the effective endoscopic images (including at least one white light endoscopic image and at least one fluorescence endoscopic image), all the effective endoscopic images are input into the effective endoscopic image feature extraction module in the trained endoscopic image processing network to extract features from the effective endoscopic images through the effective endoscopic image feature extraction module, obtaining effective endoscopic image features. When the effective endoscopic image features are the same as the white light image features, the effective endoscopic image with the effective endoscopic image features is determined as a white light endoscopic image; when the effective endoscopic image features are the same as the fluorescence image features, the effective endoscopic image with the effective endoscopic image features is determined as a fluorescence endoscopic image.

[0057] Optionally, the effective endoscopic image features include pixel brightness features for characterizing the image brightness. The white light image features include white light brightness features for characterizing the white light image brightness, and the fluorescence image features include fluorescence brightness features for characterizing the fluorescence image brightness. For example, the image brightness characterized by the effective endoscopic image features extracted from the effective endoscopic image B is 500 cd / m 2 , the white light image brightness characterized by the white light image features is 200 cd / m 2 , and the fluorescence image brightness characterized by the fluorescence image features is 500 cd / m 2 . Therefore, the effective endoscopic image with such effective endoscopic image features is determined to be a fluorescence endoscopic image.

[0058] 103. Information extraction processing is performed on the white light endoscopic image to obtain the target object region information in the white light endoscopic image.

[0059] Among them, the target object region information is information related to the target object position. Specifically, since the morphology and color of each gland and microvessel can be clearly displayed in the white light endoscopic image, the target object region information can be determined in the white light endoscopic image in advance.

[0060] In this embodiment, the target object region can be the target object abnormal region in the white light endoscopic image. Among them, the abnormal region information can be the region where the current region in the white light endoscopic image is different from the surrounding region state, or the region where the state of the current target object region in the white light endoscopic image is different from the preset target object state. For example, if the color of region A in the white light endoscopic image is different from the color of its surrounding region, it is considered an abnormal region. In this scenario, foreign objects can be detected. For example, for a foreign object protruding into the esophagus, the region position of the foreign object can be located by comparing the color of the foreign object with the color of the esophagus.

[0061] Another example is taking the microvessels as an example. If the color of the microvessel region in the white light endoscopic image is different from the color of its surrounding microvessel regions, it is considered an abnormal region, which is different from other regions or the default region color, and it is confirmed that the blood flow rate is abnormal. Another example is taking the glands as an example. If the shape of gland B region in the white light endoscopic image is different from the preset shape of gland B, it is considered an abnormal region.

[0062] Specifically, input the white-light endoscopy image into the white-light endoscopy image feature extraction module in the trained endoscopy image processing network to extract features from the white-light endoscopy image through the white-light endoscopy image feature extraction module, obtaining the white-light endoscopy image features. When the white-light endoscopy image features are the same as the target object image features, determine the white-light endoscopy image with the white-light endoscopy image features as the target object image. Then input the target object image into the image segmentation module in the trained endoscopy image processing network to perform feature segmentation on the target object image through the image segmentation module, obtaining the target object region with the target object region features and the position information of the target object region. Finally, use the position information of the target object region as the target object region information.

[0063] Optionally, the white-light endoscopy image features include the pixel chromaticity features (used to characterize the chromaticity size) of a preset-size region composed of continuous pixel points, and the shape features (used to characterize the shape type of the graph within the preset-size region in the image) of a preset-size region composed of continuous pixel points. The target object image features include the target object chromaticity features and the target object shape features, used to characterize the chromaticity size and shape type of the preset-size region.

[0064] For example, the pixel chromaticity size within the 36 mm (width) × 36 mm (height) region composed of continuous pixel points characterized by the white-light endoscopy image features extracted from the white-light endoscopy image C is (36, 255, 160), and the shape type of the graph is circular. While the chromaticity size of the target object region within the 36 mm (width) × 36 mm (height) region is (36, 255, 160), and the shape type of the target object region is serrated. Since their shape features are different, it is determined that the white-light endoscopy image C is not the target object image.

[0065] Furthermore, to more precisely determine the location of the specific target object in the target object image, input the target object image into the image segmentation module in the trained endoscopy image processing network to perform feature segmentation on the target object region in the target object image through the image segmentation module, obtaining the target object region features. The target object region features characterize the distribution region of the target object. The boundary between the target object region and the non-target object region (i.e., the contour of the target object) can be determined based on the target object region features, and the position information (such as coordinates) of the target object region can be determined based on this boundary.

[0066] For example, input the target object image into the image segmentation module in the trained endoscopy image processing network, obtain the boundary between the target object region and the non-target object region, and determine that the target object region is a closed region composed of the coordinates (56, 20), (56, 80), (62, 13), and (70, 49) based on this boundary.

[0067] 104. Determine the corresponding target object region in the fluorescence endoscopy image according to the target object region information.

[0068] Among them, since the type of the target object can be accurately identified by observing the change of the fluorescence brightness of the target object region in the fluorescence endoscopy image over time, but usually, the position of the target object cannot be accurately observed only from the fluorescence image. Therefore, in order to more intuitively observe the type of the target object, it is necessary to first determine the position of the target object in the fluorescence endoscopy image.

[0069] Specifically, in this embodiment, after determining the position information of the target object region in the above white light endoscopy image (target object image), the endoscopist uses a staining agent to mark the target object region in the human body and connects the marked points. Optionally, the staining agent includes a dye formed by mixing fluorescence and Indian ink. Then, input all the target object images with the marked target object regions and all the fluorescence endoscopy images into the image matching module in the trained endoscopy image processing network, so that the image matching module determines the fluorescence endoscopy image corresponding to the target object image with each marked target object region according to the connected marked points, and then determines the corresponding target object region in the fluorescence endoscopy image according to the target object region information.

[0070] For example, as Figure 2 , the endoscopist uses a staining agent to mark two points beside the target object region 2001 in the human body and connects these two marked points to form a line segment 2002. Then, input the target object image 201 with the marked target object region and all the fluorescence endoscopy images into the image matching module in the trained endoscopy image processing network, so that the image matching module determines the fluorescence endoscopy image 202 corresponding to the target object image 201 according to parameters such as the direction and length of the line segment 2002. Then, determine that the coordinates of the corresponding target object region in the fluorescence endoscopy image 202 are a closed region composed of the coordinates (56, 20), (56, 80), (62, 13), and (70, 49) according to the coordinates of the target object position in the target object image 201, and outline the closed region.

[0071] Further, after the above step 104, it further includes:

[0072] Determine the type of the target object according to the fluorescence brightness parameters of the corresponding target object regions in each fluorescence endoscopy image and the fluorescence brightness parameters of the non-target object regions in each fluorescence endoscopy image.

[0073] Among them, the fluorescence brightness parameter is used to characterize the intensity of fluorescence, and the target object types include malignant target objects and benign target objects. In the above process, the target object area has been outlined in the fluorescence endoscopy image. Since the blood flow velocity in the malignant target object area is slower than that in the benign target object area or the normal area, the increase rate of the fluorescence intensity in the malignant target object area is slower than that in the benign target object area. By comparing the increase rate of the fluorescence intensity of the marked target object area in the fluorescence endoscopy image with time and the increase rate of the fluorescence intensity of the non-target object area with time, it can be determined whether the target object is malignant or benign.

[0074] As Figure 3 shown, Figure 3 is another schematic flowchart of the endoscopic image processing method provided by the embodiment of the present application. The specific process can be as follows:

[0075] 301. Input the endoscopic image to be processed into the trained endoscopic image processing network for classification processing to obtain the features of the endoscopic image to be processed.

[0076] For example, input the endoscopic image A to be processed into the feature extraction module of the trained ResNet network for feature extraction to obtain the features of the endoscopic image to be processed with a resolution of 1024×768.

[0077] 302. When the features of the endoscopic image to be processed are the same as the features of the valid endoscopic image, determine that the endoscopic image to be processed with the features of the endoscopic image to be processed is a valid endoscopic image.

[0078] For example, the resolution size represented by the features of the valid endoscopic image is greater than or equal to 1024×768. Since it is the same as the features of the endoscopic image to be processed with a resolution of 1024×768, it is determined that the endoscopic image A to be processed is a valid endoscopic image.

[0079] 303. Input the valid endoscopic image into the feature extraction module of the trained endoscopic image processing network for feature extraction to obtain the features of the valid endoscopic image.

[0080] For example, input the valid endoscopic image A into the feature extraction module of the trained ResNet network for feature extraction to obtain the features of the valid endoscopic image with an image brightness of 200 cd / m 2 of the valid endoscopic image.

[0081] 304. Determine whether the features of the valid endoscopic image are the same as the features of the white light image or the fluorescence image. If they are the same as the features of the fluorescence image, execute step 305; if they are the same as the features of the white light image, execute step 306.

[0082] For example, if the brightness of the fluorescence image characterized by the fluorescence image feature is 500 cd / m 2 , and the brightness of the white light image characterized by the white light image feature is 200 cd / m 2 , since the brightness of the effective endoscopic image A characterized by the effective endoscopic image feature is equal to the brightness of the white light image characterized by the white light image feature, step 306 is executed;

[0083] 305. Determine that the effective endoscopic image is a fluorescence endoscopic image.

[0084] 306. Determine that the effective endoscopic image is a white light endoscopic image.

[0085] 307. Input the white light endoscopic image into the white light endoscopic image feature extraction module in the trained endoscopic image processing network for feature extraction to obtain the white light endoscopic image feature.

[0086] For example, input the white light endoscopic image A into the white light endoscopic image feature extraction module in the trained endoscopic image processing network for feature extraction to obtain the white light endoscopic image feature that characterizes the chromaticity size within a preset size area as (36, 255, 160) and the shape type of the graph within the preset size area as zigzag.

[0087] 308. When the white light endoscopic image feature is the same as the target object image feature, determine that the white light endoscopic image is the target object image.

[0088] For example, if the target object image feature characterizes the chromaticity size within a 36 mm (width) × 36 mm (height) area as (36, 255, 160) and the shape type as zigzag, it is determined that the white light endoscopic image A is the target object image.

[0089] 309. Input the target object image into the image segmentation module in the trained endoscopic image processing network for feature segmentation to obtain the target object area with the target object area feature and the position information of the target object area.

[0090] For example, input the target object image A into the image segmentation module in the trained endoscopic image processing network for feature segmentation to obtain the boundary between the target object area and the non-target object area, and determine the target object area as a closed area formed by connecting the coordinate points (56, 20), (56, 80), (62, 13), and (70, 49) based on this boundary.

[0091] 310. Input the target object image with the marked target object area and all fluorescence endoscopic images into the image matching module in the trained endoscopic image processing network, so that the image matching module determines the fluorescence endoscopic image corresponding to the target object image with each marked target object area according to the connected marked points.

[0092] For example, as Figure 2 , the target object image 201 with a line segment 2002 formed by two marked points pre-marked beside the target object area 2001 and all fluorescence endoscope images are input into the image matching module in the trained endoscope image processing network. The image matching module determines the fluorescence endoscope image 202 as the fluorescence endoscope image corresponding to the target object image 201 according to the direction and length of the line segment 2002 in the target object image 201.

[0093] 311. Determine the corresponding target object area in the fluorescence endoscope image according to the position information of the target object area.

[0094] For example, it is determined that the target object area 2001 in the fluorescence endoscope image 202 is a closed area formed by connecting the coordinate points (56, 20), (56, 80), (62, 13), and (70, 49).

[0095] 312. Determine the target object type according to the change of the fluorescence intensity of the target object area and the non-target object area in the fluorescence endoscope image over time.

[0096] For example, the increasing speed of the fluorescence intensity of the target object area in the fluorescence endoscope image 202 is slower than that of the non-target object area, so it is determined that the target object is a malignant target object.

[0097] As can be seen from the above, the endoscope image processing method provided by the embodiments of the present application first obtains the effective endoscope images in the to-be-processed endoscope images, then classifies the effective endoscope images to obtain white light endoscope images and fluorescence endoscope images, then performs information extraction processing on the white light endoscope images to obtain the target object area information in the white light endoscope images, and finally determines the corresponding target object area in the fluorescence endoscope image according to the target object area information. Among them, the fluorescence image uses fluorescence to outline and display each area in the image, and its display effect is clearer and more intuitive. Moreover, compared with determining the type of the target object by observing the microvessels and glandular structures on the surface of the target object, the method of identifying the target object type by observing and comparing the fluorescence intensity of the target object area and the non-target object area in the fluorescence image is simpler, effectively reducing the difficulty of identifying the target object type, and thus effectively alleviating the technical problem of the large difficulty in identifying the target object type at present.

[0098] According to the method described in the above embodiments, this embodiment will be further described from the perspective of an endoscope image processing device. The endoscope image processing device can be specifically implemented as an independent entity or integrated in an electronic device.

[0099] Please refer to Figure 4 ,Figure 4 Specifically describe the endoscopic image processing device provided by the embodiments of the present application. The endoscopic image processing device may include: an acquisition module 10, a classification module 20, an information extraction module 30, and a determination module 40, where:

[0100] (1) Acquisition module 10

[0101] The acquisition module 10 is configured to acquire a valid endoscopic image in the endoscopic image to be processed.

[0102] (2) Classification module 20

[0103] The classification module 20 is configured to perform classification processing on the valid endoscopic image to obtain a white light endoscopic image and a fluorescence endoscopic image.

[0104] (3) Information extraction module 30

[0105] The information extraction module 30 is configured to perform information extraction processing on the white light endoscopic image to obtain the target object region information in the white light endoscopic image.

[0106] (4) Determination module 40

[0107] The determination module 40 is configured to determine the corresponding target object region in the fluorescence endoscopic image according to the target object region information.

[0108] In specific implementation, each of the above modules may be implemented as an independent entity, or may be arbitrarily combined and implemented as the same or several entities. For the specific implementation of each of the above modules, reference may be made to the method embodiments described above, which will not be elaborated here.

[0109] As can be seen from the above, the endoscopic image processing device provided by the embodiments of the present application first acquires a valid endoscopic image in the endoscopic image to be processed through the acquisition module 10, then performs classification processing on the valid endoscopic image through the classification module 20 to obtain a white light endoscopic image and a fluorescence endoscopic image, then performs information extraction processing on the white light endoscopic image through the information extraction module 30 to obtain the target object region information in the white light endoscopic image, and finally determines the corresponding target object region in the fluorescence endoscopic image according to the target object region information through the determination module 40. Among them, the fluorescence image uses fluorescence to outline and display each region in the image, and its display effect is relatively clear and intuitive. Moreover, compared with determining whether it is a malignant target object by observing the microvessels and glandular structures on the surface of the target object, the method of observing and comparing the fluorescence intensity of the target object region and the fluorescence intensity of the non-target object region in the fluorescence image to identify the target object type is simpler, effectively reducing the difficulty of identifying the target object type, thereby effectively alleviating the technical problem of the large difficulty in identifying the target object type at present.

[0110] Correspondingly, an endoscopic image processing system according to an embodiment of the present invention includes any endoscopic image processing device provided by the embodiments of the present invention, and this endoscopic image processing device can be integrated into an electronic device.

[0111] Since this endoscopic image processing system can include any endoscopic image processing device provided by the embodiments of the present invention, thus, the beneficial effects achievable by any endoscopic image processing device provided by the embodiments of the present invention can be realized. For details, refer to the previous embodiments and will not be elaborated herein.

[0112] In addition, an embodiment of the present application also provides an electronic device. As Figure 5 shown, the electronic device 500 includes a processor 501 and a memory 502. Among them, the processor 501 is electrically connected to the memory 502.

[0113] The processor 501 is the control center of the electronic device 500, connecting various parts of the entire electronic device through various interfaces and lines. By running or loading application programs stored in the memory 502, and by calling data stored in the memory 502, it executes various functions of the electronic device and processes data, thereby performing overall monitoring of the electronic device.

[0114] In this embodiment, the processor 501 in the electronic device 500 will, according to the following steps, load instructions corresponding to the processes of one or more application programs into the memory 502, and the processor 501 will run the application programs stored in the memory 502 to realize various functions:

[0115] Obtain the effective endoscopic image in the endoscopic image to be processed;

[0116] Perform classification processing on the effective endoscopic image to obtain a white light endoscopic image and a fluorescence endoscopic image;

[0117] Perform information extraction processing on the white light endoscopic image to obtain the target object area information in the white light endoscopic image;

[0118] Determine the corresponding target object area in the fluorescence endoscopic image according to the target object area information.

[0119] Figure 6 Shows the specific structural block diagram of the electronic device provided by the embodiment of the present invention, and this electronic device can be used to implement the endoscopic image processing method provided in the above embodiment.

[0120] The RF circuit 610 is used to receive and transmit electromagnetic waves, realizing the mutual conversion between electromagnetic waves and electrical signals, so as to communicate with a communication network or other devices. The RF circuit 610 may include various existing circuit elements for performing these functions. For example, an antenna, a radio frequency transceiver, a digital signal processor, an encryption / decryption chip, a subscriber identity module (SIM) card, a memory, and so on. The RF circuit 610 can communicate with various networks such as the Internet, an enterprise intranet, a wireless network or communicate with other devices through a wireless network. The above-mentioned wireless network may include a cellular phone network, a wireless local area network or a metropolitan area network. The above-mentioned wireless network can use various communication standards, protocols and technologies, including but not limited to Global System for Mobile Communication (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (WCDMA), Code Division Access (CDMA), Time Division Multiple Access (TDMA), Wireless Fidelity (Wi-Fi) (such as Institute of Electrical and Electronics Engineers standards IEEE802.11a, IEEE 802.11b, IEEE802.11g and / or IEEE 802.11n), Voice over Internet Protocol (VoIP), Worldwide Interoperability for Microwave Access (Wi-Max), other protocols for email, instant messaging and short messages, and any other suitable communication protocols, and may even include those protocols that have not been developed yet.

[0121] The memory 620 can be used to store software programs and modules. The processor 680 executes various functional applications and data processing by running the software programs and modules stored in the memory 620. The memory 620 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memories. In some instances, the memory 620 may further include a memory remotely set relative to the processor 680, and these remote memories can be connected to the electronic device 600 through a network. Examples of the above-mentioned network include but are not limited to the Internet, an enterprise intranet, a local area network, a mobile communication network and their combinations.

[0122] The input unit 630 can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control. Specifically, the input unit 630 can include a touch-sensitive surface 631 and other input devices 632. The touch-sensitive surface 631, also known as a touch display screen or a touchpad, can collect touch operations of a user thereon or nearby (such as operations of the user using any suitable object or accessory such as a finger, a stylus, etc. on or near the touch-sensitive surface 631), and drive corresponding connection devices according to a preset program. Optionally, the touch-sensitive surface 631 can include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the touch position of the user, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into contact coordinates, and then sends it to the processor 680, and can receive and execute commands sent by the processor 680. In addition, various types such as resistive, capacitive, infrared, and surface acoustic wave can be used to implement the touch-sensitive surface 631. In addition to the touch-sensitive surface 631, the input unit 630 can also include other input devices 632. Specifically, the other input devices 632 can include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, etc.

[0123] The display unit 640 can be used to display information input by the user or information provided to the user and various graphical user interfaces of the electronic device 600, and these graphical user interfaces can be composed of graphics, text, icons, videos, and any combination thereof. The display unit 640 can include a display panel 641. Optionally, the display panel 641 can be configured in forms such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode). Further, the touch-sensitive surface 631 can cover the display panel 641. When the touch-sensitive surface 631 detects a touch operation thereon or nearby, it is transmitted to the processor 680 to determine the type of touch event, and then the processor 680 provides a corresponding visual output on the display panel 641 according to the type of touch event. Although in Figure 6 the touch-sensitive surface 631 and the display panel 641 are implemented as two independent components to achieve input and output functions, in some embodiments, the touch-sensitive surface 631 and the display panel 641 can be integrated to achieve input and output functions.

[0124] The electronic device 600 may further include at least one sensor 650, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor. Among them, the ambient light sensor can adjust the brightness of the display panel 641 according to the brightness of the ambient light, and the proximity sensor can turn off the display panel 641 and / or the backlight when the electronic device 600 is moved to the ear. As a kind of motion sensor, the gravity acceleration sensor can detect the magnitude of acceleration in all directions (generally three axes). When stationary, it can detect the magnitude and direction of gravity, and can be used in applications for identifying the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors such as gyroscopes, barometers, hygrometers, thermometers, and infrared sensors that the electronic device 600 may also be configured with, they will not be elaborated here.

[0125] The audio circuit 660, the speaker 661, and the microphone 662 can provide an audio interface between the user and the electronic device 600. The audio circuit 660 can transmit the electrical signal converted from the received audio data to the speaker 661, and the speaker 661 converts it into a sound signal for output; on the other hand, the microphone 662 converts the collected sound signal into an electrical signal, which is received by the audio circuit 660 and then converted into audio data. After the audio data is output to the processor 680 for processing, it is sent through the RF circuit 610 to, for example, another terminal, or the audio data is output to the memory 620 for further processing. The audio circuit 660 may also include an earphone jack to provide communication between the peripheral earphone and the electronic device 600.

[0126] The electronic device 600 can help the user send and receive emails, browse the web, and access streaming media, etc. through the transmission module 670 (such as a Wi-Fi module), which provides the user with wireless broadband Internet access. Although Figure 6 the transmission module 670 is shown, it can be understood that it does not belong to the essential components of the electronic device 600 and can be omitted entirely within the scope of not changing the essence of the invention according to needs.

[0127] The processor 680 is the control center of the electronic device 600, connecting various parts of the entire mobile phone through various interfaces and circuits. By running or executing software programs and / or modules stored in the memory 620, and by invoking the data stored in the memory 620, it executes various functions of the electronic device 600 and processes data, thereby monitoring the mobile phone as a whole. Optionally, the processor 680 may include one or more processing cores; in some embodiments, the processor 680 may integrate an application processor and a modem processor. Among them, the application processor mainly processes the operating system, user interface, application programs, etc., and the modem processor mainly processes wireless communications. It can be understood that the above-mentioned modem processor may not be integrated into the processor 680 either.

[0128] The electronic device 600 also includes a power supply 690 (such as a battery) for supplying power to each component. In some embodiments, the power supply can be logically connected to the processor 680 through a power management system, so as to realize functions such as management of charging, discharging, and power consumption management through the power management system. The power supply 690 may also include any components such as one or more DC or AC power supplies, a recharge system, a power failure detection circuit, a power converter or inverter, and a power status indicator.

[0129] Although not shown, the electronic device 600 may also include a camera (such as a front camera, a rear camera), a Bluetooth module, etc., which will not be elaborated here. Specifically, in this embodiment, the display unit of the electronic device is a touch screen display, and the electronic device also includes a memory, and one or more programs, where one or more programs are stored in the memory and are configured to be executed by one or more processors. One or more programs include instructions for performing the following operations:

[0130] Obtain the effective endoscopic image in the endoscopic image to be processed;

[0131] Perform classification processing on the effective endoscopic image to obtain a white light endoscopic image and a fluorescence endoscopic image;

[0132] Perform information extraction processing on the white light endoscopic image to obtain the target object area information in the white light endoscopic image;

[0133] Determine the corresponding target object area in the fluorescence endoscopic image according to the target object area information.

[0134] Specifically in implementation, the above-mentioned each module can be implemented as an independent entity, or can be combined arbitrarily to be implemented as the same or several entities. For the specific implementation of the above-mentioned each module, reference can be made to the previous method embodiments, which will not be elaborated here.

[0135] Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructions or by controlling relevant hardware through instructions. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor. For this purpose, an embodiment of the present invention provides a storage medium that stores multiple instructions that can be loaded by a processor to execute the steps in any of the endoscopic image processing methods provided by the embodiments of the present invention.

[0136] Among them, the storage medium may include: read-only memory (ROM, Read Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disc, etc.

[0137] Since the instructions stored in the storage medium can execute the steps in any of the endoscopic image processing methods provided by the embodiments of the present invention, the beneficial effects achievable by any of the endoscopic image processing methods provided by the embodiments of the present invention can be achieved. For details, refer to the previous embodiments and will not be elaborated here.

[0138] For the specific implementation of each of the above operations, refer to the previous embodiments and will not be elaborated here.

[0139] In summary, although the present application has been disclosed above with preferred embodiments, the above preferred embodiments are not intended to limit the present application. Those of ordinary skill in the art can make various changes and modifications without departing from the spirit and scope of the present application. Therefore, the protection scope of the present application is subject to the scope defined by the claims.

Claims

1. An endoscopic image processing method, characterized in that, it includes: Obtain the effective endoscopic image in the endoscopic image to be processed; Classify the effective endoscopic image to obtain a white light endoscopic image and a fluorescence endoscopic image; Perform information extraction processing on the white light endoscopic image to obtain the target object area information in the white light endoscopic image; Determine the corresponding target object area in the fluorescence endoscopic image according to the target object area information; Among them, the step of performing information extraction processing on the white light endoscopic image to obtain the target object area information in the white light endoscopic image includes: Input the white light endoscopic image into the white light endoscopic image feature extraction module in the trained endoscopic image processing network, so as to perform feature extraction on the white light endoscopic image through the white light endoscopic image feature extraction module to obtain white light endoscopic image features; When the white light endoscopic image features are the same as the target object image features, determine the white light endoscopic image with the white light endoscopic image features as the target object image; Input the target object image into the image segmentation module in the trained endoscopic image processing network, so as to perform feature segmentation on the target object image through the image segmentation module to obtain a target object area with target object area features and the position information of the target object area; Use the position information of the target object area as the target object area information; The step of determining the corresponding target object area in the fluorescence endoscopic image according to the target object area information includes: Input the target object images with all marked target object areas and the fluorescence endoscopic image into the image matching module in the trained endoscopic image processing network, so as to determine the corresponding relationship between each fluorescence endoscopic image and the target object image with each marked target object area through the image matching module; Determine the corresponding target object area in each fluorescence endoscopic image according to the corresponding relationship and the target object area information.

2. The endoscopic image processing method according to claim 1, characterized in that, The step of obtaining the effective endoscopic image in the endoscopic image to be processed includes: Input the endoscopic image to be processed into the trained endoscopic image processing network to obtain the endoscopic image to be processed features; When the endoscopic image to be processed features are the same as the effective endoscopic image features, determine the endoscopic image to be processed with the endoscopic image to be processed features as the effective endoscopic image.

3. The endoscopic image processing method according to claim 2, characterized in that, The step of inputting the endoscopic image to be processed into the trained endoscopic image processing network to obtain the endoscopic image to be processed features includes: Input the endoscopic image to be processed into the endoscopic image to be processed feature extraction module in the trained endoscopic image processing network, so as to perform feature extraction on the endoscopic image to be processed through the endoscopic image to be processed feature extraction module to obtain the endoscopic image to be processed features.

4. The endoscopic image processing method according to claim 2, characterized in that, The step of classifying the effective endoscopic image to obtain a white light endoscopic image and a fluorescence endoscopic image includes: Input the effective endoscopic image into the effective endoscopic image feature extraction module in the trained endoscopic image processing network, so as to extract features from the effective endoscopic image through the effective endoscopic image feature extraction module to obtain effective endoscopic image features; When the effective endoscopic image features are the same as the white light image features, determine that the effective endoscopic image with the effective endoscopic image features is a white light endoscopic image; When the effective endoscopic image features are the same as the fluorescence image features, determine that the effective endoscopic image with the effective endoscopic image features is a fluorescence endoscopic image.

5. The endoscopic image processing method according to claim 2, wherein, after the step of determining the corresponding target object area in the fluorescence endoscopic image according to the target object area information, further includes: Determine the target object type according to the fluorescence brightness parameters of the corresponding target object areas in each fluorescence endoscopic image and the fluorescence brightness parameters of the non-target object areas in each fluorescence endoscopic image.

6. An endoscopic image processing device, wherein, comprises: An acquisition module for acquiring the effective endoscopic image in the endoscopic image to be processed; A classification module for classifying the effective endoscopic image to obtain a white light endoscopic image and a fluorescence endoscopic image; An information extraction module for extracting information from the white light endoscopic image to obtain the target object area information in the white light endoscopic image; A determination module for determining the corresponding target object area in the fluorescence endoscopic image according to the target object area information; wherein, the step of extracting information from the white light endoscopic image to obtain the target object area information in the white light endoscopic image includes: Input the white light endoscopic image into the white light endoscopic image feature extraction module in the trained endoscopic image processing network, so as to extract features from the white light endoscopic image through the white light endoscopic image feature extraction module to obtain white light endoscopic image features; When the white light endoscopic image features are the same as the target object image features, determine that the white light endoscopic image with the white light endoscopic image features is a target object image; Input the target object image into the image segmentation module in the trained endoscopic image processing network, so as to perform feature segmentation on the target object image through the image segmentation module to obtain a target object area with target object area features and the position information of the target object area; Use the position information of the target object area as the target object area information; The step of determining the corresponding target object area in the fluorescence endoscopic image according to the target object area information includes: Input the target object images with all marked target object areas and the fluorescence endoscopic image into the image matching module in the trained endoscopic image processing network, so as to determine the corresponding relationship between each fluorescence endoscopic image and the target object images with each marked target object area through the image matching module; Determine the corresponding target object regions in each fluorescence endoscopy image according to the corresponding relationship and the target object region information.

7. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a plurality of instructions, and the instructions are adapted to be loaded by a processor to execute the steps in the endoscope image processing method according to any one of claims 1 to 5.

8. An electronic device, characterized in that it includes a processor and a memory, the processor is electrically connected to the memory, the memory is used to store instructions and data, and the processor is used to execute the steps in the endoscope image processing method according to any one of claims 1 to 5.

Images