Bronchoscope image processing method based on computer vision

Abstract

The invention relates to the technical field of medical image processing, and discloses a bronchoscope image processing method based on computer vision. The method comprises the following steps: respectively acquiring image data of the inner wall of the bronchus in visible light and near-infrared bands; constructing a multi-modal feature vector, and extracting texture features by using a gray-level co-occurrence matrix; and inputting the multi-modal features into a three-dimensional convolutional neural network to perform automatic recognition of a lesion area. Compared with the technical problems that in the prior art, bronchoscope image processing is insufficient in tissue detail recognition precision, and especially under the conditions of a complex tissue structure and multi-layer spectral information, accurate recognition of a lesion area cannot be achieved, the method is an innovative method of multi-band imaging and multi-modal feature fusion, and the technical problem that the accuracy of tissue detail recognition cannot be achieved is solved. Three-dimensional lesion segmentation is realized, so that the problems of detail loss and low recognition precision caused by single-band imaging are avoided, and the detection precision of the lesion area in the bronchus is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image processing, and in particular relates to a bronchoscopic image processing method based on computer vision. Background Art

[0002] At present, bronchoscopic image processing technology has significant deficiencies in accurately identifying lesion areas. For example, traditional bronchoscopic imaging mostly relies on imaging data from a single band, such as the visible light band, which makes it difficult to provide sufficient image details when encountering complex bronchial lining tissues or lesion areas. In addition, existing image processing methods generally lack the use of multimodal information such as near-infrared band data and tissue reflection characteristics, which makes it difficult to accurately identify deep tissue information and tiny lesions in the image, especially under conditions of tortuous bronchial linings, uneven illumination, and complex textures. Existing technologies are difficult to achieve accurate three-dimensional segmentation and detection of lesion areas, and cannot fully meet clinical needs. Therefore, there is an urgent need for a method that can fuse information from visible and near-infrared bands under multimodal imaging conditions and combine advanced deep learning models to process bronchoscopic images. This method needs to be able to achieve high-precision lesion area identification and three-dimensional segmentation in a complex endobronchial environment to improve the accuracy and efficiency of bronchoscopic examinations and help doctors observe lesion sites more intuitively. Summary of the invention

[0003] In view of the above-mentioned technical deficiencies, the purpose of the present invention is to propose a bronchoscopic image processing method based on computer vision, aiming to solve the technical problem that bronchoscopic image processing in the prior art is insufficient in accuracy in identifying tissue details, especially under the conditions of complex tissue structures and multi-level spectral information, and cannot achieve accurate identification of lesion areas.

[0004] In order to solve the above technical problems, the present invention adopts the following technical solution: The present invention provides a bronchoscopic image processing method based on computer vision.

[0005] The bronchoscopic image processing method based on computer vision comprises:

[0006] Step S10: k micro-structured light sensors and 1 micro-attitude sensor are installed on the bronchoscope. The micro-structured light sensor is used to collect the distance data from the bronchoscope to different points on the inner wall of the bronchus in real time. The distance from the bronchoscope lens to the i-th position point on the inner wall surface is recorded as d i ; The micro-attitude sensor is used to collect the posture data of the bronchoscope in real time, including the bronchoscope position data (x0, y0, z0) and the current viewing direction data

[0007] Step S20: Based on the bronchoscope posture data and distance di , calculate the three-dimensional coordinates of each point on the bronchial wall, using the formula:

[0008]

[0009] Among them, (x i ,y i , z i ) is the three-dimensional coordinate of the i-th position point on the inner wall surface;

[0010] All the calculated points (x i ,y i , z i ) to form preliminary three-dimensional point cloud data, and perform denoising and filtering on the preliminary three-dimensional point cloud data to obtain three-dimensional point cloud data;

[0011] Step S30: installing a multi-band light source and a corresponding imaging sensor on the bronchoscope, performing imaging in the visible light band of 400-700nm and the near infrared band of 700-900nm respectively, executing steps S10 to S20 in different bands, generating three-dimensional point cloud data in the visible light band and three-dimensional point cloud data in the near infrared band, and each band of point cloud data records the inner wall reflection intensity information of the bronchial inner wall;

[0012] Step S40: for the three-dimensional point cloud data of the visible light band and the near infrared band obtained in step S30, extract the multi-band reflection characteristics of each point, and construct a multi-modal feature vector F = [S 可见光 , S 近红外 ], the multimodal feature vector F is used to generate the multimodal grayscale feature vector H, where S 可见光 is the reflection value in the 400-700nm band, S 近红外 It is the reflection value of 700-900nm band;

[0013] Step S50: Generate visible light and near-infrared pseudo grayscale images according to the multimodal grayscale feature vector H, and use the gray level co-occurrence matrix GLCM to extract texture feature information, including contrast C and entropy E:

[0014]

[0015] Where m and n are gray levels (integers between 0 and 255), represents the gray level in the gray-level co-occurrence matrix, P(m,n) represents the joint probability of a pixel pair with gray levels m and n, C is the contrast, and is the entropy;

[0016] Combine the reflectance value, grayscale value and texture features to form the final multimodal feature vector

[0017] F multi =

[0018] [S 可见光 , S 近红外 , H 可见光 , H 近红外 , C 可见光 , C 近红外 , E 可见光 , E 近红外 ];

[0019] Among them, H 可见光 is the pseudo grayscale value generated by reflection value mapping in the visible light band, H 近红外 is the pseudo grayscale value generated by reflectance mapping in the near-infrared band, C 可见光 is the contrast eigenvalue extracted based on the gray-level co-occurrence matrix in the visible light band, C 近红外 is the contrast eigenvalue extracted based on the gray-level co-occurrence matrix in the near-infrared band, E 可见光 is the entropy eigenvalue extracted based on the gray-level co-occurrence matrix in the visible light band, E 近红外 is the entropy eigenvalue extracted based on the gray-level co-occurrence matrix in the near-infrared band;

[0020] Step S60: The final multimodal feature vector F generated in step S50 is multi The data is input into a three-dimensional convolutional neural network (3DCNN), which extracts features layer by layer by sliding the three-dimensional convolution kernel in the three-dimensional space of the feature vector to identify abnormal areas of the bronchial inner wall.

[0021] After multi-layer convolution and pooling processing, the abnormal area of ​​the bronchial wall is segmented, and the classification label of each volume pixel is output, marked as a lesion volume pixel or a normal volume pixel;

[0022] All lesion volume pixels are marked in a three-dimensional space, and a three-dimensional segmentation mask is generated, wherein the three-dimensional segmentation mask is used to distinguish the lesion area from the normal area.

[0023] Preferably, in step S40, the process of generating the multimodal grayscale feature vector H includes:

[0024] The following formula is used to generate the grayscale values ​​of the visible light band and the near-infrared band:

[0025]

[0026] Among them, H 波段 is the gray value of the corresponding band, S 波段 is the gray value of the corresponding band, S min and S max They are the minimum and maximum values ​​of the reflectance values ​​in the corresponding bands respectively;

[0027] And construct a multimodal grayscale feature vector H = [H可见光 , H 近红外 ].

[0028] Preferably, S min and S max Dynamically adjusted according to the lighting conditions and imaging contrast of the current bronchoscopic image.

[0029] Preferably, in step S30, the multi-band light source is an adjustable light source.

[0030] Preferably, in step S30, the adjustable light source includes an LED array of multiple bands, and light sources of different bands are selected according to different scenes.

[0031] Preferably, in step S50, a sliding window method is used to extract local texture features on the pseudo grayscale image.

[0032] Preferably, in step S60, the model training data of the three-dimensional convolutional neural network 3DCNN is multimodal feature vector sample data with a scale of more than 10,000 samples and corresponding labeled volume pixel sample data.

[0033] The present invention also provides a bronchoscopic image processing system based on computer vision, comprising:

[0034] The data acquisition module is used to install k micro-structured light sensors and 1 micro-attitude sensor on the bronchoscope. The micro-structured light sensor is used to collect the distance data from the bronchoscope to different points on the inner wall of the bronchus in real time. The distance from the bronchoscope lens to the i-th position point on the inner wall surface is recorded as d i ; The micro-attitude sensor is used to collect the posture data of the bronchoscope in real time, including the bronchoscope position data (x0, y0, z0) and the current viewing direction data

[0035] The 3D point cloud generation module is used to generate a 3D point cloud based on the bronchoscope’s position data and distance d i , calculate the three-dimensional coordinates of each point on the bronchial wall, using the formula:

[0036]

[0037] Among them, (x i ,y i , z i ) is the three-dimensional coordinate of the i-th position point on the inner wall surface;

[0038] All the calculated points (x i ,y i , z i ) to form preliminary three-dimensional point cloud data, and perform denoising and filtering on the preliminary three-dimensional point cloud data to obtain three-dimensional point cloud data;

[0039] A multi-band imaging module is used to install a multi-band light source and a corresponding imaging sensor on a bronchoscope, perform imaging in the visible light band of 400-700nm and the near-infrared band of 700-900nm, respectively, and perform steps S10 to S20 in different bands to generate three-dimensional point cloud data in the visible light band and three-dimensional point cloud data in the near-infrared band, wherein each band of point cloud data records the inner wall reflection intensity information of the inner wall of the bronchus;

[0040] The multimodal feature extraction module is used to extract the multi-band reflection characteristics of each point from the three-dimensional point cloud data of the visible light band and the near infrared band obtained in step S30, and construct a multimodal feature vector F = [S 可见光 , S 近红外 ], the multimodal feature vector F is used to generate the multimodal grayscale feature vector H, where S 可见光 is the reflection value in the 400-700nm band, S 近红外 It is the reflection value of 700-900nm band;

[0041] The texture feature analysis module is used to generate visible light and near-infrared pseudo grayscale images based on the multimodal grayscale feature vector H, and use the gray level co-occurrence matrix GLCM to extract texture feature information, including contrast C and entropy E:

[0042]

[0043] Where m and n are gray levels (integers between 0 and 255), represents the gray level in the gray-level co-occurrence matrix, P(m,n) represents the joint probability of a pixel pair with gray levels m and n, C is the contrast, and is the entropy;

[0044] Combine the reflectance value, grayscale value and texture features to form the final multimodal feature vector

[0045] F multi =

[0046] [S 可见光 , S 近红外 , H 可见光 , H 近红外 , C 可见光 , C 近红外 , E 可见光 , E 近红外 ];

[0047] Among them, H 可见光 is the pseudo grayscale value generated by reflection value mapping in the visible light band, H 近红外 is the pseudo grayscale value generated by reflectance mapping in the near-infrared band, C 可见光is the contrast eigenvalue extracted based on the gray-level co-occurrence matrix in the visible light band, C 近红外 is the contrast eigenvalue extracted based on the gray-level co-occurrence matrix in the near-infrared band, E 可见光 is the entropy eigenvalue extracted based on the gray-level co-occurrence matrix in the visible light band, E 近红外 is the entropy eigenvalue extracted based on the gray-level co-occurrence matrix in the near-infrared band;

[0048] The lesion region marking module is used to convert the final multimodal feature vector F generated in step S50 into miltu The data is input into a three-dimensional convolutional neural network (3DCNN), which extracts features layer by layer by sliding the three-dimensional convolution kernel in the three-dimensional space of the feature vector to identify abnormal areas of the bronchial inner wall.

[0049] After multi-layer convolution and pooling processing, the abnormal area of ​​the bronchial wall is segmented, and the classification label of each volume pixel is output, marked as a lesion volume pixel or a normal volume pixel;

[0050] All lesion volume pixels are marked in a three-dimensional space, and a three-dimensional segmentation mask is generated, wherein the three-dimensional segmentation mask is used to distinguish the lesion area from the normal area.

[0051] The present invention also provides a bronchoscopic image processing device based on computer vision, comprising a memory, a processor, and a bronchoscopic image processing program based on computer vision stored in the memory and executable on the processor, wherein the bronchoscopic image processing program based on computer vision implements the bronchoscopic image processing method based on computer vision when executed by the processor.

[0052] The present invention also provides a computer program product, comprising a bronchoscopic image processing program based on computer vision, wherein the bronchoscopic image processing program based on computer vision, when executed by a processor, implements the bronchoscopic image processing method based on computer vision.

[0053] The beneficial effect of the present invention is that compared with the prior art that bronchoscopic image processing is not accurate enough in identifying tissue details, especially under complex tissue structures and multi-level spectral information conditions, and cannot achieve the technical problem of accurate identification of lesion areas, the present application realizes three-dimensional lesion segmentation through the innovative method of multi-band imaging and multi-modal feature fusion, thereby avoiding the problems of detail loss and low recognition accuracy caused by single-band imaging, and improving the detection accuracy of endobronchial lesion areas. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0055] Figure 1 The figure is a flowchart of a first embodiment of a bronchoscopic image processing method based on computer vision of the present invention.

[0056] Figure 2 The present invention is a schematic diagram of a device for a bronchoscopic image processing method based on computer vision. DETAILED DESCRIPTION

[0057] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] Embodiment 1: Figure 1 , which is a flow chart of the first embodiment of the bronchoscopic image processing method based on computer vision of the present invention, proposes the first embodiment of the bronchoscopic image processing method based on computer vision of the present invention.

[0059] In a first embodiment, the bronchoscopic image processing method based on computer vision includes:

[0060] Step S10: k micro-structured light sensors and 1 micro-attitude sensor are installed on the bronchoscope. The micro-structured light sensor is used to collect the distance data from the bronchoscope to different points on the inner wall of the bronchus in real time. The distance from the bronchoscope lens to the i-th position point on the inner wall surface is recorded as d i ; The micro-attitude sensor is used to collect the posture data of the bronchoscope in real time, including the bronchoscope position data (x0, y0, z0) and the current viewing direction data

[0061] It should be noted that in this step, the installation positions of the micro structured light sensor and the micro attitude sensor should be accurate to ensure that the collected data is accurate and valid. Especially in the complex spatial environment of the bronchus, a slight deviation in the sensor position may affect the accuracy of the distance data and posture data.

[0062] It is understandable that the distance and posture data of each position point changes dynamically during the movement of the bronchoscope. In order to ensure the continuity and accuracy of data collection, the micro-structured light sensor should have a high response speed and be able to collect data at multiple positions in a short time.

[0063] It should be understood that the viewing direction data is the direction vector of the bronchoscope at its current position, which is used to indicate the orientation of the bronchoscope lens. The accuracy of this direction vector is crucial because it directly affects the calculation of the subsequent 3D point cloud coordinates.

[0064] For example, when the bronchoscope reaches a certain position on the inner wall of the bronchus, the distance data recorded by the microstructured light sensor may be 5 mm, the position information recorded by the micro-attitude sensor may be (x0=10, y0=15, z0=20), and the viewing direction vector is With these data, the exact coordinates of the location point in three-dimensional space can be calculated to further form a complete three-dimensional point cloud.

[0065] Step S20: Based on the bronchoscope posture data and distance d i , calculate the three-dimensional coordinates of each point on the bronchial wall, using the formula:

[0066]

[0067] Among them, (x i ,y i , z i ) is the three-dimensional coordinate of the i-th position point on the inner wall surface;

[0068] All the calculated points (x i ,y i , z i ) to form preliminary three-dimensional point cloud data, and perform denoising and filtering on the preliminary three-dimensional point cloud data to obtain three-dimensional point cloud data;

[0069] It should be noted that since the inner wall of the bronchus is usually uneven and has a curved structure, the collected point cloud data may contain noise or abnormal points, so it is necessary to perform denoising and filtering after the 3D point cloud aggregation to ensure the quality of the point cloud data. These processes help remove invalid data points caused by equipment jitter or acquisition errors.

[0070] It should be understood that filtering can be performed using a variety of methods, such as statistical filtering based on neighborhood density and mean filtering, etc. Selecting an appropriate filtering method can enhance the smoothness of the three-dimensional point cloud, ensure the accuracy of the data structure, and provide high-quality three-dimensional data for subsequent image processing and analysis.

[0071] For example, assuming that the position data of the bronchoscope at a position point is (x0=10, y0=15, z0=20), the distance d from the lens to a point on the inner wall surface is i =5mm, direction vector Then the three-dimensional coordinates of the point can be calculated as:

[0072] (x i ,y i , z i )=(10,15,20)+5×(0.6,0.8,0)=(13,19,20)

[0073] By performing similar calculations on all points and aggregating them, a complete three-dimensional point cloud model can be formed, and smoother three-dimensional point cloud data can be obtained after denoising and filtering.

[0074] Step S30: installing a multi-band light source and a corresponding imaging sensor on the bronchoscope, performing imaging in the visible light band of 400-700nm and the near infrared band of 700-900nm respectively, executing steps S10 to S20 in different bands, generating three-dimensional point cloud data in the visible light band and three-dimensional point cloud data in the near infrared band, and each band of point cloud data records the inner wall reflection intensity information of the bronchial inner wall;

[0075] It should be noted that the purpose of selecting the visible light and near-infrared bands is to obtain the reflective characteristics of the bronchial inner wall at different depths and structural levels. The visible light band can provide surface details, while the near-infrared band can penetrate deeper into the tissue structure and capture the internal reflection information. This multi-band imaging method helps to comprehensively analyze the complex tissue of the bronchial inner wall. For example, data collected in the visible light band may reflect the texture details of the surface of the bronchial inner wall, such as tiny wrinkles and irregular surfaces; while data collected in the near-infrared band can reveal abnormal reflective areas deeper in the bronchial wall. This information is of great reference value for locating the lesion area and judging the extent of the lesion.

[0076] It is understandable that in actual operation, the reflection intensity of different bands will be affected by the surface texture, thickness and tissue density of the bronchial lining. Therefore, by comparing the difference in reflection intensity under visible light and near-infrared bands, the characteristics of the lesion area can be better identified and the detection accuracy of the lesion tissue can be enhanced.

[0077] It should be understood that in order to ensure the accuracy of the data in each band, the imaging sensor needs to adjust the exposure according to the light intensity of different bands to prevent the image from being overexposed or underexposed. This ensures that clear reflection intensity information is obtained in each band, which facilitates the subsequent multimodal feature extraction.

[0078] Step S40: for the three-dimensional point cloud data of the visible light band and the near infrared band obtained in step S30, extract the multi-band reflection characteristics of each point, and construct a multi-modal feature vector F = [S 可见光 , S 近红外 ], the multimodal feature vector F is used to generate the multimodal grayscale feature vector H, where S 可见光 is the reflection value in the 400-700nm band, S 近红外 It is the reflection value of 700-900nm band;

[0079] It should be noted that the construction of multimodal feature vectors aims to fuse the reflection information of different bands to capture the detailed features of the bronchial lining at different depths and tissue structures. The visible light band provides surface texture information, while the near-infrared band can penetrate deeper and provide richer tissue structure information, thereby increasing the recognition accuracy of the lesion area.

[0080] It is understandable that, because the reflective properties of the bronchial lining vary with wavelength, the reflective data of a single band often cannot fully display the complete characteristics of the bronchial lining. Therefore, through the multimodal feature fusion of visible light and near-infrared bands, it is possible to integrate information from different bands in the three-dimensional point cloud data and improve the depth and resolution of the image.

[0081] It should be understood that when generating the multimodal grayscale feature vector H, the reflectance values ​​of different bands need to be normalized to ensure the consistency of the data. 可见光 and S 近红外 It is necessary to scale the maximum and minimum reflectance values ​​so that they are mapped into a grayscale range of 0 to 255, thereby generating a pseudo grayscale image suitable for subsequent processing.

[0082] Step S50: Generate visible light and near-infrared pseudo grayscale images according to the multimodal grayscale feature vector H, and use the gray level co-occurrence matrix GLCM to extract texture feature information, including contrast C and entropy E:

[0083]

[0084] Where m and n are gray levels (integers between 0 and 255), represents the gray level in the gray-level co-occurrence matrix, P(m,n) represents the joint probability of a pixel pair with gray levels m and n, C is the contrast, and is the entropy;

[0085] Combine the reflectance value, grayscale value and texture features to form the final multimodal feature vector

[0086] F multi =

[0087] [S 可见光 , S近红外 , H 可见光 , H 近红外 , C 可见光 , C 近红外 , E 可见光 , E 近红外 ];

[0088] Among them, H 可见光 is the pseudo grayscale value generated by reflection value mapping in the visible light band, H 近红外 is the pseudo grayscale value generated by reflectance mapping in the near-infrared band, C 可见光 is the contrast eigenvalue extracted based on the gray-level co-occurrence matrix in the visible light band, C 近红外 is the contrast eigenvalue extracted based on the gray-level co-occurrence matrix in the near-infrared band, E 可见光 is the entropy eigenvalue extracted based on the gray-level co-occurrence matrix in the visible light band, E 近红外 is the entropy eigenvalue extracted based on the gray-level co-occurrence matrix in the near-infrared band;

[0089] It should be noted that GLCM (Gray Level Co-occurrence Matrix) is a common method for extracting texture features based on the spatial relationship between gray levels. In the present invention, the contrast and entropy extracted by GLCM can help analyze the texture changes of the surface and deep structures of the bronchial inner wall, so as to better identify the diseased tissue.

[0090] It can be understood that contrast C measures the degree of grayscale variation. The higher the value, the greater the grayscale difference and the rougher the surface texture. Entropy E measures the randomness of grayscale distribution. The higher the value, the more complex the texture. These features are extracted in the visible and near-infrared bands, which can provide multi-angle texture information and help to fully describe tissue characteristics.

[0091] It should be understood that the construction of the multimodal feature vector includes reflectance value, grayscale value and texture features, integrating the information of visible light and near-infrared bands. Through this fusion, data from different bands can be effectively integrated, feature information can be enriched, and solid data support can be provided for subsequent three-dimensional segmentation and lesion area detection.

[0092] Step S60: The final multimodal feature vector F generated in step S50 is multi The data is input into a three-dimensional convolutional neural network (3DCNN), which extracts features layer by layer by sliding the three-dimensional convolution kernel in the three-dimensional space of the feature vector to identify abnormal areas of the bronchial inner wall.

[0093] After multi-layer convolution and pooling processing, the abnormal area of ​​the bronchial wall is segmented, and the classification label of each volume pixel is output, marked as a lesion volume pixel or a normal volume pixel;

[0094] All lesion volume pixels are marked in a three-dimensional space, and a three-dimensional segmentation mask is generated, wherein the three-dimensional segmentation mask is used to distinguish the lesion area from the normal area.

[0095] It should be noted that the three-dimensional convolution kernel of 3DCNN can capture the spatial correlation of multimodal feature vectors in three-dimensional space, thereby more accurately identifying the lesion area of ​​the bronchial lining. Compared with traditional two-dimensional convolution, three-dimensional convolution can extract features in the depth dimension and is more suitable for processing bronchoscopic images containing multi-level information.

[0096] It should be understood that the multi-layer convolution and pooling processing of 3DCNN helps to gradually extract high-level features, reduce noise interference, and retain the response to the characteristics of the lesion area. This process of extracting features layer by layer can effectively filter out irrelevant information and ensure that the characteristics of the lesion area can be accurately captured and segmented. The final three-dimensional segmentation mask is not only used to display the spatial location of the lesion area, but also can be used for subsequent analysis. By visualizing the lesion area in three-dimensional space, it can help doctors observe the location, size and shape of the lesion more intuitively.

[0097] For example, in a bronchial wall detection case, through 3D CNN processing, it is possible to identify certain specific areas of volume pixels with abnormal reflection and texture features, and mark these volume pixels as lesion areas. The final generated 3D segmentation mask can show the exact location of these lesion areas on the bronchial wall.

[0098] In addition, the bronchoscopic image processing system based on computer vision provided by the present invention adopts a bronchoscopic image processing method based on computer vision in the above-mentioned embodiment, which can solve a technical problem of bronchoscopic image processing based on computer vision. Compared with the prior art, the beneficial effects of the bronchoscopic image processing system based on computer vision provided by the present invention are the same as the beneficial effects of the bronchoscopic image processing method based on computer vision provided by the above-mentioned embodiment, and the other technical features of the bronchoscopic image processing system based on computer vision are the same as the features disclosed in the above-mentioned embodiment method, which will not be repeated here.

[0099] The present invention provides a bronchoscopic image processing device based on computer vision, please refer to Figure 2, a bronchoscopic image processing device based on computer vision includes: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a bronchoscopic image processing method based on computer vision in the above-mentioned embodiment 1. A bronchoscopic image processing device based on computer vision in an embodiment of the present invention may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. A bronchoscopic image processing device based on computer vision is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention. A bronchoscopic image processing device based on computer vision may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM) 1004. Various programs and data required for the operation of a bronchoscopic image processing device based on computer vision are also stored in the RAM 1004. The processing device 1001, the ROM 1002, and the RAM 1004 are connected to each other via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1003 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1009. The communication device 1009 can allow a bronchoscopic image processing device based on computer vision to communicate with other devices wirelessly or by wire to exchange data. Although a bronchoscopic image processing device based on computer vision with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or have alternatively.

[0100] The present invention also provides a computer program product, including a computer program, which implements the steps of a bronchoscopic image processing method based on computer vision as described above when the computer program is executed by a processor. The computer program product provided by the present invention can solve the technical problem of bronchoscopic image processing based on computer vision. Compared with the prior art, the beneficial effects of the computer program product provided by the present invention are the same as the beneficial effects of a bronchoscopic image processing method based on computer vision provided by the above embodiment, which will not be described in detail here.

[0101] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through a communication device, or installed from the storage device 1003, or installed from the ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present invention are executed.

[0102] It should be understood that the various parts disclosed in the present invention can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0103] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A bronchoscopic image processing method based on computer vision, characterized in that: Methods include: Step S10: k micro-structured light sensors and 1 micro-attitude sensor are installed on the bronchoscope. The micro-structured light sensor is used to collect the distance data from the bronchoscope to different points on the inner wall of the bronchus in real time. The distance from the bronchoscope lens to the i-th position point on the inner wall surface is recorded as d i ; The micro-attitude sensor is used to collect the posture data of the bronchoscope in real time, including the bronchoscope position data (x0, y0, z0) and the current viewing direction data Step S20: Based on the bronchoscope posture data and distance d i , calculate the three-dimensional coordinates of each point on the bronchial wall, using the formula: Among them, (x i ,y i , z i ) is the three-dimensional coordinate of the i-th position point on the inner wall surface; All the calculated points (x i ,y i , z i ) to form preliminary three-dimensional point cloud data, and perform denoising and filtering on the preliminary three-dimensional point cloud data to obtain three-dimensional point cloud data; Step S30: installing a multi-band light source and a corresponding imaging sensor on the bronchoscope, performing imaging in the visible light band of 400-700nm and the near infrared band of 700-900nm respectively, executing steps S10 to S20 in different bands, generating three-dimensional point cloud data in the visible light band and three-dimensional point cloud data in the near infrared band, and each band of point cloud data records the inner wall reflection intensity information of the bronchial inner wall; Step S40: for the three-dimensional point cloud data of the visible light band and the near infrared band obtained in step S30, extract the multi-band reflection characteristics of each point, and construct a multi-modal feature vector F = [S 可见光 , S 近红外 ], the multimodal feature vector F is used to generate the multimodal grayscale feature vector H, where S 可见光 is the reflection value in the 400-700nm band, S 近红外 It is the reflection value of 700-900nm band; Step S50: Generate visible light and near-infrared pseudo grayscale images according to the multimodal grayscale feature vector H, and use the gray level co-occurrence matrix GLCM to extract texture feature information, including contrast C and entropy E: Where m and n are gray levels, representing the gray levels in the gray-level co-occurrence matrix, and P(m,n) represents the joint probability of a pixel pair with gray levels m and n; Combine the reflectance value, grayscale value and texture features to form the final multimodal feature vector F multi ＝ [S 可见光 ,S 近红外 ,H 可见光 ,H 近红外 ,C 可见光 ,C 近红外 ,E 可见光 ,E 近红外 ]; Among them, H 可见光 is the pseudo grayscale value generated by reflection value mapping in the visible light band, H 近红外 is the pseudo grayscale value generated by reflectance mapping in the near-infrared band, C 可见光 is the contrast eigenvalue extracted based on the gray-level co-occurrence matrix in the visible light band, C 近红外 is the contrast eigenvalue extracted based on the gray-level co-occurrence matrix in the near-infrared band, E 可见光 is the entropy eigenvalue extracted based on the gray-level co-occurrence matrix in the visible light band, E 近红外 is the entropy eigenvalue extracted based on the gray-level co-occurrence matrix in the near-infrared band; Step S60: The final multimodal feature vector F generated in step S50 is multi The data is input into a three-dimensional convolutional neural network (3DCNN), which extracts features layer by layer by sliding the three-dimensional convolution kernel in the three-dimensional space of the feature vector to identify abnormal areas of the bronchial inner wall. After multi-layer convolution and pooling processing, the abnormal area of ​​the bronchial wall is segmented, and the classification label of each volume pixel is output, marked as a lesion volume pixel or a normal volume pixel; All lesion volume pixels are marked in a three-dimensional space, and a three-dimensional segmentation mask is generated, wherein the three-dimensional segmentation mask is used to distinguish the lesion area from the normal area.

2. A bronchoscopic image processing method based on computer vision as claimed in claim 1, characterized in that: In step S40, the process of generating the multimodal grayscale feature vector H includes: The following formula is used to generate the grayscale values ​​of the visible light band and the near-infrared band: Among them, H 波段 is the gray value of the corresponding band, S 波段 is the gray value of the corresponding band, S min and S max They are the minimum and maximum values ​​of the reflectance values ​​in the corresponding bands respectively; And construct a multimodal grayscale feature vector H = [H 可见光 , H 近红外 ].

3. A bronchoscopic image processing method based on computer vision as claimed in claim 2, characterized in that: S min and S max Dynamically adjusted according to the lighting conditions and imaging contrast of the current bronchoscopic image.

4. The bronchoscopic image processing method based on computer vision according to claim 1, characterized in that: In step S30, the multi-band light source is an adjustable light source.

5. A bronchoscopic image processing method based on computer vision as claimed in claim 4, characterized in that: In step S30, the adjustable light source includes an LED array of multiple wavelength bands, and light sources of different wavelength bands are selected according to different scenes.

6. A bronchoscopic image processing method based on computer vision as claimed in claim 1, characterized in that: In step S50, a sliding window method is used to extract local texture features from the pseudo grayscale image.

7. A bronchoscopic image processing method based on computer vision as claimed in claim 1, characterized in that: In step S60, the model training data of the three-dimensional convolutional neural network 3DCNN is multimodal feature vector sample data with a scale of more than 10,000 samples and corresponding labeled volume pixel sample data.

8. A bronchoscopic image processing system based on computer vision, characterized in that: The computer vision-based bronchoscopic image processing system comprises: The data acquisition module is used to install k micro-structured light sensors and 1 micro-attitude sensor on the bronchoscope. The micro-structured light sensor is used to collect the distance data from the bronchoscope to different points on the inner wall of the bronchus in real time. The distance from the bronchoscope lens to the i-th position point on the inner wall surface is recorded as d i ; The micro-attitude sensor is used to collect the posture data of the bronchoscope in real time, including the bronchoscope position data (x0, y0, z0) and the current viewing direction data The 3D point cloud generation module is used to generate a 3D point cloud based on the bronchoscope’s position data and distance d i , calculate the three-dimensional coordinates of each point on the bronchial wall, using the formula: Among them, (x i ,y i , z i ) is the three-dimensional coordinate of the i-th position point on the inner wall surface; All the calculated points (x i ,y i , z i ) to form preliminary three-dimensional point cloud data, and perform denoising and filtering on the preliminary three-dimensional point cloud data to obtain three-dimensional point cloud data; A multi-band imaging module is used to install a multi-band light source and a corresponding imaging sensor on a bronchoscope, perform imaging in the visible light band of 400-700nm and the near infrared band of 700-900nm, respectively, and perform steps S10 to S20 in different bands to generate three-dimensional point cloud data in the visible light band and three-dimensional point cloud data in the near infrared band, and each band of point cloud data records the inner wall reflection intensity information of the bronchial inner wall; The multimodal feature extraction module is used to extract the multi-band reflection characteristics of each point from the three-dimensional point cloud data of the visible light band and the near infrared band obtained in step S30, and construct a multimodal feature vector F = [S 可见光 , S 近红外 ], the multimodal feature vector F is used to generate the multimodal grayscale feature vector H, where S 可见光 is the reflection value in the 400-700nm band, S 近红外 It is the reflection value of 700-900nm band; The texture feature analysis module is used to generate visible light and near-infrared pseudo grayscale images based on the multimodal grayscale feature vector H, and use the gray level co-occurrence matrix GLCM to extract texture feature information, including contrast C and entropy E: Where m and n are gray levels (integers between 0 and 255), represents the gray level in the gray-level co-occurrence matrix, P(m,n) represents the joint probability of a pixel pair with gray levels m and n, C is the contrast, and is the entropy; Combine the reflectance value, grayscale value and texture features to form the final multimodal feature vector F multi ＝ [S 可见光 ,S 近红外 ,H 可见光 ,H 近红外 ,C 可见光 ,C 近红外 ,E 可见光 ,E 近红外 ]; Among them, H 可见光 is the pseudo grayscale value generated by reflection value mapping in the visible light band, H 近红外 is the pseudo grayscale value generated by reflectance mapping in the near-infrared band, C 可见光 is the contrast eigenvalue extracted based on the gray-level co-occurrence matrix in the visible light band, C 近红外 is the contrast eigenvalue extracted based on the gray-level co-occurrence matrix in the near-infrared band, E 可见光 is the entropy eigenvalue extracted based on the gray-level co-occurrence matrix in the visible light band, E 近红外 is the entropy eigenvalue extracted based on the gray-level co-occurrence matrix in the near-infrared band; The lesion region marking module is used to convert the final multimodal feature vector F generated in step S50 into multi The data is input into a three-dimensional convolutional neural network (3DCNN), which extracts features layer by layer by sliding the three-dimensional convolution kernel in the three-dimensional space of the feature vector to identify abnormal areas of the bronchial inner wall. After multi-layer convolution and pooling processing, the abnormal area of ​​the bronchial wall is segmented, and the classification label of each volume pixel is output, marked as a lesion volume pixel or a normal volume pixel; All lesion volume pixels are marked in a three-dimensional space, and a three-dimensional segmentation mask is generated, wherein the three-dimensional segmentation mask is used to distinguish the lesion area from the normal area.

9. A bronchoscopic image processing device based on computer vision, characterized in that: The bronchoscopic image processing device based on computer vision comprises: A memory, a processor, and a bronchoscopic image processing program based on computer vision stored in the memory and executable on the processor, wherein the bronchoscopic image processing program based on computer vision, when executed by the processor, implements the bronchoscopic image processing method based on computer vision described in any one of Claims 1 to 7.

10. A computer program product, characterized in that The computer program product includes a bronchoscopic image processing program based on computer vision, and when the bronchoscopic image processing program based on computer vision is executed by a processor, the bronchoscopic image processing method based on computer vision described in any one of Claims 1 to 7 is implemented.

Images