Endoscopic image processing method, device, computer equipment and storage medium

Abstract

The present application provides an endoscopic image processing method, device, equipment and storage medium, which relates to the field of image processing and is used to improve the display effect of endoscopic images. The method mainly includes: obtaining a spinal endoscopic image from a spinal endoscopic video stream; filtering the spinal endoscopic image according to the time information, angle information and position information of each frame of the spinal endoscopic image; performing image preprocessing on each frame of the filtered spinal endoscopic image, and inputting the image preprocessed spinal endoscopic image into a spinal recognition model to obtain the spinal position area corresponding to each frame of the spinal endoscopic image; extracting a spinal structure image from the spinal endoscopic image according to the spinal position area; projecting the position information and depth information of each pixel point in each frame of the spinal structure image to the corresponding position in the three-dimensional space to form point cloud data; and converting the point cloud data into a triangular mesh to obtain a three-dimensional model of the endoscopic image of the spine.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an endoscopic image processing method, device, equipment and storage medium. Background Art

[0002] With the rapid development of medical technology, endoscopes have become an important tool for surgeons in clinical diagnosis, providing doctors with more advanced technical means to detect and remove internal lesions in the human body, reducing the probability of complications. However, due to the complexity of the internal structure of the human body and the limitations of imaging technology, medical endoscope images have defects such as uneven illumination, blur, and low contrast. Therefore, how to improve the display effect of endoscope images has become an urgent problem to be solved. Summary of the invention

[0003] The embodiments of the present application provide an endoscopic image processing method, apparatus, computer equipment and storage medium for improving the display effect of endoscopic images.

[0004] An embodiment of the present invention provides an endoscopic image processing method, the method comprising:

[0005] Acquire a spinal endoscope image from a spinal endoscope video stream, wherein each frame of the spinal endoscope image in the spinal endoscope video stream corresponds to time information, angle information, and position information;

[0006] Filtering the spinal endoscope image according to the time information, angle information and position information of each frame of the spinal endoscope image;

[0007] Performing image preprocessing on each frame of the filtered spinal endoscopic image, and inputting the image preprocessed spinal endoscopic image into a spinal recognition model to obtain a spinal position area corresponding to each frame of the spinal endoscopic image, wherein the spinal position area at least includes a position area where a vertebral body, an intervertebral disc, and a nerve root are located;

[0008] Extracting a spinal structure image from the spinal endoscope image according to the spinal position region;

[0009] Determine the depth information of each pixel in the spinal structure image based on the angle information and position information corresponding to each frame of the spinal structure image;

[0010] The position information and depth information of each pixel point in each frame of the spinal structure image are projected to the corresponding position in the three-dimensional space to form point cloud data; and the point cloud data is converted into a triangular mesh to obtain a three-dimensional model of the endoscopic image of the spine.

[0011] In an optional embodiment, filtering the spinal endoscope image according to the time information, angle information and position information of each frame of the spinal endoscope image includes:

[0012] Based on the time information, acquiring spinal endoscopic images whose angle information and position information deviations within a predetermined time period are less than a predetermined value, and forming the acquired spinal endoscopic images into a spinal endoscopic image group;

[0013] Determining the clarity of each spinal endoscopic image in the spinal endoscopic image group;

[0014] The spinal endoscope image group is filtered out except the spinal endoscope image with the highest definition.

[0015] In an optional embodiment, determining the clarity of each spinal endoscopy image in the spinal endoscopy image group includes:

[0016] Identifying bubbles of each spinal endoscope image in the spinal endoscope image group by image recognition technology, and identifying shadows and reflection areas of each spinal endoscope image in the spinal endoscope image group by Gaussian blur and edge detection;

[0017] The clarity of each spinal endoscopic image in the spinal endoscopic image group is calculated based on the identified bubbles, shadows and reflection areas of the spinal endoscopic image.

[0018] In an optional embodiment, the step of calculating the clarity of each spinal endoscopic image in the spinal endoscopic image group according to the identified bubbles, shadows and reflection areas of the spinal endoscopic image comprises:

[0019] The clarity of the spinal endoscope image is calculated by the following formula:

[0020] ,

[0021] in, is the weight value, n is the number of bubbles, is the size of the area where the ith bubble is located, m is the number of shadows, is the size of the area where the jth shadow is located, q is the number of reflection areas, is the size of the area where the pth reflection area is located, and A is the total area size of the spinal endoscopy image.

[0022] In an optional embodiment, the image preprocessing of each frame of the spinal endoscopy image after filtering includes:

[0023] Converting each frame of the filtered spinal endoscope image into an HSV space, and adjusting the V channel in the HSV space according to the gamma value to achieve brightness processing of the spinal endoscope image;

[0024] The spinal endoscope image after brightness processing is reconverted into RGB format, and detail processing and contrast processing are performed on the spinal endoscope image reconverted into RGB format.

[0025] In an optional embodiment, determining the depth information of each pixel in the spinal structure image based on the angle information and position information corresponding to each frame of the spinal structure image includes:

[0026] Inputting each frame of the spinal structure image and its corresponding angle information and position information into an image depth prediction model to obtain a depth prediction value for each pixel in the spinal structure image;

[0027] Acquire matching image pixels by performing image registration on the spinal endoscopic images in the spinal endoscopic image group, estimate a disparity map between the spinal endoscopic images by using the matching image pixels, and obtain a depth estimation value of each pixel in the spinal structure image according to the disparity map;

[0028] Depth information of each pixel in the spinal structure image is determined according to the depth prediction value and the depth estimation value.

[0029] In an optional embodiment, the step of obtaining matching image pixels by performing image registration on the spinal endoscopic images in the spinal endoscopic image group, and estimating the disparity map between the spinal endoscopic images by using the matching image pixels, comprises:

[0030] Calculating the pixel matching degree between the spinal endoscopic images in the spinal endoscopic image group by normalized correlation coefficient, and determining the pixel points whose pixel matching degree is greater than a preset value as matching image pixels;

[0031] Determining a spinal endoscopic image in which the number of matched image pixel points in the spinal endoscopic image group exceeds a target number as a matched spinal endoscopic image;

[0032] The disparity map between the spinal endoscopic images is estimated by using the matched image pixel points in the matched spinal endoscopic images in the spinal endoscopic image group.

[0033] An embodiment of the present invention provides an endoscopic image processing device, the device comprising:

[0034] An acquisition module, used for acquiring a spinal endoscope image from a spinal endoscope video stream, wherein each frame of the spinal endoscope image in the spinal endoscope video stream corresponds to time information, angle information, and position information;

[0035] A filtering module, used for filtering the spinal endoscope image according to the time information, angle information and position information of each frame of the spinal endoscope image;

[0036] A recognition module, used for performing image preprocessing on each frame of the filtered spinal endoscopic image, and inputting the image preprocessed spinal endoscopic image into a spinal recognition model to obtain a spinal position area corresponding to each frame of the spinal endoscopic image, wherein the spinal position area at least includes a position area where a vertebral body, an intervertebral disc and a nerve root are located;

[0037] An extraction module, used for extracting a spinal structure image from the spinal endoscope image according to the spinal position area;

[0038] A determination module, configured to determine the depth information of each pixel in the spinal structure image based on the angle information and position information corresponding to each frame of the spinal structure image;

[0039] The projection conversion module is used to project the position information and depth information of each pixel point in each frame of the spinal structure image to the corresponding position in the three-dimensional space to form point cloud data; and convert the point cloud data into a triangular mesh to obtain a three-dimensional model of the endoscopic image of the spine.

[0040] A computer program product comprises a computer program, wherein the computer program implements the above-mentioned endoscopic image processing method when executed by a processor.

[0041] The present invention provides an endoscopic image processing method, device, computer equipment and storage medium. First, a spinal endoscopic image is obtained from a spinal endoscopic video stream. Then, the spinal endoscopic image is filtered according to the time information, angle information and position information of each frame of the spinal endoscopic image. Then, each frame of the filtered spinal endoscopic image is preprocessed, and the spinal endoscopic image after image preprocessing is input into a spinal recognition model to obtain a spinal position area corresponding to each frame of the spinal endoscopic image. According to the spinal position area, a spinal structure image is extracted from the spinal endoscopic image, and the depth information of each pixel in the spinal structure image is determined based on the angle information and position information corresponding to each frame of the spinal structure image. Finally, the position information and depth information of each pixel in each frame of the spinal structure image are projected to the corresponding position in the three-dimensional space to form point cloud data, and the point cloud data is converted into a triangular mesh to obtain a three-dimensional model of the endoscopic image of the spine. Compared with the existing imaging directly based on endoscopic video stream, the present application extracts the spinal structure image from the spinal endoscopic image after filtering and preprocessing the spinal endoscopic image, and then converts the spinal structure image into a three-dimensional model of the endoscopic image, thereby improving the display effect of the endoscopic image. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A flow chart of an endoscopic image processing method provided in this application;

[0043] Figure 2 A filtering flow chart of a spinal endoscope image provided in this application;

[0044] Figure 3 A flow chart for determining depth information in a spinal structure image provided by the present application;

[0045] Figure 4 A schematic diagram of the structure of an endoscopic image processing device provided in this application;

[0046] Figure 5 A schematic diagram of a computer device provided in this application. DETAILED DESCRIPTION

[0047] In order to better understand the above-mentioned technical scheme, the technical scheme of the embodiments of the present application is described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present application and the specific features in the embodiments are detailed descriptions of the technical scheme of the embodiments of the present application, rather than limitations on the technical scheme of the present application. In the absence of conflict, the embodiments of the present application and the technical features in the embodiments may be combined with each other.

[0048] See also Figure 1, is an endoscopic image processing method provided by an embodiment of the present invention, the method specifically includes S101-S106:

[0049] S101, obtaining a spinal endoscopic image from a spinal endoscopic video stream, wherein each frame of the spinal endoscopic image in the spinal endoscopic video stream corresponds to time information, angle information, and position information.

[0050] The spinal endoscopy video stream is a real-time video stream that observes and records the internal structure of the spine by using an endoscopic device. In this embodiment, the acquired spinal endoscopy video stream is subjected to video frame extraction to obtain multiple frames of spinal endoscopy images, and each frame of the spinal endoscopy image corresponds to shooting time information, angle information, and position information. Among them, the angle information and position information can be the relative angle and relative position of the endoscope device that shoots the spinal endoscopy video stream relative to a fixed position, or the angle and position of the endoscope device in a certain coordinate system, which is not specifically limited in this embodiment.

[0051] S102, filtering the spinal endoscope image according to the time information, angle information and position information of each frame of the spinal endoscope image.

[0052] It should be noted that the spinal endoscopy video stream contains multiple frames of spinal endoscopy images. When the endoscope device shoots at a fixed position and angle for a certain period of time, multiple frames of spinal endoscopy images with the same image content or unclear spinal endoscopy images may be extracted through the spinal endoscopy video stream. In order to reduce the number of spinal endoscopy images processed, this embodiment can first filter the spinal endoscopy images.

[0053] Specifically, this embodiment can filter spinal endoscope images with repeated contents, or can filter spinal endoscope images with poor display effects, and can also filter spinal endoscope images that are not located in a predetermined area.

[0054] like Figure 2 As shown, in an optional embodiment provided by the present application, filtering the spinal endoscope image according to the time information, angle information and position information of each frame of the spinal endoscope image includes:

[0055] S1021, based on the time information, obtaining spinal endoscopic images whose angle information and position information deviations within a predetermined time period are less than a predetermined value, and forming the obtained spinal endoscopic images into a spinal endoscopic image group.

[0056] Among them, the predetermined time period and the predetermined value are the time period and value set according to actual needs. For example, the predetermined time period can be 5 adjacent seconds, and the predetermined value includes an angle deviation value and a position deviation value. The angle deviation value can be 5 degrees, and the position deviation (the deviations of the horizontal and vertical coordinates are both less than 5). Based on the above conditions, spinal endoscopic images with angle deviations and position deviations less than 5 within 5 adjacent seconds are obtained from the spinal endoscopic images, and the acquired spinal endoscopic images are combined into a spinal endoscopic image group, so as to analyze and filter the spinal endoscopic images in the group and retain the clearest spinal endoscopic images in the group.

[0057] S1022, determining the clarity of each spinal endoscope image in the spinal endoscope image group.

[0058] In an optional embodiment provided in the present application, determining the clarity of each spinal endoscopic image in the spinal endoscopic image group includes: identifying bubbles of each spinal endoscopic image in the spinal endoscopic image group by image recognition technology, and identifying shadows and reflection areas of each spinal endoscopic image in the spinal endoscopic image group by Gaussian blur and edge detection; calculating the clarity of each spinal endoscopic image in the spinal endoscopic image group based on the identified bubbles, shadows and reflection areas of the spinal endoscopic image.

[0059] Among them, the presence of bubbles in the spinal endoscopy image may block the line of sight and affect the doctor's clear observation of the internal structure of the spine, especially when the bubbles are located near important anatomical structures, which may interfere with surgical operations or cause misjudgment. To this end, this embodiment can identify bubbles in each spinal endoscopy image in the spinal endoscopy image group through image recognition technology, such as using template matching or specific pattern recognition technology in image processing to automatically identify and locate bubbles in the spinal endoscopy image.

[0060] Shadows and reflection areas in spinal endoscopy images will affect the clarity of the image. Shadows are usually caused by occlusions caused by endoscopic instruments, surgical instruments or other tissues, resulting in certain areas that cannot be directly observed. The reflection area may be caused by the light source shining on the metal instrument or other highly reflective surface, which may cause unnecessary bright spots or interference information to appear in the image. Therefore, this embodiment needs to identify the shadows and reflection areas of each spinal endoscopy image in the spinal endoscopy image group, and then use this to calculate the image clarity and image filtering. This embodiment can identify shadows and reflection areas in spinal endoscopy images through machine learning or image processing techniques, such as Gaussian blur and edge detection. These algorithms can distinguish normal image features from abnormal shadows or reflections, thereby evaluating their impact on the clarity of the field of view.

[0061] Specifically, the clarity of the spinal endoscope image is calculated by the following formula in this embodiment:

[0062] ,

[0063] in, is the weight value, n is the number of bubbles, is the size of the area where the ith bubble is located, m is the number of shadows, is the size of the area where the jth shadow is located, q is the number of reflection areas, is the size of the area where the pth reflection area is located, and A is the total area size of the spinal endoscopy image. It can be seen from the above formula that the larger the total area of ​​the bubbles, shadows, and reflection areas in a spinal endoscopy image, the smaller the corresponding clarity D value.

[0064] S1023, filtering out the spinal endoscope image group except the spinal endoscope image with the highest definition.

[0065] S103, performing image preprocessing on each frame of the filtered spinal endoscopic image, and inputting the image preprocessed spinal endoscopic image into a spinal recognition model to obtain a spinal position area corresponding to each frame of the spinal endoscopic image.

[0066] In this embodiment, the image preprocessing can use a histogram analysis method to calculate the contrast of the image, and then by analyzing the distribution of the image brightness value, it can be determined whether the image has sufficient contrast to distinguish different tissues and structures, and then an adaptive histogram equalization (AHE) or local contrast enhancement algorithm is used to automatically adjust and optimize the contrast of the image, thereby improving the visibility of the tissue structure. Specifically, this embodiment can also automatically detect and correct color distortion in the image based on an algorithm of a color histogram and color correction technology (such as white balance) to ensure that the color in the image is close to a natural state without oversaturation or fading.

[0067] In an optional embodiment provided in the present application, the image preprocessing is performed on each frame of the spinal endoscopic image after filtering, including: converting each frame of the spinal endoscopic image after filtering to the HSV space, adjusting the V channel in the HSV space according to the gamma value to achieve brightness processing of the spinal endoscopic image, that is, merging the enhanced V channel with the H and S channels, and then converting to the RGB color space to obtain a brightness corrected image; reconverting the spinal endoscopic image that has undergone brightness processing into the RGB format, and performing detail processing and contrast processing on the spinal endoscopic image that has been reconverted into the RGB format.

[0068] The brightness features in spinal endoscopy images can be roughly divided into three categories: low light, normal light, and high light. This application can restore the brightness of low-light images by performing a gamma transform with a γ value greater than 1 and less than 1.5 on the V channel, in order to obtain an endoscopic image with better visual effects; for images with good light, the brightness of the V channel can be randomly attenuated by performing a gamma transform with a γ value less than 1 and greater than 0.5, in order to simulate endoscopic images under brightness attenuation, and finally obtain spinal endoscopy images with poor brightness vision and good brightness vision.

[0069] Optionally, in order to maintain the hue and vividness of the image while correcting the brightness of the spinal endoscope image, this embodiment adjusts the V channel in the HSV space to achieve brightness processing of the spinal endoscope image. In traditional adaptive gamma correction, since the cumulative distribution function continues to increase with the increase of the image grayscale value, the gamma parameter gradually decreases, and the brighter areas in the image will be processed using a smaller gamma value, so that the pixels move toward the highest grayscale level, resulting in over-enhancement and loss of vascular details in the brighter areas of the endoscopic image.

[0070] The gamma correction is:

[0071] , ,

[0072] in, is the brightness value of the corrected spinal endoscope image, is the maximum value of the spinal endoscopy image, is the original brightness value of the spinal endoscope image, is the gamma parameter, is the cumulative distribution function of the image, is the cumulative distribution function of the trimmed histogram, is the cutoff coefficient (can be 0.5).

[0073] In this embodiment, the specific process of performing detail processing based on the spinal endoscopy image reconverted into RGB format can be: using various spatial domain filters, such as using a sharpening filter to enhance the edges and details in the image; converting the image to the frequency domain (for example, using Fourier transform), and then enhancing the high-frequency components in the frequency domain; using a nonlinear function (such as an S-shaped curve) to adjust the dynamic range of the image while enhancing the brightness and contrast of the image without excessively amplifying the noise; combining multiple images with different focusing distances to generate a full-focus image through image fusion technology, thereby enhancing the details of the overall image.

[0074] The specific process of detail processing and contrast processing based on the spinal endoscopy image reconverted into RGB format can be: by adjusting the histogram distribution of the image, all gray levels are distributed as evenly as possible, thereby increasing the contrast of the image; performing histogram equalization on the local area of ​​the image; by linearly or nonlinearly adjusting the brightness value of the image, the contrast of the image can be stretched to cover a wider grayscale range.

[0075] In an optional embodiment provided in the present application, a Gaussian membership function is used to convert the spatial region pixels to a fuzzy domain, and the conversion formula is as follows:

[0076] ,

[0077] in, Represents the pixel position under the c channel Pixel value, is the minimum value in the pixel area, is the image variance. This conversion formula can effectively convert the pixels of the regional pixel block into fuzzy values.

[0078] Then, the pixels in the blurred domain are enhanced as follows using the scale with the most complete eigenvalue linear structure at multiple scales:

[0079] , ,

[0080] Among them, the eigenvalues ​​of the Hessian matrix of the known image and It can be obtained by calculation, that is and is the eigenvalue corresponding to the maximum linear structure scale. The difference in the size of the eigenvalue can represent different information structures in the image; is a hyperparameter (taken as 0.5).

[0081] The enhanced blur plane is then converted to a spatial plane using the following conversion formula:

[0082] ,

[0083] The spinal recognition model in this embodiment is a pre-trained neural network model, which is trained using spinal endoscopy sample images, the sample labels of which are spinal position areas marked in the image, and the spinal position areas at least include the position areas where the vertebral bodies, intervertebral discs, and nerve roots are located. The spinal recognition model in this embodiment can automatically identify and distinguish spinal structures, such as vertebral bodies, intervertebral discs, and nerve roots. Image segmentation technology, such as graph-based image segmentation methods (Graph-Based Segmentation), can also be applied to automatically separate the spinal position area from the background.

[0084] S104: extracting a spinal column structure image from the spinal column endoscopic image according to the spinal column position area.

[0085] S105, determining depth information of each pixel in the spinal structure image based on the angle information and position information corresponding to each frame of the spinal structure image.

[0086] like Figure 3 As shown, in an optional embodiment provided by the present application, the depth information of each pixel in the spinal structure image is determined based on the angle information and position information corresponding to each frame of the spinal structure image, including:

[0087] S1051, input each frame of the spinal structure image and its corresponding angle information and position information into an image depth prediction model to obtain a depth prediction value of each pixel in the spinal structure image.

[0088] Among them, the image depth prediction model is a model trained based on sample data, that is, the sample data input into the image depth prediction model includes the spinal structure sample image and its corresponding angle information and position information. The depth prediction value of each pixel in the spinal structure image is obtained through model prediction, and then the loss value is calculated based on the depth prediction value and the actual depth value of the spinal structure sample image. When the loss value is less than a certain value, the training of the image depth prediction model is completed.

[0089] S1052, obtaining matching image pixels by performing image registration on the spinal endoscopic images in the spinal endoscopic image group, estimating the disparity map between the spinal endoscopic images by the matching image pixels, and obtaining the depth estimation value of each pixel in the spinal structure image according to the disparity map.

[0090] Specifically, the method of obtaining matching image pixels by performing image registration on the spinal endoscopic images in the spinal endoscopic image group and estimating the disparity map between the spinal endoscopic images through the matching image pixels includes: calculating the pixel matching degree between the spinal endoscopic images in the spinal endoscopic image group through the normalized correlation coefficient, and determining the pixels whose pixel matching degree is greater than a preset value as matching image pixels; determining the spinal endoscopic image in which the number of matched image pixels in the spinal endoscopic image group exceeds the target number as the matched spinal endoscopic image; and estimating the disparity map between the spinal endoscopic images through the matching image pixels in the matching spinal endoscopic images in the spinal endoscopic image group.

[0091] S1053, determining the depth information of each pixel in the spinal structure image according to the depth prediction value and the depth estimation value.

[0092] Specifically, in this embodiment, the depth prediction value and the depth estimation value may be averaged or weighted to obtain the depth information of each pixel in the spinal structure image.

[0093] S106, projecting the position information and depth information of each pixel point in each frame of the spinal structure image to a corresponding position in a three-dimensional space to form point cloud data; and converting the point cloud data into a triangular mesh to obtain a three-dimensional model of the endoscopic image of the spine.

[0094] The present embodiment provides an endoscopic image processing method, which first acquires a spinal endoscopic image from a spinal endoscopic video stream, then filters the spinal endoscopic image according to the time information, angle information and position information of each frame of the spinal endoscopic image, then performs image preprocessing on each filtered frame of the spinal endoscopic image, and inputs the image preprocessed spinal endoscopic image into a spinal recognition model to obtain a spinal position area corresponding to each frame of the spinal endoscopic image, extracts a spinal structure image from the spinal endoscopic image according to the spinal position area, and determines the depth information of each pixel in the spinal structure image based on the angle information and position information corresponding to each frame of the spinal structure image; finally, the position information and depth information of each pixel in each frame of the spinal structure image are projected to the corresponding position in the three-dimensional space to form point cloud data, and the point cloud data is converted into a triangular mesh to obtain a three-dimensional model of the endoscopic image of the spine. Compared with the existing imaging directly based on endoscopic video stream, the present application extracts the spinal structure image from the spinal endoscopic image after filtering and preprocessing the spinal endoscopic image, and then converts the spinal structure image into a three-dimensional model of the endoscopic image, thereby improving the display effect of the endoscopic image.

[0095] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0096] In one embodiment, an endoscopic image processing device is provided, and the endoscopic image processing device corresponds one-to-one to the endoscopic image processing method in the above embodiment. Figure 4 As shown, the functional modules of the endoscopic image processing device are described in detail as follows:

[0097] An acquisition module 41 is used to acquire a spinal endoscope image from a spinal endoscope video stream, wherein each frame of the spinal endoscope image in the spinal endoscope video stream corresponds to time information, angle information, and position information;

[0098] A filtering module 42, configured to filter the spinal endoscope image according to time information, angle information and position information of each frame of the spinal endoscope image;

[0099] The recognition module 43 is used to perform image preprocessing on each frame of the spinal endoscopic image after filtering, and input the image preprocessed spinal endoscopic image into the spinal recognition model to obtain the spinal position area corresponding to each frame of the spinal endoscopic image, wherein the spinal position area at least includes the position area where the vertebral body, intervertebral disc and nerve root are located;

[0100] An extraction module 44, configured to extract a spinal structure image from the spinal endoscope image according to the spinal position region;

[0101] A determination module 45, configured to determine the depth information of each pixel in the spinal structure image based on the angle information and position information corresponding to each frame of the spinal structure image;

[0102] The projection conversion module 46 is used to project the position information and depth information of each pixel point in each frame of the spinal structure image to the corresponding position in the three-dimensional space to form point cloud data; and convert the point cloud data into a triangular mesh to obtain a three-dimensional model of the endoscopic image of the spine.

[0103] In an optional embodiment, the filtering module 42 is specifically configured to:

[0104] Based on the time information, acquiring spinal endoscopic images whose angle information and position information deviations within a predetermined time period are less than a predetermined value, and forming the acquired spinal endoscopic images into a spinal endoscopic image group;

[0105] Determining the clarity of each spinal endoscopic image in the spinal endoscopic image group;

[0106] The spinal endoscope image group is filtered out except the spinal endoscope image with the highest definition.

[0107] In an optional embodiment, the filtering module 42 is specifically configured to:

[0108] Identifying bubbles of each spinal endoscope image in the spinal endoscope image group by image recognition technology, and identifying shadows and reflection areas of each spinal endoscope image in the spinal endoscope image group by Gaussian blur and edge detection;

[0109] The clarity of each spinal endoscopic image in the spinal endoscopic image group is calculated based on the identified bubbles, shadows and reflection areas of the spinal endoscopic image.

[0110] In an optional embodiment, the filtering module 42 calculates the clarity of the spinal endoscope image by the following formula:

[0111] ,

[0112] in, is the weight value, n is the number of bubbles, is the size of the area where the ith bubble is located, m is the number of shadows, is the size of the area where the jth shadow is located, q is the number of reflection areas, is the size of the area where the pth reflection area is located, and A is the total area size of the spinal endoscopy image.

[0113] In an optional embodiment, the identification module 43 is specifically configured to:

[0114] Converting each frame of the filtered spinal endoscope image into an HSV space, and adjusting the V channel in the HSV space according to the gamma value to achieve brightness processing of the spinal endoscope image;

[0115] The spinal endoscope image after brightness processing is reconverted into RGB format, and detail processing and contrast processing are performed on the spinal endoscope image reconverted into RGB format.

[0116] In an optional embodiment, the determination module 45 is specifically configured to:

[0117] Inputting each frame of the spinal structure image and its corresponding angle information and position information into an image depth prediction model to obtain a depth prediction value for each pixel in the spinal structure image;

[0118] Acquire matching image pixels by performing image registration on the spinal endoscopic images in the spinal endoscopic image group, estimate a disparity map between the spinal endoscopic images by using the matching image pixels, and obtain a depth estimation value of each pixel in the spinal structure image according to the disparity map;

[0119] Depth information of each pixel in the spinal structure image is determined according to the depth prediction value and the depth estimation value.

[0120] In an optional embodiment, the determination module 45 is specifically configured to:

[0121] Calculating the pixel matching degree between the spinal endoscopic images in the spinal endoscopic image group by normalized correlation coefficient, and determining the pixel points whose pixel matching degree is greater than a preset value as matching image pixels;

[0122] Determining a spinal endoscopic image in which the number of matched image pixel points in the spinal endoscopic image group exceeds a target number as a matched spinal endoscopic image;

[0123] The disparity map between the spinal endoscopic images is estimated by using the matched image pixel points in the matched spinal endoscopic images in the spinal endoscopic image group.

[0124] For the specific definition of the endoscopic image processing device, please refer to the definition of the endoscopic image processing method above, which will not be repeated here. Each module in the above-mentioned device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0125] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an endoscopic image processing method is implemented.

[0126] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the following steps are implemented:

[0127] Acquire a spinal endoscope image from a spinal endoscope video stream, wherein each frame of the spinal endoscope image in the spinal endoscope video stream corresponds to time information, angle information, and position information;

[0128] Filtering the spinal endoscope image according to the time information, angle information and position information of each frame of the spinal endoscope image;

[0129] Performing image preprocessing on each frame of the filtered spinal endoscopic image, and inputting the image preprocessed spinal endoscopic image into a spinal recognition model to obtain a spinal position area corresponding to each frame of the spinal endoscopic image, wherein the spinal position area at least includes a position area where a vertebral body, an intervertebral disc, and a nerve root are located;

[0130] Extracting a spinal structure image from the spinal endoscope image according to the spinal position region;

[0131] Determine the depth information of each pixel in the spinal structure image based on the angle information and position information corresponding to each frame of the spinal structure image;

[0132] The position information and depth information of each pixel point in each frame of the spinal structure image are projected to the corresponding position in the three-dimensional space to form point cloud data; and the point cloud data is converted into a triangular mesh to obtain a three-dimensional model of the endoscopic image of the spine.

[0133] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0134] Acquire a spinal endoscope image from a spinal endoscope video stream, wherein each frame of the spinal endoscope image in the spinal endoscope video stream corresponds to time information, angle information, and position information;

[0135] Filtering the spinal endoscope image according to the time information, angle information and position information of each frame of the spinal endoscope image;

[0136] Performing image preprocessing on each frame of the filtered spinal endoscopic image, and inputting the image preprocessed spinal endoscopic image into a spinal recognition model to obtain a spinal position area corresponding to each frame of the spinal endoscopic image, wherein the spinal position area at least includes a position area where a vertebral body, an intervertebral disc, and a nerve root are located;

[0137] Extracting a spinal structure image from the spinal endoscope image according to the spinal position region;

[0138] Determine the depth information of each pixel in the spinal structure image based on the angle information and position information corresponding to each frame of the spinal structure image;

[0139] The position information and depth information of each pixel point in each frame of the spinal structure image are projected to the corresponding position in the three-dimensional space to form point cloud data; and the point cloud data is converted into a triangular mesh to obtain a three-dimensional model of the endoscopic image of the spine.

[0140] In one embodiment, a computer program product is provided, the computer program product comprising a computer program, the computer program being executed by a processor to implement the following steps:

[0141] Acquire a spinal endoscope image from a spinal endoscope video stream, wherein each frame of the spinal endoscope image in the spinal endoscope video stream corresponds to time information, angle information, and position information;

[0142] Filtering the spinal endoscope image according to the time information, angle information and position information of each frame of the spinal endoscope image;

[0143] Performing image preprocessing on each frame of the filtered spinal endoscopic image, and inputting the image preprocessed spinal endoscopic image into a spinal recognition model to obtain a spinal position area corresponding to each frame of the spinal endoscopic image, wherein the spinal position area at least includes a position area where a vertebral body, an intervertebral disc, and a nerve root are located;

[0144] Extracting a spinal structure image from the spinal endoscope image according to the spinal position region;

[0145] Determine the depth information of each pixel in the spinal structure image based on the angle information and position information corresponding to each frame of the spinal structure image;

[0146] The position information and depth information of each pixel point in each frame of the spinal structure image are projected to the corresponding position in the three-dimensional space to form point cloud data; and the point cloud data is converted into a triangular mesh to obtain a three-dimensional model of the endoscopic image of the spine.

[0147] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0148] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0149] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.

Claims

1. An endoscopic image processing method, characterized in that: The method comprises: Acquire a spinal endoscope image from a spinal endoscope video stream, wherein each frame of the spinal endoscope image in the spinal endoscope video stream corresponds to time information, angle information, and position information; Filtering the spinal endoscope image according to the time information, angle information and position information of each frame of the spinal endoscope image; Performing image preprocessing on each frame of the filtered spinal endoscopic image, and inputting the image preprocessed spinal endoscopic image into a spinal recognition model to obtain a spinal position area corresponding to each frame of the spinal endoscopic image, wherein the spinal position area at least includes a position area where a vertebral body, an intervertebral disc, and a nerve root are located; Extracting a spinal structure image from the spinal endoscope image according to the spinal position region; Determine the depth information of each pixel in the spinal structure image based on the angle information and position information corresponding to each frame of the spinal structure image; The position information and depth information of each pixel point in each frame of the spinal structure image are projected to the corresponding position in the three-dimensional space to form point cloud data; and the point cloud data is converted into a triangular mesh to obtain a three-dimensional model of the endoscopic image of the spine.

2. The method according to claim 1, characterized in that The filtering of the spinal endoscope image according to the time information, angle information and position information of each frame of the spinal endoscope image comprises: Based on the time information, acquiring spinal endoscopic images whose angle information and position information deviations within a predetermined time period are less than a predetermined value, and forming the acquired spinal endoscopic images into a spinal endoscopic image group; Determining the clarity of each spinal endoscopic image in the spinal endoscopic image group; The spinal endoscope image group is filtered out except the spinal endoscope image with the highest definition.

3. The method according to claim 2, characterized in that Determining the clarity of each spinal endoscope image in the spinal endoscope image group includes: Identifying bubbles of each spinal endoscope image in the spinal endoscope image group by image recognition technology, and identifying shadows and reflection areas of each spinal endoscope image in the spinal endoscope image group by Gaussian blur and edge detection; The clarity of each spinal endoscopic image in the spinal endoscopic image group is calculated based on the identified bubbles, shadows and reflection areas of the spinal endoscopic image.

4. The method according to claim 3, characterized in that The step of calculating the clarity of each spinal endoscope image in the spinal endoscope image group according to the identified bubbles, shadows and reflection areas of the spinal endoscope image comprises: The clarity of the spinal endoscope image is calculated by the following formula: ， in, is the weight value, n is the number of bubbles, is the size of the area where the ith bubble is located, m is the number of shadows, is the size of the area where the jth shadow is located, q is the number of reflection areas, is the size of the area where the pth reflection area is located, and A is the total area size of the spinal endoscopy image.

5. The method according to any one of claims 2 to 4, characterized in that: The image preprocessing of each frame of the spinal endoscope image after filtering includes: Converting each frame of the filtered spinal endoscope image into an HSV space, and adjusting the V channel in the HSV space according to the gamma value to achieve brightness processing of the spinal endoscope image; The spinal endoscope image after brightness processing is reconverted into RGB format, and detail processing and contrast processing are performed on the spinal endoscope image reconverted into RGB format.

6. The method according to claim 5, characterized in that The determining the depth information of each pixel in the spinal structure image based on the angle information and position information corresponding to each frame of the spinal structure image includes: Inputting each frame of the spinal structure image and its corresponding angle information and position information into an image depth prediction model to obtain a depth prediction value for each pixel in the spinal structure image; Acquire matching image pixels by performing image registration on the spinal endoscopic images in the spinal endoscopic image group, estimate a disparity map between the spinal endoscopic images by using the matching image pixels, and obtain a depth estimation value of each pixel in the spinal structure image according to the disparity map; Depth information of each pixel in the spinal structure image is determined according to the depth prediction value and the depth estimation value.

7. The method according to claim 6, characterized in that The method of obtaining matching image pixels by performing image registration on the spinal endoscopic images in the spinal endoscopic image group, and estimating the disparity map between the spinal endoscopic images by using the matching image pixels, comprises: Calculating the pixel matching degree between the spinal endoscopic images in the spinal endoscopic image group by normalized correlation coefficient, and determining the pixel points whose pixel matching degree is greater than a preset value as matching image pixels; Determining a spinal endoscopic image in which the number of matched image pixel points in the spinal endoscopic image group exceeds a target number as a matched spinal endoscopic image; The disparity map between the spinal endoscopic images is estimated by using the matched image pixel points in the matched spinal endoscopic images in the spinal endoscopic image group.

8. An endoscope image processing device, characterized in that: The device comprises: An acquisition module, used for acquiring a spinal endoscope image from a spinal endoscope video stream, wherein each frame of the spinal endoscope image in the spinal endoscope video stream corresponds to time information, angle information, and position information; A filtering module, used for filtering the spinal endoscope image according to the time information, angle information and position information of each frame of the spinal endoscope image; A recognition module, used for performing image preprocessing on each frame of the filtered spinal endoscopic image, and inputting the image preprocessed spinal endoscopic image into a spinal recognition model to obtain a spinal position area corresponding to each frame of the spinal endoscopic image, wherein the spinal position area at least includes a position area where a vertebral body, an intervertebral disc and a nerve root are located; An extraction module, used for extracting a spinal structure image from the spinal endoscope image according to the spinal position area; A determination module, configured to determine the depth information of each pixel in the spinal structure image based on the angle information and position information corresponding to each frame of the spinal structure image; The projection conversion module is used to project the position information and depth information of each pixel point in each frame of the spinal structure image to the corresponding position in the three-dimensional space to form point cloud data; and convert the point cloud data into a triangular mesh to obtain a three-dimensional model of the endoscopic image of the spine.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the endoscopic image processing method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the endoscopic image processing method according to any one of claims 1 to 7 is implemented.

Images