Endoscope image adaptive sampling method, system and device and storage medium

Abstract

The invention discloses an endoscope image adaptive sampling method, system and device and a storage medium, and the method comprises the steps: obtaining a historical image set of an endoscope under a multi-scene condition, and carrying out the data processing and feature extraction, and obtaining an initial feature set; performing feature engineering on the initial feature set to obtain a target feature set; the method comprises the following steps: pre-constructing an initial adaptive sampling model comprising a CNN learning module, a reinforcement learning module and an integrated learning module; manufacturing a model data set according to the target feature set and extracting a training set from the model data set; inputting the training set into the initial adaptive sampling model for model training to obtain a target adaptive sampling model; and performing real-time image sampling on the current scene of the endoscope according to the target adaptive sampling model to obtain an adaptive sampling image of the endoscope. According to the invention, self-adaptive sampling can be carried out aiming at the real working environment of the endoscope, the system can adapt to real-time changing and complex imaging scenes, the improvement of endoscope images is really realized, and resource waste is avoided.

Description

Technical Field

[0001] The present invention relates to the field of medical treatment and image processing technology, and in particular to an endoscopic image adaptive sampling method, system, device and storage medium. Background Art

[0002] Endoscopic imaging technology plays a vital role in medical diagnosis and surgery, especially in minimally invasive surgery, where the real-time images provided by the endoscope are the key basis for doctors to perform precise operations.

[0003] However, endoscopic imaging is often affected by factors such as low light, complex tissue reflections, and dynamic environments, resulting in reduced image quality. In order to improve image quality, in traditional image processing technology, due to its fixed processing flow, the entire image is uniformly sampled and processed in a fixed sampling manner. Although this method can improve image quality to a certain extent, it does not perform adaptive sampling for the real real-time working environment of the endoscope. It can only improve the quality of part of the collected image, and the improvement is limited. It cannot ensure the quality of the endoscopic image obtained during the entire real-time sampling process, and cannot adapt to real-time changing and complex imaging scenes. In addition, uniform sampling and uniform processing in a fixed sampling method can easily lead to a waste of system resources. Summary of the invention

[0004] In view of this, the present invention provides an endoscopic image adaptive sampling method, system, device and storage medium to solve the problem that the existing endoscopic image processing technology does not perform adaptive sampling for the actual real-time working environment of the endoscope, resulting in the inability to improve the image quality in real time, limited improvement and easy waste of system resources.

[0005] The present invention provides an endoscopic image adaptive sampling method, the method comprising:

[0006] Acquire a historical image set of the endoscope under multiple scene conditions, and perform data processing and feature extraction on the historical image set to obtain an initial feature set corresponding to the historical image set;

[0007] Performing feature engineering on the initial feature set to obtain a target feature set;

[0008] Pre-constructing an initial adaptive sampling model, wherein the initial adaptive sampling model includes a CNN learning module, a reinforcement learning module and an integrated learning module;

[0009] A model data set is created according to the target feature set, and a training set is extracted from the model data set;

[0010] Inputting the training set into the initial adaptive sampling model, and sequentially performing model training using the CNN learning module, the reinforcement learning module, and the integrated learning module to obtain a target adaptive sampling model;

[0011] The target adaptive sampling model is used to perform real-time image sampling on the current scene of the endoscope to obtain an endoscope adaptive sampling image.

[0012] Optionally, performing real-time image sampling on the current scene of the endoscope according to the target adaptive sampling model to obtain an endoscope adaptive sampling image includes:

[0013] Obtain a current frame image in the current scene of the endoscope; assume that the current frame image is the i-th frame image, where i is a positive integer;

[0014] Using the target adaptive sampling model to predict the importance of partitions of the i-th frame image, and obtain a first importance score of the i-th frame image under multiple partitions;

[0015] Allocate sampling points to each partition of the i-th frame image according to all the first importance scores to obtain a first sampling point density of the i-th frame image in each partition;

[0016] The calculation formula of the first sampling point density is specifically:

[0017]

[0018] in, is the first sampling point density of the i-th frame image in the j-th partition, is the importance score of the i-th frame image in the j-th partition, and n is the total number of partitions;

[0019] According to all the first sampling point densities, acquiring the (i+1)th frame image of the current scene of the endoscope;

[0020] Using the target adaptive sampling model to predict the importance of partitions of the i+1th frame image, and obtaining the second importance score of the i+1th frame image in each partition;

[0021] Based on the first importance score of the i-th frame image in each partition and the second importance score of the i+1-th frame image in each partition, the first sampling point density of each partition is adjusted to obtain the second sampling point density of the i+1-th frame image in each partition;

[0022] According to all the second sampling point densities, acquiring the (i+2)th frame image in the current scene of the endoscope;

[0023] By analogy, each frame image of the endoscope in the current scene is obtained, and the endoscope adaptive sampling image is obtained according to all frame images of the endoscope in the current scene.

[0024] Optionally, the inputting the training set into the initial adaptive sampling model, sequentially using the CNN learning module, the reinforcement learning module and the integrated learning module to perform model training to obtain a target adaptive sampling model, comprises:

[0025] Inputting the training set into the CNN learning module in the initial adaptive sampling model, performing image recognition using the CNN learning module, and extracting a key feature library;

[0026] Inputting the key feature library into the reinforcement learning module in the initial adaptive sampling model, and using the reinforcement learning module to perform strategy learning to obtain a sampling strategy;

[0027] The sampling strategy is input into the integrated learning module in the initial adaptive sampling model, and the integrated learning model is used to perform strategy optimization to obtain a target adaptive sampling model.

[0028] Optionally, after obtaining the target adaptive sampling model, the method further includes:

[0029] Extracting a validation set from the model dataset;

[0030] Using the validation set to validate the target adaptive sampling model;

[0031] If the verification is passed, a grid search or Bayesian optimization method is used to adjust the model hyperparameters of the target adaptive sampling model;

[0032] If the verification fails, the model data set is re-made, and the training set is re-extracted, the re-extracted training set is input into the initial adaptive sampling model, and the model is re-trained using the CNN learning module, the reinforcement learning module and the integrated learning module in sequence until the target adaptive sampling model obtained again passes the verification; or, the initial adaptive sampling model is adjusted, and the training set is input into the adjusted initial adaptive sampling model, and the model is re-trained using the CNN learning module, the reinforcement learning module and the integrated learning module in the adjusted initial adaptive sampling model in sequence until the target adaptive sampling model obtained again passes the verification.

[0033] Optionally, performing feature engineering on the initial feature set to obtain a target feature set includes:

[0034] Using a feature importance evaluation method based on a tree model to obtain the feature importance of each feature in the initial feature set;

[0035] According to the importance of all the features, all the features in the initial feature set are filtered to obtain a first feature set;

[0036] Using a cross feature combination method, all features in the important feature set are combined to obtain a second feature set;

[0037] The second feature set is subjected to feature dimension reduction by using a principal component analysis method or a linear discriminant analysis method to obtain the target feature set.

[0038] Optionally, performing data processing and feature extraction on the historical image set to obtain an initial feature set corresponding to the historical image set includes:

[0039] Performing labeling processing on the historical image set to obtain a first image set; wherein the historical image set specifically includes historical working images of the endoscope under each scene condition in a continuous time;

[0040] Performing image cleaning and image enhancement on the first image set to obtain a second image set corresponding to the first image set;

[0041] The local features, global features and spatial features of each image in the second image set are extracted respectively, and the initial feature sets corresponding to the second image set are obtained respectively according to the local features, global features and spatial features of all images in the second image set.

[0042] Optionally, the spatial features of each image in the second image set include depth information between the endoscope and the target area;

[0043] Extracting the spatial features of each image in the second image set specifically includes:

[0044] Extracting the t-th frame image and the t+1-th frame image under any scene condition from the second image set, and extracting at least one feature point from the t-th frame image using a corner point detection method;

[0045] The Lucas-Kanade optical flow estimation method is used to solve the optical flow vector of each feature point between the t-th frame image and the t+1-th frame image;

[0046] According to the optical flow vector of each feature point between the t-th frame image and the t+1-th frame image, the distance between the endoscope and the target area at each feature point is calculated respectively;

[0047] The calculation formula of the distance between the endoscope and the target area at the feature point (x, y) is as follows:

[0048]

[0049] Among them, Z x,y is the distance between the endoscope and the target area at the feature point (x, y), Δx and Δy are the horizontal displacement and vertical displacement of the optical flow vector corresponding to the feature point (x, y) between the t-th frame image and the t+1-th frame image, f is the focal length of the endoscope, and B is the known movement distance of the endoscope between the t-th frame image and the t+1-th frame image;

[0050] According to the distance between the endoscope and the target area under all feature points, the depth information between the endoscope and the target area is calculated.

[0051] Optionally, the spatial features of each image in the second image set further include angle information between the endoscope and the target area;

[0052] Extracting the spatial features of each image in the second image set further specifically includes:

[0053] Extracting a k-th frame image under any scene condition from the second image set, performing edge detection on the k-th frame image, and obtaining a gradient direction vector of each pixel point in the k-th frame image;

[0054] According to the gradient direction vector of each pixel point in the k-th frame image, the gradient direction angle of each pixel point in the k-th frame image is calculated respectively;

[0055] Obtain multiple local windows of the k-th frame image, and calculate the main direction of each local window according to the gradient direction angles of all pixels in each local window;

[0056] The main directions of all local windows are weighted averaged to obtain the global main direction of the k-th frame image; or the RANSAC random sampling method is used to determine the global main direction of the k-th frame image from the main directions of all local windows;

[0057] According to the global main direction of the k-th frame image, the angle information between the endoscope and the target area is calculated;

[0058] The calculation formula of the angle information between the endoscope and the target area is as follows:

[0059] α=θ global -θ ref ;

[0060] Among them, α is the angle information between the endoscope and the target area, θ global is the global main direction of the k-th frame image, θ ref is the reference direction.

[0061] Optionally, each scene condition includes corresponding tissue site data, pathological state data, imaging environment data and imaging mode data.

[0062] In addition, the present invention also provides an endoscopic image adaptive sampling system, which is applied to the aforementioned endoscopic image adaptive sampling method, and the system comprises:

[0063] An image acquisition module, used to acquire a historical image set of the endoscope under multiple scene conditions;

[0064] A feature extraction module, used for performing data processing and feature extraction on the historical image set to obtain an initial feature set corresponding to the historical image set;

[0065] A feature processing module, used for performing feature engineering on the initial feature set to obtain a target feature set;

[0066] A model building module, used to pre-build an initial adaptive sampling model, wherein the initial adaptive sampling model includes a CNN learning module, a reinforcement learning module and an integrated learning module;

[0067] A model training module is used to produce a model data set according to a target feature set, and extract a training set from the model data set; input the training set into the initial adaptive sampling model, and perform model training using the CNN learning module, the reinforcement learning module, and the integrated learning module in sequence to obtain a target adaptive sampling model;

[0068] The model application module performs real-time image sampling on the current scene of the endoscope according to the target adaptive sampling model to obtain an endoscope adaptive sampling image.

[0069] In addition, the present invention also provides an endoscopic image adaptive sampling device, including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the method steps in the aforementioned endoscopic image adaptive sampling method when executed.

[0070] In addition, the present invention also provides a computer storage medium, which includes: at least one instruction, which implements the method steps in the aforementioned endoscopic image adaptive sampling method when the instruction is executed by a computer.

[0071] The beneficial effects of the present invention are as follows: by acquiring a historical image set of the endoscope under multiple scene conditions, the training set used for subsequent model training can meet the diversity, thereby ensuring that the target adaptive sampling model obtained through training has wide applicability; by processing the data of the historical image set, the quality and consistency of the data can be ensured, facilitating the subsequent feature extraction process, and then by extracting the features of the historical image set after data processing, it is convenient for subsequent model training, avoiding irrelevant factors in the image from participating in the model training, and effectively ensuring the efficiency and accuracy of the model training; by the feature engineering of the extracted initial feature set, the key features directly related to the endoscope sampling strategy can be extracted, and the feature space can be simplified and the calculation complexity can be reduced. The effect of complexity can further improve the efficiency and accuracy of subsequent model training; by pre-building an initial adaptive sampling model including CNN learning module, reinforcement learning module and integrated learning module, and then using the training set extracted from the produced model data set for model training, the complexity and real-time nature of endoscopic imaging can be fully considered to ensure that an ideal and accurate target adaptive sampling model is obtained; finally, the target adaptive sampling model is applied to the actual process, that is, in the current scene of the endoscope, real-time image sampling of the current scene is performed, and the image can be adaptively sampled according to the adaptive sampling strategy provided by the target adaptive model, so as to realize the adaptive sampling of endoscopic images and truly improve the image quality in real time;

[0072] The method, system, device and storage medium for adaptive sampling of endoscopic images of the present invention can adaptively sample the actual real-time working environment of the endoscope, can effectively ensure the quality of the endoscopic images obtained during the entire real-time sampling process, adapt to real-time changing and complex imaging scenes, and truly improve the quality of endoscopic images with greater improvement, without causing waste of system resources under the sampling mode of unified sampling and unified processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] The features and advantages of the present invention will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the present invention in any way. In the accompanying drawings:

[0074] Figure 1 A flow chart of an endoscopic image adaptive sampling method in Embodiment 1 of the present invention is shown;

[0075] Figure 2 A flow chart showing data processing and feature extraction of a historical image set in Embodiment 1 of the present invention is shown;

[0076] Figure 3 A flow chart showing feature engineering of an initial feature set in Embodiment 1 of the present invention is shown;

[0077] Figure 4A flow chart showing real-time image sampling of the current scene of the endoscope in the first embodiment of the present invention is shown;

[0078] Figure 5 The structure diagram of an endoscope image adaptive sampling system in the second embodiment of the present invention is shown. DETAILED DESCRIPTION

[0079] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0080] Embodiment 1

[0081] This embodiment provides an endoscope image adaptive sampling method, such as Figure 1 As shown, the method includes:

[0082] S1: Acquire a historical image set of an endoscope under multiple scene conditions, and perform data processing and feature extraction on the historical image set to obtain an initial feature set corresponding to the historical image set;

[0083] S2: Perform feature engineering on the initial feature set to obtain a target feature set;

[0084] S3: pre-constructing an initial adaptive sampling model, wherein the initial adaptive sampling model includes a CNN learning module, a reinforcement learning module and an integrated learning module;

[0085] S4: creating a model data set according to the target feature set, and extracting a training set from the model data set;

[0086] S5: inputting the training set into the initial adaptive sampling model, and sequentially performing model training using the CNN learning module, the reinforcement learning module, and the integrated learning module to obtain a target adaptive sampling model;

[0087] S6: Perform real-time image sampling on the current endoscope scene according to the target adaptive sampling model to obtain an endoscope adaptive sampling image.

[0088] In this embodiment, by acquiring a historical image set of the endoscope under multiple scene conditions, the training set used for subsequent model training can meet the diversity, thereby ensuring that the target adaptive sampling model obtained through training has wide applicability; through data processing of the historical image set, the quality and consistency of the data can be ensured, which is convenient for the subsequent feature extraction process, and then through feature extraction of the historical image set after data processing, it is convenient for subsequent model training, avoiding irrelevant factors in the image from participating in model training, and effectively ensuring the efficiency and accuracy of model training; through feature engineering of the extracted initial feature set, it is possible to extract key features directly related to the endoscope sampling strategy, and also to simplify the feature space and reduce the complexity of calculation. The efficiency and accuracy of subsequent model training can be further improved by pre-building an initial adaptive sampling model including CNN learning module, reinforcement learning module and ensemble learning module, and then using the training set extracted from the produced model data set for model training, which can fully take into account the complexity and real-time nature of endoscopic imaging, and ensure that an ideal and accurate target adaptive sampling model is obtained; finally, the target adaptive sampling model is applied to the actual process, that is, in the current scene of the endoscope, real-time image sampling of the current scene is performed, and the image can be adaptively sampled according to the adaptive sampling strategy provided by the target adaptive model, so as to realize the adaptive sampling of endoscopic images and truly improve the image quality in real time.

[0089] The adaptive sampling method for endoscopic images of the present embodiment can perform adaptive sampling according to the actual real-time working environment of the endoscope, can effectively ensure the quality of the endoscopic images obtained during the entire real-time sampling process, adapt to real-time changing and complex imaging scenarios, and truly improve the quality of endoscopic images with greater improvement, without causing waste of system resources under the sampling methods of unified sampling and unified processing.

[0090] Each step of the method for adaptive sampling of endoscope images in this embodiment is described in detail below.

[0091] Specifically, in the present embodiment S1, each scene condition includes corresponding tissue site data, pathological state data, imaging environment data and imaging mode data.

[0092] Tissue part data refers to the tissue part to which an image in the historical image set belongs, that is, what tissue part the image is collected from. The data includes types such as the digestive tract, respiratory tract, and urinary system.

[0093] Pathological state data refers to the pathological state corresponding to the tissue part to which the image in the historical image set belongs, that is, the image is an image of the collected tissue part under a certain pathological state. The data includes types such as normal state and pathological state (such as tumor, ulcer, inflammation, etc.).

[0094] Imaging environment data refers to the imaging environment to which the image in the historical image set belongs, that is, the imaging environment in which the image was collected. The data includes low-light environment, strong-light environment, light source type, light source brightness, light source equipment parameters (such as light source angle, lens angle), dynamic scene, static scene, etc.

[0095] The imaging mode data refers to the imaging mode of an image in a historical image set, that is, the imaging mode in which the image was acquired. The data includes ultrasound imaging, fluorescence imaging, and the like.

[0096] Through the above-mentioned scene conditions, it can be effectively ensured that the acquired historical image set covers a variety of different scenes and imaging conditions, so that the training set used for subsequent model training meets the diversity, and truly ensures that the target adaptive sampling model obtained through training has wide applicability.

[0097] Specifically, in the present embodiment S1, the historical image set specifically includes historical working images of the endoscope in a continuous period of time under each scene condition.

[0098] In the above-mentioned historical image set, by collecting historical working images in continuous time under each scene condition, it is convenient to subsequently analyze the dynamic characteristics of the endoscopic image and obtain features with dynamic characteristics; since the dynamic characteristics are closely related to the sampling environment of the endoscope, it can play a decisive role in the sampling strategy of the endoscopic image. Therefore, obtaining features with dynamic characteristics based on historical working images in continuous time can ensure the accuracy and reliability of subsequent model training.

[0099] Specifically, the above-mentioned historical image set in this embodiment can be obtained through working images collected by the endoscope during the historical period, and can also be obtained in a big data network through search methods such as web crawlers, and this embodiment does not impose any restrictions.

[0100] Preferably, in this embodiment S1, if Figure 2 As shown, data processing and feature extraction are performed on the historical image set to obtain an initial feature set corresponding to the historical image set, including:

[0101] S11: performing labeling processing on the historical image set to obtain a first image set;

[0102] S12: performing image cleaning and image enhancement on the first image set to obtain a second image set corresponding to the first image set;

[0103] S13: extracting local features, global features and spatial features of each image in the second image set respectively, and obtaining initial feature sets corresponding to the second image set according to the local features, global features and spatial features of all images in the second image set respectively.

[0104] By annotating the historical image set, the target area in each image (such as diseased tissue, important anatomical structure, textured area, etc.) can be clearly identified, and high-quality and low-quality images in the historical image set can be preliminarily screened out to provide a reference for subsequent feature extraction and model training; image cleaning can be used to remove low-quality images that are blurred, too dark, or contain obvious artifacts, so as to improve the quality of the subsequent model data set, and on the other hand, it is convenient to delete duplicate image samples to avoid model overfitting; image enhancement can improve the visual effect of the image and enhance the effect of subsequent model training; finally, by extracting the local features, global features, and spatial features of each image respectively, it is convenient to capture important features related to the sampling strategy, which helps to improve the accuracy and efficiency of subsequent model training.

[0105] Specifically, in S11 of this embodiment, the labeling process specifically adopts a manual labeling method, which can better combine expert experience to determine the target area of ​​the image and the high-quality image and the low-quality image.

[0106] In S12 of this embodiment, the image enhancement may adopt a histogram equalization method, a contrast adjustment method and a denoising method. This embodiment specifically adopts a histogram equalization method, and its corresponding processing formula is:

[0107]

[0108] Where I(x,y) is the initial brightness of the pixel at (x,y) in the first image of the first image set after image cleaning, and I new (x, y) is the enhanced brightness of the pixel at (x, y) of the first image in the first image set after image cleaning, CDF is the cumulative distribution function of the first image in the first image set after image cleaning, min(CDF) is the minimum value in the cumulative distribution function of the first image in the first image set after image cleaning, N and M are the horizontal and vertical sizes of the first image in the first image set after image cleaning, respectively.

[0109] In the present embodiment S13, the local features include texture information, edge information, brightness gradient, etc. Texture information is the arrangement pattern of pixels in the local area of ​​the image in the second image set, and the extraction methods used include gray level co-occurrence matrix (GLCM) extraction method, local binary pattern (LBP) extraction method and wavelet transform extraction method, etc. Edge information refers to the place where the brightness of the image in the second image set changes dramatically, usually corresponding to the boundary of the tissue part, and the extraction methods used include Canny edge detection method and Sobel operator detection method, etc. Brightness extraction describes the speed and direction of brightness changes in the image in the second image set, which can be obtained by calculating the first-order or second-order derivative of the image.

[0110] In the present embodiment S13, the global features include overall brightness, contrast, and hue distribution. Overall brightness reflects the average brightness level of the image in the second image set, which can be obtained by calculating the average brightness of all pixels in the image. Contrast describes the difference between the brightest and darkest parts of the image in the second image set, which can be evaluated by calculating the standard deviation of the image brightness or the distribution range of the histogram. Hue distribution can reflect the color information of different tissue parts, which can be analyzed by color histogram.

[0111] Preferably, the spatial features of each image in the second image set include depth information between the endoscope and the target area;

[0112] In this embodiment S13, extracting the spatial features of each image in the second image set specifically includes:

[0113] S131: extracting the t-th frame image and the t+1-th frame image under any scene condition from the second image set, and extracting at least one feature point from the t-th frame image using a corner point detection method;

[0114] S132: using the Lucas-Kanade optical flow estimation method, solving the optical flow vector of each feature point between the t-th frame image and the t+1-th frame image;

[0115] S133: calculating the distance between the endoscope and the target area at each feature point according to the optical flow vector between the t-th frame image and the t+1-th frame image;

[0116] The calculation formula of the distance between the endoscope and the target area at the feature point (x, y) is as follows:

[0117]

[0118] Among them, Z x,y is the distance between the endoscope and the target area at the feature point (x, y), Δx and Δy are the horizontal displacement and vertical displacement of the optical flow vector corresponding to the feature point (x, y) between the t-th frame image and the t+1-th frame image, f is the focal length of the endoscope, and B is the known movement distance of the endoscope between the t-th frame image and the t+1-th frame image;

[0119] S134: Calculate the depth information between the endoscope and the target area according to the distance between the endoscope and the target area at all feature points.

[0120] By extracting two consecutive frames of images (i.e., the t-th frame image and the t+1-th frame image) under any scene condition in the second image set, the viewing angles of the endoscope at different positions within a short time interval under the scene condition can be better reflected, and the subsequent optical flow calculation can be better facilitated, and then the depth information between the endoscope and the target area (the pathological tissue, important anatomical structure, textured area and other areas of interest) can be inferred based on the calculated optical flow vector; at least one feature point can be extracted by the corner detection method, and the feature points with strong response in the image can be captured, which can better reflect the motion characteristics of the endoscope within a short time interval; the Lucas-Kanade optical flow estimation method is a A popular optical flow estimation algorithm, which is based on the basic assumption of optical flow: in two consecutive frames of images, the brightness (or grayscale value) of the same pixel or feature point is constant, and the local area around the pixel point also follows this assumption. The optical flow vector is estimated by minimizing the square sum of the pixel brightness changes in the local window. The optical flow vector estimated by this method can reflect the displacement of the feature point in the two frames of images, and then the motion pattern of the entire image is analyzed to obtain the distance between the endoscope and the target area at each feature point; finally, the distance between the endoscope and the target area at all feature points can be used to estimate the average distance between the endoscope and the target area in the entire image (i.e., depth information).

[0121] Specifically, in S131 of this embodiment, before using the corner detection method to extract feature points, the extracted t-th frame image and t+1-th frame image are first grayed out, which can make the subsequent optical flow calculation more stable.

[0122] Specifically, in this embodiment S131, the corner detection method used is specifically the Shi-Tomasi corner detection method. The Shi-Tomasi corner detection algorithm determines whether a point is a corner point (i.e., a feature point) by calculating the eigenvalue of the autocorrelation matrix (also called the gradient covariance matrix) of each pixel in the image. Specifically, it focuses on the two eigenvalues ​​λ1 and λ2 of the matrix:

[0123] When both λ1 and λ2 are smaller than a certain threshold λmin, the region is considered flat.

[0124] When both λ1 and λ2 are greater than λmin, the point is considered a corner point.

[0125] When only one of λ1 and λ2 is greater than λmin, the region is considered to be an edge.

[0126] This method uses a simpler corner response function, that is, directly taking the smaller of the two eigenvalues ​​as the response strength. When this value is greater than a certain threshold, it is considered that a corner is detected. This method is simpler and more efficient while ensuring accuracy.

[0127] The specific operation method of the Lucas-Kanade optical flow estimation method in S132 of this embodiment adopts a conventional operation method, and its specific operation process will not be repeated here.

[0128] In S134 of this embodiment, when the distances between the endoscope and the target area at all feature points are calculated, an average value or a weighted sum method may be used to obtain the final depth information.

[0129] Preferably, the spatial features of each image in the second image set further include angle information between the endoscope and the target area;

[0130] In this embodiment S13, extracting the spatial features of each image in the second image set further specifically includes:

[0131] S135: extracting a k-th frame image under any scene condition from the second image set, performing edge detection on the k-th frame image, and obtaining a gradient direction vector of each pixel point in the k-th frame image;

[0132] S136: Calculate the gradient direction angle of each pixel point in the k-th frame image according to the gradient direction vector of each pixel point in the k-th frame image;

[0133] S137: Acquire multiple local windows of the k-th frame image, and calculate the main direction of each local window according to the gradient direction angles of all pixels in each local window;

[0134] S138: performing a weighted average operation on the main directions of all local windows to obtain the global main direction of the k-th frame image; or using a RANSAC random sampling method to determine the global main direction of the k-th frame image from the main directions of all local windows;

[0135] S139: Calculate the angle information between the endoscope and the target area according to the global main direction of the k-th frame image;

[0136] The calculation formula of the angle information between the endoscope and the target area is as follows:

[0137] α=θ global -θ ref ;

[0138] Among them, α is the angle information between the endoscope and the target area, θ global is the global main direction of the k-th frame image, θ ref is the reference direction.

[0139] By extracting the kth frame image under any scene condition and performing edge detection, the contour and edge in the image can be extracted, and then the edge or surface texture of the tissue part can be determined; when the gradient direction vector of each pixel point is obtained, the corresponding gradient direction angle can be calculated, which is convenient for subsequent main direction analysis of the local area. The main direction of the local area represents the direction in which the brightness or intensity changes most dramatically in the local area, and different endoscope angles will cause the brightness or intensity of the local area of ​​the image to change, that is, different endoscope angles will cause the main direction of the local area to change. Therefore, this embodiment can facilitate subsequent global direction analysis of the contour or texture by determining the main direction of the local area, and then accurately estimate the inclination angle of the endoscope (that is, the angle information between the endoscope and the target area). In the process of estimating the main direction of the local area, all gradient directions of the area are counted to determine the main direction of the area; and in the process of global direction analysis, the main directions of all local areas can be integrated through weighted average operation or RANSAC random sampling to achieve estimation of the global main direction.

[0140] Specifically, in S135 of this embodiment, before edge detection, the k-th frame image is first grayscaled, and the grayscaled image still retains brightness information, which simplifies subsequent processing.

[0141] In this embodiment S135, the Sobel operator detection method, the Canny edge detection method and other methods are specifically used to perform edge detection. The specific operation methods of the Sobel operator detection method and the Canny edge detection method all adopt conventional operation methods, and the specific details are not repeated here.

[0142] In S135, for any pixel point (x, y) in the k-th frame image, its corresponding gradient direction vector is In S236, the specific formula for calculating the corresponding gradient direction angle is:

[0143]

[0144] Where θx,y is the gradient direction angle at the pixel point (x,y) in the k-th frame image, and I is the brightness at the pixel point (x,y).

[0145] In S137 of this embodiment, for each local window, a histogram may be used to count the gradient direction angles of all pixels in the local window, and then the gradient direction angle corresponding to the histogram peak is selected as the main direction of the local window.

[0146] Preferably, if Figure 3 As shown, this embodiment S2 includes:

[0147] S21: using a feature importance evaluation method based on a tree model to obtain the feature importance of each feature in the initial feature set;

[0148] S22: filtering all the features in the initial feature set according to the importance of all the features to obtain a first feature set;

[0149] S23: using a cross feature combination method to combine all the features in the important feature set to obtain a second feature set;

[0150] S24: Performing feature dimension reduction on the second feature set using a principal component analysis method or a linear discriminant analysis method to obtain the target feature set.

[0151] By using a feature importance evaluation method based on a tree model to evaluate feature importance and perform filtering, we can further extract important features that are closely related to the sampling strategy, while also reducing computational costs and avoiding overfitting. Then, through feature combination, we can generate new features with higher information content, which can more effectively capture the complex relationships between the original features and improve the performance of the model. Finally, feature dimensionality reduction can be performed to reduce the dimension of features by mapping or transformation while maintaining the main information of the data set. This can not only reduce computational complexity, but also help reduce the risk of overfitting.

[0152] Specifically, before S21 of this embodiment, the initial feature set can also be standardized to eliminate the influence of dimension; the initial feature set after standardization can also be rotated, scaled, translated, flipped and other data enhancement processes to effectively improve the robustness of subsequent models.

[0153] In this embodiment S3, the CNN learning module (convolutional neural network learning module) in the initial adaptive sampling model can automatically identify and classify the diseased areas in the endoscopic image, and has stronger image recognition capabilities; the intensity learning module in the initial adaptive sampling model can learn the optimal adoption strategy to improve the efficiency and quality of image acquisition; the integrated learning module in the initial adaptive sampling model can better adapt to multimodal data analysis and better adapt to training sets under multi-scenario conditions, thereby improving the model prediction ability and obtaining a target adaptive sampling model with higher accuracy.

[0154] In this embodiment S4, after the model data set is produced, the model data set is divided into a training set, a test set and a validation set according to a preset ratio (such as a ratio of 8:1:1).

[0155] Preferably, this embodiment S5 includes:

[0156] S51: inputting the training set into the CNN learning module in the initial adaptive sampling model, performing image recognition using the CNN learning module, and extracting a key feature library;

[0157] S52: inputting the key feature library into the reinforcement learning module in the initial adaptive sampling model, and performing strategy learning using the reinforcement learning module to obtain a sampling strategy;

[0158] S53: Input the sampling strategy into the integrated learning module in the initial adaptive sampling model, and use the integrated learning model to optimize the strategy to obtain a target adaptive sampling model.

[0159] Through the above model training process, a target adaptive model that can output the optimal sampling strategy can be gradually learned, so that the sampling density can be intelligently adjusted during the endoscopic imaging process.

[0160] Among them, in S51, the training set is input into the convolution layer, pooling layer and other structures of the CNN learning module to automatically extract high-level feature representations, namely the key feature library, from the preprocessed image. In S52, based on the key feature library extracted by the CNN learning module and the reward function defined by the reinforcement learning module, the sampling strategy is trained using the reinforcement learning module. During the training process, new sampling images can be continuously input into the CNN learning module for feature extraction to update the key feature library and the reward function, thereby obtaining the sampling strategy. In S53, while or after the reinforcement learning model is trained, an integrated learning model is used to evaluate and optimize multiple sampling strategies. The training process composed of the above three learning modules is an iterative optimization process, and finally a target adaptive sampling model for endoscopic image sampling is obtained that combines CNN school, reinforcement learning and integrated learning.

[0161] Preferably, after S53, the method further includes:

[0162] S54: extracting a validation set from the model data set;

[0163] S55: Verify the target adaptive sampling model using the verification set;

[0164] S56A: If the verification is passed, a grid search or Bayesian optimization method is used to adjust the model hyperparameters of the target adaptive sampling model;

[0165] S56B: If the verification fails, the model data set is re-made, and the training set is re-extracted, and the re-extracted training set is input into the initial adaptive sampling model, and the model is re-trained using the CNN learning module, the reinforcement learning module and the integrated learning module in sequence until the target adaptive sampling model obtained again passes the verification; or, the initial adaptive sampling model is adjusted, and the training set is input into the adjusted initial adaptive sampling model, and the model is re-trained using the CNN learning module, the reinforcement learning module and the integrated learning module in the adjusted initial adaptive sampling model in sequence until the target adaptive sampling model obtained again passes the verification.

[0166] Through the above verification process, on the one hand, the effectiveness of the target adaptive sampling model can be ensured, ensuring its performance, and on the other hand, its robustness can be verified to reduce the risk of overfitting or underfitting. When the verification is passed, further adjustment of the model hyperparameters can further optimize the model so that the model has the best performance. If the verification fails, the model is retrained and verified until the verification is passed, truly ensuring the performance and robustness of the target adaptive sampling model.

[0167] Specifically, in S55, on the one hand, the performance and generalization ability of the model are measured by calculating indicators such as accuracy, precision, recall and F1-score (i.e., the harmonic mean of precision and recall); on the other hand, the robustness of the model is verified by the k-fold cross-validation method.

[0168] Specifically, in S56A, the hyperparameters of the model include learning rate, number of layers, number of nodes, regularization parameters, etc. The specific operation method of the grid search or Bayesian optimization method adopts the conventional operation method, and the specific details are not repeated here.

[0169] Preferably, if Figure 4 As shown, this embodiment S6 includes:

[0170] S61: Acquire a current frame image in the current scene of the endoscope; assume that the current frame image is the i-th frame image, where i is a positive integer;

[0171] S62: using the target adaptive sampling model to perform partition importance prediction on the i-th frame image, and obtain a first importance score of the i-th frame image under multiple partitions;

[0172] S63: Allocate sampling points to each partition of the i-th frame image according to all the first importance scores to obtain a first sampling point density of the i-th frame image in each partition;

[0173] The calculation formula of the first sampling point density is specifically:

[0174]

[0175] in, is the first sampling point density of the i-th frame image in the j-th partition, is the importance score of the i-th frame image in the j-th partition, and n is the total number of partitions;

[0176] S64: acquiring the (i+1)th frame image of the current scene of the endoscope according to all the first sampling point densities;

[0177] S65: using the target adaptive sampling model to perform partition importance prediction on the i+1th frame image, and obtain the second importance score of the i+1th frame image in each partition;

[0178] S66: Based on the first importance score of the i-th frame image in each partition and the second importance score of the i+1-th frame image in each partition, adjusting the first sampling point density of each partition to obtain the second sampling point density of the i+1-th frame image in each partition;

[0179] S67: Acquire the (i+2)th frame image of the current scene of the endoscope according to all the second sampling point densities;

[0180] S68: Similarly, each frame image of the endoscope in the current scene is obtained, and the endoscope adaptive sampling image is obtained according to all frame images of the endoscope in the current scene.

[0181] The scene conditions in which the endoscope is actually applied are the current scene. First, the current frame image (set as the i-th frame image) is input into the target adaptive sampling model, and the importance score of each partition of the image (i.e., the first importance score) can be output. The sampling point density of each partition can be calculated based on the importance score of each partition (i.e., the first sampling point density) to achieve the sampling point allocation of the current frame image. These first sampling point densities can realize the adaptive sampling of the next frame image corresponding to the current frame image (i.e., the i+1-th frame image), that is, the i+1-th frame image is obtained, and can also be used as the basis for the allocation of sampling points of the next frame image. When the i+1-th frame image is obtained, the importance score of each partition of the i+1-th frame image can be output in the same way as the first importance score. Importance score (i.e., second importance score); based on the first importance score and the second importance score under each partition, the change of partition importance during the dynamic sampling process of endoscopic images can be captured, and then the sampling points of each partition are adjusted, that is, the first sampling point density of the previous frame is adjusted to obtain a new sampling point density (i.e., second sampling point density) for each partition; these second sampling point densities can, on the one hand, realize the adaptive sampling of the next frame image (i+2 frame) corresponding to the i+1 frame image, that is, obtain the i+2 frame image, and on the other hand, can also be used as the basis for allocating sampling points for the i+3 frame image; and so on, the sampling point allocation of each subsequent frame image can be realized and the corresponding image sampling can be realized, truly realizing the adaptive sampling of endoscopic images.

[0182] Specifically, in S63, the higher the first importance score of the partition, the more details of the target area exist in the corresponding partition, and more sampling points need to be allocated (i.e., the first sampling point density is higher). Correspondingly, the lower the first importance score of the partition, the fewer sampling points need to be allocated (i.e., the first sampling point density is lower). The more important the area, the more sampling points are allocated, and the less important the area, the fewer sampling points are allocated. On the basis of ensuring that sufficient details are mastered in the key areas, computing resources can be effectively saved, and the quality of the endoscopic image can be truly improved, and the improvement is greater, without causing waste of system resources under the sampling mode of unified sampling and unified processing.

[0183] Specifically, in S66, the first sampling point density is adjusted based on the first importance score and the second importance score, and a suitable adjustment strategy can be selected according to the actual situation. For example, when the importance scores of a partition in the two frames of images are both greater than the preset threshold, the corresponding first sampling point density is increased according to the preset ratio as the corresponding new second sampling point density; when only one of the importance scores of a partition in the two frames of images is greater than the preset threshold, the corresponding first sampling point density is maintained unchanged as the corresponding second sampling point density; when the importance scores of a partition in the two frames of images are both less than or equal to the preset threshold, the corresponding first sampling point density is reduced according to the preset ratio as the corresponding new second sampling point density. In the above adjustment strategy, the preset threshold and the preset ratio can be pre-set and adjusted according to the actual situation, and are not limited here.

[0184] Embodiment 2

[0185] An endoscopic image adaptive sampling system is applied to the endoscopic image adaptive sampling method of embodiment 1, such as Figure 5 As shown, the system comprises:

[0186] An image acquisition module, used to acquire a historical image set of the endoscope under multiple scene conditions;

[0187] A feature extraction module, used for performing data processing and feature extraction on the historical image set to obtain an initial feature set corresponding to the historical image set;

[0188] A feature processing module, used for performing feature engineering on the initial feature set to obtain a target feature set;

[0189] A model building module, used to pre-build an initial adaptive sampling model, wherein the initial adaptive sampling model includes a CNN learning module, a reinforcement learning module and an integrated learning module;

[0190] A model training module is used to produce a model data set according to a target feature set, and extract a training set from the model data set; input the training set into the initial adaptive sampling model, and perform model training using the CNN learning module, the reinforcement learning module, and the integrated learning module in sequence to obtain a target adaptive sampling model;

[0191] The model application module performs real-time image sampling on the current scene of the endoscope according to the target adaptive sampling model to obtain an endoscope adaptive sampling image.

[0192] In this embodiment, by utilizing the image acquisition module, by acquiring the historical image sets of the endoscope under multiple scene conditions, the training sets subsequently used for model training can meet the diversity, thereby ensuring that the target adaptive sampling model obtained through training has wide applicability; by utilizing the feature extraction module, by processing the data of the historical image sets, the quality and consistency of the data can be ensured, and the subsequent feature extraction process is convenient. Then, by extracting the features of the historical image sets after data processing, subsequent model training is facilitated, and irrelevant factors in the image are avoided from participating in the model training, effectively ensuring the efficiency and accuracy of the model training; by utilizing the feature processing module, by the feature engineering of the extracted initial feature sets, the key features directly related to the endoscope sampling strategy can be extracted, and the feature space can be simplified and the calculation complexity can be reduced. The efficiency and accuracy of subsequent model training can be further improved by using the model construction module, by pre-building an initial adaptive sampling model including a CNN learning module, a reinforcement learning module and an integrated learning module, and then by using the model training module, using the training set extracted from the produced model data set for model training, which can fully consider the complexity and real-time nature of endoscopic imaging, and ensure that an ideal and accurate target adaptive sampling model is obtained; finally, using the model application module, the target adaptive sampling model is applied to the actual process, that is, in the current scene of the endoscope, real-time image sampling of the current scene is performed, and the image can be adaptively sampled according to the adaptive sampling strategy provided by the target adaptive model, so as to realize the adaptive sampling of endoscopic images and truly improve the image quality in real time.

[0193] The endoscopic image adaptive sampling system of the present embodiment can perform adaptive sampling according to the actual real-time working environment of the endoscope, can effectively ensure the quality of the endoscopic image obtained during the entire real-time sampling process, adapt to real-time changing and complex imaging scenes, and truly improve the quality of endoscopic images with greater improvement, without causing waste of system resources under the sampling mode of unified sampling and unified processing.

[0194] The functions of each module in the endoscopic image adaptive sampling system described in this embodiment are the same as the method steps of the endoscopic image adaptive sampling method described in Example 1. Therefore, for details not included in this embodiment, please refer to Example 1 and Example 2. Figure 1 and Figure 4 The detailed description will not be repeated here.

[0195] Embodiment 3

[0196] This embodiment also provides an endoscopic image adaptive sampling device, including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the method steps in the endoscopic image adaptive sampling method of the first embodiment when executed.

[0197] Through the computer program stored in the memory and running on the processor, adaptive sampling can be performed according to the actual real-time working environment of the endoscope, which can effectively ensure the quality of the endoscopic image obtained in the entire real-time sampling process, adapt to real-time changing and complex imaging scenes, and truly improve the quality of endoscopic images with greater improvement, without causing waste of system resources under the sampling mode of unified sampling and unified processing.

[0198] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of a computer device, and uses various interfaces and lines to connect various parts of the entire computer device.

[0199] The memory can be used to store computer programs and / or models. The processor realizes various functions of the computer device by running or executing the computer programs and / or models stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system and at least one application required for a function (such as a sound playback function, an image playback function, etc.); the data storage area can store data created according to the use of the mobile phone (such as audio data, video data, etc.). In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0200] It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by a computer program. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0201] These computer programs may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0202] These computer programs can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0203] This embodiment further provides a computer storage medium, which includes: at least one instruction, which implements the method steps in the endoscopic image adaptive sampling method of embodiment 1 when the instruction is executed by a computer.

[0204] By executing a computer storage medium containing at least one instruction, adaptive sampling can be performed according to the actual real-time working environment of the endoscope, which can effectively ensure the quality of the endoscopic image obtained during the entire real-time sampling process, adapt to real-time changing and complex imaging scenes, and truly improve the quality of the endoscopic image with greater improvement, without causing waste of system resources under the sampling method of unified sampling and unified processing.

[0205] Similarly, for details not yet provided in this embodiment, please refer to Embodiment 1, Embodiment 2 and Figures 1 to 5 The detailed description will not be repeated here.

[0206] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. An endoscopic image adaptive sampling method, characterized in that: The method comprises: Acquire a historical image set of the endoscope under multiple scene conditions, and perform data processing and feature extraction on the historical image set to obtain an initial feature set corresponding to the historical image set; Performing feature engineering on the initial feature set to obtain a target feature set; Pre-constructing an initial adaptive sampling model, wherein the initial adaptive sampling model includes a CNN learning module, a reinforcement learning module and an integrated learning module; A model data set is created according to the target feature set, and a training set is extracted from the model data set; Inputting the training set into the initial adaptive sampling model, and sequentially performing model training using the CNN learning module, the reinforcement learning module, and the integrated learning module to obtain a target adaptive sampling model; The target adaptive sampling model is used to perform real-time image sampling on the current scene of the endoscope to obtain an endoscope adaptive sampling image.

2. The method for adaptive sampling of endoscope images according to claim 1, characterized in that: Performing real-time image sampling on the current scene of the endoscope according to the target adaptive sampling model to obtain an endoscope adaptive sampling image includes: Obtain a current frame image in the current scene of the endoscope; assume that the current frame image is the i-th frame image, where i is a positive integer; Using the target adaptive sampling model to predict the importance of partitions of the i-th frame image, and obtain a first importance score of the i-th frame image under multiple partitions; Allocate sampling points to each partition of the i-th frame image according to all the first importance scores to obtain a first sampling point density of the i-th frame image in each partition; The calculation formula of the first sampling point density is specifically: in, is the first sampling point density of the i-th frame image in the j-th partition, is the importance score of the i-th frame image in the j-th partition, and n is the total number of partitions; According to all the first sampling point densities, acquiring the (i+1)th frame image of the current scene of the endoscope; Using the target adaptive sampling model to predict the importance of partitions of the i+1th frame image, and obtaining the second importance score of the i+1th frame image in each partition; Based on the first importance score of the i-th frame image in each partition and the second importance score of the i+1-th frame image in each partition, the first sampling point density of each partition is adjusted to obtain the second sampling point density of the i+1-th frame image in each partition; According to all the second sampling point densities, acquiring the (i+2)th frame image in the current scene of the endoscope; By analogy, each frame image of the endoscope in the current scene is obtained, and the endoscope adaptive sampling image is obtained according to all frame images of the endoscope in the current scene.

3. The method for adaptive sampling of endoscope images according to claim 1, characterized in that: The step of inputting the training set into the initial adaptive sampling model, and sequentially performing model training using the CNN learning module, the reinforcement learning module, and the integrated learning module to obtain a target adaptive sampling model includes: Inputting the training set into the CNN learning module in the initial adaptive sampling model, performing image recognition using the CNN learning module, and extracting a key feature library; Inputting the key feature library into the reinforcement learning module in the initial adaptive sampling model, and using the reinforcement learning module to perform strategy learning to obtain a sampling strategy; The sampling strategy is input into the integrated learning module in the initial adaptive sampling model, and the integrated learning model is used to perform strategy optimization to obtain a target adaptive sampling model.

4. The method for adaptive sampling of endoscope images according to claim 1, characterized in that: After obtaining the target adaptive sampling model, the method further includes: Extracting a validation set from the model dataset; Using the validation set to validate the target adaptive sampling model; If the verification is passed, a grid search or Bayesian optimization method is used to adjust the model hyperparameters of the target adaptive sampling model; If the verification fails, the model data set is re-made, and the training set is re-extracted, the re-extracted training set is input into the initial adaptive sampling model, and the model is re-trained using the CNN learning module, the reinforcement learning module and the integrated learning module in sequence until the target adaptive sampling model obtained again passes the verification; or, the initial adaptive sampling model is adjusted, and the training set is input into the adjusted initial adaptive sampling model, and the model is re-trained using the CNN learning module, the reinforcement learning module and the integrated learning module in the adjusted initial adaptive sampling model in sequence until the target adaptive sampling model obtained again passes the verification.

5. The method for adaptive sampling of endoscope images according to claim 1, characterized in that: The performing feature engineering on the initial feature set to obtain a target feature set includes: Using a feature importance evaluation method based on a tree model to obtain the feature importance of each feature in the initial feature set; According to the importance of all the features, all the features in the initial feature set are filtered to obtain a first feature set; Using a cross feature combination method, all features in the important feature set are combined to obtain a second feature set; The second feature set is subjected to feature dimension reduction by using a principal component analysis method or a linear discriminant analysis method to obtain the target feature set.

6. The method for adaptive sampling of endoscope images according to claim 1, characterized in that: The performing data processing and feature extraction on the historical image set to obtain an initial feature set corresponding to the historical image set includes: Performing labeling processing on the historical image set to obtain a first image set; wherein the historical image set specifically includes historical working images of the endoscope under each scene condition in a continuous time; Performing image cleaning and image enhancement on the first image set to obtain a second image set corresponding to the first image set; The local features, global features and spatial features of each image in the second image set are extracted respectively, and the initial feature sets corresponding to the second image set are obtained respectively according to the local features, global features and spatial features of all images in the second image set.

7. The method for adaptive sampling of endoscope images according to claim 6, characterized in that: The spatial features of each image in the second image set include depth information between the endoscope and the target area; Extracting the spatial features of each image in the second image set specifically includes: Extracting the t-th frame image and the t+1-th frame image under any scene condition from the second image set, and extracting at least one feature point from the t-th frame image using a corner point detection method; The Lucas-Kanade optical flow estimation method is used to solve the optical flow vector of each feature point between the t-th frame image and the t+1-th frame image; According to the optical flow vector of each feature point between the t-th frame image and the t+1-th frame image, the distance between the endoscope and the target area at each feature point is calculated respectively; The calculation formula of the distance between the endoscope and the target area at the feature point (x, y) is as follows: Among them, Z x,y is the distance between the endoscope and the target area at the feature point (x, y), Δx and Δy are the horizontal displacement and vertical displacement of the optical flow vector corresponding to the feature point (x, y) between the t-th frame image and the t+1-th frame image, f is the focal length of the endoscope, and B is the known movement distance of the endoscope between the t-th frame image and the t+1-th frame image; According to the distance between the endoscope and the target area under all feature points, the depth information between the endoscope and the target area is calculated.

8. The method for adaptive sampling of endoscope images according to claim 6, characterized in that: The spatial features of each image in the second image set also include angle information between the endoscope and the target area; Extracting the spatial features of each image in the second image set further specifically includes: Extracting a k-th frame image under any scene condition from the second image set, performing edge detection on the k-th frame image, and obtaining a gradient direction vector of each pixel point in the k-th frame image; According to the gradient direction vector of each pixel point in the k-th frame image, the gradient direction angle of each pixel point in the k-th frame image is calculated respectively; Obtain multiple local windows of the k-th frame image, and calculate the main direction of each local window according to the gradient direction angles of all pixels in each local window; The main directions of all local windows are weighted averaged to obtain the global main direction of the k-th frame image; or the RANSAC random sampling method is used to determine the global main direction of the k-th frame image from the main directions of all local windows; According to the global main direction of the k-th frame image, the angle information between the endoscope and the target area is calculated; The calculation formula of the angle information between the endoscope and the target area is as follows: a=θ global -θ ref ; Among them, α is the angle information between the endoscope and the target area, θ global is the global main direction of the k-th frame image, θ ref is the reference direction.

9. The method for adaptive sampling of endoscopic images according to any one of claims 1 to 8, characterized in that: Each scene condition includes corresponding tissue location data, pathological state data, imaging environment data and imaging mode data.

10. An endoscope image adaptive sampling system, characterized in that: Applied to the method for adaptive sampling of endoscopic images according to any one of claims 1 to 9, the system comprises: An image acquisition module, used to acquire a historical image set of the endoscope under multiple scene conditions; A feature extraction module, used for performing data processing and feature extraction on the historical image set to obtain an initial feature set corresponding to the historical image set; A feature processing module, used for performing feature engineering on the initial feature set to obtain a target feature set; A model building module, used to pre-build an initial adaptive sampling model, wherein the initial adaptive sampling model includes a CNN learning module, a reinforcement learning module and an integrated learning module; A model training module is used to produce a model data set according to a target feature set, and extract a training set from the model data set; input the training set into the initial adaptive sampling model, and perform model training using the CNN learning module, the reinforcement learning module, and the integrated learning module in sequence to obtain a target adaptive sampling model; The model application module performs real-time image sampling on the current scene of the endoscope according to the target adaptive sampling model to obtain an endoscope adaptive sampling image.

11. An endoscope image adaptive sampling device, characterized in that: The method comprises a processor, a memory and a computer program stored in the memory and executable on the processor, wherein the computer program implements the method steps in the method for adaptive sampling of endoscopic images as claimed in any one of claims 1 to 9 when executed.

12. A computer storage medium, characterized in that: The computer storage medium comprises: at least one instruction, which implements the method steps in the endoscopic image adaptive sampling method according to any one of claims 1 to 9 when the instruction is executed by a computer.

Images