Arthroscopic image real-time abnormality recognition system based on deep learning

Abstract

The application relates to the technical field of deep learning, in particular to an arthroscopic image real-time abnormality recognition system based on deep learning. The system identifies the high light performance degree of different regions in the image during model recognition, analyzes the high light performance and the probability value deviation of interframe pixel points, extracts the dynamic change trend of the target region pixel points in continuous frames, and obtains the consistent degree of performance in time sequence. The system further analyzes the covering influence of high light on the region in the time sequence video, and then adjusts the segmentation confidence degree by using the segmentation degree of the pixel points in the initial target region which is influenced by the high light, compensates the region information which is partially blocked or blurred through multi-frame fusion, and obtains the final target region for model training. Through interframe high light feature and region fusion processing, the application reduces error accumulation and high light interference, obtains a more accurate and complete abnormal region, and effectively improves the recognition precision of deep learning.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep learning, in particular to a deep learning-based arthroscopic image real-time anomaly recognition system. BACKGROUND

[0002] Arthroscopic surgery, as a minimally invasive and efficient diagnosis and treatment method, is widely used in orthopedic clinics. Through a few small incisions on the skin, surgical instruments are inserted into the joint cavity, and endoscopic images are obtained by shooting in the joint cavity through arthroscopic equipment to observe the state of tissue structures such as cartilage, synovial membrane, and meniscus. Deep learning uses deep neural network models to automatically learn complex texture and boundary features from a large amount of data, and is often used for region recognition and segmentation in medical images.

[0003] In the prior art, a deep learning-based convolutional neural network is usually used to automatically identify and segment abnormal regions of arthroscopic images, and a single-frame image static model is generally used, which is extremely unstable under high exposure interference. However, due to the mirror highlights caused by the liquid, fat droplets and tissue surface reflection in the joint cavity, the real boundaries of the small abnormal parts will be covered or distorted after the highlights occupy the field of view, making it difficult for the model to focus on the boundary details during training, thereby reducing the recognition accuracy of the small region boundaries, and leading to the model misjudgment of the highlight region and the abnormal region, and insufficient recognition accuracy of the model training. SUMMARY

[0004] In order to solve the technical problems in the prior art that the highlights will cover or distort the real boundaries of the small abnormal parts, making it difficult for the model to focus on the boundary details during training, thereby reducing the recognition accuracy of the small region boundaries, and leading to insufficient recognition accuracy of the model training, the purpose of the present application is to provide a deep learning-based arthroscopic image real-time anomaly recognition system, and the technical solution adopted is as follows:

[0005] The present application provides a deep learning-based arthroscopic image real-time anomaly recognition system, which comprises:

[0006] A data preparation module for region segmentation of each frame of arthroscopic image in the training set by an anomaly recognition model, obtaining different regions and determining an initial target region, and a probability value of each pixel point;

[0007] An exposure performance analysis module for obtaining the exposure performance degree of each frame of arthroscopic image according to the consistency degree of the highlight performance of the pixel value distribution between different regions in each frame of arthroscopic image;

[0008] The segmentation performance analysis module is configured to obtain time sequence consistency of each pixel point according to a deviation degree of a probability value of each pixel point in the initial target region between each two adjacent frames; obtain segmentation reliability of the pixel point by a local pixel continuous change deviation degree of each pixel point in the initial target region; and obtain a segmentation influence index of the pixel point according to a change correlation between the segmentation reliability and the exposure performance degree of each pixel point on the continuous frame arthroscopy images.

[0009] The confidence adjustment module is configured to obtain a fusion probability value of the pixel point according to the segmentation influence index, the segmentation reliability and the time sequence consistency of each pixel point in the initial target region, and combine the probability values of the pixel point in different frame arthroscopy images.

[0010] The model training module is configured to obtain a final target region by the fusion probability value of the pixel point, and train an abnormality recognition model by taking the final target region as training data.

[0011] Further, the exposure performance degree acquisition method comprises:

[0012] For any one frame of arthroscopy image, a highlight performance degree of each region is obtained according to a uniform distribution degree of pixel values in each region in the arthroscopy image.

[0013] A mean value of the highlight performance degrees of all regions in the arthroscopy image is taken as a highlight uniformity of the arthroscopy image.

[0014] After calculating the highlight performance degree difference between each two different regions in the arthroscopy image, a mean value of all highlight performance degree differences is negatively correlated to obtain a performance consistency of the arthroscopy image.

[0015] The highlight uniformity and the performance consistency of the arthroscopy image are combined to obtain the exposure performance degree of the arthroscopy image.

[0016] Further, the highlight performance degree acquisition method comprises:

[0017] All pixel value variances in each region in the arthroscopy image are calculated and negatively correlated to obtain a pixel uniformity of each region.

[0018] A mean value of all pixel values in each region in the arthroscopy image is taken as a highlight degree of each region.

[0019] The pixel uniformity and the highlight degree of each region are combined to obtain the highlight performance degree of each region in the arthroscopy image.

[0020] Further, the time sequence consistency acquisition method comprises:

[0021] For a pixel point in an initial target region in any frame of arthroscopic images, in a preset neighborhood range in the time sequence of the arthroscopic images, the probability value difference of the pixel point in each adjacent two frames of arthroscopic images is calculated, and then the mean value of all probability value differences is negatively correlated to obtain the time sequence consistency of the pixel point.

[0022] Further, the segmentation reliability acquisition method comprises:

[0023] For a pixel point in an initial target region in any frame of arthroscopic images, the direction of the line segment between the pixel point and the center point of the initial target region is taken as the continuity direction of the pixel point;

[0024] The line segment between the center point as the starting point and the intersection point of the continuity direction and the boundary of the initial target region is taken as a change line segment, and the pixel point is on the change line segment;

[0025] The pixel values of the continuously distributed pixel points on the change line segment are linearly fitted, and the slope of the fitted straight line is taken as the overall change degree of the change line segment;

[0026] The pixel values of the continuously distributed pixel points in the preset local range of the pixel point on the change line segment are linearly fitted, and the slope of the fitted straight line is taken as the local change degree of the pixel point;

[0027] The difference between the local change degree and the overall change degree of the pixel point is negatively correlated to obtain the segmentation reliability of the pixel point in the arthroscopic images.

[0028] Further, the segmentation influence index acquisition method comprises:

[0029] For a pixel point in any initial target region, the segmentation reliability of the pixel point in each frame of arthroscopic images is negatively correlated to obtain the segmentation influence degree of the pixel point in each frame of arthroscopic images, and the segmentation influence degree of the pixel point is arranged in the order of the time sequence of the continuous frames of arthroscopic images to obtain the segmentation influence sequence of the pixel point;

[0030] The exposure performance degree is arranged in the order of the time sequence of the continuous frames of arthroscopic images to obtain the exposure sequence;

[0031] The correlation between the segmentation influence sequence and the exposure sequence of the pixel point is calculated to obtain the segmentation influence index of the pixel point.

[0032] Further, the fusion probability value acquisition method comprises:

[0033] For a pixel point in an initial target region in any frame of arthroscopic images, the ratio of the time sequence consistency and the segmentation influence index of the pixel point is taken as the time sequence reliability;

[0034] The product of the timing reliability and the segmentation reliability of the pixel point is taken as a reliable adjustment degree; and a sum value of the reliable adjustment degree and the segmentation reliability is normalized as the confidence of the pixel point;

[0035] The probability values of the pixel points are weighted and summed based on the confidence of the pixel points in each arthroscopic image to obtain the fusion probability value of the pixel point.

[0036] Further, the method for obtaining the final target region comprises:

[0037] The region composed of the pixel points with the fusion probability value higher than the preset segmentation threshold is taken as the final target region.

[0038] Further, the training of the abnormality recognition model with the final target region as the training data comprises:

[0039] The final target region in the arthroscopic image is marked with a label as the training data to input a deep neural network for training of the abnormality recognition model, and a cross-entropy loss function is used to evaluate the difference between the label and a real label.

[0040] Further, the method for obtaining the probability value comprises:

[0041] In the process of identifying the region in each arthroscopic image by the abnormality recognition model through the deep learning neural network, the logits of each pixel are output by the last layer of the network, and the probability value of each pixel point is converted by Sigmoid.

[0042] The present application has the following beneficial effects:

[0043] The present application analyzes the high light performance and the probability value deviation of the pixel points between frames by identifying the high light performance degree of different regions in the image during model recognition, extracts the dynamic change trend of the pixel points of the target region in the continuous frames, and obtains the consistent degree of performance in time sequence. The influence of high light on the covering of the region in the time sequence video is further analyzed, and then the segmentation confidence degree is adjusted by using the segmentation degree of the pixel points in the initial target region affected by the high light and extracting the redundant information between frames. The partially occluded or blurred region information is compensated by multi-frame fusion, the dynamic weighting of the segmentation results of each frame abnormal region is realized, the high confidence final target region with the clearest abnormal boundary and the most complete details is screened out, and the recognition model trained subsequently has higher precision. The present application reduces the error accumulation and high light interference through the inter-frame high light feature and region fusion processing, obtains more accurate and complete abnormal regions, and effectively improves the recognition precision of deep learning. BRIEF DESCRIPTION OF DRAWINGS

[0044] In order to more clearly illustrate the technical solutions and advantages of the embodiments of the present application or the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0045] Figure 1 A structure diagram of an arthroscopic image real-time abnormality recognition system based on deep learning provided by an embodiment of the present application;

[0046] Figure 2 An exposed arthroscopic image schematic diagram provided by an embodiment of the present application;

[0047] Figure 3 An arthroscopic image schematic diagram containing a final target area provided by an embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to further illustrate the technical means and effects adopted by the present application to achieve the predetermined purpose, the specific implementation, structure, features and effects of the arthroscopic image real-time abnormality recognition system based on deep learning according to the present application are described in detail as follows. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. In addition, the specific features, structures or characteristics in one or more embodiments can be combined in any suitable form.

[0049] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs.

[0050] The specific scheme of the arthroscopic image real-time abnormality recognition system based on deep learning provided by the present application is described in detail below with reference to the drawings.

[0051] Before the neural network is trained, the training data is preprocessed to help the neural network learn the features of the data better. If the data set is directly trained, the abnormal area is extracted by the convolutional neural network, and due to the real-time exposure during the arthroscopic operation process, errors may exist in the edge details of the abnormal part segmentation. In order to optimize the model training and improve the accuracy of real-time image recognition, the arthroscopic image needs to be preprocessed to eliminate the exposure effect and determine clearer and more accurate data for model training.

[0052] Please refer to Figure 1The diagram illustrates a structure of a real-time anomaly recognition system for arthroscopic images based on deep learning, according to an embodiment of the present invention. The system includes: a data preparation module 101, an exposure performance analysis module 102, a segmentation performance analysis module 103, a confidence adjustment module 104, and a model training module 105.

[0053] The data preparation module 101 is used to perform region segmentation on each frame of arthroscopic images in the training set through an anomaly recognition model, obtain different regions, determine the initial target region, and the probability value of each pixel.

[0054] Arthroscopic video sequences are obtained from a public database, segmented into image sequences by frame, and white balance and image preprocessing are performed. In this embodiment of the invention, the image data format can be JPEG. The image preprocessing can specifically include image grayscale processing and image denoising processing, such as using grayscale weighting for grayscale conversion and bilateral filtering for denoising. White balance and image preprocessing are well known techniques in the art and are not limited thereto.

[0055] During arthroscopic surgery, illumination and surface reflections from structures within the joint cavity can create specular highlights, which can obscure or distort the true boundaries of the joint, leading to errors in the identification of abnormal areas. To eliminate the impact of highlights on the identification of abnormal parts in arthroscopic images, it is necessary to first analyze the exposure of each frame to quantify the effect of exposure on segmentation performance. Please refer to [link to relevant documentation]. Figure 2 The illustration shows a schematic diagram of an exposed arthroscopic image provided by an embodiment of the present invention.

[0056] To observe the exposure levels of arthroscopic images, it is necessary to analyze the impact of exposure through regional contrast analysis. When there is severe exposure, all areas of the arthroscope may be affected, resulting in each area remaining bright and having poor contrast. Therefore, deep learning neural networks can be used to segment arthroscopic images and initially identify the regions with abnormalities, which are the initial target regions.

[0057] During arthroscopic surgery, the highlighted areas move as the arthroscope moves. Therefore, the probability of the target region is obtained for each pixel in each frame of the arthroscopic image. When a deep learning neural network is used to identify abnormal regions in the arthroscopic image, the last layer of the network outputs the logits of each pixel, which reflects the original evaluation score of the network model, that is, the probability that the pixel is an abnormal region. Therefore, it is converted into a probability value for each pixel by Sigmoid. The probability value obtained by Sigmoid normalization represents the degree to which each pixel is likely to be classified as an abnormal region.

[0058] The exposure performance analysis module 102 is configured to obtain the exposure performance degree of each arthroscopic image according to the consistency degree of the pixel value distribution high light performance between different regions in each arthroscopic image.

[0059] The high light part in the arthroscopic image is generally characterized by high overall brightness, forming a saturated white spot and having no internal texture. When the arthroscopic image is seriously exposed, the high light affects most or even all regions, and the high light performance degree of the image in the overall region is consistent and high. Therefore, the exposure degree of the image is obtained by the high light performance of the multiple regions.

[0060] Preferably, in the embodiment of the present application, the exposure performance degree is obtained by the following method:

[0061] For any arthroscopic image, the high light performance degree of each region is obtained according to the uniform distribution degree of the pixel value in each region. When the pixel value in the region is higher and more uniform, it indicates that the high light degree in the region is stronger. Therefore, in the embodiment of the present application, the variance of all pixel values in each region of the arthroscopic image is calculated and negatively correlated to obtain the pixel uniformity of each region. When the pixel uniformity is larger, the high light degree is higher. Furthermore, the mean value of all pixel values in each region of the arthroscopic image is taken as the high light degree of each region. When the high light degree is larger, the region has a higher possibility of high light distribution.

[0062] It should be noted that the negative correlation mapping is a technical means familiar to those skilled in the art, which can adopt a negative exponential function or an inverse proportional value form, and the like, which is not limited and described herein.

[0063] Therefore, the high light performance degree of each region in the arthroscopic image is obtained by combining the pixel uniformity and the high light degree of each region. In the embodiment of the present application, the product of the pixel uniformity and the high light degree is taken as the high light performance degree of each region in the arthroscopic image. When the pixel value distribution in the region is higher and more uniform, the high light performance is more significant.

[0064] Furthermore, the overall region of each image is comprehensively analyzed to reflect the exposure influence degree of the image. The mean value of the high light performance degrees of all regions in the arthroscopic image is taken as the high light uniformity of the arthroscopic image. When the high light uniformity is higher, it indicates that the image is more seriously exposed.

[0065] After calculating the high light performance degree difference between each two different regions in the arthroscopic image, the mean value of all high light performance degree differences is negatively correlated to obtain the performance consistency degree of the arthroscopic image. Through the high light performance degree difference between different regions, when the overall difference is smaller, it indicates that the high light degree is more consistent, and at this time, the image is more seriously exposed.

[0066] Finally, the high light uniformity and the performance consistency are combined to obtain the exposure performance of the arthroscopic image. In the embodiment of the present application, the product of the high light uniformity and the performance consistency is taken as the exposure performance of the arthroscopic image. When the high light uniformity and the performance consistency are both higher, the exposure of the arthroscopic image will be more affected.

[0067] The segmentation performance analysis module 103 is configured to obtain the time sequence consistency of each pixel point in the initial target region according to the deviation degree of the probability value of each pixel point between each two adjacent frames; obtain the segmentation reliability of the pixel point in the initial target region by the deviation degree of the local pixel continuous change of each pixel point; and obtain the segmentation influence index of the pixel point according to the change correlation between the segmentation reliability and the exposure performance of the pixel point on the continuous frame arthroscopic images.

[0068] In the continuous movement process, the high light has different influences on different regions in the joint cavity when it moves, resulting in different degrees of high light artifacts and detail loss in different frames of arthroscopic images, thereby directly affecting the recognition accuracy of the abnormal contour by the deep segmentation model. By quantifying the exposure of each frame and the corresponding segmentation recognition performance, the influence on the pixel is determined, and therefore the segmentation influence on the pixel point is further analyzed by the probability value.

[0069] Therefore, the probability value change degree of the pixel point in the initial target region on the continuous adjacent frames is analyzed first. Preferably, in the embodiment of the present application, the method for obtaining the time sequence consistency comprises:

[0070] For a pixel point in the initial target region in any frame of arthroscopic image, the time sequence analysis is performed on each pixel point preliminarily identified as abnormal. In a preset neighborhood range on the time sequence of the arthroscopic image, the probability value difference of the pixel point in each two adjacent frames of arthroscopic images is calculated, and then the mean value of all probability value differences is negatively correlated to obtain the time sequence consistency of the pixel point. On the continuous time sequence in the neighborhood range, the more the probability value changes between the pixel points in the adjacent two frames, that is, the greater the difference, the greater the influence on the abnormal division of the pixel point, and the lower the consistency in the time sequence. Therefore, the time sequence consistency is obtained by the negative correlation of all adjacent frame differences.

[0071] In the embodiment of the present application, the preset neighborhood range is set to a range of two frames before and after each frame of arthroscopic image in the time sequence, that is, a range of a total of 5 frames centered on the arthroscopic image. The specific range setting implementer can adjust it according to the specific implementation condition, which is not limited herein.

[0072] Meanwhile, considering as a part of a region, even if the pixel point changes, the change is continuous and not conspicuous, so the local continuous change of each pixel point is analyzed to determine the credibility of the pixel point as a part of the divided abnormal region. In the embodiment of the present application, the method for obtaining the segmentation credibility comprises:

[0073] For a pixel point in an initial target region in any frame of arthroscopic image, the direction of the line connecting the pixel point and the center point of the initial target region is taken as the continuous direction of the pixel point, and the line segment between the starting point of the center point and the intersection point of the initial target region boundary along the continuous direction is taken as the change line segment, which reflects the main direction of the continuous change of the pixel point. The pixel point is on the change line segment, that is, the intersection point and the center point are taken as the change line segment after the center point passes through the pixel point and intersects with the initial target region boundary.

[0074] Since the abnormal region part will have a clear boundary with the normal part, the more consistent the local pixel change of the pixel point is with the overall change, the higher the degree of the pixel point as a region center point is reflected. Therefore, the pixel values of the continuously distributed pixel points on the change line segment are linearly fitted, the slope of the fitted straight line is taken as the overall change degree of the change line segment, and then the pixel values of the continuously distributed pixel points in the preset local range of the pixel point are linearly fitted, the slope of the fitted straight line is taken as the local change degree of the pixel point, and the pixel change on the overall line segment and the local change degree of the pixel point are obtained respectively.

[0075] In the embodiment of the present application, the preset local range is set to the range of the left and right adjacent two pixel points on the change line segment centered on the pixel point, and the specific range can be adjusted by the implementer. It should be noted that linear fitting is a well-known technical means for those skilled in the art, which will not be described here.

[0076] Finally, the difference between the local change degree of the pixel point and the overall change degree is negatively correlated, and the segmentation credibility of the pixel point in the arthroscopic image is obtained. The lower the degree of difference, the higher the credibility of the pixel point as an abnormal region division.

[0077] By analyzing the change of the segmentation credibility of the pixel point caused by the exposure condition, the degree of influence of the exposure on the pixel point in the region identification is reflected. Preferably, in the embodiment of the present application, the method for obtaining the segmentation influence index comprises:

[0078] For a pixel point in any initial target region, the segmentation credibility of the pixel point in each frame of arthroscopic image is negatively correlated, and the segmentation influence degree of the pixel point in each frame of arthroscopic image is obtained, which reflects the untrustworthy situation of the pixel point being divided as abnormal.

[0079] The segmentation influence degree of the pixel point is arranged in the order of the joint arthroscopy images in time sequence, to obtain a segmentation influence sequence of the pixel point, and the exposure performance degree is arranged in the order of the joint arthroscopy images in time sequence, to obtain an exposure sequence, and the distribution is arranged according to the arrangement of the continuous frames in time sequence, to respectively represent the change of the untrusted degree of the pixel point in the continuous frames and the change of the exposure performance of different frames.

[0080] Further, the correlation between the segmentation influence sequence and the exposure sequence of the pixel point is calculated, to obtain a segmentation influence index of the pixel point, when the exposure performance increases, the untrusted degree also increases, which reflects that the influence of the pixel point being divided is higher, which is caused by the exposure influence, and therefore the segmentation influence index is larger. In the embodiment of the present application, the correlation between the sequences can be calculated by using the Pearson correlation coefficient, and the sequence correlation degree calculation method is a technical means known to those skilled in the art, and can also be selected from dynamic time warping algorithm and the like, which is not limited and described here.

[0081] The confidence adjustment module 104 is used for obtaining a fusion probability value of the pixel point according to the segmentation influence index, the segmentation trusted degree and the time sequence consistency of each pixel point in the initial target region, and combining the probability value of the pixel point in different frame joint arthroscopy images.

[0082] Through multi-frame fusion, the region is more complete and accurate, the pixel point which is less affected by exposure and more reliable in segmentation has a higher priority and is more reliable and has a higher weight in fusion. In the embodiment of the present application, the method for obtaining the fusion probability value includes:

[0083] For a pixel point in the initial target region in any frame joint arthroscopy image, the ratio of the exposure time sequence consistency and the segmentation influence index of the pixel point is taken as the time sequence reliability, and when the consistency degree is higher and the affected degree is lower, the pixel point has a larger reliability in fusion.

[0084] The product of the time sequence reliability and the segmentation trusted degree of the pixel point is taken as the trusted adjustment degree, and the sum value of the trusted adjustment degree and the segmentation trusted degree is normalized as the confidence of the pixel point. Through the adjustment of the time sequence reliability, when the time sequence reliability is higher and the segmentation trusted degree is larger, the pixel point is more reliable in the frame joint arthroscopy image, and has a larger weight.

[0085] It should be noted that the normalization is a technical means known to those skilled in the art, and the normalization can be linear normalization or standard normalization, and the specific normalization method is not limited here.

[0086] Finally, the probability values of the pixel points are weighted and summed based on the confidence of the pixel points in each arthroscopic image to obtain a fusion probability value of the pixel points. The confidence is used as a weight to multiply the probability value and then summed to reflect the final probability degree.

[0087] When the final fusion probability value of the pixel point is higher, the pixel point is more reliable as an abnormal division. Therefore, the determination of the final target region is determined by the fusion probability value. In the embodiment of the present application, the region composed of the pixel points with a fusion confidence higher than a preset segmentation threshold is regarded as the final target region, wherein the preset segmentation threshold can be set to 0.7, and the specific value can be adjusted by the implementer according to the specific implementation condition, which is not limited herein. Please refer to Figure 3 which shows an arthroscopic image schematic diagram containing a final target region provided by an embodiment of the present application.

[0088] In the embodiment of the present application, after pre-processing the video images in the database, a more complete and accurate structure image is obtained, and an expert standard mask of the final target region in the image can be obtained, that is, the label of the final target region in the arthroscopic image. The labeled image and the mask label are used as training data to input a deep neural network to train an abnormality recognition model. The deep neural network can use a U-Net model, and a cross-entropy loss function is used to evaluate the difference between the label and the true label. The network parameters are continuously optimized through back propagation to improve the segmentation ability of the model to the abnormal region. It should be noted that the training of the neural network model is a technology known to those skilled in the art, and will not be described here.

[0089] After the training is completed, the embodiment of the present application can also use an independent validation set to evaluate the performance of the model. By calculating multiple indicators such as accuracy (Accuracy), recall (Recall), precision (Precision) and F1 score, the performance of the model under different abnormal conditions and different image qualities is comprehensively measured to ensure its generalization ability and robustness on unseen data. According to the verification result, the model structure or training strategy is further fine-tuned to improve the overall recognition accuracy.

[0090] In summary, the application analyzes the high light performance and analyzes the probability value deviation of the pixel points between frames by identifying the high light performance degree of different regions in the image, extracts the dynamic change trend of the target region pixel points in the continuous frames, and obtains the consistent degree of performance in time sequence. Further analysis is made on the covering effect of high light on the region in the time sequence video, and then the segmentation confidence degree is adjusted by using the frame redundancy information to extract the segmentation degree of the pixel points in the initial target region affected by the high light, the partially occluded or blurred region information is compensated by multi-frame fusion, the dynamic weighting of the segmentation results of each frame abnormal region is realized, the most clear and complete high confidence final target region is screened out from the abnormal boundary, and the subsequent training recognition model has higher precision. Through the inter-frame high light feature and region fusion processing, the error accumulation and high light interference are reduced, more accurate and complete abnormal regions are obtained, and the recognition accuracy of deep learning is effectively improved.

[0091] It should be noted that the above-mentioned embodiment sequence of the application is only for description, and does not represent the advantages and disadvantages of the embodiments. The processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multi-task processing and parallel processing are also possible or may be advantageous.

[0092] Each embodiment in the specification is described in a progressive manner, and the same and similar parts between each embodiment can be referred to each other. Each embodiment mainly describes the difference from other embodiments.

Claims

1. A real-time anomaly recognition system for arthroscopic images based on deep learning, characterized in that, The system includes: The data preparation module is used to perform region segmentation on each frame of arthroscopic images in the training set using an anomaly detection model, to obtain different regions and determine the initial target region, as well as the probability value of each pixel. The exposure performance analysis module is used to obtain the exposure performance of each frame of arthroscopic image based on the consistency of the highlight performance of different regions in the pixel value distribution. The segmentation performance analysis module is used to obtain the temporal consistency of each pixel based on the deviation of the probability value of each pixel in the initial target region between each two adjacent frames; in the initial target region, the segmentation confidence of each pixel is obtained by the deviation of the local continuous pixel change; and the segmentation influence index of each pixel is obtained based on the correlation between the segmentation confidence of each pixel in consecutive frame arthroscopic images and the exposure performance. The confidence adjustment module is used to obtain the fusion probability value of each pixel based on the segmentation impact index, segmentation confidence and temporal consistency of each pixel in the initial target region, combined with the probability value of the pixel in different frames of arthroscopic images; The model training module is used to obtain the final target region through the fusion probability value of pixels; and to train the anomaly recognition model using the final target region as training data.

2. The real-time anomaly recognition system for arthroscopic images based on deep learning according to claim 1, characterized in that, The method for obtaining the exposure performance includes: For any frame of arthroscopic image, the highlight performance of each region is obtained based on the uniformity of the pixel value distribution in each region of the arthroscopic image. The mean of the highlight intensity of all regions in the arthroscopic image is taken as the highlight uniformity of the arthroscopic image. After calculating the difference in highlight performance between every two different regions in the arthroscopic image, the mean of all highlight performance differences is negatively correlated to obtain the performance consistency of the arthroscopic image. By combining the highlight uniformity and performance consistency of the arthroscopic image, the exposure performance of the arthroscopic image is obtained.

3. The real-time anomaly recognition system for arthroscopic images based on deep learning according to claim 2, characterized in that, The method for obtaining the highlight performance includes: Calculate the variance of all pixel values ​​in each region of the arthroscopic image and perform negative correlation mapping to obtain the pixel mean of each region; The mean value of all pixels in each region of the arthroscopic image is used as the highlight of each region. By combining the pixel uniformity and highlight intensity of each region, the highlight performance of each region in the arthroscopic image is obtained.

4. The real-time anomaly recognition system for arthroscopic images based on deep learning according to claim 1, characterized in that, The method for obtaining timing consistency includes: For a pixel in the initial target region of any frame of arthroscopic image, within a preset neighborhood of the temporal sequence of the arthroscopic image, the probability value difference of the pixel in each of two adjacent frames of arthroscopic images is calculated, and the mean of all probability value differences is negatively correlated to obtain the temporal consistency of the pixel.

5. The real-time anomaly recognition system for arthroscopic images based on deep learning according to claim 1, characterized in that, The method for obtaining the segmentation confidence level includes: For any pixel in the initial target region of any frame of arthroscopic image, the direction of the line connecting the pixel to the center point of the initial target region is taken as the continuous direction of the pixel. The line segment between the intersection of the center point and the boundary of the initial target area along a continuous direction is called the changing line segment; the pixel point is on the changing line segment. A straight line is fitted to the pixel values ​​of continuously distributed pixels on the changing line segment, and the slope of the fitted line is used as the overall degree of change of the changing line segment. On the changing line segment, the pixel values ​​of continuously distributed pixels within a preset local range of the pixel are fitted with a straight line, and the slope of the fitted line is used as the local degree of change of the pixel. By performing a negative correlation mapping between the local variability of the pixel and the overall variability, the segmentation confidence of the pixel in the arthroscopic image is obtained.

6. The real-time anomaly recognition system for arthroscopic images based on deep learning according to claim 1, characterized in that, The method for obtaining the segmentation impact index includes: For any pixel in an initial target region, negative correlation mapping is performed on the segmentation confidence of the pixel in each frame of arthroscopic image to obtain the segmentation influence of the pixel in each frame of arthroscopic image; the segmentation influence of the pixel is arranged in the order of consecutive frames of arthroscopic images in time sequence to obtain the segmentation influence sequence of the pixel. The exposure performance is arranged in the order of consecutive frames of arthroscopic images in time sequence to obtain the exposure sequence; Calculate the correlation between the segmentation impact sequence and the exposure sequence of the pixel to obtain the segmentation impact index of the pixel.

7. The real-time anomaly recognition system for arthroscopic images based on deep learning according to claim 1, characterized in that, The method for obtaining the fusion probability value includes: For any pixel in the initial target region of an arthroscopic image frame, the ratio of the temporal consistency of that pixel to the segmentation impact index is taken as the temporal reliability. The product of the temporal reliability and segmentation confidence of the pixel is used as the confidence adjustment degree; the sum of the confidence adjustment degree and the segmentation confidence degree is normalized and used as the confidence degree of the pixel. The probability values ​​of a pixel are weighted and summed based on its confidence level in each frame of arthroscopic images to obtain the fusion probability value of that pixel.

8. The real-time anomaly recognition system for arthroscopic images based on deep learning according to claim 1, characterized in that, The method for obtaining the final target region includes: The region consisting of pixels with a fusion probability value higher than the preset segmentation threshold is taken as the final target region.

9. The real-time anomaly recognition system for arthroscopic images based on deep learning according to claim 1, characterized in that, The step of training the anomaly detection model using the final target region as training data includes: The final target region in the arthroscopic image is labeled and used as training data to train the deep neural network for anomaly recognition model. The cross-entropy loss function is used to evaluate the difference between the label and the true label.

10. The real-time anomaly recognition system for arthroscopic images based on deep learning according to claim 1, characterized in that, The method for obtaining the probability value includes: In the process of the anomaly detection model identifying regions in each frame of arthroscopic image through a deep learning neural network, the last layer of the network outputs the logits of each pixel, which is then converted into a probability value for each pixel using a sigmoid function.

Images