Endoscope type identification method, device and equipment and endoscope system

Abstract

The invention is suitable for the technical field of medical treatment, and provides an endoscope type identification method, device and equipment and an endoscope system. The endoscope type identification method comprises the following steps: acquiring a plurality of frames of endoscope images acquired by an endoscope; determining classification data in one-to-one correspondence with each frame of the endoscope image; processing the classified data, improving the data quality of the classified data, and obtaining to-be-identified data; and under the condition that the to-be-identified data does not meet a preset type identification condition, determining the type of the endoscope according to the classification data. According to the embodiment of the invention, the detection efficiency and the detection accuracy of the endoscope can be considered.

Description

Technical Field

[0001] The present application belongs to the field of medical technology, and in particular relates to a method, device, equipment and endoscope system for identifying the type of an endoscope. Background Art

[0002] An endoscope is a diagnostic and auxiliary treatment device that is widely used in clinical practice. The device can enter the human body through natural orifices or surgical incisions, clearly and realistically display the anatomical structure, and assist doctors in seeing lesions in the human body that cannot be displayed by X-rays.

[0003] Endoscopes can be divided into digestive tract endoscopes, urogenital tract endoscopes, respiratory tract endoscopes, body cavity endoscopes, and head organ endoscopes according to their uses. When using an endoscope system, the relevant technology requires medical staff to manually select the corresponding type of endoscope on the equipment according to the detection requirements to adapt to the corresponding purpose. This method not only increases the workload of medical staff and reduces the efficiency of endoscope detection, but also easily introduces erroneous information due to forgetting to select or selecting incorrectly, resulting in a decrease in the accuracy of endoscope detection, which is not conducive to the application of endoscopes. Summary of the invention

[0004] The embodiments of the present application provide a method, apparatus, device and storage medium for identifying the type of an endoscope, which can take into account both the endoscope detection efficiency and the detection accuracy.

[0005] A first aspect of an embodiment of the present application provides a method for identifying the type of an endoscope, comprising: acquiring a plurality of frames of endoscopic images collected by an endoscope; determining classification data corresponding to each frame of the endoscopic image; processing the classification data to improve the data quality of the classification data to obtain data to be identified; and if the data to be identified does not meet a preset type identification condition, determining the type of the endoscope based on the classification data.

[0006] In some embodiments of the first aspect, after obtaining the data to be identified, the method for identifying the type of the endoscope further includes: identifying the type of the endoscope based on the data to be identified; when the type identification of the endoscope based on the data to be identified fails, determining that the data to be identified does not meet the type identification condition.

[0007] In some embodiments of the first aspect, the data to be identified includes a classification result of the corresponding endoscopic image;

[0008] After obtaining the data to be identified, the method for identifying the type of endoscope includes: accumulating the counting values ​​of the classification results associated with the same endoscope type according to the classification results of each frame of the endoscope image, to obtain a first cumulative value corresponding to each endoscope type; when the first cumulative value does not meet the cumulative value condition, confirming that the data to be identified does not meet the type identification condition.

[0009] In some embodiments of the first aspect, the processing of the classified data to improve the data quality of the classified data to obtain the data to be identified includes: improving the data quality of the classified data multiple times to obtain multiple copies of the data to be identified, wherein the first copy of the data to be identified is obtained by improving the data quality of the classified data, the Nth copy of the data to be identified is obtained by improving the data quality of the N-1th copy of the data to be identified, and N is a positive integer greater than 1.

[0010] In some embodiments of the first aspect, the cumulative value condition includes at least one threshold condition, and each portion of the data to be identified corresponds to at least one threshold condition; after obtaining the first cumulative value corresponding to each type of endoscope, the endoscope type identification method also includes: detecting whether the first cumulative value accumulated by the Nth portion of the data to be identified meets the corresponding threshold condition; when the first cumulative value accumulated according to the Nth portion of the data to be identified does not meet the corresponding threshold condition, detecting whether the first cumulative value accumulated according to the N-1th portion of the data to be identified meets the corresponding threshold condition, and so on, until the first cumulative value accumulated according to the first portion of the data to be identified does not meet the corresponding threshold condition, confirming that the first cumulative value does not meet the cumulative value condition.

[0011] In some embodiments of the first aspect, the threshold condition corresponding to the Nth portion of the data to be identified includes a first condition; detecting whether the first cumulative value obtained by accumulating the Nth portion of the data to be identified meets the corresponding threshold condition includes: determining the highest second cumulative value among the first cumulative values ​​calculated based on the Nth portion of the data to be identified; and confirming that the first cumulative value obtained by accumulating the Nth portion of the data to be identified meets the first condition when the second cumulative value is greater than or equal to the cumulative value threshold.

[0012] In some embodiments of the first aspect, the threshold condition corresponding to the Nth portion of the data to be identified also includes a second condition; when the first cumulative value is less than the cumulative value threshold, detecting whether the first cumulative value obtained by accumulating the Nth portion of the data to be identified meets the corresponding threshold condition also includes: determining the second highest third cumulative value in the first cumulative value calculated based on the Nth portion of the data to be identified; determining a first difference between the second cumulative value and the third cumulative value; and when the first difference is greater than the difference threshold, confirming that the first cumulative value obtained by accumulating the Nth portion of the data to be identified meets the second condition.

[0013] In some embodiments of the first aspect, the threshold condition corresponding to the N-1th portion of the data to be identified includes a third condition; detecting whether the first cumulative value accumulated by adding the N-1th portion of the data to be identified meets the corresponding threshold condition includes: determining the highest fourth cumulative value and the second highest fifth cumulative value among the first cumulative values ​​calculated based on the N-1th portion of the data to be identified; determining a second difference between the fourth cumulative value and the fifth cumulative value; and when the second difference is greater than the difference threshold, confirming that the first cumulative value accumulated by adding the N-1th portion of the data to be identified meets the third condition.

[0014] In some embodiments of the first aspect, according to the classification result of each frame of the endoscopic image, the counting values ​​of the classification results associated with the same endoscope type are accumulated to obtain a first cumulative value corresponding to each endoscope type, including: in frame order, if the endoscope type associated with the classification result of the endoscopic image of the Mth frame is a target type, and in the first M-1 frames of the endoscopic images, the number of the endoscopic images belonging to the target classification result is greater than the quantity threshold, then the first cumulative value of the target type is increased by a first numerical value, wherein M is a positive integer greater than 0, and the target classification result is a classification result associated with the classification result of the endoscopic image of the Mth frame; otherwise, the first cumulative value of the target type is increased by a second numerical value, and the second numerical value is less than the first numerical value, and the second numerical value is greater than 0.

[0015] In some embodiments of the first aspect, after obtaining the first cumulative value corresponding to each type of endoscope, the method for identifying the type of endoscope further includes: if there is the first cumulative value that satisfies the cumulative value condition, confirming that the data to be identified satisfies the type identification condition, and taking the endoscope type corresponding to the first cumulative value as the type of the endoscope.

[0016] In some embodiments of the first aspect, when the data to be identified does not meet a preset type identification condition, determining the type of the endoscope according to the classification data includes: detecting the duration of the current entry of the endoscope into the body; when the data to be identified does not meet the type identification condition and the duration is less than a duration threshold, reacquiring a new endoscopic image to update the data to be identified; when the updated data to be identified does not meet the type identification condition and the duration is less than the duration threshold, returning to execute the step of reacquiring a new endoscopic image to update the data to be identified until the updated data to be identified meets the type identification condition; or, until the updated data to be identified does not meet the type identification condition and the duration is greater than or equal to the duration threshold, determining the type of the endoscope according to the classification data.

[0017] In some embodiments of the first aspect, after obtaining the data to be identified, the method for identifying the type of endoscope further includes: accumulating count values ​​of the endoscopic images belonging to the in vivo frame classification according to the classification result of each frame of the endoscopic image, to obtain a sixth cumulative value of the in vivo frame classification, and the sixth cumulative value is used to determine the duration.

[0018] In some embodiments of the first aspect, the processing of the classified data to improve the data quality of the classified data to obtain the data to be identified includes: performing data filtering on the classified data to obtain the data to be identified that corresponds one-to-one to each frame of the endoscopic image.

[0019] In some embodiments of the first aspect, the classification data includes multiple preset classifications and classification confidences corresponding to each preset classification; the data filtering of the classification data to obtain the data to be identified corresponding to each frame of the endoscopic image includes: for the preset classifications among the multiple preset classifications whose classification confidence is lower than the target confidence threshold, updating the classification confidence to a preset confidence value to obtain the corresponding data to be identified.

[0020] In some embodiments of the first aspect, before updating the classification confidence to a preset confidence value, the endoscope type identification method also includes: obtaining a default confidence threshold; determining the first classification with the highest classification confidence and the second classification with the second highest classification confidence in the classification data; and when the scene corresponding to the first classification and the scene corresponding to the second classification are interrelated scenes, increasing the default confidence threshold to obtain the target confidence threshold.

[0021] In some embodiments of the first aspect, the data filtering is performed on the classification data to obtain the data to be identified corresponding to each frame of the endoscopic image, including: for the classification data corresponding to each frame of the endoscopic image, determining the first classification with the highest classification confidence; in the first classification corresponding to consecutive i frames of endoscopic images, determining the third classification with the highest number of occurrences and the number of target endoscopic images, wherein the target endoscopic image is the endoscopic image of the first classification with the highest classification confidence and the third classification, and i is a positive integer greater than 2; if the number of images is greater than a number threshold, then the classification results of the consecutive i frames of endoscopic images are all updated to the third classification to obtain the data to be identified corresponding to the consecutive i frames of endoscopic images.

[0022] In some embodiments of the first aspect, after determining the type of the endoscope, the method for identifying the type of the endoscope further includes: determining an algorithm library corresponding to the type of the endoscope based on the type of the endoscope.

[0023] A second aspect of an embodiment of the present application provides a type identification device for an endoscope, comprising: an image acquisition unit, used to acquire a plurality of frames of endoscopic images collected by the endoscope; an image classification unit, used to determine classification data for each frame of the endoscopic image; a data quality improvement unit, used to process the classification data, improve the data quality of the classification data, and obtain data to be identified; and a type identification unit, used to determine the type of the endoscope according to the classification data when the data to be identified does not meet the type identification conditions.

[0024] A third aspect of an embodiment of the present application provides an image processing device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned endoscope type identification method when executing the computer program.

[0025] A fourth aspect of an embodiment of the present application provides an endoscope system, comprising: a light source, an endoscope, an image processing device, and a display; wherein the image processing device is used to execute the steps of the endoscope type identification method as described in the first aspect.

[0026] A fifth aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned endoscope type identification method are implemented.

[0027] A sixth aspect of an embodiment of the present application provides a computer program product, which, when executed on an image processing device, enables the image processing device to execute the steps of the above-mentioned endoscope type identification method.

[0028] In an embodiment of the present application, after acquiring several frames of endoscopic images collected by an endoscope and determining the classification data of each frame of endoscopic images, the data quality of the classification data is improved to obtain data to be identified. The data to be identified with higher data quality can be used to identify the type of endoscope, and when the data to be identified does not meet the type identification conditions, the type of endoscope can be determined in a timely manner based on the original classification data, and the type of endoscope currently in use can be identified in a timely and automatic manner. Therefore, both the efficiency and accuracy of endoscopic detection can be taken into account. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0030] Figure 1 is a schematic structural diagram of an endoscope system provided in an embodiment of the present application;

[0031] Figure 2 It is a schematic diagram of the implementation flow of a method for identifying the type of an endoscope provided in an embodiment of the present application;

[0032] Figure 3 It is a schematic diagram of a specific implementation process for confirming whether the data to be identified meets the type identification condition provided in an embodiment of the present application;

[0033] Figure 4 It is a schematic diagram of a specific implementation process of determining the type of an endoscope provided in an embodiment of the present application;

[0034] Figure 5 is a schematic structural diagram of a type identification device for an endoscope provided in an embodiment of the present application;

[0035] Figure 6 It is a structural schematic diagram of the image processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work are protected by the present application.

[0037] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0038] In the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0039] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0040] When using an endoscope system, the relevant technology requires medical staff to manually select the corresponding type of endoscope on the equipment according to the detection requirements to adapt to the corresponding purpose. This method not only increases the workload of medical staff and reduces the efficiency of endoscope detection, but also easily introduces erroneous information due to forgetting to select or selecting incorrectly, resulting in a decrease in the accuracy of endoscope detection, which is not conducive to the application of endoscopes.

[0041] In view of this, the present application proposes a method for identifying the type of endoscope, which can accurately identify the type of the current endoscope without the need for medical staff to select the type, and can complete the type identification in a timely manner to ensure the efficiency of endoscopic detection.

[0042] In order to illustrate the technical solution of the present application, a specific embodiment is provided below for illustration.

[0043] Please refer to Figure 1 , Figure 1 1 is a schematic diagram of an endoscope system provided in an embodiment of the present application. The endoscope system may include: a light source device 101 , an endoscope 102 (or a scope), an image processing device 103 and a display 104 .

[0044] The light source device 101 may include a light source and an optical element, wherein the light source may generate a light beam of a specific wavelength, and the optical element includes but is not limited to a collimating lens, a focusing lens, etc. The light source device 101 may be used to converge the light beam generated by the light source and inject it into the end face of the light guide of the endoscope 102, so that the light beam is emitted from the front end of the endoscope 102, and provides illumination light for the endoscope 102.

[0045] The endoscope 102 may include an insertion portion in an elongated shape. The insertion portion may be equipped with a built-in imaging element. The endoscope 102 may collect images of the front end of the endoscope 102 through the imaging element, and feed the collected images back to the image processing device 103 through a connecting line. More specifically, the doctor may insert the insertion portion into the patient's body and collect images of the patient's body through the imaging element. It should be understood that an operating portion may also be connected to one end of the endoscope 102, and the operating portion may generate a control signal for controlling the insertion portion under the operation of the doctor.

[0046] The image processing device 103 may be a computer or a dedicated processing device for an endoscope system. The image processing device 103 can perform image processing based on the image collected by the endoscope 102 and send the image processing result to the display 104. It is understood that image processing includes but is not limited to: image recognition, image cropping, and image enhancement.

[0047] It is understandable that the type recognition of the endoscope is an image recognition algorithm executed on the image processing device 103. According to different uses, the endoscope 102 can be divided into types such as digestive tract endoscopes, urogenital tract endoscopes, respiratory tract endoscopes, body cavity endoscopes and head organ endoscopes. Among them, digestive tract endoscopes can also be specifically divided into different types such as colonoscopes and gastroscopes. For endoscope systems, endoscopes for different purposes usually need to adaptively call different algorithms. Through the type recognition of the endoscope, the image processing device 103 can identify the type of the current endoscope among the aforementioned different types of endoscopes, so as to facilitate the call of the algorithm corresponding to the type.

[0048] The display 104 may be a liquid crystal display (LCD), a light-emitting diode (LED) display or other display devices with display capabilities. The display 104 may display according to the image processing result provided by the image processing device 103 .

[0049] In some examples, the display 104 can display in real time the image captured by the camera element of the endoscope 102 and the recognition result of the image processing device 103 performing image recognition (such as lesion recognition). In other examples, the display 104 can also display the three-dimensional model obtained by the image processing device 103 performing three-dimensional reconstruction based on the image.

[0050] It should be noted that Figure 1 This does not constitute a limitation on the endoscope system. In actual applications, the endoscope system may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the endoscope system may also include a power supply device, an AI processing device specifically used to execute an artificial intelligence (AI) algorithm, etc., which is not limited in this application.

[0051] Please refer to Figure 2 , Figure 2 A schematic diagram of the implementation flow of a method for identifying the type of endoscope provided in an embodiment of the present application is shown. The method can be applied to the aforementioned image processing device 103 and is suitable for situations where both the efficiency and accuracy of endoscopic detection need to be taken into account.

[0052] Specifically, the above-mentioned method for identifying the type of endoscope may include the following steps S201 to S204.

[0053] Step S201, obtaining a plurality of frames of endoscopic images collected by an endoscope.

[0054] The endoscopic image is an image captured by the imaging element of the endoscope 102. In order to accurately identify the type of the endoscope, the image processing device 103 may acquire several frames of endoscopic images. The several frames of endoscopic images may refer to multiple frames of images acquired during the same endoscopic examination.

[0055] It should be understood that several frames of endoscopic images can be collected in real time by the endoscope 102, or they can be pre-stored in the memory of the image processing device 103 after collection and read from the memory when the endoscope type needs to be identified. This application does not impose any restrictions on this.

[0056] Step S202, determining the classification data corresponding to each frame of the endoscopic image.

[0057] In the implementation of the present application, the classification data is obtained by classifying the endoscopic images. Specifically, the endoscopic images can be classified in different ways, for example, according to whether the endoscopic images are in vivo images (i.e., images taken inside the body) or in vitro images (i.e., images taken outside the body), or according to the parts / organs (e.g., intestines, throat, buttocks, etc.) captured by them.

[0058] In an embodiment of the present application, the classification of endoscopic images can be implemented using an image classification algorithm based on deep learning. Exemplarily, the features of each frame of endoscopic images can be extracted by a convolutional neural network (CNN), and then the classification probability and corresponding confidence are obtained using pooling operations and fully connected layers on these features. Commonly used CNN networks include but are not limited to deep residual networks (ResNet), densely connected networks (Densely Connected Convolutional Networks, DenseNet), and variants of these two types of networks.

[0059] In the implementation of the present application, the classification of endoscopic images can also be implemented using a video classification algorithm based on deep learning. The video classification algorithm can use methods such as spatiotemporal network CNN-LSTM, CNN-Transformer, etc., where LSTM (Long Short-Term Memory) represents a long short-term memory network, which can model the temporal change information between consecutive frames, and Transformer represents a transformer or transformer, which has a similar function to LSTM and can use the self-attention mechanism to learn the temporal information in the video, and finally classify the features to complete the classification and recognition of each frame of the endoscopic image in the video.

[0060] Specifically, for each type of endoscope, the landmark scene data in vitro and in vivo can be collected in advance as sample data. In order to ensure the accuracy of recognition and reduce the interference of non-information frames, fuzzy or ambiguous in vivo or in vitro data can also be collected as sample data. For the sample data, preprocessing can be performed to facilitate feature extraction and improve the calculation speed. For the specific scene of endoscope type identification, the black border of the image in the sample data can be cut off. Among them, the black border records some invalid information that is not referenced for endoscope type identification, such as shooting time, hospital, and patient age. Since the proportion of black borders in endoscopic images is usually large, cutting off the black border can improve the calculation speed, reduce the calculation memory, and facilitate the algorithm to extract useful information. Based on the preprocessed data, model training can be performed to obtain the aforementioned network for endoscopic image classification. When it is necessary to identify the type of endoscope, the image processing device 103 can perform the same preprocessing (such as cutting the black border) on the acquired endoscopic image, and input it into the trained network to obtain classification data.

[0061] It is understandable that the above two classification methods are only examples, and this application does not limit the classification method of endoscopic images.

[0062] In some specific embodiments, the classification data corresponding to a single-frame endoscopic image may include scene classification results. Specifically, the scene classification results may be expressed as: irrelevant in vitro (such as dim pictures, pictures of equipment with no differentiation, etc.), relevant in vitro (pictures in vitro that help distinguish the types of endoscopes, such as pictures of buttocks and noses), irrelevant in vivo (close-up, in vivo pictures without obvious structures), and relevant in vivo (pictures in vivo that help distinguish the types of endoscopes, such as pictures of the throat, esophagus, intestines, etc.).

[0063] For in vivo correlation and in vitro correlation, the classification data may also include the classification confidence of each preset classification in a plurality of preset classifications. The preset classification is a classification that may appear in the endoscopic image, and each preset classification may correspond to a part / organ.

[0064] As an example, the classification of a single frame endoscopic image can be expressed as: {in vivo related; intestine: 0.8, throat: 0.1, tongue: 0, esophagus: 0.1, ...}.

[0065] Step S203, processing the classified data to improve the data quality of the classified data and obtain data to be identified.

[0066] Among them, improving the data quality of the classified data may mean improving the credibility of the classified data. The data to be identified is the classified data after the data quality is improved.

[0067] It should be understood that as the data quality improves, the obtained data to be identified can more accurately identify the type of endoscope, which helps to improve the accuracy of identifying the endoscope type.

[0068] Step S204: when the data to be identified does not meet the preset type identification condition, the type of the endoscope is determined according to the classification data.

[0069] The preset type recognition condition is used to determine whether the data to be recognized can identify the type of the endoscope 102 .

[0070] When the data to be identified meets the preset type identification condition, it means that the data to be identified can identify the type of the endoscope 102, and the type of the endoscope 102 can be determined based on the data to be identified. At this time, the image processing device 103 can identify the type of the endoscope 102 based on the data to be identified with higher credibility.

[0071] When the data to be identified does not meet the preset type identification condition, it means that the data to be identified cannot identify the type of the endoscope 102. At this time, the type of the endoscope 102 can be determined according to the classification data. That is, when the data to be identified is insufficient to support the type identification of the endoscope 102, the classification data with relatively low credibility can be used to identify the type of the endoscope 102.

[0072] Specifically, based on the aforementioned classification confidence, the type of the endoscope 102 can be determined. For example, the classification with the highest classification confidence can be determined, and the endoscope type corresponding to the classification is used as the type of the current endoscope 102. For example, in the classification data, the classification confidence of the intestine is the highest, and the corresponding endoscope type "colonoscope" is the type of the current endoscope 102.

[0073] In an embodiment of the present application, after acquiring several frames of endoscopic images collected by the endoscope 102 and determining the classification data of each frame of the endoscopic image, the data quality of the classification data is improved to obtain data to be identified. The data to be identified with higher data quality can be used to identify the type of the endoscope 102. Moreover, when the data to be identified does not meet the type identification conditions, the type of the endoscope 102 can be determined in a timely manner based on the original classification data, and the type of the endoscope 102 currently in use can be identified in a timely and automatic manner. Therefore, both the efficiency and accuracy of endoscopic detection can be taken into account.

[0074] In some embodiments of the present application, the image processing device 103 may identify the type of the endoscope 102 based on the data to be identified. When the type of the endoscope 102 fails to be identified based on the data to be identified, it may be determined that the data to be identified does not meet the type identification condition.

[0075] Accordingly, when the type of the endoscope 102 is successfully identified based on the data to be identified, the image processing device 103 may determine that the data to be identified meets the type identification condition.

[0076] As an example, after improving the data quality of the classified data corresponding to each frame of the endoscopic image, the data to be identified corresponding to each frame of the endoscopic image can be obtained, and the data to be identified can be represented as the classification with the highest classification confidence after the data quality is improved. If there is a classification of data to be identified that exceeds a preset proportion in the corresponding data to be identified in each endoscopic image, it can be confirmed that the type of the endoscope 102 is successfully identified, and the endoscope type corresponding to the classification is used as the type of the current endoscope 102. Otherwise, it can be confirmed that the type of the endoscope has failed to be identified.

[0077] As another example, the data to be identified can be represented as classification confidences of different preset classifications after data quality is improved. Among all the data to be identified, the classification confidences of the same classification are accumulated. If the highest value among the accumulated confidence values ​​exceeds a preset threshold, it can be confirmed that the type of the endoscope 102 is successfully identified, and the endoscope type corresponding to the classification with the highest value is used as the type of the current endoscope 102. Otherwise, it can be confirmed that the type of the endoscope 102 has failed to be identified.

[0078] In addition to the above two exemplary methods, the present application also provides a more specific determination method.

[0079] Please refer to Figure 3 After obtaining the data to be identified, the type identification method of the endoscope 102 may include the following steps S301 to S302.

[0080] Step S301 , according to the classification result of each frame of the endoscope image, the classification results associated with the same endoscope type are counted and accumulated to obtain a first accumulated value corresponding to each endoscope type.

[0081] Specifically, the data to be identified of each frame of the endoscopic image may include a corresponding classification result. The classification result may be one of the aforementioned multiple preset classifications, and each preset classification may correspond to at least one endoscope type. For example, the intestine may correspond to a colonoscope, and the tongue may correspond to a gastroscope and a laryngoscope.

[0082] In the classification results of all endoscopic images, the count values ​​of the classification results associated with the same endoscope type are accumulated to obtain the first cumulative value corresponding to each endoscope type. For example, the classification results of three endoscopic images are tongue (corresponding to gastroscope and laryngoscope), intestine (corresponding to colonoscope), and intestine (corresponding to colonoscope), then the first cumulative value corresponding to the colonoscope is 2, and the first cumulative values ​​corresponding to the gastroscope and laryngoscope are both 1.

[0083] Considering that there is a correlation between multiple frames of endoscopic images corresponding to the same type of endoscope, in some embodiments of the present application, if the endoscope type associated with the classification result of the Mth frame of endoscopic image is the target type, and in the first M-1 frames of endoscopic images, the number of endoscopic images belonging to the target classification result is greater than the quantity threshold, then the first cumulative value of the target type is increased by a first value. Otherwise, the first cumulative value of the target type is increased by a second value.

[0084] Wherein, M is a positive integer greater than 0, the target classification result is a classification result associated with the classification result of the Mth frame of the endoscopic image, the second value is less than the first value, and the second value is greater than 0.

[0085] It should be understood that the correlation of classification results may refer to the correlation (or connectivity) between two classification results in terms of human body structure. In other words, after the endoscope 102 enters the body, it can continuously collect multiple organs, and the multiple organs are correlated with each other as classification results.

[0086] For example, if a certain number of endoscopic images in the first M-1 frames have been classified as tongue, then if the classification result of the endoscopic image of the Mth frame is throat, the first cumulative value of the gastroscope can be increased by 2 (first value). Otherwise, if the classification result of the endoscopic image of the Mth frame is throat, the first cumulative value of the gastroscope can be increased by 1 (second value).

[0087] Step S302: when the first cumulative value does not satisfy the cumulative value condition, confirm that the data to be identified does not satisfy the type identification condition.

[0088] In the implementation of the present application, the first cumulative value corresponding to each endoscope type can indicate the possibility that the current endoscope belongs to the endoscope type. When the first cumulative value does not meet the cumulative value condition, for example, when the first cumulative value is less than a certain threshold, it can be confirmed that the data to be identified does not meet the type identification condition, so that the classification data can be used to identify the endoscope type.

[0089] In order to ensure the accuracy of endoscope type recognition as much as possible, in step S102, the image processing device 103 may perform multiple data quality improvements on the classified data to obtain multiple sets of data to be recognized.

[0090] The first set of data to be identified is obtained by improving the data quality of the classified data, the Nth set of data to be identified is obtained by improving the data quality of the N-1th set of data to be identified, and N is a positive integer greater than 1.

[0091] That is to say, after the first data quality improvement of the classified data, each data quality improvement can be achieved based on the result of the previous data quality improvement. In this way, each piece of data to be identified is more credible than the previous piece of data to be identified.

[0092] For each piece of data to be identified, a corresponding first cumulative value can be accumulated. At this time, the cumulative value condition can include at least one threshold condition, and each piece of data to be identified corresponds to at least one threshold condition.

[0093] Correspondingly, the image processing device 103 may detect whether the first accumulated value obtained by accumulating the Nth piece of data to be identified meets the corresponding threshold condition.

[0094] When the first cumulative value obtained by accumulating the Nth portion of data to be identified meets the corresponding threshold condition, it can be confirmed that the first cumulative value meets the cumulative value condition, and then the type of the current endoscope 102 is determined by the first cumulative value obtained by accumulating the Nth portion of data to be identified.

[0095] When the first cumulative value accumulated based on the Nth portion of data to be identified does not meet the corresponding threshold condition, detect whether the first cumulative value accumulated based on the N-1th portion of data to be identified meets the corresponding threshold condition, and so on, until the first cumulative value accumulated based on the first portion of data to be identified does not meet the corresponding threshold condition, confirming that the first cumulative value does not meet the cumulative value condition.

[0096] Specifically, the threshold condition corresponding to the Nth portion of the data to be identified includes a first condition. The above-mentioned detection of whether the first cumulative value accumulated by adding the Nth portion of the data to be identified meets the corresponding threshold condition may include: determining the highest second cumulative value among the first cumulative values ​​calculated based on the Nth portion of the data to be identified; and confirming that the first cumulative value accumulated by adding the Nth portion of the data to be identified meets the first condition when the second cumulative value is greater than or equal to the cumulative value threshold.

[0097] The cumulative value threshold can be set according to actual needs, for example, it can be set to 3x (x>1). When the second cumulative value is greater than or equal to the cumulative value threshold, it means that the current type of endoscope corresponding to the second cumulative value is more likely to be large, and it can be confirmed that the first cumulative value accumulated by the Nth portion of the data to be identified meets the first condition, and the Nth portion of the data to be identified can give a highly reliable identification result. At this time, the endoscope type can be identified by the first cumulative value accumulated by the Nth portion of the data to be identified.

[0098] Considering that the Nth piece of data to be identified is the data to be identified with the highest data quality, the threshold condition corresponding to the Nth piece of data to be identified may also include a second condition. When the first cumulative value is less than the cumulative value threshold, the image processing device 103 may determine the second highest third cumulative value in the first cumulative value calculated according to the Nth piece of data to be identified. Then, the first difference between the second cumulative value and the third cumulative value is determined, and when the first difference is greater than the difference threshold, it is confirmed that the first cumulative value accumulated by the Nth piece of data to be identified meets the second condition.

[0099] The difference threshold can be set according to actual needs, for example, it can be set to x.

[0100] That is to say, if the first cumulative value exceeding the cumulative value threshold cannot be found, analysis can be performed based on the first difference between the second cumulative value and the third cumulative value. If the first difference between the second cumulative value and the third cumulative value is greater than the difference threshold, it means that the possibility of the endoscope type corresponding to the second cumulative value is significantly greater than the possibility of other endoscope types. In this case, the endoscope type is identified by the first cumulative value obtained by accumulating the Nth portion of the data to be identified.

[0101] If the first difference is less than or equal to the difference threshold, it can be confirmed that the first cumulative value accumulated by the Nth portion of the data to be identified does not meet the second condition. At this time, it can be detected whether the first cumulative value accumulated by the N-1th portion of the data to be identified meets the corresponding threshold condition.

[0102] Specifically, the threshold condition corresponding to the N-1th piece of data to be identified may include the third condition. The image processing device 103 may determine the highest fourth cumulative value and the second highest fifth cumulative value in the first cumulative value calculated based on the N-1th piece of data to be identified, and determine the second difference between the fourth cumulative value and the fifth cumulative value. When the second difference is greater than the difference threshold, it is confirmed that the first cumulative value accumulated by the N-1th piece of data to be identified meets the third condition.

[0103] That is to say, if the second difference is greater than the difference threshold, it means that the possibility of the endoscope type corresponding to the fourth cumulative value is significantly greater than the possibility of other endoscope types. At this time, the endoscope type can be identified by the first cumulative value obtained by accumulating the N-1th data to be identified.

[0104] When N>2, for the other copies of data to be identified except the Nth and N-1th copies, they can be identified by referring to the N-1th copy of data to be identified.

[0105] If the first cumulative value accumulated until the first piece of data to be identified still does not meet the corresponding threshold condition, it can be confirmed that the first cumulative value does not meet the cumulative value condition, so as to use the classification data to identify the type of endoscope.

[0106] Correspondingly, when the first cumulative value satisfies the cumulative value condition, it can be confirmed that the data to be identified satisfies the type identification condition, so as to determine the type of the endoscope 102 based on the first cumulative value.

[0107] Specifically, if there is a first cumulative value that satisfies the cumulative value condition, it is confirmed that the data to be identified meets the type identification condition, and the endoscope type corresponding to the first cumulative value that satisfies the cumulative value condition is used as the type of the endoscope 102 .

[0108] In this way, the endoscope type can be identified based on the data with the highest credibility, and when the identification fails, the credibility of the data is gradually reduced to obtain the endoscope type identification.

[0109] In view of the need to promptly identify the type of endoscope 102 during real-time endoscopic examination, such as Figure 4 As shown, when the data to be identified does not meet the preset type identification conditions, the type of the endoscope 102 is determined according to the classification data, which may specifically include steps S401 to S403.

[0110] Step S401, detecting the duration of the endoscope currently entering the body.

[0111] Specifically, the classification result of each frame of endoscopic image in the data to be identified may include the classification results of in vivo frame classification and in vitro frame classification. Taking the aforementioned scene classification as an example, in vivo related and in vivo irrelevant both belong to in vivo frame classification, and in vitro related and in vitro irrelevant both belong to in vitro frame classification. According to the classification result of each frame of endoscopic image, the count values ​​of the endoscopic images belonging to the in vivo frame classification can be accumulated to obtain the sixth cumulative value of the in vivo frame classification.

[0112] In some embodiments of the present application, the endoscopic image can be collected at a preset frame rate fps, and when the sixth cumulative value reaches 2*fps, it can be confirmed that the endoscope has entered the body. The larger the sixth cumulative value, the more in-vivo images captured, which means that the endoscope 102 has been in the body for a longer time, so the sixth cumulative value can be used to determine the above-mentioned duration.

[0113] Step S402: when the data to be identified does not meet the type identification condition and the duration is less than the duration threshold, a new endoscopic image is acquired again to update the data to be identified.

[0114] The duration threshold may be set according to actual conditions, for example, the duration threshold may be set to 20 seconds.

[0115] If the data to be identified does not meet the type identification conditions and the duration is less than the duration threshold, the image processing device 103 can re-acquire a new endoscopic image to update the data to be identified, and then improve the possibility of endoscopic identification based on the data to be identified through the information supplemented by the new endoscopic image.

[0116] Step S403, when the updated data to be identified does not meet the type identification condition and the duration is less than the duration threshold, return to execute the step of reacquiring a new endoscopic image to update the data to be identified until the updated data to be identified meets the type identification condition, or, until the updated data to be identified does not meet the type identification condition and the duration is greater than or equal to the duration threshold, determine the type of endoscope according to the classification data.

[0117] Among them, the method of judging whether the updated data to be identified meets the type identification conditions can be the same as the method of judging whether the data to be identified meets the type identification conditions in the previous text, and this application will not elaborate on this.

[0118] That is to say, if the updated data to be identified does not meet the type identification condition, but the duration is less than the duration threshold, the image can continue to be collected and the data can be updated to increase the possibility of endoscope identification based on the data to be identified, until the updated data to be identified meets the type identification condition, indicating that the identification result of the endoscope type can be given based on the updated data to be identified. If the updated data to be identified has not met the type identification condition, and the duration is greater than or equal to the duration threshold, the type of endoscope 102 can be determined based on the classification data to ensure that the identification result can be given within the duration threshold.

[0119] In this way, the problem of missed lesions or missed parts due to providing identification results too late can be avoided.

[0120] In addition, in the implementation manner of the present application, the method for improving data quality can be selected according to actual conditions.

[0121] In some embodiments of the present application, data to be identified corresponding to each frame of the endoscopic image can be obtained by performing data filtering on the classified data.

[0122] Specifically, data filtering can be classified into confidence filtering and time series filtering.

[0123] The process of confidence filtering is as follows: for a preset classification whose classification confidence is lower than a target confidence threshold among multiple preset classifications, the classification confidence is updated to a preset confidence value to obtain the corresponding data to be identified.

[0124] Specifically, the classification data may include multiple preset classifications and the classification confidence corresponding to each preset classification. If the classification confidence is lower than the target confidence threshold (for example, it can be set to 0.5), it means that the preset classification is not available. The classification confidence can be updated to a preset confidence value (for example, -1) to avoid the preset classification affecting the type recognition of the endoscope and obtain the corresponding data to be identified.

[0125] If the classification confidence of a preset classification is higher than the target confidence threshold, the classification confidence can be maintained in the data to be identified.

[0126] Considering that there is a high similarity between some parts (such as the esophagus and the intestine), in order to reduce the errors caused by such similarity, the image processing device 103 can obtain a default confidence threshold. Then, the first classification with the highest classification confidence and the second classification with the second highest classification confidence in the classification data are determined. In the case where the scene corresponding to the first classification and the scene corresponding to the second classification are interrelated scenes, the default confidence threshold is increased to obtain the target confidence threshold. Otherwise, the default confidence threshold is used as the target confidence threshold.

[0127] Specifically, the user can pre-configure the association relationship between different scenes. If two scenes are highly similar, they can be confirmed as mutually related scenes. If the scene corresponding to the first classification and the scene corresponding to the second classification are mutually related scenes, in order to avoid the two similar scenes affecting the accuracy of image classification and endoscope recognition, the default confidence threshold can be increased to ensure the reliability of the results.

[0128] For example, the first category with the highest classification confidence is the intestine, and the second category with the second highest classification confidence is the esophagus. In this case, the default confidence threshold of 0.5 can be increased to 0.6.

[0129] The process of temporal filtering is as follows: for the classification data corresponding to each frame of endoscopic image, determine the first classification with the highest classification confidence, determine the third classification with the highest number of occurrences in the first classification corresponding to consecutive i frames of endoscopic images, and the number of target endoscopic images. If the number of images is greater than the number threshold, the classification results of consecutive i frames of endoscopic images are all updated to the third classification, and the data to be identified corresponding to consecutive i frames of endoscopic images are obtained.

[0130] Among them, the target endoscopic image is the endoscopic image of the first classification with the highest classification confidence, and i is a positive integer greater than 2.

[0131] That is to say, for the first category corresponding to consecutive i frames of endoscopic images, such as i=10, filtering is performed by determining a threshold t (t<=i, such as t=8). If the number of images j corresponding to the mode m (third category) of the i first categories is greater than t, the recognition results corresponding to the remaining ij endoscopic images are considered to be wrong, and the recognition results corresponding to the ij endoscopic images are all changed to the mode m (third category).

[0132] In addition to confidence filtering and time series filtering, other methods for improving data quality (such as supplementing missing values ​​and removing discrete values) can be applied to this application, and this application does not impose any restrictions on this.

[0133] After determining the type of the endoscope 102, the image processing device 103 can determine an algorithm library corresponding to the type of the endoscope 102 based on the type of the endoscope 102. The algorithm library includes one or more algorithms, such as the light source adaptation algorithm of the light source device 101, the target recognition algorithm (including the lesion recognition algorithm and the part recognition algorithm) of the image processing device 103, and the display algorithm of the display 104. By determining the algorithm library corresponding to the type of the endoscope 102, the purpose of the endoscope 102 can be adapted, making the endoscope system more intelligent.

[0134] It is understandable that the target recognition algorithm can be combined with the images related to the body and the body as previously identified for recognition, which helps to remove non-information frames and improve accuracy.

[0135] For ease of understanding, the data quality improvement times N=2 are used as an example for explanation. Based on several frames of endoscopic images (for a preset acquisition time, such as 10s), classified data can be identified, and the preset classification with the highest classification confidence in the classified data is recorded as data1. Based on data1, confidence filtering can be performed to obtain data2, and based on data2, time series filtering can be performed to obtain data3.

[0136] For data3, the first cumulative value of the endoscope type and the sixth cumulative value of the in-vivo frame classification can be counted respectively. If data3 does not meet the first condition: that is, the first cumulative value of a certain endoscope type in data3 does not reach 3x, and the duration of entering the body does not reach the duration threshold, then change to the second condition: that is, the difference between the first cumulative value of a certain endoscope type in data3 and the first cumulative value of other endoscope types exceeds x, and the class is selected as the final recognition result. If the second condition is still not met, change to the third condition: count the first cumulative value of each endoscope type from data2, and the difference between the first cumulative value of a certain endoscope type and the cumulative value of other endoscope types exceeds x, then the endoscope recognition algorithm can be terminated and the class can be selected as the final recognition result. If the third condition cannot be met, the acquisition time can be adjusted (for example, the acquisition time plus 2), a 2s endoscope image is added, and the judgment of the previous three conditions is repeated.

[0137] In order to avoid unlimited time superposition, a duration threshold max_n is set (such as 20 seconds after entering the body). If the duration reaches the duration threshold max_n and the previous three conditions are still not met, the conditions are further relaxed: the first cumulative value of each endoscope type is counted from data1, and the one with the highest first cumulative value is selected as the final judgment result.

[0138] The embodiments of the present application can filter out and correct false positive points through time series filtering, and reduce the interference of complex scenes on the model through confidence filtering, so as to achieve a higher recognition accuracy. When the data to be recognized cannot identify the type of endoscope within the time threshold, the recognition result can be given in a timely manner based on the classification data to prevent the missed detection of lesions or parts due to the long time of endoscope type recognition. Therefore, it can better balance the recognition accuracy and detection efficiency.

[0139] It should be noted that, for the sake of simplicity of description, the aforementioned method embodiments are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited to the described order of actions, because according to the present application, certain steps can be performed in other orders.

[0140] like Figure 5 The figure is a schematic structural diagram of a device 500 for identifying a type of endoscope provided in an embodiment of the present application. The device 500 for identifying a type of endoscope is configured on an image processing device 103 .

[0141] Specifically, the endoscope type identification device 500 may include:

[0142] An image acquisition unit 501 is used to acquire a plurality of frames of endoscopic images collected by an endoscope;

[0143] An image classification unit 502, configured to determine classification data of each frame of the endoscopic image;

[0144] A data quality improvement unit 503 is used to process the classified data to improve the data quality of the classified data and obtain data to be identified;

[0145] The type identification unit 504 is used to determine the type of the endoscope according to the classification data when the data to be identified does not meet the type identification condition.

[0146] In some embodiments of the present application, the above-mentioned type identification unit 504 can also be specifically used to: identify the type of the endoscope based on the data to be identified; when the type identification of the endoscope based on the data to be identified fails, determine that the data to be identified does not meet the type identification condition.

[0147] In some embodiments of the present application, the data to be identified includes the classification result of the corresponding endoscopic image; the above-mentioned type identification unit 504 can also be specifically used to: according to the classification result of each frame of the endoscopic image, accumulate the counting values ​​of the classification results associated with the same endoscope type to obtain a first cumulative value corresponding to each endoscope type; when the first cumulative value does not meet the cumulative value condition, confirm that the data to be identified does not meet the type identification condition.

[0148] In some embodiments of the present application, the above-mentioned data quality improvement unit 503 can be specifically used to: perform multiple data quality improvements on the classified data to obtain multiple copies of the data to be identified, wherein the first copy of the data to be identified is obtained by performing data quality improvement on the classified data, and the Nth copy of the data to be identified is obtained by performing data quality improvement on the N-1th copy of the data to be identified, and N is a positive integer greater than 1.

[0149] In some embodiments of the present application, the cumulative value condition includes at least one threshold condition, and each portion of the data to be identified corresponds to at least one threshold condition; the above-mentioned type identification unit 504 can also be specifically used to: detect whether the first cumulative value accumulated by the Nth portion of the data to be identified meets the corresponding threshold condition; in the case that the first cumulative value accumulated according to the Nth portion of the data to be identified does not meet the corresponding threshold condition, detect whether the first cumulative value accumulated by the N-1th portion of the data to be identified meets the corresponding threshold condition, and so on, until the first cumulative value accumulated by the first portion of the data to be identified does not meet the corresponding threshold condition, confirming that the first cumulative value does not meet the cumulative value condition.

[0150] In some embodiments of the present application, the threshold condition corresponding to the Nth portion of the data to be identified includes a first condition; the above-mentioned type identification unit 504 can also be specifically used to: determine the highest second cumulative value in the first cumulative value calculated based on the Nth portion of the data to be identified; when the second cumulative value is greater than or equal to the cumulative value threshold, confirm that the first cumulative value obtained by accumulating the Nth portion of the data to be identified satisfies the first condition.

[0151] In some embodiments of the present application, the threshold condition corresponding to the Nth portion of the data to be identified also includes a second condition; the above-mentioned type identification unit 504 can also be specifically used to: determine the second highest third cumulative value in the first cumulative value calculated based on the Nth portion of the data to be identified; determine the first difference between the second cumulative value and the third cumulative value; when the first difference is greater than the difference threshold, confirm that the first cumulative value obtained by accumulating the Nth portion of the data to be identified meets the second condition.

[0152] In some embodiments of the present application, the threshold condition corresponding to the N-1th portion of the data to be identified includes a third condition; the above-mentioned type identification unit 504 can also be specifically used to: determine the highest fourth cumulative value and the second highest fifth cumulative value in the first cumulative value calculated based on the N-1th portion of the data to be identified; determine the second difference between the fourth cumulative value and the fifth cumulative value; and when the second difference is greater than the difference threshold, confirm that the first cumulative value obtained by accumulating the N-1th portion of the data to be identified satisfies the third condition.

[0153] In some embodiments of the present application, the type identification unit 504 may also be specifically used to: in frame sequence, if the endoscope type associated with the classification result of the endoscopic image in the Mth frame is a target type, and in the endoscopic images of the first M-1 frames, the number of endoscopic images belonging to the target classification result is greater than a quantity threshold, then a first value is added to the first cumulative value of the target type, wherein M is a positive integer greater than 0, and the target classification result is a classification result associated with the classification result of the endoscopic image in the Mth frame; otherwise, a second value is added to the first cumulative value of the target type, and the second value is less than the first value, and the second value is greater than 0.

[0154] In some embodiments of the present application, the above-mentioned type identification unit 504 can also be specifically used to: if there is the first cumulative value that satisfies the cumulative value condition, confirm that the data to be identified satisfies the type identification condition, and use the endoscope type corresponding to the first cumulative value as the type of the endoscope.

[0155] In some embodiments of the present application, the type identification unit 504 can also be specifically used to: detect the duration of the current endoscope entering the body; when the data to be identified does not meet the type identification condition and the duration is less than the duration threshold, reacquire a new endoscopic image to update the data to be identified; when the updated data to be identified does not meet the type identification condition and the duration is less than the duration threshold, return to execute the step of reacquiring a new endoscopic image to update the data to be identified until the updated data to be identified meets the type identification condition; or, until the updated data to be identified does not meet the type identification condition and the duration is greater than or equal to the duration threshold, determine the type of the endoscope according to the classification data.

[0156] In some embodiments of the present application, the type identification unit 504 may also be specifically used to: accumulate the count values ​​of the endoscopic images belonging to the in vivo frame classification according to the classification result of each frame of the endoscopic image, to obtain a sixth cumulative value of the in vivo frame classification, and the sixth cumulative value is used to determine the duration.

[0157] In some embodiments of the present application, the data quality improvement unit 503 may be specifically used to: perform data filtering on the classified data to obtain the data to be identified corresponding to each frame of the endoscopic image.

[0158] In some embodiments of the present application, the classification data includes multiple preset classifications and classification confidences corresponding to each preset classification; the above-mentioned data quality improvement unit 503 can be specifically used to: for the preset classifications among the multiple preset classifications whose classification confidences are lower than the target confidence threshold, update the classification confidences to preset confidence values ​​to obtain the corresponding data to be identified.

[0159] In some embodiments of the present application, the above-mentioned data quality improvement unit 503 can be specifically used to: obtain a default confidence threshold; determine the first category with the highest classification confidence and the second category with the second highest classification confidence in the classification data; when the scene corresponding to the first category and the scene corresponding to the second category are interrelated scenes, increase the default confidence threshold to obtain the target confidence threshold.

[0160] In some embodiments of the present application, the data quality improvement unit 503 may be specifically used to: determine the first classification with the highest classification confidence for the classification data corresponding to each frame of the endoscopic image; determine the third classification with the highest number of occurrences and the number of target endoscopic images in the first classification corresponding to consecutive i frames of endoscopic images, wherein the target endoscopic image is the endoscopic image of the first classification with the highest classification confidence, and i is a positive integer greater than 2; if the number of images is greater than a number threshold, then update the classification results of consecutive i frames of endoscopic images to the third classification, and obtain the data to be identified corresponding one-to-one to consecutive i frames of endoscopic images.

[0161] In some embodiments of the present application, the device 500 for identifying the type of endoscope may further include a selection unit, which is specifically used to determine an algorithm library corresponding to the type of endoscope based on the type of the endoscope.

[0162] It should be noted that, for the convenience and simplicity of description, the specific working process of the endoscope type identification device 500 can be referred to Figures 1 to 4 The corresponding process of the method will not be repeated here.

[0163] like Figure 6 , which is a schematic diagram of an image processing device 103 provided in an embodiment of the present application. Specifically, the image processing device 103 may include: a processor 1030, a memory 1031, and a computer program 1032 stored in the memory 1031 and executable on the processor 1030, such as an endoscope type recognition program. When the processor 1030 executes the computer program 1032, the steps in the above-mentioned endoscope type recognition method embodiments are implemented, such as Figure 2 Alternatively, when the processor 1030 executes the computer program 1032, the functions of each module / unit in the above-mentioned device embodiments are realized, for example Figure 5 The functions of the image acquisition unit 501, the image classification unit 502, the data quality improvement unit 503 and the type identification unit 504 are shown.

[0164] The computer program may be divided into one or more modules / units, which are stored in the memory 1031 and executed by the processor 1030 to complete the present application. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program in the image processing device 103.

[0165] For example, the computer program can be divided into: an image acquisition unit, an image classification unit, a data quality improvement unit, and a type recognition unit. The specific functions of each unit are as follows: an image acquisition unit, used to acquire a plurality of frames of endoscopic images collected by an endoscope; an image classification unit, used to determine the classification data of each frame of the endoscopic image; a data quality improvement unit, used to process the classification data, improve the data quality of the classification data, and obtain the data to be recognized; a type recognition unit, used to determine the type of the endoscope according to the classification data when the data to be recognized does not meet the type recognition condition.

[0166] The image processing device 103 may include, but is not limited to, a processor 1030 and a memory 1031. Those skilled in the art will appreciate that Figure 6 It is only an example of the image processing device 103 and does not constitute a limitation of the image processing device 103. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the image processing device 103 may also include input and output devices, network access devices, buses, etc.

[0167] The processor 1030 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), off-the-shelf programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0168] The memory 1031 may be an internal storage unit of the image processing device 103, such as a hard disk or memory of the image processing device 103. The memory 1031 may also be an external storage device of the image processing device 103, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the image processing device 103. Further, the memory 1031 may also include both an internal storage unit of the image processing device 103 and an external storage device. The memory 1031 is used to store the computer program and other programs and data required by the image processing device 103. The memory 1031 may also be used to temporarily store data that has been output or is to be output.

[0169] It should be noted that, for the convenience and brevity of description, the structure of the above-mentioned image processing device 103 can also refer to the specific description of the structure in the method embodiment, and will not be repeated here.

[0170] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0171] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0172] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0173] In the embodiments provided in the present application, it should be understood that the disclosed devices / equipment and methods can be implemented in other ways. For example, the device / equipment embodiments described above are merely schematic, for example, the division of the modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0174] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0175] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0176] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0177] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for identifying the type of an endoscope, characterized in that: include: Acquire a plurality of frames of endoscopic images collected by the endoscope; Determining classification data corresponding to each frame of the endoscopic image; Processing the classified data to improve the data quality of the classified data and obtain data to be identified; In the case that the data to be identified does not meet the preset type identification condition, the type of the endoscope is determined according to the classification data.

2. The method for identifying the type of endoscope according to claim 1, characterized in that: After obtaining the data to be identified, the method for identifying the type of endoscope further includes: identifying the type of the endoscope based on the data to be identified; When the type identification of the endoscope based on the data to be identified fails, it is determined that the data to be identified does not satisfy the type identification condition.

3. The method for identifying the type of endoscope according to claim 1, characterized in that: The data to be identified includes the classification result of the corresponding endoscopic image; After obtaining the data to be identified, the method for identifying the type of endoscope includes: According to the classification result of each frame of the endoscope image, the classification results associated with the same endoscope type are counted and accumulated to obtain a first cumulative value corresponding to each endoscope type; When the first cumulative value does not satisfy the cumulative value condition, it is confirmed that the data to be identified does not satisfy the type identification condition.

4. The method for identifying the type of endoscope according to claim 3, characterized in that: The processing of the classified data to improve the data quality of the classified data to obtain the data to be identified includes: The classified data is subjected to multiple data quality improvements to obtain multiple copies of the data to be identified, wherein the first copy of the data to be identified is obtained by improving the data quality of the classified data, the Nth copy of the data to be identified is obtained by improving the data quality of the N-1th copy of the data to be identified, and N is a positive integer greater than 1.

5. The method for identifying the type of endoscope according to claim 4, characterized in that: The cumulative value condition includes at least one threshold condition, and each piece of the to-be-identified data corresponds to at least one of the threshold conditions; After obtaining the first accumulated value corresponding to each endoscope type, the endoscope type identification method further includes: Detecting whether the first cumulative value obtained by accumulating the Nth portion of the data to be identified meets the corresponding threshold condition; When the first cumulative value accumulated according to the Nth portion of the data to be identified does not satisfy the corresponding threshold condition, detect whether the first cumulative value accumulated according to the N-1th portion of the data to be identified satisfies the corresponding threshold condition, and so on, until the first cumulative value accumulated according to the first portion of the data to be identified does not satisfy the corresponding threshold condition, confirming that the first cumulative value does not satisfy the cumulative value condition.

6. The method for identifying the type of endoscope according to claim 5, characterized in that: The threshold condition corresponding to the Nth piece of the data to be identified includes the first condition; The detecting whether the first accumulated value obtained by accumulating the Nth portion of the data to be identified satisfies the corresponding threshold condition includes: Determining the highest second cumulative value among the first cumulative values ​​calculated based on the Nth piece of the data to be identified; When the second cumulative value is greater than or equal to the cumulative value threshold, it is confirmed that the first cumulative value accumulated by adding the Nth portion of the data to be identified meets the first condition.

7. The method for identifying the type of endoscope according to claim 6, characterized in that: The threshold condition corresponding to the Nth piece of the data to be identified also includes a second condition; When the first cumulative value is less than the cumulative value threshold, the detecting whether the first cumulative value accumulated by adding the Nth portion of the data to be identified satisfies the corresponding threshold condition further includes: Determine the second highest third cumulative value among the first cumulative values ​​calculated based on the Nth piece of the data to be identified; determining a first difference between the second accumulated value and the third accumulated value; When the first difference is greater than a difference threshold, it is confirmed that the first cumulative value obtained by accumulating the Nth portion of the data to be identified satisfies the second condition.

8. The method for identifying the type of endoscope according to claim 7, characterized in that: The threshold condition corresponding to the N-1th piece of the data to be identified includes the third condition; The detecting whether the first accumulated value obtained by accumulating the N-1th piece of the to-be-identified data satisfies the corresponding threshold condition includes: Determine the highest fourth cumulative value and the second highest fifth cumulative value among the first cumulative values ​​calculated based on the N-1th piece of the data to be identified; determining a second difference between the fourth accumulated value and the fifth accumulated value; When the second difference is greater than the difference threshold, it is confirmed that the first cumulative value obtained by accumulating the N-1th portion of the data to be identified meets the third condition.

9. The method for identifying the type of endoscope according to claim 3, characterized in that: According to the classification result of each frame of the endoscopic image, the classification results associated with the same endoscope type are counted and accumulated to obtain a first cumulative value corresponding to each endoscope type, including: In frame order, if the endoscope type associated with the classification result of the endoscopic image of the Mth frame is a target type, and in the endoscopic images of the first M-1 frames, the number of the endoscopic images belonging to the target classification result is greater than the quantity threshold, then the first cumulative value of the target type is increased by a first value, wherein M is a positive integer greater than 0, and the target classification result is a classification result associated with the classification result of the endoscopic image of the Mth frame; Otherwise, a second value is added to the first accumulated value of the target type, where the second value is less than the first value and greater than zero.

10. The method for identifying the type of endoscope according to claim 3, characterized in that: After obtaining the first accumulated value corresponding to each type of endoscope, the method for identifying the type of endoscope further includes: If the first cumulative value that satisfies the cumulative value condition exists, it is confirmed that the data to be identified satisfies the type identification condition, and the endoscope type corresponding to the first cumulative value is used as the type of the endoscope.

11. The method for identifying the type of an endoscope according to any one of claims 1 to 10, characterized in that: When the data to be identified does not meet the preset type identification condition, determining the type of the endoscope according to the classification data includes: Detecting the duration of the endoscope entering the body; When the data to be identified does not meet the type identification condition and the duration is less than the duration threshold, reacquire a new endoscopic image to update the data to be identified; In the case that the updated data to be identified does not satisfy the type identification condition and the duration is less than the duration threshold, return to the step of reacquiring a new endoscopic image to update the data to be identified until the updated data to be identified satisfies the type identification condition; or, until the updated data to be identified does not satisfy the type identification condition and the duration is greater than or equal to the duration threshold, determine the type of the endoscope according to the classification data.

12. The method for identifying the type of endoscope according to claim 11, characterized in that: After obtaining the data to be identified, the method for identifying the type of endoscope further includes: According to the classification result of each frame of the endoscopic image, the count values ​​of the endoscopic images belonging to the in vivo frame classification are accumulated to obtain a sixth cumulative value of the in vivo frame classification, and the sixth cumulative value is used to determine the duration.

13. The method for identifying the type of an endoscope according to any one of claims 1 to 10, characterized in that: The processing of the classified data to improve the data quality of the classified data to obtain the data to be identified includes: The classified data is filtered to obtain the data to be identified corresponding to each frame of the endoscopic image.

14. The method for identifying the type of endoscope according to claim 13, characterized in that: The classification data includes a plurality of preset classifications and classification confidences corresponding to each preset classification; The filtering of the classified data to obtain the data to be identified corresponding to each frame of the endoscopic image comprises: For the preset classifications among the multiple preset classifications whose classification confidence is lower than the target confidence threshold, the classification confidence is updated to a preset confidence value to obtain the corresponding data to be identified.

15. The method for identifying the type of endoscope according to claim 14, characterized in that: Before updating the classification confidence to a preset confidence value, the method for identifying the type of endoscope further includes: Get the default confidence threshold; Determine a first classification with the highest classification confidence and a second classification with the second highest classification confidence in the classification data; In the case where the scene corresponding to the first classification and the scene corresponding to the second classification are mutually related scenes, the default confidence threshold is increased to obtain the target confidence threshold.

16. The method for identifying the type of endoscope according to claim 13, characterized in that: The filtering of the classified data to obtain the data to be identified corresponding to each frame of the endoscopic image comprises: For the classification data corresponding to each frame of the endoscopic image, determining a first classification with the highest classification confidence; In the first classification corresponding to the endoscopic images of consecutive i frames, determine the third classification with the highest number of occurrences and the number of target endoscopic images, wherein the target endoscopic image is the endoscopic image of the first classification with the highest classification confidence, and i is a positive integer greater than 2; If the number of images is greater than the number threshold, the classification results of the i consecutive frames of endoscopic images are all updated to the third classification, so as to obtain the data to be identified that correspond one to one to the i consecutive frames of endoscopic images.

17. The method for identifying the type of an endoscope according to any one of claims 1 to 10, characterized in that: After determining the type of the endoscope, the method for identifying the type of the endoscope further includes: Based on the type of the endoscope, an algorithm library corresponding to the type of the endoscope is determined.

18. A type identification device for an endoscope, characterized in that: include: An image acquisition unit, used to acquire a plurality of frames of endoscopic images collected by the endoscope; An image classification unit, used to determine classification data of each frame of the endoscopic image; A data quality improvement unit, used for processing the classified data to improve the data quality of the classified data and obtain data to be identified; A type identification unit is used to determine the type of the endoscope according to the classification data when the data to be identified does not meet the type identification condition.

19. An image processing device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method for identifying the type of an endoscope according to any one of claims 1 to 17 are implemented.

20. An endoscope system, characterized in that: include: Light sources, endoscopes, image processing equipment, and displays; The image processing device is used to execute the steps of the method for identifying the type of endoscope as described in any one of claims 1 to 17.

Images