Method and device for screening lesions by using computer, and medium

Abstract

The invention provides a method and a device for screening lesions by using a computer, and a medium. Obtaining a training sample set of the medical images with lesion category labels and a category similarity matrix representing the similarity between the categories; in the deep learning network training process, intra-class and inter-class similarity distances of lesion classes are obtained based on an output feature matrix of a convolutional layer, a first loss function is obtained by combining corresponding elements in a class similarity matrix, the intra-class similarity distance is in positive correlation with the first loss function, and the inter-class similarity distance is in negative correlation with the first loss function; obtaining a second loss function based on the probability matrix of the category to which the training sample output by the deep learning network belongs and the lesion category label; training the deep learning network based on the first loss function and the second loss function; and predicting the lesion category of the medical image by using the trained deep learning network. And the prediction accuracy is improved through lesion category similarity priori knowledge.

Description

Technical Field

[0001] The present application belongs to the technical field of medical image analysis and processing, and in particular relates to a method, device and medium for screening lesions using a computer. Background Art

[0002] In recent years, with the development of artificial intelligence technology, lesions in medical images can be automatically identified and intelligently screened and analyzed through technologies such as deep learning networks, thereby effectively reducing the workload of doctors' diagnosis. However, the accuracy of lesion identification and classification directly affects the reliability of medical diagnosis. Therefore, the classification accuracy of deep learning networks has always been a technical indicator that focuses on the field of automatic medical image processing.

[0003] In existing segmented methods, a feature representation of the medical image to be classified or the identified lesion is typically first performed using a convolutional neural network, and then a classifier is used based on the feature representation to identify the lesion and predict its category. In other technologies, during the subsequent classification process using a classifier, optimization operations such as feature grouping can be combined with specific similarity metrics to achieve a more accurate prediction of the category to which the lesion belongs. However, these existing technologies only perform similarity measurement in the latter stage and use the features after the similarity measurement for category prediction, losing the rich feature information in the image feature representation in the previous stage.

[0004] It can be seen that how to make more full use of the rich information contained in the feature representation after image feature encoding is an important technical direction to further optimize the classification accuracy of deep learning networks. Summary of the Invention

[0005] This application is proposed to address the aforementioned problems existing in the prior art. This application intends to provide a computer-implemented method, apparatus, and medium for screening lesions, which can further optimize the accuracy of lesion classification prediction using a deep learning network by utilizing prior knowledge about lesion classification and the rich information contained in the feature representation after image feature encoding.

[0006] According to the first scheme of the present application, a method for screening lesions using a computer is provided, comprising obtaining a training sample set of medical images with lesion category annotations and a category similarity matrix representing the similarity between each lesion category and other lesion categories; a processor, in the process of training a deep learning network including a convolutional layer based on the training sample set: inputting the medical image training samples in the training sample set into the deep learning network; obtaining the intra-class similarity distance of each lesion category and the inter-class similarity distance of each lesion category based on the output feature matrix of the convolutional layer; obtaining the intra-class similarity distance, the inter-class similarity distance and the category similarity matrix based on the intra-class similarity distance, the inter-class similarity distance and the category similarity matrix. A first loss function is provided, so that the intra-class similarity distance is positively correlated with the first loss function, the inter-class similarity distance is negatively correlated with the first loss function, and the weight coefficient of each inter-class similarity distance is set in association with the corresponding element value in the class similarity matrix; based on the probability matrix of the category to which the medical image training sample belongs output by the deep learning network and the lesion category labeling of the medical image training sample, a second loss function is obtained; the deep learning network is trained based on a joint loss function considering the first loss function and the second loss function; and the trained deep learning network is used to predict the lesion category to which the medical image of the lesion category to be screened belongs.

[0007] According to a second embodiment of the present application, a computer-implemented apparatus for screening for lesions is provided. The apparatus includes an interface configured to obtain a training sample set of medical images labeled with lesion categories and a category similarity matrix representing the similarity between each lesion category and other lesion categories. The apparatus also includes at least one processor configured to execute the computer-implemented method for screening for lesions described in various embodiments of the present application.

[0008] According to the third embodiment of the present application, a non-temporary computer-readable storage medium is provided, on which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, the method for screening lesions using a computer implemented in various embodiments of the present application is executed.

[0009] According to the computer-implemented method, device, and medium for screening lesions in accordance with various embodiments of the present application, by introducing intra-class / inter-class similarity distances at the feature graph output by the convolutional layer, and introducing a category similarity matrix containing prior knowledge about the lesion category into the calculation of the loss function, the deep learning network can learn a feature graph distribution with high aggregation within the convolutional layer output class and low coupling between classes, thereby making different lesion categories easier to distinguish accurately, and the prediction results of the lesion category are more in line with the prior rules. The trained deep learning network will also have a significant improvement in prediction performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In the drawings, which are not necessarily drawn to scale, the same reference numerals may describe similar components in different views. The same reference numerals with letter suffixes or different letter suffixes may represent different instances of similar components. The accompanying drawings generally illustrate various embodiments by way of example and not limitation, and together with the description and claims, serve to illustrate the disclosed embodiments. Where appropriate, the same reference numerals are used throughout the drawings to refer to the same or similar parts. Such embodiments are illustrative and are not intended to be exhaustive or exclusive of the embodiments of the present apparatus or method.

[0011] Figure 1 A flowchart of a method for screening lesions using a computer according to an embodiment of the present application is shown.

[0012] FIG2( a ) shows the distribution of feature maps output by a deep learning network that does not include distance metric learning according to an embodiment of the present application.

[0013] FIG2( b ) shows the distribution of feature maps output by a deep learning network including distance metric learning according to an embodiment of the present application.

[0014] Figure 3 A schematic diagram of the structure of a device 300 for screening lesions according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0015] In order to enable those skilled in the art to better understand the technical solution of the present disclosure, the present disclosure is described in detail below in conjunction with the accompanying drawings and specific embodiments. The embodiments of the present disclosure are further described in detail below in conjunction with the accompanying drawings and specific embodiments, but are not intended to limit the present disclosure.

[0016] The terms “first”, “second” and similar terms used in this application do not indicate any order, quantity or importance, but are only used to distinguish different parts. The expressions “first” and “second” are only used for the convenience of expression to distinguish between numbers, and are not intended to imply that the “first component” and the “second component” must have different physical properties. In fact, the “first component” and the “second component” may have the same or different structures, and this is not limited here, as long as the “first component” and the “second component” are discrete components. Furthermore, if the context provides sufficient explanation, the “first component” and the “second component” may not even be discrete components, but may be integrated into the same component, or may be interchangeable with each other.

[0017] In this application, when a specific device is described as being located between a first device and a second device, an intervening device may or may not be present between the specific device and the first device or the second device. When a specific device is described as being connected to another device, the specific device may be directly connected to the other device without an intervening device, or may be not directly connected to the other device but with an intervening device.

[0018] Words such as "include" or "comprising" imply that the elements preceding the word include the elements listed after it, and do not exclude the possibility of additional elements being included. Words such as "upper," "lower," "left," and "right" are used only to indicate relative positions. If the absolute position of the object being described changes, the relative position may also change accordingly.

[0019] Figure 1 A flowchart of a method for screening lesions using a computer according to an embodiment of the present application is shown.

[0020] like Figure 1 As shown, in step 101, a training sample set of medical images with lesion category annotations and a category similarity matrix representing the similarity between each lesion category and other lesion categories are first obtained. In other embodiments, medical images of the lesion category to be screened should also be obtained.

[0021] In some embodiments, the medical images in the training sample set and the medical images of the lesion category to be screened may include, but are not limited to, coronary CTA images, lung CT images, head and neck CTA images, etc.

[0022] In some embodiments, the category similarity matrix S For example, it can be expressed as the following formula (1): (1) in, n is the number of lesion categories, ,…, represents the similarity between lesion category 1 and lesion category 1~lesion category n. Similarly, ,…, It represents the similarity between lesion category n and lesion category 1~lesion category n. Assuming that the similarity value is normalized to the interval [0, 1], the category similarity matrix S is a matrix whose diagonal elements are all 1.

[0023] In the embodiment of the present application, the medical image may be, for example, a coronary CTA image, a head and neck CTA image, a lung CT image, etc. Taking the coronary CTA image as an example, assuming that the lesion categories to be predicted only include plaque (lesion category 1 is lesion) and no plaque (lesion category 2 is no lesion), then, assuming that the number of images of the two categories is equal, the feature distribution within each category of the images is basically close, and normalization processing is performed, the and to the same value, such as 1, and and Set to the same value, for example, 0, so that the category similarity matrix It can be expressed as the following formula (2): (2) In other embodiments, in order to screen a specific lesion category or a specific group of lesion categories, the specific lesion category / a group of lesion categories can be defined as having lesions, and the corresponding medical image as a whole is defined as a positive sample (Positive), while all other situations that do not contain the specific lesion category / a group of lesion categories are defined as having no lesions as a whole, and the corresponding medical images are defined as negative samples (Negative). In this way, according to the above-mentioned method of including the two lesion categories of having lesions and having no lesions, the medical image can be accurately screened to determine whether it belongs to a specific lesion category / a group of lesion categories and meets the user's requirements.

[0024] In other cases, it is assumed that not only the presence of lesions is to be predicted, but also the subtypes of plaques need to be further distinguished, for example, vulnerable plaques (lesion category 1), calcified plaques (lesion category 2), mixed plaques (lesion category 3) and no plaques (lesion category 4). In this case, the similarity between each lesion category and other lesion categories can be pre-defined based on prior knowledge associated with the lesion category, for example, it can be pre-defined by doctors based on their field experience. The following formula (3) gives a category similarity matrix: Example: (3) It is worth noting that the category similarity matrix of formula (3) The values ​​of each similarity can also be set after theoretical calculation or inference, and / or statistically analyzing relevant supporting data in a certain data set using other algorithms or tools. This can more reasonably and accurately reflect the similarities between different lesion categories based on the inherent correlation between lesion categories, and provide a reference for doctor users to more accurately distinguish lesion subtypes.

[0025] In other embodiments, for example, it is desired to predict different development stages of lesions contained in medical images, such as tumor staging (such as T1 to T4 stages), degree of atherosclerosis (mild, moderate, severe, etc.), that is, lesions at different development stages are used as corresponding lesion categories. In this case, the category similarity matrix can be defined based on the actual similarity between lesions at different development stages. S ,Therefore, different development stages of the lesions can be better distinguished, providing more valuable reference for doctor users.

[0026] Next, in step 102, the processor trains the deep learning network based on the training sample set, wherein the deep learning network includes at least a convolutional layer, for example, it can be a deep convolutional neural network of the type of fully convolutional neural network, ResNet (residual network), VGG (visual geometry group), Hrnet‌HRNet (high resolution network), etc. The specific network architecture is not limited in this application.

[0027] The training process of a deep learning network may include the following steps: Step 1021: Input the medical image training samples in the training sample set into the deep learning network.

[0028] Step 1022: Based on the output feature matrix after the convolution layer performs deep convolution on the medical image, a distance metric is calculated to obtain the intra-class similarity distance of each lesion category and the inter-class similarity distance of each lesion category.

[0029] Step 1023: Based on the intra-class similarity distance, the inter-class similarity distance, and the class similarity matrix, a first loss function is obtained, such that the intra-class similarity distance is positively correlated with the first loss function, the inter-class similarity distance is negatively correlated with the first loss function, and the weight coefficients of the respective inter-class similarity distances are set in association with the corresponding element values ​​in the class similarity matrix. In other words, the optimization direction of the first loss function is to make the intra-class similarity distance smaller and the inter-class similarity distance larger, and the prior knowledge about the inter-class similarity contained in the class similarity matrix is ​​introduced into the first loss function to participate in the training of the deep learning network.

[0030] Step 1024: Based on the probability matrix of the category to which the medical image training samples belong and the lesion category labels of the medical image training samples output by the deep learning network, obtain a second loss function. In some embodiments, the lesion category labels of the medical image training samples in the training sample set can be represented as a one-dimensional vector, in which the value of the element corresponding to the lesion category to which the sample belongs is 1, and the other elements are 0. Thus, the lesion category labels of each medical image training sample can be represented as a matrix composed of multiple one-dimensional arrays, and the second loss function can be constructed based on the cross entropy between the probability matrix of the category to which each medical image training sample belongs and the lesion category label matrix of the medical image training samples output by the deep learning network, in order to achieve performance optimization between classifier categories in the deep learning network. In other embodiments, the second loss function can also be constructed using any applicable method such as mean square error (MSE), mean absolute error (MAE), etc., and this application does not impose any restrictions on this.

[0031] There is no order requirement for executing the above steps 1023 and 1024, and both can be executed simultaneously.

[0032] Step 1025: Training the deep learning network based on a joint loss function that considers the first loss function and the second loss function. As an example only, the joint loss function may be, for example, a weighted loss function of the first loss function and the second loss function. The deep learning network may be trained, for example, by performing the joint loss function shown in the following equation (4): ) is minimized to achieve: (4) in, is the first loss function, is the second loss function, is the weighted coefficient of the loss function, which is a value in the range of (0,1) and can be set as needed. This application does not impose any specific restrictions. In the training optimization process, for example, a back propagation optimization algorithm or an improved variant of a gradient optimization algorithm including a momentum method and an adaptive learning rate may be used, and this application does not limit this.

[0033] After the deep learning network training is complete, in step 103, the trained deep learning network is used to predict the lesion category of the medical image to be screened. The medical image to be screened can be pre-loaded or acquired through an interface via data transmission from a medical imaging device such as a CT machine, but this application does not limit this.

[0034] In some embodiments, the prediction results output by the trained deep learning network can be, for example, a vector composed of probability values ​​of the medical image belonging to various lesion categories. Based on this vector, an applicable algorithm can be used to determine the lesion category to which the final medical image belongs. The specific method can be, for example, selecting the lesion category corresponding to the maximum probability value, etc. This application does not limit this.

[0035] Figure 2(a) shows the distribution of feature maps output by the convolutional layer when there is no intra-class / inter-class similarity distance learning, and Figure 2(b) shows the distribution of feature maps output by the convolutional layer when intra-class / inter-class similarity distance learning is included according to an embodiment of the present application. Figures 2(a) and 2(b) both take lung images with pneumonia and non-pneumonia as examples, where dots represent pneumonia (positive samples) and stars represent non-pneumonia (negative samples). It can be clearly seen from Figures 2(a) and 2(b) that although in Figure 2(a), most of the pneumonia features and non-pneumonia features are located in different areas and can be basically distinguished, there are still a large number of image features at the intersection. These features will have great uncertainty in classification and may even lead to misclassification. In Figure 2(b), due to the addition of learning optimization of the intra-class / inter-class similarity distance metric for the features encoded by the convolutional layer, not only is the boundary between the two lesion categories clear and the feature spacing significantly increased, but the features within the same lesion category are also more clustered and compact, making it easier for the deep learning network to classify samples and achieving higher classification accuracy. In particular, the classification performance will be better for those image features that were originally at the intersection. Taking the medical image as a lung CT image, the lesion categories to be predicted include calcified pulmonary nodules, ground glass pulmonary nodules, solid pulmonary nodules, and semi-solid pulmonary nodules. In existing lesion category prediction methods, lesion categories with smaller differences such as solid pulmonary nodules and semi-solid pulmonary nodules are often not well distinguished. However, in the embodiments of the present application, since the distance between different lesion categories is maximized during the learning process of the deep learning network, the prediction results will also have higher accuracy, higher recall rate, and better overall prediction performance.

[0036] In addition, according to the computer-implemented method for screening lesions in various embodiments of the present application, when providing prior knowledge for neural network training and learning, it is not necessary to re-label or supplement the samples in the training sample set one by one, but only to provide an additional category similarity matrix. In other words, the method of the embodiment of the present application does not excessively increase the user's additional workload. By introducing the category similarity matrix when constructing the first loss function, not only can the distribution trend of the image features extracted by the convolution layer be optimized, but the deep learning network will also be trained at the coding feature level to generate image coding features that are as consistent as possible with the prior knowledge in the category similarity matrix, so that when classifying lesion categories, not only the overall classification performance is higher from a statistical point of view, but also the probability of the occurrence of adverse conditions that violate prior knowledge when the deep learning network is trained as a black box can be reduced to a certain extent.

[0037] There are many ways to introduce the category similarity matrix into the construction of the first loss function. Taking as an example, the specific method of setting the weight coefficients of the similarity distances between each class in association with the corresponding elements in the class similarity matrix is ​​described.

[0038] First, we can use the category similarity matrix Calculate the inter-class distance matrix representing the distance between each lesion class and other lesion classes , as shown in the following formula (5): (5) in, n is the number of lesion categories, For elements all 1 n × n Matrix. From the inter-class distance matrix The calculation method shows that the higher the similarity between the lesion category and other lesion categories, the smaller the corresponding inter-class distance. Therefore, the inter-class distance matrix In fact, it reflects the inter-class distance between each lesion class and other lesion classes.

[0039] Then, based on the inter-class distance matrix The first proportional relationship between the inter-class distances between each lesion category and other lesion categories , pre-set the second proportional relationship between the weight coefficients of the corresponding inter-class similarity distance in the first loss function The inter-class distance matrix shown in formula (5) For example, the first proportional relationship It can be expressed as the following formula (6): (6) In some embodiments, according to the first proportional relationship Presetting the second proportional relationship between the weight coefficients of the corresponding inter-class similarity distances in the first loss function For example, the above inter-class distance matrix can be directly The corresponding inter-class distance element value in is used as its weight coefficient, that is, and Exactly the same, as shown in the following formula (7): (7) Alternatively, in some other embodiments, the weight coefficients corresponding to the inter-class similarity distances in the first loss function can also be used as learnable parameters in the deep learning network, that is, in the deep learning network training process, according to the inter-class distance matrix The first proportional relationship between the distance between each lesion category and other lesion categories , dynamically adjust the weight coefficient of the corresponding inter-class similarity distance in the first loss function, so that after the deep learning network training is completed, the ranking of the similarity distances between each class is consistent with the inter-class distance matrix The deviation of the order of the corresponding elements in does not exceed the preset range. For example, in the inter-class distance matrix shown in formula (5) In the example, the distances between classes are sorted from large to small according to the element values. As shown in the following formula (8): (8) The values ​​of the elements in {} are equal, so the order of the elements is not differentiated.

[0040] After the deep learning network training is completed, the ranking of the similarity distances between the classes is The deviation between them should not exceed the preset range, that is, their sorting should roughly conform to The order in , or, only a few elements deviate slightly The specific acceptable element sorting, such as the number of deviated elements and the range of deviation, can be determined based on the training cost of the deep learning network and / or the needs of the user, and this application does not limit it here.

[0041] An exemplary method for obtaining the intra-class similarity distance of each lesion category and the inter-class similarity distance of each lesion category based on the output feature matrix of the convolutional layer is as follows: In one aspect, based on the output feature matrix of the convolutional layer, the feature mean of the medical image training samples with the same lesion category label is calculated as the feature mean of the corresponding lesion category. Then, based on the medical image training samples in each lesion category and the feature mean of that lesion category, the intra-class similarity distance of that lesion category is calculated. In some embodiments, the specific method for calculating the feature mean can be, for example, calculating the arithmetic mean of each feature vector, or using methods such as harmonic mean, normalization, and then calculating the mean, which are not specifically limited in this application.

[0042] On the other hand, based on the medical image training samples in each lesion category and the feature means of other lesion categories except the lesion category, the inter-class similarity distance between the lesion category and other lesion categories is calculated.

[0043] Taking the lesion category as an example, which only includes lesion and no lesion, the intra-class similarity distance of each lesion category includes the cosine distance of the positive sample class. pp ) and the cosine distance within the negative sample class (Distance nn ), as shown in formula (9) and formula (10) respectively: (9) (10) In formula (9) and formula (10),<a,b> represents the cosine distance between a and b, ||·|| represents the modulus of the eigenvector; represents the number of samples in the positive class, represents the number of samples in the negative sample class, Indicates the convolutional layer output feature matrix corresponding to the The feature vector of positive samples, represents the mean of all feature vectors in the positive class (that is, the centroid of the positive class), Indicates the convolutional layer output feature matrix corresponding to the The feature vector of negative samples, represents the mean of all feature vectors in the negative class (that is, the centroid of the negative class).

[0044] The inter-class similarity distance of each lesion category includes the cosine distance between positive and negative samples. np ) and the cosine distance between negative and positive samples (Distance pn ), as shown in formula (11) and formula (12) respectively: (11) (12) The meanings of the symbols and numbers in the above formula (11) and formula (12) are the same as those in formula (9) or formula (10) and are not repeated here.

[0045] It is worth noting that in the calculation process of the above-mentioned intra-class distances and inter-class distances, in addition to the cosine distance, measurement methods such as Euclidean distance (such as L2 distance), Manhattan distance (such as L1 distance), Manhattan square distance, Chebyshev distance, etc. can also be used. This application does not make specific restrictions on this.

[0046] Based on the calculated intra-class similarity distance of each lesion category and the inter-class similarity distance between each lesion category and other lesion categories, the first loss function can be further calculated according to the following formula (13): : (13) Among them, k1 and k2 are weighted coefficients of the cosine distance between classes, taking values ​​between (0, 1), and k1 and k2 are determined in association with the similarity between the positive sample class and the negative sample class, and the similarity between the negative sample class and the positive sample class in the class similarity matrix.

[0047] In general, the ratio between k1 and k2 should be consistent with the ratio between the similarity between negative sample classes and the similarity between negative sample classes and positive sample classes in the category similarity matrix, or should not exceed a preset deviation. More specifically, the category similarity matrix in formula (2) above can be combined with Let’s take an example. First, calculate the similarity matrix with the category The corresponding inter-class distance matrix , as shown in formula (14): (14) in, For elements all 1 2 × 2 matrix.

[0048] From formula (14), we can see that The first proportional relationship of the corresponding inter-class distance As shown in formula (15): =1:1 (15) Therefore, the second proportional relationship between k1 and k2 can be , that is, when k1 and k2 are normalized, k1 and k2 can be set to the same value, that is, k1 and k2 are both set to 0.5.

[0049] In some other embodiments, the category similarity matrix It is not always a symmetric matrix. For example, if the number of images corresponding to different lesion categories is quite different, and / or the intra-class feature distribution (such as dispersion index) of images corresponding to different lesion categories is quite different, the category similarity matrix can be set according to the specific situation of the image sample set and the lesion category. The specific values ​​of the elements in are not limited in this application, as long as they can fully reflect the similarity between each lesion category and other lesion categories.

[0050] Table 1 shows a comparison of the classification performance of deep learning networks on a test set using the methods of the present invention and a conventional method from the prior art (which does not include a class similarity matrix or intra-class / inter-class distance metrics in the convolutional layer output feature matrix). The experiments in Table 1 used a multi-center private dataset. The medical images in the dataset were coronary CTA images, and the lesion categories included plaque and non-plaque. The experiments were conducted using 802 training data samples, 100 validation data samples, and 100 test samples in the test set.

[0051] Table 1 Comparison of classification performance between the method of the present application and conventional methods in the prior art

[0052] The method in the embodiment of the present application introduces the intra-class / inter-class similarity distance at the feature map output by the convolution layer, and introduces the category similarity matrix containing the prior knowledge about the lesion category into the calculation of the loss function, so that the deep learning network can learn the feature map distribution with high aggregation within the convolution layer output class and low coupling between classes, thereby making different lesion categories easier to distinguish accurately, and the prediction results of the lesion category are more in line with the prior rules. It can be seen from the three performance index data in Table 1 that the method in the embodiment of the present application improves the accuracy index from 85.49% of the conventional method to 86.52%, and the recall rate index is also improved from 75.51% to 78.57%. The F1 value reflecting the comprehensive performance is improved from 80.19% to 82.35%. Therefore, it can be considered that the method in the embodiment of the present application is comprehensively superior to the conventional method in the prior art.

[0053] According to an embodiment of the present application, a device for screening lesions using a computer is also provided. Figure 3 A schematic diagram of the structure of a device for screening lesions according to an embodiment of the present application is shown.

[0054] like Figure 3As shown, the apparatus 300 includes at least an interface 301 and at least one processor 302. In some embodiments, the interface 301 can be configured to obtain a training sample set of medical images with lesion category annotations and a category similarity matrix representing the similarity between each lesion category and other lesion categories. In other embodiments, for example, medical images of the lesion category to be screened can also be obtained via the interface 301. The at least one processor 302 can be configured to execute the computer-implemented method for screening lesions as described in various embodiments of the present application.

[0055] In some embodiments, the processor 302 may be, for example, a processing device including one or more general-purpose processing devices, such as a microprocessor, a central processing unit (CPU), a graphics processing unit (GPU), etc. More specifically, the processor may be a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a processor that executes other instruction sets, or a processor that executes a combination of instruction sets. The processor 302 may also be one or more special-purpose processing devices, such as an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP), a system on a chip (SoC), etc.

[0056] The exemplary methods described herein may be at least partially implemented by a machine or computer. Furthermore, a non-transitory computer-readable storage medium is provided according to an embodiment of the present application, on which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, the computer-implemented method for screening for lesions as described in various embodiments of the present application is executed.

[0057] This application describes various operations or functions that can be implemented as software code or instructions or defined as software code or instructions. Such content can be source code or differential code ("incremental" or "patch" code) that can be directly executed ("object" or "executable" form). Software code or instructions can be stored in a computer-readable storage medium and, when executed, can cause a machine to perform the described functions or operations, and include any mechanism for storing information in a form accessible to a machine (e.g., a computing device, an electronic system, etc.), such as recordable or non-recordable media (e.g., read-only memory (ROM), random access memory (RAM), magnetic disk storage media, optical storage media, flash memory devices, etc.).

[0058] The implementation of the computer-implemented method for screening for lesions according to an embodiment of the present application may include software code, such as microcode, assembly language code, high-level language code, etc. Various software programming techniques can be used to create various programs or program modules. For example, program parts or program modules can be designed in or with the aid of Java, Python, C, C++, assembly language, or any other known programming language. One or more of such software parts or modules can be integrated into a computer system and / or computer-readable media. Such software code may include computer-readable instructions for performing various methods. The software code may form part of a computer program product or computer program module. In addition, in an example, the software code may be tangibly stored on one or more volatile, non-transitory, or non-volatile tangible computer-readable media, such as during execution or at other times. Examples of such tangible computer-readable media may include, but are not limited to, hard disks, removable disks, removable optical disks (such as optical disks and digital video disks), cassette tapes, memory cards or memory sticks, random access memory (RAM), read-only memory (ROM), etc.

[0059] Furthermore, although exemplary embodiments have been described herein, the scope includes any and all embodiments based on the present application with equivalent elements, modifications, omissions, combinations (e.g., solutions that intersect various embodiments), adaptations, or changes. The elements of the claims are to be interpreted broadly based on the language employed in the claims and are not limited to the examples described in this specification or during the prosecution of the application, which examples are to be construed as non-exclusive. Therefore, it is intended that this specification and examples be considered merely as examples, with the true scope and spirit being indicated by the following claims and their full scope of equivalents.

[0060] The above description is intended to be illustrative and not restrictive. For example, the above examples (or one or more of their embodiments) may be used in combination with each other. For example, a person of ordinary skill in the art may use other embodiments when reading the above description. In addition, in the above detailed description, various features may be grouped together to simplify the application. This should not be interpreted as an intention that a disclosed feature that is not claimed for protection is essential to any claim. On the contrary, the subject matter of the present application may have less than all the features of a particular disclosed embodiment. Thus, the claims are incorporated herein into the detailed description as examples or embodiments, with each claim independently serving as a separate embodiment, and it is contemplated that these embodiments may be combined with each other in various combinations or arrangements. The scope of the present application should be determined with reference to the appended claims and the full scope of equivalents to which such claims are entitled.

[0061] The above embodiments are merely exemplary embodiments of the present application and are not intended to limit the scope of the present application. The scope of protection of the present application is defined by the claims. Those skilled in the art may make various modifications or equivalent substitutions to the present application within the essence and scope of protection of the present application, and such modifications or equivalent substitutions shall also be deemed to fall within the scope of protection of the present application.

Claims

1. A method for screening lesions using a computer, characterized in that: include: Obtaining a training sample set of medical images with lesion category annotations and a category similarity matrix representing the similarity between each lesion category and other lesion categories; The processor, in a process of training a deep learning network including a convolutional layer based on the training sample set: Inputting the medical image training samples in the training sample set into the deep learning network; Based on the output feature matrix of the convolutional layer, obtaining the intra-class similarity distance of each lesion category and the inter-class similarity distance of each lesion category; Based on the intra-class similarity distance, the inter-class similarity distance, and the class similarity matrix, obtaining a first loss function, such that the intra-class similarity distance is positively correlated with the first loss function, the inter-class similarity distance is negatively correlated with the first loss function, and a weight coefficient of each inter-class similarity distance is set in association with a corresponding element value in the class similarity matrix; Obtaining a second loss function based on the probability matrix of the category to which the medical image training sample belongs and the lesion category labeling of the medical image training sample output by the deep learning network; Training the deep learning network based on a joint loss function that considers the first loss function and the second loss function; The trained deep learning network is used to predict the lesion category of medical images to be screened.

2. The method according to claim 1, characterized in that The weight coefficients of the inter-class similarity distances are associated with corresponding element values ​​in the class similarity matrix, specifically including calculating an inter-class distance matrix representing the distance between each lesion class and other lesion classes based on the class similarity matrix, and: According to the first proportional relationship between the distances between each lesion category and other lesion categories in the inter-class distance matrix, a second proportional relationship between the weight coefficients of the corresponding inter-class similarity distances in the first loss function is pre-set; or, During the training process of the deep learning network, the weight coefficient of the corresponding inter-class similarity distance in the first loss function is dynamically adjusted based on the first proportional relationship between the distances between each lesion category and other lesion categories in the inter-class distance matrix, so that after the training of the deep learning network is completed, the deviation between the ranking of each inter-class similarity distance and the ranking of the corresponding elements in the inter-class distance matrix does not exceed a preset range.

3. The method according to claim 1 or 2, characterized in that Based on the output feature matrix of the convolutional layer, obtaining the intra-class similarity distance of each lesion category and the inter-class similarity distance of each lesion category further includes: Based on the output feature matrix of the convolutional layer, calculating the feature mean of the medical image training samples with the same lesion category label as the feature mean of the corresponding lesion category; Based on each medical image training sample in each lesion category and the feature mean of the lesion category, the intra-class similarity distance of the lesion category is calculated; Based on each medical image training sample in each lesion category and the feature means of other lesion categories except the lesion category, the inter-class similarity distance between the lesion category and other lesion categories is calculated.

4. The method according to claim 1 or 2, characterized in that Obtaining a second loss function based on a probability matrix of the category to which the medical image training sample belongs and a lesion category labeling matrix of the medical image training sample output by the deep learning network further includes: The cross entropy of the probability matrix of the category to which the medical image training sample belongs and the lesion category labeling matrix of the medical image training sample output by the deep learning network is used as the second loss function.

5. The method according to claim 2, characterized in that In the case where the lesion category only includes lesion and no lesion, the intra-class similarity distance of each lesion category includes the intra-class distance of positive samples and the intra-class distance of negative samples; The inter-class similarity distance of each lesion category includes the positive sample-negative sample inter-class distance and the negative sample-positive sample inter-class distance.

6. The method according to claim 5, characterized in that Based on the intra-class similarity distance and the inter-class similarity distance, obtaining a first loss function further includes: Calculate the intra-class cosine distance of positive samples, the intra-class cosine distance of negative samples, the inter-class cosine distance of positive samples and negative samples, and the inter-class cosine distance of negative samples and positive samples respectively; The first loss function is calculated based on the intra-class cosine distance of the positive sample, the intra-class cosine distance of the negative sample, the inter-class cosine distance between the positive sample and the negative sample, and the inter-class cosine distance between the negative sample and the positive sample, and in combination with the class similarity matrix.

7. The method according to claim 6, characterized in that The first loss function is calculated based on the intra-class cosine distance of the positive sample, the intra-class cosine distance of the negative sample, the inter-class cosine distance between the positive sample and the negative sample, and the inter-class cosine distance between the negative sample and the positive sample, further including calculating the first loss function according to formula (9) to formula (13): （9） （10） （11） （12） （13） In formulas (9) to (13), is the first loss function, Distance pp Represents the cosine distance within the positive sample class, Distance nn Represents the cosine distance within the negative sample class, Distance np Represents the cosine distance between positive and negative sample classes, Distance pn Represents the cosine distance between negative and positive sample classes,<a,b> represents the cosine distance between a and b, ||·|| represents the modulus of the eigenvector; represents the number of samples in the positive class, represents the number of samples in the negative sample class, Indicates the convolutional layer output feature matrix corresponding to the The feature vector of positive samples, represents the mean of all feature vectors in the positive sample class, Indicates the convolutional layer output feature matrix corresponding to the The feature vector of negative samples, represents the mean of all eigenvectors in the negative sample class, k1 and k2 are weighted coefficients of the cosine distance between classes, which take values ​​between (0, 1), and k1 and k2 are determined in association with the similarity between the positive sample class and the negative sample class in the class similarity matrix, as well as the similarity between the negative sample class and the positive sample class.

8. The method according to claim 1 or 2, characterized in that The deep learning network includes one of a fully convolutional neural network, ResNet, and VGG.

9. The method according to claim 1 or 2, characterized in that The medical image is one of a coronary CTA image and a lung CT image; In the case that the medical image is a coronary CTA image, the lesion category includes plaque and no plaque, or the lesion category includes vulnerable plaque, calcified plaque, mixed plaque, and no plaque.

10. A device for screening lesions using a computer, characterized in that: The device comprises: An interface configured to: obtain a training sample set of medical images with lesion category annotations and a category similarity matrix representing the similarity between each lesion category and other lesion categories; At least one processor is configured to execute the method for screening lesions using a computer as described in any one of claims 1 to 9.

11. A non-transitory computer-readable storage medium having computer-executable instructions stored thereon, wherein when the computer-executable instructions are executed by a processor, the method for screening lesions using a computer as described in any one of claims 1 to 9 is performed.

Images