Model training apparatus and recognition apparatus for target recognition in medical images

Abstract

The present disclosure describes a model training device and a recognition device for target recognition in medical images. The model training device comprises an acquisition module, a construction module and a training module; the acquisition module is configured to acquire a medical image as a training sample and a labeled region corresponding to a target in the training sample; the construction module is configured to determine a target region of the target within the labeled region, and construct a training set using the training sample, the labeled region and the target region, pixels within the target region in the training sample being determined to belong to the target; and the training module is configured to train a to-be-trained model based on the training set, and optimize the to-be-trained model using a training loss function to obtain a trained model, wherein in the training loss function, a spatial weight is used to reduce the negative impact of pixels in a first region in the training sample on the to-be-trained model, the first region being a region outside the target region of the target within the labeled region in the training sample. Thus, small targets can be effectively recognized.

Description

Technical Field

[0001] This disclosure relates to the field of image processing based on artificial intelligence, and specifically to a model training device and a recognition device for target recognition in medical images. Background Technology

[0002] In recent years, artificial intelligence technology has made tremendous achievements in the field of computer vision. For example, deep learning technology is increasingly widely used in semantic segmentation, image classification, and object recognition. Especially in the medical field, it is often used to segment, identify, or classify objects in medical images to assist in object analysis.

[0003] Currently, deep learning-based target recognition technology achieves high accuracy for large targets, but its performance on small targets (such as thin or small objects) is less than satisfactory, easily leading to false alarms and missed detections. Furthermore, distinguishing the categories of small targets is also difficult. For example, in fundus images, small targets such as punctate hemorrhages and microaneurysms are difficult to detect and differentiate using deep learning due to their small size, light color, and similar color to other objects. Therefore, effective methods for identifying small targets remain to be studied. Summary of the Invention

[0004] This disclosure is made in view of the above-mentioned state of the prior art, and its purpose is to provide a model training device and a recognition device for target recognition in medical images that can effectively identify small targets.

[0005] To this end, the first aspect of this disclosure provides a model training apparatus for target recognition in medical images, including an acquisition module, a construction module, and a training module. The acquisition module is configured to acquire the medical image as a training sample and the labeled regions corresponding to the target in the training sample. The construction module is configured to determine the target region of the target within the labeled regions and construct a training set using the training samples, the labeled regions, and the target regions, wherein pixels within the target regions in the training samples are determined to belong to the target. The training module is configured to train a model to be trained based on the training set and optimize the model to be trained using a training loss function to obtain a trained model. In the training loss function, spatial weights are used to reduce the negative impact of pixels in a first region in the training samples on the model to be trained. The first region is a region outside the target region of the target within the labeled regions in the training samples. In this case, by acquiring the target regions where all determined pixels within the labeled regions of the training samples belong to the target to identify pixels with undetermined categories within the labeled regions, and by combining spatial weights to train the model to be trained to reduce the negative impact of pixels with undetermined categories within the labeled regions on the model to be trained, the accuracy of the prediction results of the trained model for the input image can be improved. This allows for the effective identification of small targets.

[0006] Additionally, in the model training apparatus according to the first aspect of this disclosure, optionally, the model training apparatus further includes a preprocessing module configured to preprocess the training samples before constructing the training set. In the preprocessing, the training samples are thresholded to obtain region-of-interest (ROI) segmentation results for identifying the RPI of the training samples, and the RPI is used to crop the training samples. This ensures that the training samples are of uniform size and contain RPIs.

[0007] Furthermore, in the model training apparatus according to the first aspect of this disclosure, optionally, the target region is determined by a region segmentation result, wherein the region segmentation result is obtained by under-segmenting the image data within the labeled region. Thus, pixels within the labeled region whose category is not determined can be identified based on the region segmentation result obtained through under-segmentation.

[0008] Furthermore, in the model training apparatus according to the first aspect of this disclosure, optionally, the construction module includes a threshold segmentation module, which is configured to: obtain image data to be segmented based on image data corresponding to the labeled regions in the training samples, or obtain the image data to be segmented based on image data corresponding to the labeled regions in the training samples and image data corresponding to the labeled regions in the segmentation results of interest, wherein the segmentation results of interest are binary images used to identify the regions of interest in the training samples; and perform threshold segmentation on the image data to be segmented to obtain the region segmentation result, wherein the region segmentation result is a binary image. In this case, the target region in the image data to be segmented can be identified through threshold segmentation, and when the labeled regions include regions other than the regions of interest, noise outside the regions of interest can be eliminated.

[0009] Additionally, in the model training apparatus according to the first aspect of this disclosure, optionally, the threshold segmentation module obtains a target segmentation threshold according to the threshold acquisition method of the label category to which the target belongs, and performs threshold segmentation on the image data to be segmented using the target segmentation threshold. The threshold acquisition method for each label category is determined by the average area and average color of each label category. The threshold acquisition method includes a first method and a second method, where the average area of ​​the label category corresponding to the first method is greater than the average area of ​​the label category corresponding to the second method, and the average color of the label category corresponding to the first method is lighter than the average color of the label category corresponding to the second method. For the first method, a threshold is found such that the area of ​​pixels with gray values ​​greater than the threshold in the image data to be segmented is less than a preset multiple of the area of ​​the image data to be segmented, and the threshold is used as the target segmentation threshold, where the preset multiple is greater than 0 and less than 1. For the second method, if the length of the smallest side of the image data to be segmented is less than a preset length, the average gray value of the pixels in the image data to be segmented is taken as the target segmentation threshold; otherwise, the target segmentation threshold is determined based on the gray values ​​of the four corner regions and the center region of the image data to be segmented. In this scenario, the target segmentation threshold can be obtained based on the characteristics of the target's corresponding labeled category. This improves the accuracy of threshold segmentation.

[0010] Additionally, in the model training apparatus according to the first aspect of this disclosure, optionally, the threshold segmentation module further includes an erosion module, the erosion module being configured to perform an erosion operation on the threshold segmentation result of the image data to be segmented before obtaining the region segmentation result, wherein the erosion kernel... Satisfying the formula: ,in, Indicates the height and sum of the labeled area Indicates the width of the labeled area. This represents the height of the training sample. This represents the width of the training sample. This represents the preset hyperparameters. In this case, an appropriately sized erosion kernel can be obtained based on the size of the training samples, the size of the labeled region, and the preset hyperparameters. This helps to suppress excessive erosion.

[0011] Furthermore, in the model training apparatus according to the first aspect of this disclosure, the training loss function may optionally employ weighted balanced cross-entropy. In this case, the imbalance between positive and negative samples can be suppressed, thereby further improving the accuracy of the model under training in recognizing small targets.

[0012] Furthermore, in the model training apparatus according to the first aspect of this disclosure, optionally, in the spatial weights, pixels in the first region of the training samples are assigned a first weight, wherein the first weight is 0. In this case, samples with undetermined categories can be ignored to reduce the negative impact of undetermined categories on the model to be trained.

[0013] Furthermore, in the model training apparatus according to the first aspect of this disclosure, optionally, pixels in the first, second, third, and fourth regions of the training samples are assigned a first weight, a second weight, a third weight, and a fourth weight, respectively. The second region is the target region, the third region is a region within the region of interest that does not belong to the labeled region, and the fourth region is a region outside the region of interest. The first weight is less than the second weight and less than the third weight, and the fourth weight is less than the second weight and less than the third weight. In this case, the negative impact of pixels with undetermined categories and pixels outside the region of interest on the model being trained can be suppressed, while the positive impact of non-target regions within the target region and within the region of interest on the model being trained can be increased. Therefore, the accuracy of the model can be improved.

[0014] Furthermore, in the model training apparatus according to the first aspect of this disclosure, optionally, the model to be trained is a semantic segmentation model, and the prediction result of the model to be trained is the semantic segmentation result of the training samples. Thus, small targets can be identified.

[0015] Furthermore, in the model training apparatus according to the first aspect of this disclosure, the shape of the labeled region may optionally be rectangular. This reduces the difficulty of labeling.

[0016] A second aspect of this disclosure provides a target recognition device for medical images, comprising: an acquisition module configured to acquire the medical image as an input image; and a prediction module configured to determine prediction results of each trained model for the input image based on at least one trained model obtained by the model training device of the first aspect of this disclosure, and to acquire a target prediction result based on the prediction results of the at least one trained model.

[0017] Furthermore, in the recognition device according to the second aspect of this disclosure, optionally, the prediction results of each trained model include the probability that each pixel in the input image belongs to a corresponding labeled category. The prediction module integrates the prediction results of the at least one trained model according to the labeled category and the pixel to obtain the integrated probability that each pixel in the input image belongs to the corresponding labeled category. Based on the integrated probability, a connected region is determined, and based on the connected region, the target prediction result corresponding to each labeled category is obtained. If only one trained model exists, the probability is used as the integrated probability; otherwise, the average of the prediction results of multiple trained models is calculated to obtain the average probability that each pixel in the input image belongs to the corresponding labeled category, and this average is used as the integrated probability. In this case, obtaining the target prediction result based on the integrated probability can further improve the accuracy of the target prediction result.

[0018] Additionally, in the recognition device according to the second aspect of this disclosure, the medical image may optionally be a fundus image. In this case, the trained model is able to recognize small targets in the fundus image.

[0019] Additionally, in the identification device according to the second aspect of this disclosure, the target may optionally include microaneurysms, punctate hemorrhages, patchy hemorrhages, and linear hemorrhages. In this case, the trained model is able to identify small targets in fundus images.

[0020] According to this disclosure, a model training device and a recognition device for target recognition in medical images that can effectively identify small targets are provided. Attached Figure Description

[0021] This disclosure will now be explained in further detail by way of example only with reference to the accompanying drawings, in which:

[0022] Figure 1 This is a schematic diagram illustrating an example of identifying a target environment as described in this disclosure.

[0023] Figure 2 This is a block diagram illustrating an example of a model training apparatus as described in this disclosure.

[0024] Figure 3This is a schematic diagram showing the labeled areas of some examples involved in this disclosure.

[0025] Figure 4 This is a schematic diagram illustrating the region segmentation results of some examples involved in this disclosure.

[0026] Figure 5 This is a block diagram illustrating an example of a building module involved in this disclosure.

[0027] Figure 6 This is a flowchart illustrating an example of a threshold segmentation module obtaining region segmentation results as described in this disclosure.

[0028] Figure 7 This is an architecture diagram illustrating an example of a trainable model employing the U-Net architecture as described in this disclosure.

[0029] Figure 8 This is a schematic diagram illustrating several areas of some examples involved in this disclosure.

[0030] Figure 9 This is a flowchart illustrating an example of the model training method involved in this disclosure.

[0031] Figure 10 This is a block diagram illustrating an example of a recognition device for target recognition in medical images, as described in this disclosure.

[0032] Figure 11 This is a flowchart illustrating an example of a method for target recognition in medical images as described in this disclosure. Detailed Implementation

[0033] The preferred embodiments of this disclosure are described in detail below with reference to the accompanying drawings. In the following description, the same reference numerals are used for the same components, and repeated descriptions are omitted. Furthermore, the drawings are merely schematic diagrams, and the proportions of the components or the shapes of the components may differ from actual figures. It should be noted that the terms "comprising" and "having," and any variations thereof, in this disclosure, do not necessarily limit the process, method, system, product, or apparatus to the explicitly listed steps or units, but may include or have other steps or units not explicitly listed or inherent to these processes, methods, products, or apparatuses. All methods described in this disclosure may be performed in any suitable order unless otherwise indicated herein or clearly contradicted by the context.

[0034] The term "circuit" in this document can refer to hardware circuitry and / or a combination of hardware circuitry and software. The term "model" in this disclosure refers to a device capable of processing input and providing corresponding output. In this document, the terms "neural network," "deep neural network," "model," "network," and "neural network model" are used interchangeably. Furthermore, when referring to the rectangular properties (e.g., sides, width, height, and dimension) of a relevant object (e.g., labeled regions, image data to be segmented, and targets), unless otherwise specified, the rectangular properties of the object's bounding rectangle may be assumed.

[0035] Existing deep learning-based target recognition schemes employ various bounding box annotation methods (i.e., annotation methods that do not require accurate boundaries) to identify small targets. However, as briefly mentioned above, this scheme's performance in identifying small targets in images is unsatisfactory, with a significant risk of false alarms and missed detections. This is because small targets have a small area, few extractable features, and are easily affected by noise and other tissues. Therefore, a better approach is to segment small targets using deep learning-based target segmentation to achieve their recognition. However, this approach requires precise annotation of the small target boundaries, making image annotation difficult. To overcome the shortcomings of the above two schemes, this disclosure obtains a target region in the annotation area where all pixels are determined to belong to the target, and uses the annotated region of the determined target region as the gold standard for segmenting medical images, thereby achieving accurate identification of small targets. In particular, this disclosure uses a spatial weighting method to handle the negative impact of pixels of undetermined categories located within the annotation region but outside the target region on image segmentation. In this case, small targets can be effectively identified.

[0036] Therefore, this disclosure presents an example of a training model and a scheme for target recognition in medical images to address one or more of the aforementioned problems and / or other potential problems. The scheme employs an image segmentation method for target recognition (i.e., first obtaining target regions in the labeled areas of the training samples where all pixels are determined to belong to the target, and then post-processing the labeled areas of the determined target regions to obtain the target recognition result). Specifically, this scheme identifies pixels of undetermined categories within the labeled areas by identifying target regions in the labeled areas of the training samples where all pixels are determined to belong to the target, and trains a neural network model using spatial weights (i.e., the weights can be set in relation to the pixel position) to reduce the negative impact of undetermined categories of pixels in the labeled areas on the neural network model, thereby improving the accuracy of the trained model's predictions of input images (e.g., medical images). Furthermore, the trained model can be a trained neural network model (i.e., a trained neural network model), such as a trained semantic segmentation model. This allows for optimization of the performance of the trained model and improves the accuracy of the model in recognizing small targets. In some examples, the trained model can be the optimal neural network model obtained after training.

[0037] The examples disclosed herein relate to a training model and a scheme for target recognition in medical images, which effectively identifies small targets. The model training device for target recognition in medical images disclosed herein may be simply referred to as a model training device or a training device. It should be noted that the scheme disclosed herein is also applicable to the recognition of large targets.

[0038] The images involved in the examples of this disclosure can be from cameras, CT scans, PET-CT scans, SPECT scans, MRI, ultrasound, X-rays, angiography, fluorescein images, capsule endoscopy images, or combinations thereof. In some examples, the images can be medical images. For example, medical images can include, but are not limited to, fundus images, lung images, stomach images, chest images, and brain images. This allows for the identification of small targets in medical images. In some examples, the images can be natural images. Natural images can be images observed or captured in natural scenes. This allows for the identification of small targets in natural images. The following description uses a fundus image from a medical image as an example to illustrate the examples of this disclosure, and such description does not limit the scope of this disclosure. Those skilled in the art can use other types of images without limitation.

[0039] Examples of this disclosure will now be described in detail with reference to the accompanying drawings. Figure 1 This is a schematic diagram illustrating an example of identifying a target environment 100 as described in this disclosure. Figure 1As shown, the target environment 100 may include a computing device 110. The computing device 110 can be any device with computing capabilities. For example, the computing device 110 can be a cloud server, a personal computer, a mainframe, or a distributed computing system.

[0040] The computing device 110 can acquire input 120 and generate an output 140 corresponding to input 120 using a neural network model 130 (sometimes simply referred to as the model to be trained 130 or model 130). In some examples, input 120 can be the image described above, and output 140 can be a prediction result, training parameters (e.g., weights), or performance metrics (e.g., accuracy and error rate). In some examples, neural network model 130 can be, but is not limited to, a semantic segmentation model (e.g., U-Net), or other image processing-related models. Furthermore, neural network model 130 can be implemented using any suitable network architecture, such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), and deep neural networks (DNNs).

[0041] In some examples, the target environment 100 may also include a model training device and a model application device (not shown). The model training device can be used to implement a training method for training the neural network model 130 to obtain a trained model. The model application device can be used to implement a correlation method using the trained model to obtain prediction results for target recognition in an image. Additionally, during the model training phase, the neural network model 130 may be a model to be trained 130. During the model application phase, the neural network model 130 may be a trained model.

[0042] Figure 2 This is a block diagram illustrating an example of a model training apparatus 200 as described in this disclosure. For example, the model training apparatus 200 may be composed of... Figure 1 The computing device 110 shown is used for implementation. Additionally, the model training device 200 can be used to train a model for target recognition in medical images.

[0043] like Figure 2 As shown, the model training device 200 may include an acquisition device 210. The acquisition device 210 may be configured to acquire medical images used as training samples and labeled regions corresponding to targets in the training samples. That is, during the training phase, medical images can be acquired as training samples. This enables the identification of targets in the medical images. In some examples, the medical images may be color images. This improves the accuracy of identifying small targets.

[0044] Furthermore, medical images may contain specific targets, which can belong to at least one category of interest (i.e., the category to be identified). In some examples, for fundus images, targets may include small features such as microaneurysms, punctate hemorrhages, patchy hemorrhages, and linear hemorrhages. In such cases, the trained model can identify these small targets in the fundus image.

[0045] Figure 3 This is a schematic diagram showing the labeled areas of some examples involved in this disclosure.

[0046] In some examples, targets in the training samples can be labeled to obtain labeled regions. Furthermore, the shape of the labeled region can be rectangular, circular, or a shape that matches the shape of the target in the training samples (e.g., the shape of the labeled region can be the outline of the target). Preferably, the shape of the labeled region can be rectangular. This reduces the difficulty of labeling. As an example, Figure 3 The image shows a labeled region D1 in a fundus image. The shape of the labeled region D1 is rectangular, and the target within the labeled region D1 is a patchy hemorrhage.

[0047] Additionally, the labeled area can have a corresponding label (i.e., the target's label category), which can be used to distinguish the target's category. The label category can correspond one-to-one with the target category. For example, for fundus images, the target category and label category can be, but are not limited to, microaneurysms, punctate hemorrhages, patchy hemorrhages, and linear hemorrhages. In some examples, the corresponding label category can be represented by numbers. This facilitates calculations by the computing device 110. Furthermore, the labeled area and its corresponding label can be referred to as the labeling result.

[0048] like Figure 2 As shown, the model training device 200 may further include a construction module 230. The construction module 230 may be configured to determine the target region of an object within a labeled region in the training samples, and construct a training set using the training samples, the labeled regions, and the target regions.

[0049] Furthermore, pixels within the target region in the training samples are determined to belong to the target. In some examples, the target region can be determined by the region segmentation result. That is, the region segmentation result can be used to determine the target region of the target within the labeled region. In this case, the target region within the labeled region can be identified, and thus pixels with undetermined categories can be determined based on the target region. Specifically, pixels outside the target region within the labeled region in the training samples can be pixels with undetermined categories. However, the examples in this disclosure are not limited to this; in other examples, any other method that can determine the target region can be used.

[0050] Furthermore, the region segmentation result can be any form of data capable of identifying the target region (e.g., an image). In some examples, the region segmentation result can be a binary image. In some examples, for region segmentation results of binary images, the region corresponding to a pixel with a value of 1 can be designated as the target region (that is, if the pixel value is 1, it can indicate that the pixel at the corresponding position in the training sample belongs to the target; if the pixel value is 0, it can indicate that the pixel at the corresponding position in the training sample is a pixel with an undetermined category). In this case, the negative impact of pixels with undetermined categories on the untrained model 130 can be reduced.

[0051] Figure 4 This is a schematic diagram illustrating the region segmentation results of some examples involved in this disclosure.

[0052] As an example, Figure 4 It shows Figure 3 The region segmentation result A1 corresponds to the labeled region D1, where D2 is the target region. Additionally, to make the region segmentation result A1 clearer, Figure 4 The region segmentation result A1 is a proportionally enlarged result and does not imply any limitation on this disclosure. Figure 4 The actual size of the region segmentation result A1 can be the same as that of the labeled region D1.

[0053] In some examples, the building module 230 can be configured to perform under-segmentation on the image data within the labeled regions of the training samples to obtain region segmentation results (i.e., under-segmentation can be used to segment out the target region corresponding to the target within the labeled regions to obtain region segmentation results). Thus, pixels with undetermined categories within the labeled regions can be identified based on the region segmentation results obtained through under-segmentation. Generally, under-segmentation refers to a situation where a foreground target is missegmented as background, but the background is not missegmented as a foreground target. Here, under-segmentation can be defined as pixels belonging to the target in the labeled regions being missegmented as non-targets, but pixels not belonging to the target in the labeled regions not being missegmented as targets. In this case, pixels within the target region in the region segmentation results can be determined to belong to the target. Furthermore, pixels outside the target region but within the labeled regions are not necessarily targets (i.e., they can be pixels with undetermined categories). In this case, by performing under-segmentation on the image data within the labeled regions of the training samples to identify pixels with undetermined categories within the labeled regions, the training model 130 can be subsequently trained using spatial weights to reduce the negative impact of pixels with undetermined categories within the labeled regions on the training model 130, thereby improving the accuracy of the prediction results of the trained model 130 on the input image. This allows for the effective identification of small targets.

[0054] In some examples, the construction module 230 can be configured to determine the region segmentation result corresponding to the labeled region based on the training samples and the image data corresponding to the labeled regions in the aforementioned segmentation results of interest. Specifically, the construction module 230 can perform a product operation on the image data corresponding to the labeled regions in the training samples (hereinafter referred to as the first image data) and the image data corresponding to the labeled regions in the aforementioned segmentation results of interest (i.e., the segmentation results of interest can be a binary image used to identify the region of interest in the training samples) to obtain the image data to be segmented (i.e., the image data within the labeled region), and perform under-segmentation on the image data to be segmented to determine the region segmentation result corresponding to the labeled region. In this case, when the labeled region includes regions outside the region of interest, noise outside the region of interest can be eliminated.

[0055] Figure 5 This is a block diagram illustrating an example of the building module 230 involved in this disclosure.

[0056] In some examples, such as Figure 5 As shown, the construction module 230 may include a threshold segmentation module 231. The threshold segmentation module 231 can be configured to obtain region segmentation results based on threshold segmentation.

[0057] Figure 6 This is a flowchart illustrating an example of how the threshold segmentation module 231 of this disclosure obtains region segmentation results. That is, it shows some examples of the process for obtaining region segmentation results in this disclosure.

[0058] like Figure 6 As shown, obtaining the region segmentation result may include step S102. In step S102, the image data to be segmented can be obtained based on the labeled regions. As mentioned above, the first image data can be the image data corresponding to the labeled regions in the training samples, and the second image data can be the image data corresponding to the labeled regions in the segmentation results of interest. That is, the first image data and / or the second image data can be obtained based on the labeled regions, and then the image data to be segmented can be obtained based on the first image data, or the first image data and the second image data.

[0059] In some examples, the image data to be segmented can be obtained based on the first image data. In some examples, the image data to be segmented can be obtained based on the color channels (e.g., red, green, and blue channels) of the first image data. Taking a fundus image as an example, the image data to be segmented can be obtained based on the green channel of the first image data. Specifically, the first image data corresponding to the labeled region can be obtained (e.g., cropped) from the training samples, and then the green channel (i.e., the G channel) of the first image data can be taken to obtain the image data to be segmented. In some examples, the corresponding color channel (green channel) of the first image data can be used as the image data to be segmented. In addition, the color space and color channels can be selected according to the characteristics of the medical image itself, and this disclosure does not impose any particular limitations.

[0060] In other examples, the image data to be segmented can be obtained based on the first image data and the second image data. In this case, noise outside the region of interest can be eliminated when the labeled region includes a region outside the region of interest. In some examples, the image data to be segmented can be obtained based on the color channels of the first image data and the second image data. Specifically, the color channels of the first image data can be represented as follows: The second image data is represented as Then the image data to be segmented can be represented as ,in, It can represent the image data to be segmented. It can represent the product operation of elements (that is, the gray values ​​of pixels).

[0061] It should be noted that the first image data, the second image data, and the image data to be segmented can represent image data (e.g., pixel data, data stream, or image) of the corresponding region. In practice, the pixel values ​​or pixel location markers of the corresponding region can be stored in the appropriate medium (e.g., memory or disk) as needed to form image data in the appropriate form, thus facilitating processing. Furthermore, the shapes of the first image data, the second image data, and the image data to be segmented can match the shape of the labeled region, or they can be the bounding rectangle of the labeled region, depending on the method used to obtain the region segmentation results.

[0062] Furthermore, during the process of obtaining region segmentation results, if it is necessary to utilize the rectangular characteristics of the image data to be segmented (e.g., sides, length, width, height, and four corners) and the shape of the labeled region is not rectangular, the image data to be segmented can be obtained based on the region corresponding to the bounding rectangle of the labeled region. That is, after converting the shape of the labeled region into a rectangle, the image data to be segmented can be obtained based on the converted labeled region.

[0063] like Figure 6As shown, obtaining the region segmentation result may further include step S104. In step S104, threshold segmentation may be performed on the image data to be segmented to obtain the region segmentation result. However, the examples of this disclosure are not limited to this; in other examples, other methods may be used to perform under-segmentation on the image data to be segmented to obtain the region segmentation result.

[0064] In some examples, in step S104, a target segmentation threshold (described later) can be used to perform threshold segmentation on the image data to be segmented, thereby obtaining a region segmentation result. Thus, target regions in the image data to be segmented can be identified through threshold segmentation. In some examples, during threshold segmentation, pixels in the image data to be segmented with grayscale values ​​not less than the target segmentation threshold can be set to 1, while other pixels can be set to 0, thereby obtaining a region segmentation result.

[0065] Furthermore, in threshold segmentation of image data to be segmented, there can be multiple methods for obtaining the target segmentation threshold. For example, the target segmentation threshold can be obtained according to the common Otsu method. In some examples, at least one of the methods described in the examples of this disclosure can be selected to obtain the target segmentation threshold.

[0066] In some examples, the target segmentation threshold can be obtained based on the annotation category to which the target belongs. In other examples, the target segmentation threshold can be obtained based on a thresholding method that determines the target's annotation category. In this case, the target segmentation threshold can be obtained based on the characteristics of the annotation category itself. This improves the accuracy of threshold segmentation. Furthermore, the thresholding method for obtaining the annotation category can include both the first and second methods. Additionally, the annotation category to which the target belongs in the training samples can be known. For example, the annotation category to which the target belongs in the training samples can be the annotation labels in the annotation results.

[0067] In some examples, the threshold method for obtaining each annotation category can be obtained through the features of each annotation category. In some examples, the threshold method can be determined based on the average area and average color of each annotation category. However, the examples disclosed herein are not limited to these; in other examples, the threshold method for obtaining annotation categories can also be determined empirically. For example, for fundus images, patchy hemorrhages in fundus images can use the first method, while microaneurysms, punctate hemorrhages, and linear hemorrhages in fundus images can use the second method.

[0068] In some examples, the average area and average color of each labeled category can be fixed values, obtained statistically from the sample data. For example, the average area and average color can be obtained by averaging the area and color of targets of the same category (e.g., for training samples, the same category can refer to the same labeled category) in the sample data (e.g., training samples). In other examples, the fixed values ​​can also be empirical values.

[0069] In some examples, in methods for determining the threshold for label categories based on their average area and average color, the average area of ​​the label category corresponding to the first method can be greater than that of the label category corresponding to the second method, and the average color of the label category corresponding to the first method can be lighter than that of the label category corresponding to the second method. For example, the first method can be used for targets with large areas and light colors (e.g., patchy hemorrhages in fundus images). The second method can be used for targets with small areas and dark colors (e.g., microaneurysms, punctate hemorrhages, and linear hemorrhages in fundus images).

[0070] In some examples, in the method of determining the acquisition threshold of a label category based on the average area and average color of each label category, the acquisition threshold method used for the label category can be determined by a first preset area and preset color value. Therefore, the acquisition threshold method used for the label category can be automatically obtained.

[0071] In some examples, if the average area of ​​the labeled category is greater than the first preset area and the average color is less than the preset color value (that is, the area of ​​the target of the labeled category is relatively large and the color is relatively light), the labeled category can be determined to use the first method. Otherwise, if the average area of ​​the labeled category is not greater than the first preset area and the average color is not less than the preset color value (that is, the area of ​​the target of the labeled category is relatively small and the color is relatively dark), the labeled category can be determined to use the second method.

[0072] In some examples, the first preset area and preset color value can be adjusted based on the region segmentation results. In other examples, the first preset area and preset color value can be fixed values, which can be obtained by statistical analysis of sample data. That is, statistical methods can be used to analyze a small amount of sample data with different first preset area and preset color values ​​to determine the optimal first preset area and preset color values ​​for classification.

[0073] As mentioned above, the target segmentation threshold can be obtained based on the threshold acquisition method of the target's annotation category. In some examples, the target segmentation threshold can be obtained based on the threshold acquisition method of the target's annotation category and the image data to be segmented corresponding to the training samples.

[0074] In some examples, for the first method (i.e., the method for obtaining the threshold of the labeled category of the target is the first method), a threshold can be found such that the area of ​​pixels with gray values ​​greater than the threshold in the image data to be segmented is less than a preset multiple of the area of ​​the image data to be segmented. This threshold is used as the target segmentation threshold, where the preset multiple can be greater than 0 and less than 1. Taking medical images as 8-bit quantized images as an example, thresholds from 0 to 255 can be iterated to find a threshold such that the area of ​​pixels with gray values ​​greater than the threshold in the image data to be segmented is less than a preset multiple of the area of ​​the image data to be segmented. This threshold is then used as the target segmentation threshold. Furthermore, the preset multiple can be any value that prevents the target region from being segmented. For example, the preset multiple can be a relatively small value to prevent the target region from being segmented. In some examples, the preset multiple can be determined empirically based on the shape of the target.

[0075] In some examples, for the second method (that is, the second method is used to obtain the threshold of the labeled category to which the target belongs), the mean gray value of the pixels in the image data to be segmented can be used as the target segmentation threshold, or the target segmentation threshold can be determined based on the gray values ​​of the four corner regions and the center region of the image data to be segmented.

[0076] In some examples, for the second method, if the length of the smallest side of the image data to be segmented is less than a preset length, the average grayscale value of the pixels in the image data to be segmented can be used as the target segmentation threshold. In some examples, the preset length can be any value that prevents the target region from being segmented. In some examples, the preset length can be a first preset proportion of the smallest side of the training samples. Specifically, the preset length can be expressed as... ,in, This can represent the first preset ratio. It can represent the height of the training sample. It can represent the width of the training samples.

[0077] In some examples, the first preset scale can be a fixed value. In some examples, the first preset scale can be determined based on the average size of targets of the same category in medical images. In some examples, the first preset scale can be determined based on the average width and average height of targets of the same category in medical images. In some examples, the first preset scale can satisfy the formula:

[0078] ,

[0079] in, and These can represent the average width and average height of targets of the same category in medical images, respectively. and The standard deviations of width and height can be represented separately. and These can represent the average width and average height of the medical image, respectively. Here, the medical image can be an image from a data source used to obtain the first preset scale. In some examples, the data source can be training data. Additionally, the relevant parameters related to the first preset scale can be obtained in a similar manner to obtaining the relevant parameters related to the preset hyperparameters, which will not be elaborated here.

[0080] In some examples, for the second method, if the length of the smallest side of the image data to be segmented is not less than a preset length, the target segmentation threshold can be determined based on the gray values ​​of the four corner regions and the center region of the image data to be segmented. Specifically, the image data to be segmented can be divided into a preset number of parts (e.g., 9 parts), and the target segmentation threshold can be determined based on the gray values ​​of the four corner regions and the center region of the image data to be segmented. In some examples, determining the target segmentation threshold based on the gray values ​​of the four corner regions and the center region of the image data to be segmented can be done by taking the average of the gray values ​​of the pixels in each of the four corner regions and the average gray value of the pixels in the center region as the target segmentation threshold.

[0081] Return to reference Figure 5 In some examples, the threshold segmentation module 231 may include an erosion module 231a. The erosion module 231a may be configured to perform an erosion operation on the threshold segmentation result of the image data to be segmented.

[0082] In some examples, the erosion module 231a can be configured to perform an erosion operation on the thresholded segmentation result (i.e., the initial segmentation result) of the image data to be segmented before obtaining the region segmentation result. In this case, the probability of isolated pixels in the thresholded segmentation result due to noise can be reduced.

[0083] In some examples, during the erosion operation performed by the erosion module 231a on the threshold segmentation result of the image data to be segmented, the erosion kernel... The formula can be satisfied:

[0084] ,

[0085] in, It can represent the height and sum of the labeled regions (i.e., the labeled regions corresponding to the image data to be segmented). It can represent the width of the labeled area. It can represent the height of the training sample. It can represent the width of the training samples. Preset hyperparameters can be used. In this case, an appropriately sized erosion kernel can be obtained based on the size of the training samples, the size of the labeled region, and the preset hyperparameters. This helps suppress over-erosion. In some examples, the preset hyperparameters can be used to adjust the size of the erosion kernel. In this case, a smaller erosion kernel can be used for particularly small targets. This helps prevent over-erosion operations from causing the target region of particularly small targets to disappear.

[0086] In some examples, the preset hyperparameters can be fixed values. In some examples, the preset hyperparameters can be determined based on the average size of similar targets in medical images. In some examples, the preset hyperparameters can be determined based on the average width and average height of similar targets in medical images. In some examples, the preset hyperparameters... The formula can be satisfied:

[0087] ,

[0088] in, and These can represent the average width and average height of targets of the same category in medical images, respectively. and The standard deviations of width and height can be represented separately. and The average width and average height of the medical image can be represented separately. Here, the medical image can be an image from a data source used to obtain the preset hyperparameters. In some examples, the width and height of targets of the same category in multiple training samples, as well as the width and height of the training samples, can be statistically analyzed to obtain the relevant parameters of the preset hyperparameters. That is, the data source can be training data. In some examples, when obtaining the preset hyperparameters in a medical image with labeled regions (e.g., training samples), the width and height of the target can also be the width and height of the corresponding labeled regions. Thus, the width and height of the target can be easily obtained.

[0089] Generally, the threshold segmentation result of the image data to be segmented may contain multiple connected regions. In some examples, the erosion module 231a can perform an erosion operation on the threshold segmentation result of the image data to be segmented to obtain at least one connected region, and select the connected region whose center is closest to the center of the image data to be segmented as the region segmentation result. In addition, the connected region closest to the center of the image data to be segmented can represent the identified target region. Thus, an accurate target region can be obtained. In some examples, contours can be searched on the erosion result (i.e., at least one connected region), and the contours with the largest area (e.g., 3) can be selected as candidates, and the connected region corresponding to the contour whose center is closest to the center of the image data to be segmented is retained as the region segmentation result.

[0090] Return to reference Figure 2 As described above, the construction module 230 can be configured to construct a training set using training samples, labeled regions, and target regions. That is, the construction module 230 can construct a training set based on training samples and labeled regions of at least one determined target region corresponding to the training samples. It should be noted that if the target region is determined by region segmentation results, the construction module 230 can also be configured to construct a training set using training samples and region segmentation results. That is, the construction module 230 can construct a training set based on training samples and at least one region segmentation result corresponding to the training samples.

[0091] In some examples, the training set may include training samples and a gold standard for the training samples. In some examples, the construction module 230 may obtain the gold standard for the training samples based on the labeled regions of the determined target regions. In some examples, the construction module 230 may obtain the gold standard for the training samples based on the region segmentation results. That is, the construction module 230 may identify the target regions based on the region segmentation results, and then determine the true category of the pixels in the training samples based on the labeled regions of the determined target regions. Thus, the gold standard for the training samples can be obtained.

[0092] In some examples, the true category may include at least one of the target's labeled category (e.g., for fundus images, this could include microaneurysms, punctate hemorrhages, patchy hemorrhages, and linear hemorrhages), no target category, and undetermined category. This is specifically related to the process of optimizing the model 130 to be trained.

[0093] Additionally, the labeled category of the target in the true category can be the category to which pixels within the target region of the target in the labeled region of the training sample (i.e., the second region described later) belong. Furthermore, the undetermined category in the true category can be the category to which pixels within the region other than the target region of the target in the labeled region of the training sample (i.e., the first region described later) belong. Additionally, the targetless category in the true category can be the category to which pixels outside the labeled region of the training sample belong. In some examples, the region outside the labeled region of the training sample can include regions within the region of interest but not belonging to the labeled region (i.e., the third region described later). For example, for medical images, regions within the region of interest but not belonging to the labeled region can be regions corresponding to targetless tissue in the medical image. In some examples, the region outside the labeled region of the training sample can include regions within the region of interest but not belonging to the labeled region, and regions outside the region of interest (i.e., the fourth region described later).

[0094] In some examples, the building module 230 can also use training samples, labeled regions, and target regions to build validation and test sets.

[0095] In some examples, the model training device 200 may also include a preprocessing module (not shown). The preprocessing module may be configured to preprocess the training samples appropriately before the construction module 230 constructs the training set, based on actual conditions (e.g., the quality of the training samples does not meet the training requirements or the size of the training samples is inconsistent).

[0096] In some examples, preprocessing the training samples may include standardizing their size. For example, the training samples may be standardized to 1024×1024 or 2048×2048. This disclosure does not limit the size of the training samples. In some examples, preprocessing the training samples may include cropping them. In some examples, cropping the training samples may involve identifying the region of interest (ROI) and using it to crop the samples. This ensures that the training samples are of uniform size and contain the ROI. In some examples, the ROI may be a region where a target can exist (also known as a foreground region). For example, for fundus images, the ROI may be the fundus region.

[0097] In some examples, the preprocessing module can segment the training samples to obtain regions of interest (ROIs). In other examples, the preprocessing module can perform thresholding segmentation on the training samples to obtain ROI segments, which can then be used to identify the ROIs within the training samples. This allows for the identification of ROIs. Furthermore, the ROI segments obtained through thresholding can be binary images (also known as binarized images). It is understandable that while the above example uses thresholding to obtain ROI segments, other methods suitable for obtaining ROI segments are also applicable. For example, ROI segments can be obtained using neural networks.

[0098] In some examples, the preprocessing module performs threshold segmentation on the training samples by dividing the training samples into a preset number of parts (e.g., 9 parts), determining the segmentation threshold based on the gray values ​​of the four corner regions and the center region, and then performing threshold segmentation on the training samples based on this threshold to obtain the segmentation result of interest. In other examples, determining the segmentation threshold based on the gray values ​​of the four corner regions and the center region can be achieved by taking the average gray value of the pixels in each of the four corner regions and the average gray value of the pixels in the center region as the segmentation threshold, thus obtaining the segmentation result of interest.

[0099] Furthermore, before obtaining the segmentation result of interest in threshold segmentation, the preprocessing module can perform an erosion operation on the threshold segmentation result corresponding to the training samples (i.e., the initial segmentation result) to obtain the segmentation result of interest. For example, the threshold segmentation result of the training samples can be eroded twice to obtain the segmentation result of interest, where the size of the erosion kernel can be 5. This can eliminate noise at the edges of the region of interest (e.g., the fundus region).

[0100] Return to reference Figure 2 The model training device 200 may further include a training module 250. The training module 250 may be configured to train the model to be trained 130 based on the training set and optimize the model to be trained 130 using a training loss function to obtain a trained model.

[0101] In some examples, the model 130 to be trained may include, but is not limited to, a semantic segmentation model. Furthermore, the prediction result of the model 130 to be trained may include, but is not limited to, the semantic segmentation result of the training samples. Thus, small targets can be identified. For example, in the example described above where the input 120 is image data to be semantically segmented and the model 130 to be trained is a semantic segmentation model, the prediction result could be the semantic segmentation result of the image data. Additionally, the input 120 described above could be color image data.

[0102] In some examples, high-dimensional feature information can be added to the model 130 to be trained. This can improve the accuracy of identifying small targets. In some examples, feature information of different dimensions can be extracted from medical images (e.g., training samples) in the model 130 to be trained, and feature information of a preset dimension closest to the highest dimension can be fused with the highest dimension feature information to increase high-dimensional feature information.

[0103] Figure 7 This is an architecture diagram illustrating an example of a trainable model 130 employing the U-Net architecture as described in this disclosure.

[0104] As an example, Figure 7 The image shows a trainable model 130 using the U-Net architecture. Common network layers in the U-Net architecture are not explained in detail here. Figure 7 As shown, the preset dimension can be 2. The feature information of the two dimensions can include feature information 131a and feature information 131b. Feature information 131a can be fused with the feature information of the highest dimension through upsampling layer 132a, and feature information 131b can be fused with the feature information of the highest dimension through upsampling layer 132b. In addition, the convolution size of upsampling layer 132a and upsampling layer 132b can be any value that makes the feature information (e.g., feature information 131a and feature information 131b) consistent with the size of the feature information of the highest dimension after upsampling.

[0105] In some examples, during the training of the model to be trained 130, the training module 250 can obtain the prediction results corresponding to the training samples based on the training samples of the training set, and then construct a training loss function based on the gold standard of the training samples and the prediction results. Furthermore, the training loss function can represent the degree of difference between the gold standard of the training samples and the corresponding prediction results.

[0106] As described above, the gold standard for training samples can be obtained based on the labeled regions of the determined target region or based on the region segmentation results. In some examples, the region segmentation results (which can be equivalent to the labeled regions of the determined target region) can be directly used as the gold standard for training samples. In some examples, the region segmentation results can be used as the gold standard for pixels within the labeled regions corresponding to the target in the training samples to obtain the gold standard for the training samples. Furthermore, the gold standard for pixels outside the labeled regions corresponding to the target in the training samples can be set as needed. For example, it can be fixed to a single category (e.g., the no-target category involved in this disclosure). Alternatively, it can be set by manually labeling the training samples or by automatically labeling the training samples using artificial intelligence algorithms. This disclosure does not impose any particular restrictions on how the gold standard for pixels outside the labeled regions corresponding to the target in the training samples is set.

[0107] In some examples, weights can be assigned to pixels of undetermined class in the training samples within the training loss function to reduce the negative impact of undetermined class pixels on the model 130 to be trained. This improves the accuracy of the model 130. In some examples, spatial weights can be used in the training loss function to reduce the negative impact of undetermined class pixels in the training samples on the model 130 to be trained.

[0108] In some examples, in spatial weighting, the training samples can be divided into several regions (or at least one region), and the weights can be used to adjust the influence of each region in the several regions on the model to be trained 130.

[0109] In some examples, several regions may include a first region. The first region can be a region of pixels in the training samples whose category is undetermined (i.e., a region outside the target region within the labeled region of the training samples). In some examples, spatial weights can be used in the training loss function to reduce the negative impact of pixels in the first region of the training samples on the model 130 to be trained. In some examples, in spatial weighting, pixels in the first region of the training samples can be assigned a first weight to reduce the negative impact on the model 130 to be trained.

[0110] Additionally, the first weight can be any value that reduces the negative impact on the untrained model 130. In some examples, the first weight can be a fixed value. In some examples, the first weight can be 0. In this case, samples with undetermined classes can be ignored to reduce the negative impact of undetermined classes on the untrained model 130.

[0111] In some examples, several regions may include a second region. The second region can be a target region of the training samples. In some examples, pixels in the second region may be assigned a second weight in the spatial weighting. In some examples, the first weight may be less than the second weight. Additionally, the second weight can be any value that increases the positive impact of pixels in the second region on the model to be trained. In some examples, the second weight can be a fixed value. In some examples, the second weight can be 1.

[0112] In some examples, several regions may include a third region. The third region can be a region within the region of interest in the training samples that is not part of the labeled region. In some examples, pixels in the third region can be assigned a third weight in the spatial weighting. In some examples, the first weight can be less than the third weight. Furthermore, the principles for setting the third weight can be similar to those for the second weight.

[0113] In some examples, several regions may include a fourth region. The fourth region can be a region outside the region of interest in the training samples. In some examples, pixels in the fourth region can be assigned a fourth weight in the spatial weighting. In some examples, the fourth weight can be less than the second weight. Furthermore, the principles for setting the fourth weight can be similar to those for the first weight.

[0114] In some examples, several regions may simultaneously include a first region, a second region, a third region, and a fourth region. Pixels in the first, second, third, and fourth regions can be assigned a first weight, a second weight, a third weight, and a fourth weight, respectively. The first weight can be less than the second weight and less than the third weight, and the fourth weight can be less than the second weight and less than the third weight. In this case, the negative impact of pixels with undetermined categories and pixels outside the region of interest on the training model 130 can be suppressed, while the positive impact of non-target regions within the target region and within the region of interest on the training model 130 can be increased. This improves the accuracy of the model. Preferably, the first weight can be 0, the second weight can be 1, the third weight can be 1, and the fourth weight can be 0. In this case, the negative impact of pixels with undetermined categories and pixels outside the region of interest on the training model 130 can be avoided, while the positive impact of non-target regions within the target region and within the region of interest on the training model 130 can be increased. This improves the accuracy of the model 130 to be trained.

[0115] However, the examples disclosed herein are not limited thereto. In other examples, the regions may include any combination of a first region, a second region, a third region, and a fourth region.

[0116] Figure 8This is a schematic diagram illustrating several regions of some examples involved in this disclosure. Additionally, to clearly describe these regions, Figure 8 This is a schematic diagram showing the various regions of binarization, and does not limit this disclosure to necessarily dividing them into... Figure 8 All regions are shown. Among them, D3 can represent the first region, D4 can represent the second region, D5 can represent the third region, and D6 can represent the fourth region.

[0117] As mentioned above, in some examples, in spatial weighting, the training samples can be divided into several regions, and the influence of each region in the several regions on the untrained model 130 can be adjusted using weights.

[0118] In some examples, the loss can be calculated by category in the training loss function. As mentioned above, the true category can include at least one of the labeled category of the target, the no-target category, and the undetermined category. In some examples, the category in the training loss function can be derived from the true categories mentioned above. That is, the category in the training loss function can include the labeled category of the target and the no-target category, or the category in the training loss function can include the labeled category of the target, the no-target category, and the undetermined category. The specific category in the training loss function depends on the samples selected in the training loss function.

[0119] In some examples, in the training loss function, if samples (i.e., pixels) of each class in the training samples belong to corresponding regions among several regions, the loss of the corresponding sample can be multiplied by the weight of the corresponding region. In this case, the training loss function can be determined based on spatial weights, thereby adjusting the influence of pixels in different regions on the model to be trained 130.

[0120] In some examples, the impact of samples from each class on the trained model 130 can be adjusted based on the weights of each class in the training loss function. This allows for the adjustment of the influence of samples from different classes on the trained model 130.

[0121] In some examples, the impact of samples on the model to be trained 130 can be adjusted simultaneously based on spatial weights and class weights in the training loss function. This allows for adjusting the impact of samples on the model to be trained 130 by region and class.

[0122] In some examples, the training loss function can employ weighted balanced cross-entropy. In this case, it is possible to suppress the imbalance between positive and negative samples, thereby further improving the accuracy of the training model 130 in recognizing small targets. In some examples, when training the training model 130, a training loss function based on weighted balanced cross-entropy can be used, and spatial weights can be utilized to control the negative impact of pixels with undetermined categories on the training model 130.

[0123] The following example illustrates a training loss function based on weighted equilibrium cross-entropy, where the first weight of the first region is 0, the second weight of the second region is 1, the third weight of the third region is 1, and the fourth weight of the fourth region is 0. It should be noted that this does not constitute a limitation of the present disclosure; those skilled in the art can design a training loss function based on weighted equilibrium cross-entropy by freely combining the weights of each region and each category, depending on the circumstances. Training Loss Function Based on Weighted Equilibrium Cross-Entropy This satisfies the formula (i.e., it's equivalent to ignoring the losses in the first and fourth regions by setting the first and fourth weights to 0):

[0124] (1)

[0125] in, It can represent the number of categories. It can represent the first The weights of each category, It can represent the first The number of samples in each category This can represent the gold standard of the above training samples. The first category The true value of each sample It can represent the first in the prediction result The first category The predicted value of the nth sample (i.e., the nth sample) The sample belongs to the first (The probability of each category). Additionally, samples of each category can be pixels of the corresponding category from the training samples. Furthermore, samples of a single category can be determined based on the gold standard of the aforementioned training samples. As mentioned above, the category weights can adjust the influence of samples of each category on the model to be trained 130.

[0126] Furthermore, in formula (1), by setting the first weight and the fourth weight to 0, the samples in the first and fourth regions are ignored. The categories in the training loss function can include the labeled category of the target and the no-target category. The labeled category of the target can be the category to which the pixels in the second region of the training samples belong, and the no-target category can be the category to which the pixels in the third region of the training samples belong. Taking fundus images as an example, the categories in the training loss function of formula (1) can include microaneurysms, punctate hemorrhages, patchy hemorrhages, linear hemorrhages, and the no-target category.

[0127] The following combination Figure 9 The training method disclosed herein is described. Furthermore, the training method can train a model for object recognition in medical images. The training method can be implemented by a model training device 200. Figure 9This is a flowchart illustrating an example of the training method involved in this disclosure.

[0128] In some examples, such as Figure 9 As shown, the training method may include acquiring medical images as training samples and labeled regions corresponding to targets in the training samples (step S202), determining the target regions of targets within the labeled regions in the training samples, constructing a training set using the training samples, labeled regions, and target regions (step S204), training the model to be trained 130 based on the training set, and optimizing the model to be trained 130 using a training loss function to obtain a trained model (step S206). For details, please refer to the relevant description of the model training device 200.

[0129] The following combination Figure 10 This disclosure describes a target recognition device 300 (also referred to simply as recognition device 300 or model application device) for identifying targets in medical images. Furthermore, the recognition device 300 can be used for target recognition in medical images. Figure 10 This is a block diagram illustrating an example of a recognition device 300 for target recognition in medical images as described in this disclosure.

[0130] like Figure 10 As shown, the recognition device 300 may include an acquisition module 310. The acquisition module 310 may be configured to acquire a medical image as an input image. In some examples, the input image may undergo the same preprocessing as the training samples described above before being input into the trained model.

[0131] like Figure 10 As shown, the recognition device 300 may further include a prediction module 330. The prediction module 330 may be configured to utilize at least one trained model to determine the prediction results of each trained model for the input image, and to obtain a target prediction result based on the prediction results of the at least one trained model, wherein the at least one trained model may be obtained by the model training device 200 described above. Furthermore, the at least one trained model may be a model based on the same type of network architecture (e.g., U-Net), but with different network structures and / or parameters. For example, some branches or network layers may be added or removed to form at least one trained model. However, the examples in this disclosure are not limited to this; in other examples, the at least one trained model may not be based on the same type of network architecture. Additionally, the prediction results of each trained model may include the probability that each pixel in the input image belongs to a corresponding labeled category. The labeled category may be the labeled category of the aforementioned target.

[0132] In some examples, the prediction module 330 can integrate the prediction results of at least one trained model by label category and pixel to obtain the integration probability that each pixel of the input image belongs to the corresponding label category, determine the connected regions based on the integration probabilities, and obtain the target prediction results corresponding to each label category based on the connected regions. In this case, obtaining the target prediction results based on the integration probabilities can further improve the accuracy of the target prediction results.

[0133] In some examples, when the prediction module 330 obtains the ensemble probability, if there is only one trained model, the probability that each pixel in the input image belongs to the corresponding labeled category in the prediction result of the trained model can be used as the ensemble probability. Otherwise, the prediction results of multiple trained models can be averaged to obtain the average probability that each pixel in the input image belongs to the corresponding labeled category (that is, the probability average can be calculated at the pixel level according to the labeled category).

[0134] In some examples, the prediction module 330 can determine connected regions based on the ensemble probability and a classification threshold for each labeled category. Specifically, pixels with an ensemble probability not less than the classification threshold can be set to 1, while other pixels can be set to 0. In some examples, the classification threshold can be determined based on a validation set and using performance metrics. Furthermore, if connected regions exist, the number of connected regions can be one or more.

[0135] In some examples, the prediction module 330 can obtain the bounding rectangle of each connected region in obtaining the target prediction result based on the connected region. If the area of ​​the bounding rectangle is greater than the second preset area, it can be said that there is a target at the bounding rectangle; otherwise, it can be said that there is no target at the bounding rectangle.

[0136] In some examples, the second preset area can be a second preset proportion of the area of ​​the training samples. Specifically, the second preset area can be expressed as... ,in, This can represent a second preset ratio. It can represent the height of the input image. It can represent the width of the input image.

[0137] In some examples, the second preset scale can be a fixed value. In some examples, the second preset scale can be determined based on the median area of ​​similar targets in a medical image. In some examples, the second preset scale... The formula can be satisfied:

[0138] ,

[0139] in, These can represent the median area of ​​targets of the same category in medical images. It can represent the standard deviation of the area of ​​targets of the same category in medical images. and These can represent the average width and average height of the medical image, respectively. Here, the medical image can be an image from a data source used to obtain the second preset scale. In some examples, the data source can be training data. Additionally, the relevant parameters related to the second preset scale can be obtained in a similar manner to obtaining the relevant parameters related to the preset hyperparameters, which will not be elaborated here.

[0140] The following combination Figure 11 This disclosure describes a method for target recognition in medical images (hereinafter referred to as the recognition method). Furthermore, the recognition method can identify targets in medical images. The recognition method can be implemented by a recognition device 300. Figure 11 This is a flowchart illustrating an example of a method for target recognition in medical images as described in this disclosure.

[0141] like Figure 11 As shown, the recognition method may include step S302. In step S302, a medical image can be acquired as the input image. For details, please refer to the relevant description of the acquisition module 310.

[0142] like Figure 11 As shown, the recognition method may further include step S304. In step S304, at least one trained model can be used to determine the prediction results of each trained model for the input image, and the target prediction result is obtained based on the prediction results of at least one trained model. The at least one trained model can be obtained by the model training device 200 described above. For details, please refer to the relevant description of the prediction module 330.

[0143] The model training apparatus 200 and recognition apparatus 300 for target recognition in medical images disclosed herein identify unclassified pixels within the labeled regions of the training samples by acquiring target regions where all pixels within the labeled regions belong to the target. Spatial weights are then used to train the model 130 to reduce the negative impact of unclassified pixels within the labeled regions on the model 130, thereby improving the accuracy of the prediction results of the trained model 130 on the input image. This enables effective recognition of small targets.

[0144] While the present disclosure has been specifically described above in conjunction with the accompanying drawings and examples, it is to be understood that the foregoing description does not limit the present disclosure in any way. Those skilled in the art can make modifications and variations to the present disclosure as needed without departing from its essential spirit and scope, and all such modifications and variations shall fall within the scope of the present disclosure.

Claims

1. A model training device for target recognition in medical images, characterized in that, It includes an acquisition module, a construction module, and a training module; the acquisition module is configured to acquire the medical image used as a training sample and the labeled region corresponding to the target in the training sample; The construction module is configured to determine the target region of the target within the labeled region, and construct a training set using the training samples, the labeled region, and the target region, wherein pixels within the target region in the training samples are determined to belong to the target; and the training module is configured to train a model to be trained based on the training set, and optimize the model to be trained using a training loss function to obtain a trained model, wherein spatial weights are used in the training loss function to reduce the negative impact of pixels in a first region in the training samples on the model to be trained, the first region being the region outside the target region of the target within the labeled region in the training samples; the training samples also include a second region, a third region, and... The fourth region, wherein the second region is the target region, the third region is a region within the region of interest that does not belong to the labeled region, and the fourth region is a region outside the region of interest; pixels in the first, second, third, and fourth regions are respectively assigned a first weight, a second weight, a third weight, and a fourth weight, wherein the first weight is less than the second weight and less than the third weight, and the fourth weight is less than the second weight and less than the third weight; in the training loss function, the influence of samples of each category on the model to be trained is adjusted based on the weights of each category, wherein the category includes at least one of the labeled category of the target, the category without a target, and the undetermined category; wherein the first weight is 0.

2. The model training device according to claim 1, characterized in that: The model training device further includes a preprocessing module, which is configured to preprocess the training samples before constructing the training set. In the preprocessing, the training samples are thresholded to obtain the region of interest segmentation result for identifying the region of interest of the training samples, and the training samples are cropped using the region of interest.

3. The model training device according to claim 1, characterized in that: The target region is determined by the region segmentation result, wherein the region segmentation result is obtained by under-segmenting the image data within the labeled region.

4. The model training device according to claim 3, characterized in that, The construction module includes a threshold segmentation module, which is configured to: obtain image data to be segmented based on image data corresponding to the labeled regions in the training samples, or obtain the image data to be segmented based on image data corresponding to the labeled regions in the training samples and image data corresponding to the labeled regions in the interest segmentation results, wherein the interest segmentation results are binary images used to identify the regions of interest in the training samples; and perform threshold segmentation on the image data to be segmented to obtain the region segmentation results, wherein the region segmentation results are binary images.

5. The model training device according to claim 4, characterized in that: The threshold segmentation module obtains a target segmentation threshold according to the threshold acquisition method of the label category to which the target belongs, and performs threshold segmentation on the image data to be segmented using the target segmentation threshold. The threshold acquisition method for each label category is determined by the average area and average color of each label category. The threshold acquisition method includes a first method and a second method. The average area of ​​the label category corresponding to the first method is greater than the average area of ​​the label category corresponding to the second method, and the average color of the label category corresponding to the first method is lighter than the average color of the label category corresponding to the second method. For the first method, a threshold is found such that the area of ​​pixels with gray values ​​greater than the threshold in the image data to be segmented is less than a preset multiple of the area of ​​the image data to be segmented. This threshold is used as the target segmentation threshold, where the preset multiple is greater than 0 and less than 1. For the second method, if the length of the smallest side of the image data to be segmented is less than a preset length, the average gray value of the pixels in the image data to be segmented is taken as the target segmentation threshold; otherwise, the target segmentation threshold is determined based on the gray values ​​of the four corner regions and the center region of the image data to be segmented.

6. The model training apparatus according to claim 4, characterized in that: The threshold segmentation module further includes an erosion module, which is configured to perform an erosion operation on the threshold segmentation result of the image data to be segmented before obtaining the region segmentation result, wherein the erosion kernel... Satisfying the formula: ， in, Indicates the height and sum of the labeled area Indicates the width of the labeled area. This represents the height of the training sample. This represents the width of the training sample. This represents the preset hyperparameters.

7. The model training device according to claim 1, characterized in that: The training loss function uses weighted balanced cross-entropy.

8. The model training device according to claim 1, characterized in that: The model to be trained is a semantic segmentation model, and the prediction result of the model to be trained is the semantic segmentation result of the training samples.

9. The model training device according to claim 1, characterized in that: The marked area is rectangular in shape.

10. A recognition device for target identification in medical images, characterized in that, include: An acquisition module configured to acquire the medical image as an input image; Furthermore, the prediction module is configured to obtain at least one trained model from the model training apparatus according to any one of claims 1 to 9, determine the prediction results of each trained model for the input image, and obtain the target prediction result based on the prediction results of the at least one trained model.

11. The identification device according to claim 10, characterized in that: The prediction results of each trained model include the probability that each pixel in the input image belongs to the corresponding labeled category. The prediction module integrates the prediction results of at least one trained model according to the labeled category and the pixel to obtain the integrated probability that each pixel in the input image belongs to the corresponding labeled category. Based on the integrated probability, a connected region is determined, and based on the connected region, the target prediction result corresponding to each labeled category is obtained. If there is only one trained model, the probability is used as the integrated probability; otherwise, the average of the prediction results of multiple trained models is calculated to obtain the average probability that each pixel in the input image belongs to the corresponding labeled category, and this average is used as the integrated probability.

12. The identification device according to claim 10, characterized in that: The medical image is a fundus image.

13. The identification device according to claim 12, characterized in that: The targets include microaneurysms, punctate hemorrhages, patchy hemorrhages, and linear hemorrhages.

Images