Image classification method and training method of image classification network

Abstract

The application provides an image classification method and device, a training method and device of an image classification network, an electronic device, and a storage medium. The image classification method comprises: obtaining a first feature map of an original image, wherein the original image is a CT image of an abdomen; determining a second feature map based on the first feature map through an image classification network, wherein the image classification network comprises a residual network, a plurality of convolution networks, and a fusion network, the first feature map is input to the residual network, the plurality of convolution networks are connected in cascade, a first convolution network in the plurality of convolution networks is connected to the residual network, each convolution network in the plurality of convolution networks comprises a feature pyramid network, and the plurality of convolution networks are connected to the fusion network; and performing image classification on the original image based on the second feature map, wherein the image classification is a distinction between a water accumulation pixel and an organ pixel in the original image. In this way, the application can improve the efficiency of diagnosing pancreatic water accumulation.

Description

Technical Field

[0001] This application relates to, but is not limited to, the field of medical image processing technology, and in particular to an image classification method and apparatus, an image classification network training method and apparatus, an electronic device, and a storage medium. Background Technology

[0002] Currently, diagnosing hydrops requires doctors to rely on abdominal CT (computed tomography) images. On one hand, this demands a certain level of experience from the doctor. If the doctor lacks experience, or if other factors influence the diagnosis, the accuracy cannot be guaranteed. On the other hand, the diagnosis takes time. This can consume too much of the doctor's time and potentially delay the patient's treatment to some extent.

[0003] Therefore, improving the accuracy and speed of diagnosis of hydrops pancreatitis is an urgent problem to be solved. Summary of the Invention

[0004] This application provides an image classification method and apparatus, an image classification network training method and apparatus, an electronic device, and a storage medium, thereby improving the efficiency of diagnosing hydrops.

[0005] In a first aspect, this application provides an image classification method. The image classification method includes: acquiring a first feature map of an original image, wherein the original image is a CT image of an abdomen; determining a second feature map based on the first feature map using an image classification network, wherein the image classification network includes a residual network, multiple convolutional networks, and a fusion network; the first feature map is input to the residual network; the multiple convolutional networks are cascaded; the first convolutional network in the multiple convolutional networks is connected to the residual network; each of the multiple convolutional networks includes a feature pyramid network; and all the multiple convolutional networks are connected to the fusion network; and performing image classification on the original image based on the second feature map, wherein the image classification distinguishes between water accumulation pixels and organ pixels in the original image.

[0006] In some possible implementations, the operation of determining a second feature map based on the first feature map via an image classification network may include: determining a third feature map based on the first feature map via a residual network; determining a fourth feature map based on the input feature map via each convolutional network, wherein the input feature map of the first convolutional network is the third feature map, and the input feature map of any convolutional network other than the first convolutional network is the fourth feature map determined by the preceding convolutional network; and fusing multiple fourth feature maps from multiple convolutional networks via a fusion network to obtain the second feature map.

[0007] In some possible implementations, the feature pyramid network may include a first bottom-up path, a first top-down path, a second bottom-up path, and a second top-down path. The first bottom-up path, the first top-down path, the second bottom-up path, and the second top-down path are cascaded together. The first bottom-up path and the first top-down path are laterally connected. The first top-down path and the second bottom-up path are laterally connected. The second bottom-up path and the first bottom-up path are laterally connected.

[0008] In some possible implementations, the image classification network may further include a preprocessing network. The preprocessing network includes at least one residual structure. The operation of determining a second feature map based on the first feature map using the image classification network may further include: preprocessing the first feature map using the preprocessing network, wherein the preprocessed first feature map represents the location information of hydrops pancreatica.

[0009] In some possible implementations, the original image may include at least three consecutive CT images.

[0010] In some possible implementations, the ratio between the number of water-filled pixels and the number of non-water-filled pixels in the original image can be 3:1.

[0011] In some possible implementations, the operation of obtaining a first feature map of the original image may include: obtaining the original image; determining a gray-level co-occurrence matrix based on the original image; and fusing the original image and the gray-level co-occurrence matrix to obtain the first feature map.

[0012] In a second aspect, this application provides a training method for an image classification network. The training method includes: acquiring a first feature map of a sample image, wherein the sample image is a CT image of the abdomen; determining a second feature map based on the first feature map using an image classification network, wherein the image classification network includes a residual network, multiple convolutional networks, and a fusion network; the first feature map is input to the residual network; the multiple convolutional networks are cascaded; the first convolutional network is connected to the residual network; each of the multiple convolutional networks includes a feature pyramid network; and all convolutional networks are connected to the fusion network; and training the image classification network according to the second feature map.

[0013] In some possible implementations, training the image classification network based on the second feature map may include: employing a multiple loss function to determine a loss based on the second feature map and a fifth feature map, wherein the fifth feature map is the ground truth feature map corresponding to the sample image, and the multiple loss function is a combination of the Dice loss function and a distance-based loss function; and training the image classification network based on the determined loss.

[0014] In some possible implementations, the multiple loss function is expressed as:

[0015]

[0016] Here, p represents the true value, and q represents the predicted value; P Oi P represents the probability that the i-th pixel is a tumor. 1i q represents the probability that the i-th pixel is not a tumor; Oi A value of 1 indicates that the i-th pixel is a lesion voxel, and a value of 0 indicates that the i-th pixel is a non-lesion voxel; q 1i A value of 0 indicates that the i-th pixel is a diseased voxel, and a value of 1 indicates that the i-th pixel is a non-diseased voxel; α and β are coefficients; Indicates the actual distance. This indicates the predicted distance.

[0017] In a third aspect, this application provides an image classification apparatus. The image classification apparatus includes a first acquisition module, a first feature module, and a classification module. The first acquisition module is configured to acquire a first feature map of an original image. The original image is a CT image of an abdomen. The first feature module is configured to determine a second feature map based on the first feature map using an image classification network. The image classification network includes a residual network, multiple convolutional networks, and a fusion network. The first feature map is input to the residual network. The multiple convolutional networks are cascaded. The first convolutional network of the multiple convolutional networks is connected to the residual network. Each of the multiple convolutional networks includes a feature pyramid network. All of the multiple convolutional networks are connected to the fusion network. The classification module is configured to perform image classification on the original image based on the second feature map. Image classification distinguishes between water accumulation pixels and organ pixels in the original image.

[0018] In some possible implementations, the first feature module can be configured to: determine a third feature map based on a first feature map through a residual network; determine a fourth feature map based on an input feature map through each convolutional network, wherein the input feature map of the first convolutional network is the third feature map, and the input feature map of any convolutional network other than the first convolutional network is the fourth feature map determined by the preceding convolutional network; and fuse multiple fourth feature maps from multiple convolutional networks through a fusion network to obtain a second feature map.

[0019] In some possible implementations, the feature pyramid network may include a first bottom-up path, a first top-down path, a second bottom-up path, and a second top-down path. The first bottom-up path, the first top-down path, the second bottom-up path, and the second top-down path are cascaded together. The first bottom-up path and the first top-down path are laterally connected. The first top-down path and the second bottom-up path are laterally connected. The second bottom-up path and the first bottom-up path are laterally connected.

[0020] In some possible implementations, the image classification network may further include a preprocessing network. The preprocessing network includes at least one residual structure. The first feature module may also be configured to preprocess the first feature map through the preprocessing network.

[0021] In some possible implementations, the original image may include at least three consecutive CT images.

[0022] In some possible implementations, the ratio between the number of water-filled pixels and the number of non-water-filled pixels in the original image can be 3:1.

[0023] In some possible implementations, the first acquisition module may be configured to: acquire an original image; determine a gray-level co-occurrence matrix based on the original image; and fuse the original image and the gray-level co-occurrence matrix to obtain a first feature map.

[0024] In a fourth aspect, this application provides a training apparatus for an image classification network. The training apparatus includes a second acquisition module, a second feature module, and a training module. The second acquisition module is configured to acquire a first feature map of a sample image. The sample image is a CT image of an abdomen. The second feature module is configured to determine a second feature map based on the first feature map using an image classification network. The image classification network includes a residual network, multiple convolutional networks, and a fusion network. The first feature map is input to the residual network. The multiple convolutional networks are cascaded. The first convolutional network of the multiple convolutional networks is connected to the residual network. Each of the multiple convolutional networks includes a feature pyramid network. All of the multiple convolutional networks are connected to the fusion network. The training module is configured to train the image classification network based on the second feature map.

[0025] In some possible implementations, the training module can be configured to: employ a multiple loss function to determine a loss based on a second feature map and a fifth feature map, wherein the fifth feature map is the ground truth feature map corresponding to the sample image, and the multiple loss function is a combination of the Dice loss function and a distance-based loss function; and train the image classification network based on the determined loss.

[0026] In some possible implementations, the multiple loss function is expressed as:

[0027]

[0028] Here, p represents the true value, and q represents the predicted value; P Oi P represents the probability that the i-th pixel is a tumor. 1i q represents the probability that the i-th pixel is not a tumor; Oi A value of 1 indicates that the i-th pixel is a lesion voxel, and a value of 0 indicates that the i-th pixel is a non-lesion voxel; q 1i A value of 0 indicates that the i-th pixel is a diseased voxel, and a value of 1 indicates that the i-th pixel is a non-diseased voxel; α and β are coefficients; Indicates the actual distance. This indicates the predicted distance.

[0029] In a fifth aspect, this application provides an electronic device. The electronic device includes a processor and a memory. The memory is connected to the processor and configured to store executable instructions. When the processor is configured to execute the executable instructions, it implements an image classification method as described in any one of the first aspects and their possible embodiments, or a training method for an image classification network as described in any one of the third aspects and their possible embodiments.

[0030] In a sixth aspect, this application provides a computer-readable storage medium. The storage medium stores executable instructions. When executed by a processor, the executable instructions implement an image classification method as described in any one of the first aspects and their possible embodiments, or a training method for an image classification network as described in any one of the third aspects and their possible embodiments.

[0031] In a seventh aspect, this application provides a computer program product. The computer program product includes executable instructions. When executed by a processor, the executable instructions implement an image classification method as described in any one of the first aspects and their possible embodiments, or a training method for an image classification network as described in any one of the third aspects and their possible embodiments.

[0032] In this application, multiple convolutional networks are constructed within an image classification network. Each convolutional network includes a feature pyramid network, and these networks are cascaded. The outputs of these multiple convolutional networks are then fused through a fusion network. The feature pyramid network within the convolutional networks generates feature maps at different scales, enhancing information flow between networks and enabling the fusion of information at different depths. Simultaneously, the cascading of multiple convolutional networks increases the number of layers in the neural network, ensuring a high degree of abstraction in the feature maps. Therefore, when applying the image classification network of this application to process abdominal CT images, it can extract features of hydrops pancreatic fluid more accurately and quickly, thereby improving the efficiency of diagnosing hydrops pancreatic fluid.

[0033] Furthermore, the distribution of pixels in effusion is non-uniform in CT images. The acquired CT images include three or more consecutive images. These CT images are spatially continuous slices. Thus, by using three or more CT images, spatial information can be effectively increased, which is more conducive to the identification of features of pancreatic hydrops.

[0034] Furthermore, the ratio of the number of pixels with water accumulation to the number of pixels without water accumulation in the original image can follow a probability distribution of 3:1. This allows the original image to include pancreatic fluid accumulation in edge regions, thereby improving image classification performance.

[0035] Furthermore, this application obtains the corresponding gray-level co-occurrence matrix based on the original image, and then fuses and processes the original image with the gray-level co-occurrence matrix. Here, the gray-level co-occurrence matrix can extract the texture information of the original image, thereby helping to overcome the problem of difficulty in identifying hydrops pancreatitis.

[0036] Additionally, a preprocessing network is placed before the residual network and is used to preprocess the fused original image and the gray-level co-occurrence matrix. The first feature map after preprocessing represents the location information of the hydrops pancreatic fluid, thus initially obtaining the location of the hydrops pancreatic fluid.

[0037] Finally, the image classification network in this application employs a multiple loss function to calculate the loss during training. Since the multiple loss function combines a distance-based loss function and a Dice loss function, it can accelerate the convergence speed of the image classification network and improve its overall performance and robustness.

[0038] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0039] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0040] Figure 1 A flowchart of an image classification method according to an embodiment of this application is shown.

[0041] Figure 2 A structural diagram of an image classification network according to an embodiment of this application is shown.

[0042] Figure 3 This illustration shows a schematic diagram of the implementation process of the convolutional network determining the second feature map in an embodiment of this application.

[0043] Figure 4 A structural diagram of a convolutional network according to an embodiment of this application is shown.

[0044] Figure 5 A structural diagram of the first fused convolutional unit in the convolutional network of an embodiment of this application is shown.

[0045] Figure 6 A structural diagram of the second fused convolutional unit in the convolutional network of an embodiment of this application is shown.

[0046] Figure 7 A structural diagram of a preprocessing network according to an embodiment of this application is shown.

[0047] Figure 8 A structural diagram of an image classification device according to an embodiment of this application is shown.

[0048] Figure 9 A flowchart illustrating a training method for an image classification network according to an embodiment of this application is shown.

[0049] Figure 10 A structural diagram of a training device for an image classification network according to an embodiment of this application is shown.

[0050] Figure 11 A structural diagram of an electronic device according to an embodiment of this application is shown. Detailed Implementation

[0051] The embodiments of this application are described below with reference to the accompanying drawings. In the following description, reference is made to the accompanying drawings, which form part of this application and illustrate specific aspects of the embodiments of this application or to which specific aspects of the embodiments of this application may be used. It should be understood that the embodiments of this application may be used in other aspects and may include structural or logical variations not depicted in the drawings. Therefore, the following detailed description should not be construed in a limiting sense, and the scope of this application is defined by the appended claims. For example, it should be understood that the disclosure of the described methods is equally applicable to corresponding devices or systems for performing the methods, and vice versa. For example, if one or more specific method steps are described, the corresponding device may include one or more units, such as functional units, to perform the described one or more method steps (e.g., one unit performs one or more steps, or multiple units, each performing one or more of multiple steps), even if such one or more units are not explicitly described or illustrated in the drawings. On the other hand, for example, if a specific apparatus is described based on one or more units such as functional units, the corresponding method may include a step to perform the functionality of one or more units (e.g., a step to perform the functionality of one or more units, or multiple steps, each of which performs the functionality of one or more units among a plurality of units), even if such one or more steps are not explicitly described or illustrated in the accompanying drawings. Furthermore, it should be understood that, unless otherwise expressly stated, features of the various exemplary embodiments and / or aspects described herein can be combined with each other.

[0052] This application provides an image classification method and apparatus, an image classification network training method and apparatus, an electronic device, and a storage medium.

[0053] This application provides an image classification method. Figure 1 A flowchart of an image classification method according to an embodiment of this application is shown. Figure 1 As shown, the image classification method includes operations S110 to S130.

[0054] In operation S110, the first feature map of the original image is obtained.

[0055] Here, the original image is a CT image of the abdomen. In practical applications, CT images are obtained by a CT scanner performing continuous cross-sectional scans of specific parts of the body. Typically, multiple CT images can be obtained through cross-sectional scanning.

[0056] The first feature map is a combination of the original image and the gray-level co-occurrence matrix. The gray-level co-occurrence matrix is ​​obtained based on the original image and is used to characterize the texture information of the original image.

[0057] In one embodiment, operation S110 may include the following three operations.

[0058] In the first operation, the original image is acquired.

[0059] In one embodiment, the original image can be a CT image obtained from a CT scan of the abdomen of a human body. In one example, the original image may include the entire contents of a CT image. In this case, the original image can be obtained by: globally normalizing the CT image; spatially concatenating multiple consecutive CT images. In another example, the original image may include a portion of a CT image. In this case, the original image can be obtained by: globally normalizing the CT image; spatially concatenating multiple consecutive CT images; or cropping the CT image. For example, the number of spatially concatenated CT images may include at least three CT images. As another example, global normalization of the CT image may be performed uniformly on all CT images obtained from all scans. This means that all CT images have a uniform normalization standard. Alternatively, global normalization of the CT image may be performed individually on each CT image in the entire CT image set. It should be noted that in some cases, the original image may consist of only one CT image. In this case, the CT image can be obtained through global normalization and / or cropping without the need for spatial concatenation.

[0060] In the process of cropping CT images, the cropping center is first determined, and then the cropping area is determined according to a preset size. CT images contain both water-accumulated and non-water-accumulated pixels. When determining the cropping center, it is necessary to ensure that the number of water-accumulated pixels and non-water-accumulated pixels within the cropping area conforms to a certain ratio. For example, the ratio between the number of water-accumulated pixels and non-water-accumulated pixels can be 2:1, 3:1, 4:1, 3:2, or other ratios. In practical applications, water-accumulated pixels and / or non-water-accumulated pixels in CT images can be labeled. In this case, the cropping center can be determined based on the number of labels for water-accumulated pixels and / or non-water-accumulated pixels. In one embodiment, the cropping center can be randomly determined. In one embodiment, the cropping area can be square, rectangular, or other shapes. For example, the cropping area can be square. That is, the cropping area can have the same width and height. For instance, the width and height of the cropping area can be in the range of 400 pixels to 600 pixels. Another example is that the cropping area can be rectangular. In other words, the width and height of the cropped area can range from 400 pixels to 600 pixels, and the width and height are different.

[0061] Understandably, in some cases, after obtaining the cropped area, it can be further scaled. For example, the cropped area can be scaled to a predetermined size, such as 512 pixels × 512 pixels.

[0062] In the second operation, the gray-level co-occurrence matrix is ​​determined based on the original image.

[0063] After obtaining the original image, a gray-level co-occurrence matrix (GLCM) is determined based on the original image. The GLCM is used to characterize the texture information of the original image. In one embodiment, the GLCM is used to characterize at least one of the following texture information of the original image: orientation, adjacent spacing, and variation amplitude. In one embodiment, the GLCM may be obtained based on the contrast feature values ​​of the original image.

[0064] It is understandable that a gray-level co-occurrence matrix (GLCM) can be generated for a single CT image. Therefore, if the original image includes multiple CT images, multiple GLCMs can be obtained. For example, the number of GLCMs can be the same as the number of CT images.

[0065] In the third operation, the original image and the gray-level co-occurrence matrix are fused to obtain the first feature map.

[0066] Specifically, the original image obtained in the first operation and the gray-level co-occurrence matrix obtained in the second operation are concatenated.

[0067] The original image and the gray-level co-occurrence matrix (GLCM) can have different sizes. For example, the GLCM can be larger than the original image. Thus, before fusion, the original image and the GLCM can be scaled to the same size. Specifically, the original image and / or the GLCM can be enlarged or reduced so that the original image and the GLCM have the same size.

[0068] In one embodiment, when the original image is obtained from a single CT image, the original image and the gray-level co-occurrence matrix can be directly stitched together to obtain the first feature map. In another embodiment, when the original image is obtained from multiple CT images, different stitching methods can be used. For example, multiple original images and multiple gray-level co-occurrence matrices can be stitched together as two parts. That is, one part includes multiple original images, and the other part includes multiple gray-level co-occurrence matrices. Another example is that multiple original images and multiple gray-level co-occurrence images can be stitched together alternately. That is, gray-level co-occurrence matrices are inserted between multiple original images to obtain the first feature map. It should be noted that the fusion between the original image and the gray-level co-occurrence matrix can also be achieved in other ways, and this application does not specifically limit this method.

[0069] In operation S120, a second feature map is determined based on the first feature map using an image classification network.

[0070] Here, the first feature map obtained in operation S110 is input into an image classification network to obtain a second feature map. The image classification network is used to extract features from the first feature map. The second feature map is used to characterize hydrops pancreatic fluid. For example, the second feature map may include location information of the fluid-filled pixels.

[0071] Image classification networks are deep neural networks (DNNs) used to extract features from a first feature map. Figure 2 A structural diagram of an image classification network according to an embodiment of this application is shown. Figure 2 As shown, the image classification network 200 includes a residual network 210, multiple convolutional networks 220, and a fusion network 230. A first feature map is input to the residual network 210. The multiple convolutional networks 220 are cascaded. The first convolutional network 220 is connected to the residual network 210. The output of one convolutional network 220 is connected to the input of the next convolutional network 220. All convolutional networks 220 are connected to the fusion network 230.

[0072] Operation 120 will be explained below. Figure 3 This diagram illustrates the implementation flow of a convolutional network determining a second feature map in an embodiment of this application. The implementation flow is based on... Figure 2 The image classification network shown is based on this implementation. In this embodiment, the image classification network is a trained neural network. In one embodiment, referencing... Figure 2 and Figure 3 Operation 120 may include operations S310, S320 and S330.

[0073] In operation S310, the third feature map is determined based on the first feature map through the residual network.

[0074] The residual network 210 can serve as the backbone network of the image classification network 200. The residual network 210 is constructed by adding shortcut connections to a plain network to form a residual structure. Each shortcut connection corresponds to a residual structure. The residual network 210 avoids the problems of gradient vanishing and network degradation, thus ensuring that the entire image classification network 200 has sufficient depth and good performance.

[0075] In one embodiment, the residual network 210 may have different numbers of layers. For example, the residual network 210 may be resnet18, resnet34, resnet50, resnet101, resnet152, etc.

[0076] In operation S320, a fourth feature map is determined based on the input feature map through each convolutional network.

[0077] Here, the input feature map of the first convolutional network 220 is the third feature map, and the input feature map of any convolutional network other than the first convolutional network 220 is the fourth feature map determined by the previous convolutional network 220.

[0078] In the multiple convolutional networks 220, each convolutional network 220 takes a feature map as input and performs convolution on the input feature map to obtain a fourth feature map. On the one hand, since the multiple convolutional networks 220 are cascaded, the input feature map of the first convolutional network 220 is the third feature map from the residual network 210, and the input feature map of each subsequent convolutional network 220 is the fourth feature map output by the previous convolutional network 220.

[0079] It should be noted that the number of convolutional networks 220 in the image classification network 200 is greater than one, and can be set to any value as needed. For example, the image classification network 200 may include at least three convolutional networks 220. Alternatively, the image classification network 200 may include four convolutional networks 220.

[0080] In one embodiment, each of the plurality of convolutional networks 220 may include a feature pyramid network. Figure 4 A structural diagram of a convolutional network according to an embodiment of this application is shown. Figure 4 As shown, the feature pyramid network 400 may include a first bottom-up path 410, a first top-down path 420, a second bottom-up path 430, and a second top-down path 440. The first bottom-up path 410, the first top-down path 420, the second bottom-up path 430, and the second top-down path 440 are cascaded together. The first bottom-up path 410 and the first top-down path 420 are laterally connected. The first top-down path 420 and the second bottom-up path 430 are laterally connected. The second bottom-up path 430 and the second top-down path 440 are laterally connected. The second top-down path 440 and the first bottom-up path 410 are laterally connected.

[0081] It is understandable that the first bottom-up path 410 and the first top-down path 420 can constitute a magnification enhancement module. The second bottom-up path 430 and the second top-down path 440 can constitute a reduction enhancement module.

[0082] Here, the processing procedure of the input feature map by each convolutional network 220 is explained in conjunction with the structure of each convolutional network 220. In this explanation, an input feature map of size 512×512 is used as an example. It is understood that the feature map can also have other sizes, and this embodiment does not specifically limit this.

[0083] In the first bottom-up path 410, the input feature map is downsampled three times to obtain three feature maps with sizes of 256×256, 128×128, and 64×64, respectively.

[0084] The first top-down path 420 includes three first fusion convolutional units. Each first fusion convolutional unit is used to perform feature map fusion and convolution processing. Figure 5 A structural diagram of the first fused convolutional unit in the convolutional network of an embodiment of this application is shown. Figure 5 As shown, the first fusion convolutional unit 500 includes an upsampling layer 510, a fusion layer 520, and a convolutional layer 530. The first fusion convolutional unit 500 can take two feature maps as input. The upsampling layer 510 upsamples one feature map. For example, the upsampling layer 510 can double the size of a feature map. Then, the fusion layer 520 can fuse the upsampled feature map with the other feature map to obtain a fused feature map. Then, the convolutional layer 530 can perform convolution processing on the fused feature map from the fusion layer 520. For example, the convolutional layer 530 can use a 3×3 convolutional kernel to perform convolution processing on the fused feature map. Furthermore, the first fusion convolutional unit 500 may also include a processing layer 540. The processing layer 540 performs convolution processing, batch normalization processing, and activation function processing on the feature map from the convolutional layer 530. The convolution processing performed by the processing layer 540 can use a 1×1 convolutional kernel. The activation function used by the processing layer 540 can be the ReLU function. In the first top-down path 420, a first first fusion convolutional unit 500 is used to upsample the 64×64 feature map from the first bottom-up path 410, and then fuse and convolve it with a 128×128 feature map that is laterally connected from the first bottom-up path 410 to obtain a 128×128 feature map. Next, a second first fusion convolutional unit 500 is used to upsample the obtained 128×128 feature map, and then fuse and convolve it with a 256×256 feature map that is laterally connected from the first bottom-up path 410 to obtain a 256×256 feature map. Then, a third first fusion convolutional unit 500 is used to upsample the obtained 256×256 feature map, and then fuse and convolve it with a 512×512 feature map that is laterally connected from the first bottom-up path 410 to obtain a 512×512 feature map.

[0085] The second bottom-up path 430 includes three second fusion convolutional units. Each second fusion convolutional unit is used to perform feature map fusion and convolution processing. Figure 6A structural diagram of the second fused convolutional unit in the convolutional network of an embodiment of this application is shown. Figure 6 As shown, the second fusion convolutional unit 600 includes an upsampling layer 610, a fusion layer 620, and a convolutional layer 630. The second fusion convolutional unit 600 can take two feature maps as input. The upsampling layer 610 upsamples one feature map. For example, the upsampling layer 610 can double the size of a feature map. Then, the fusion layer 620 can fuse the upsampled feature map with the other feature map to obtain a fused feature map. Then, the convolutional layer 630 can perform convolution processing on the fused feature map from the fusion layer 620. For example, the convolutional layer can use a 3×3 convolutional kernel to perform convolution processing on the fused feature map. Furthermore, the first fusion convolutional unit 600 may also include a processing layer 640. The processing layer 640 performs convolution processing, batch normalization processing, and activation function processing on the feature map from the convolutional layer 630. The convolution processing performed by the processing layer 640 can use a 1×1 convolutional kernel. The activation function used by the processing layer 640 can be the ReLU function. In the second bottom-up path 430, a first second fusion convolutional unit 600 is used to upsample a 256×256 feature map that is laterally connected from the first top-down path 420, and then fuses and convolves it with a 512×512 feature map from the first top-down path 420 to obtain a 256×256 feature map. Next, a second second fusion convolutional unit 600 is used to upsample a 128×128 feature map that is laterally connected from the first top-down path 420, and then fuses and convolves it with the resulting 256×256 feature map to obtain a 128×128 feature map. Then, a third second fusion convolutional unit 600 is used to upsample a 64×64 feature map that is laterally connected from the first top-down path 420, and then fuses and convolves it with the resulting 128×128 feature map to obtain a 64×64 feature map.

[0086] In the second top-down path 440, the 64×64 feature map from the second bottom-up path 430 is upsampled and then fused with the 128×128 feature map from the first bottom-up path 410 to obtain a 128×128 feature map. Next, the obtained 128×128 feature map is upsampled and then fused with the 256×256 feature map from the first bottom-up path 410 to obtain a 256×256 feature map. Then, the obtained 256×256 feature map is upsampled and then fused with the 512×512 feature map from the first bottom-up path 410 to obtain a 512×512 feature map. In this case, the 512×512 feature map obtained from the second top-down path 440 can be used as the fourth feature map determined by the convolutional network 220.

[0087] As can be seen, multiple feature maps of different sizes are generated in the first top-down path 420, the second bottom-up path 430, and the second top-down path 440. Smaller feature maps can represent deeper information, while larger feature maps can represent shallower information. Through the lateral connections between the various paths, the flow of information in the convolutional network 220 is enhanced, and the combination of information at different depths in the various feature maps is achieved.

[0088] It should be noted that in the convolutional network 220, downsampling can employ convolution, pooling, or other downsampling methods, while upsampling can employ interpolation, unpooling, transposed convolution, or other upsampling methods. The sampling rates for downsampling and upsampling can be selected according to actual needs. Furthermore, the size and parameters of the convolutional kernels used in the convolutional layers of the first fusion convolutional unit 500 and the second fusion convolutional unit 600 can be selected according to actual needs. It is understood that the sampling rate and convolutional kernels can be different in different convolutional networks 220.

[0089] In operation S330, multiple fourth feature maps from multiple convolutional networks are fused through a fusion network to obtain a second feature map.

[0090] Fourth feature maps from multiple convolutional networks 220 are input into a fusion network 230. In one embodiment, these fourth feature maps are added point-to-point in the fusion network 230. Specifically, these fourth feature maps are concatenated together in a point-to-point manner.

[0091] Thus, by operating S310, S320 and S330, the second feature map can be obtained.

[0092] In one embodiment, the image classification network 200 may further include a preprocessing network 240. The preprocessing network 240 is positioned before the residual network 210. Specifically, a first feature map is input into the preprocessing network 240, and the output of the preprocessing network 240 is connected to the residual network 210.

[0093] In one embodiment, the preprocessing network 240 may include at least one residual structure. Figure 8 A structural diagram of a preprocessing network according to an embodiment of this application is shown. Figure 7 As shown, the preprocessing network 240 may include a first convolutional layer 241, a pooling layer 242, and three residual structures 243. The first convolutional layer 241 may use a 7×7 convolutional kernel. The pooling layer 242 may use max pooling, average pooling, or other pooling methods.

[0094] In one embodiment, the residual structure 243 includes two convolutional layers. The first and second convolutional layers are cascaded. The feature map input to the first convolutional layer is simultaneously fused with the feature map output from the second convolutional layer via skip connections. The convolutional kernels used in the two convolutional layers can be 3×3 kernels.

[0095] Correspondingly, in one embodiment, operation S120 may include operation S340. In operation S340, the first feature map is preprocessed using a preprocessing network. The preprocessed first feature map represents the location information of hydrops pancreatica.

[0096] It should be noted that the number of residual structures 243 in the preprocessing network 240, the number of convolutional layers in each residual structure 243, and the size and parameters of the convolutional kernels of the convolutional layers can all be set according to actual needs. This application embodiment does not impose specific limitations on this.

[0097] In operation S130, image classification is performed on the original image based on the second feature map.

[0098] Here, image classification distinguishes between water-accumulated pixels and organ pixels in the original image. In the original image, besides water-accumulated pixels, non-water-accumulated pixels mainly include organ pixels. Of course, non-water-accumulated pixels can include other pixels.

[0099] In one embodiment, the argmax function can be used to process the second feature map to achieve image classification. In another embodiment, the softmax function can be used to process the second feature map to achieve image classification. In yet another embodiment, the softmax and argmax functions can be used sequentially to process the second feature map to achieve image classification. It is understood that other methods can also be used to process the second feature map to achieve image classification.

[0100] Based on the same inventive concept, embodiments of this application provide an image classification device. For example... Figure 8 As shown, the image classification device 800 includes: a first acquisition module 810, a first feature module 820, and a classification module 830. The first acquisition module 810 is configured to acquire a first feature map of an original image. The original image is a CT image of the abdomen. The first feature module 820 is configured to determine a second feature map based on the first feature map using an image classification network. The image classification network includes a residual network, multiple convolutional networks, and a fusion network. The first feature map is input to the residual network. The multiple convolutional networks are cascaded. The first convolutional network in the multiple convolutional networks is connected to the residual network. Each of the multiple convolutional networks includes a feature pyramid network. All of the multiple convolutional networks are connected to the fusion network. The classification module 830 is configured to perform image classification on the original image based on the second feature map. Image classification distinguishes between water accumulation pixels and organ pixels in the original image.

[0101] In some possible implementations, the first feature module 820 may be configured to: determine a third feature map based on a first feature map through a residual network; determine a fourth feature map based on an input feature map through each convolutional network, wherein the input feature map of the first convolutional network is the third feature map, and the input feature map of any convolutional network other than the first convolutional network is the fourth feature map determined by the preceding convolutional network; and fuse multiple fourth feature maps of multiple convolutional networks through a fusion network to obtain a second feature map.

[0102] In some possible implementations, the feature pyramid network may include a first bottom-up path, a first top-down path, a second bottom-up path, and a second top-down path. The first bottom-up path, the first top-down path, the second bottom-up path, and the second top-down path are cascaded together. The first bottom-up path and the first top-down path are laterally connected. The first top-down path and the second bottom-up path are laterally connected. The second bottom-up path and the first bottom-up path are laterally connected.

[0103] In some possible implementations, the image classification network may further include a preprocessing network. The preprocessing network includes at least one residual structure. The first feature module may also be configured to preprocess the first feature map through the preprocessing network.

[0104] In some possible implementations, the original image may include at least three consecutive CT images.

[0105] In some possible implementations, the ratio between the number of water-filled pixels and the number of non-water-filled pixels in the original image can be 3:1.

[0106] In some possible implementations, the first acquisition module 810 may be configured to: acquire an original image; determine a gray-level co-occurrence matrix based on the original image; and fuse the original image and the gray-level co-occurrence matrix to obtain a first feature map.

[0107] It should be noted that the image classification device 800 in this embodiment is used to implement the image classification method described above. Therefore, the specific implementation details of the image classification device 800 can be found in the above description of the image classification method, and will not be repeated here.

[0108] Based on the same inventive concept, embodiments of this application provide a training method for an image classification network. Figure 9 A flowchart illustrating a training method for an image classification network according to an embodiment of this application is shown. Figure 9 As shown, the training methods for the image classification network include operations S910, S920, and S930.

[0109] In operation S910, the first feature map of the sample image is acquired.

[0110] Here, the sample image is a CT image of the abdomen. In practical applications, CT images are obtained by a CT scanner performing continuous cross-sectional scans of specific parts of the human body. Typically, multiple CT images can be obtained through cross-sectional scanning. Furthermore, the sample image can be obtained from multiple individuals. For example, the sample image can be obtained from a sample library. A sample library can store a large number of sample images from patients who have been diagnosed and / or treated.

[0111] The first feature map is a combination of the sample image and the gray-level co-occurrence matrix. The gray-level co-occurrence matrix is ​​obtained based on the sample image and is used to characterize the texture information of the sample image.

[0112] In one embodiment, operation S910 may include the following three operations.

[0113] In the first operation, a sample image is acquired.

[0114] In one embodiment, the sample image can be obtained based on a CT scan of the abdomen of a human body. In one example, the sample image may include the entire content of a CT image. In this case, the sample image can be obtained by: globally normalizing the CT image; spatially concatenating multiple consecutive CT images. In another example, the sample image may include a portion of a CT image. In this case, the sample image can be obtained by: globally normalizing the CT image; spatially concatenating multiple consecutive CT images; or cropping the CT image. For example, the number of spatially concatenated CT images may include at least three CT images. As another example, global normalization of the CT image may be performed uniformly on all CT images obtained from all scans. This means that all CT images have a uniform normalization standard. Alternatively, global normalization of the CT image may be performed individually on each CT image in the entire CT image set. It should be noted that in some cases, the sample image may include only one CT image. In this case, the CT image can be obtained through global normalization and / or cropping without the need for spatial concatenation.

[0115] In the process of cropping CT images, the cropping center is first determined, and then the cropping area is determined according to a preset size. CT images contain both water-accumulated and non-water-accumulated pixels. When determining the cropping center, it is necessary to ensure that the number of water-accumulated pixels and non-water-accumulated pixels within the cropping area conforms to a certain ratio. For example, the ratio between the number of water-accumulated pixels and non-water-accumulated pixels can be 2:1, 3:1, 4:1, 3:2, or other ratios. In practical applications, water-accumulated pixels and / or non-water-accumulated pixels in CT images can be labeled. In this case, the cropping center can be determined based on the number of labels for water-accumulated pixels and / or non-water-accumulated pixels. In one embodiment, the cropping center can be randomly determined. In one embodiment, the cropping area can be square, rectangular, or other shapes. For example, the cropping area can be square. That is, the cropping area can have the same width and height. For instance, the width and height of the cropping area can be in the range of 400 pixels to 600 pixels. Another example is that the cropping area can be rectangular. In other words, the width and height of the cropped area can range from 400 pixels to 600 pixels, and the width and height are different.

[0116] Understandably, in some cases, after obtaining the cropped area, it can be further scaled. For example, the cropped area can be scaled to a predetermined size, such as 512 pixels × 512 pixels.

[0117] In the second operation, the gray-level co-occurrence matrix is ​​determined based on the sample images.

[0118] After obtaining the sample image, a gray-level co-occurrence matrix (GLCM) is determined based on the sample image. The GLCM is used to characterize the texture information of the sample image. In one embodiment, the GLCM is used to characterize at least one of the following texture information of the sample image: orientation, adjacent spacing, and variation amplitude. In one embodiment, the GLCM may be obtained based on the contrast feature values ​​of the sample image.

[0119] It is understandable that a corresponding gray-level co-occurrence matrix can be generated for a single CT image. Therefore, when the sample images include multiple CT images, multiple gray-level co-occurrence matrices can be obtained. For example, the number of gray-level co-occurrence matrices can be the same as the number of CT images.

[0120] In the third operation, the sample image and the gray-level co-occurrence matrix are fused to obtain the first feature map.

[0121] Specifically, the sample image obtained in the first operation and the gray-level co-occurrence matrix obtained in the second operation are concatenated.

[0122] The sample image and the gray-level co-occurrence matrix (GLCM) can have different sizes. For example, the GLCM can be larger than the original image. Thus, before fusion, the sample image and GLCM can be scaled to the same size. Specifically, the sample image and / or GLCM can be enlarged or reduced so that the sample image and GLCM have the same size.

[0123] In one embodiment, when the sample image is obtained based on a single CT image, the sample image and the gray-level co-occurrence matrix can be directly concatenated to obtain the first feature map. In another embodiment, when the sample image is obtained based on multiple CT images, different methods can be used for concatenation. For example, multiple sample images and multiple gray-level co-occurrence matrices can be concatenated as two parts. That is, one part includes multiple sample images, and the other part includes multiple gray-level co-occurrence matrices. Another example is that multiple sample images and multiple gray-level co-occurrence images can be concatenated alternately. That is, gray-level co-occurrence matrices are inserted between the multiple sample images to obtain the first feature map. It should be noted that the fusion between the sample image and the gray-level co-occurrence matrix can also be achieved in other ways, and this embodiment does not specifically limit this method.

[0124] In operation S920, a second feature map is determined based on the first feature map through an image classification network.

[0125] It is understandable that the processing of the first feature map obtained from the sample image by the image classification network in operation S920 is similar to the processing of the first feature map obtained from the original image by the image classification network in operation S120. Therefore, the specific details of operation S920 can be found in the previous description of operation S120, and will not be repeated here.

[0126] It should be noted that the image classification network used in operating the S920 can be either an untrained network or a network that is being trained. For example, in the untrained case, the image classification network can have a preset structure and parameter values.

[0127] In operation S930, the image classification network is trained based on the second feature map.

[0128] Here, training the image classification network involves adjusting the parameters of the entire or part of the image classification network. In one embodiment, operation S930 can be implemented through the following two operations.

[0129] In the first operation, a multiple loss function is used to determine the loss based on the second and fifth feature maps.

[0130] Here, the fifth feature map is the ground truth feature map corresponding to the sample image. It is understood that the fifth feature map can be acquired and stored in the sample database simultaneously with the sample image, and it is used to characterize the true image classification of the sample image. For example, the fifth feature map is used to characterize the true hydrops of the pancreas in the sample image. For example, the fifth feature map may include the true location information of the hydrops pixels. Such true location information can be confirmed by a doctor or other relevant personnel.

[0131] In one embodiment, before calculating the loss using the multiple loss function, the second feature map can be calculated using a softmax function and / or an argmax function. For example, a softmax layer and / or an argmax layer can be used to process the second feature map. This softmax layer and / or argmax layer can be placed after the image classification network or within the image classification network; this embodiment does not specifically limit this.

[0132] The multiple loss function is a combination of the Dice loss function and a distance-based loss function. On one hand, the Dice loss function is commonly used in the training of image classification models. The Dice loss function L1 proposed in this embodiment can be expressed by formula (1):

[0133]

[0134] In formula (1), A represents the predicted value, B represents the actual value, and α and β are coefficients.

[0135] On the other hand, the morphology of hydrops pancreatis is fixed yet variable, exhibiting a connected form with an indeterminate shape. Therefore, employing a distance-based loss function that incorporates distance can effectively improve the accuracy of image classification. The Dice loss function L2 proposed in this embodiment can be expressed by formula (2):

[0136]

[0137] In formula (2), p represents the true value and q represents the predicted value; Indicates the actual distance. This indicates the predicted distance.

[0138] In one embodiment, the multiple loss function Loss(p,q) obtained by combining formulas (1) and (2) can be expressed by formula (3):

[0139]

[0140] In formula (3), P Oi P represents the probability that the i-th pixel is a tumor. 1i q represents the probability that the i-th pixel is not a tumor; Oi A value of 1 indicates that the i-th pixel is a lesion voxel, and a value of 0 indicates that the i-th pixel is a non-lesion voxel; q 1i A value of 0 indicates that the i-th pixel is a diseased voxel, and a value of 1 indicates that the i-th pixel is a non-diseased voxel; α and β are coefficients; Indicates the actual distance. This indicates the predicted distance.

[0141] In the second operation, the image classification network is trained based on the determined loss.

[0142] Specifically, after calculating the loss using multiple loss functions, the training of the image classification network can be determined based on the loss.

[0143] Understandably, this loss is used to determine whether the image classification network is sufficiently convergent. For example, if the loss is greater than a threshold, it indicates that the second feature map obtained by the image classification network for the sample image cannot achieve image classification well, meaning the image classification network is not sufficiently convergent. In this case, training the image classification network is decided. Conversely, if the loss is less than the threshold, it indicates that the second feature map obtained by the image classification network for the sample image can achieve image classification well, meaning the image classification network is sufficiently convergent. In this case, training the image classification network is no longer necessary.

[0144] In one embodiment, a training method can be implemented once for each sample image. In this way, during the training process based on a certain number of sample images, the image classification network can gradually converge until the performance of the image classification network is satisfactory.

[0145] Based on the same inventive concept, embodiments of this application provide a training device for an image classification network. Figure 10 This diagram illustrates a structural arrangement of a training apparatus for an image classification network according to an embodiment of this application. The image classification network training apparatus 1000 includes a second acquisition module 1010, a second feature module 1020, and a training module 1030. The second acquisition module 1010 is configured to acquire a first feature map of a sample image. The sample image is a CT image of the abdomen. The second feature module 1020 is configured to determine a second feature map based on the first feature map using the image classification network. The image classification network includes a residual network, multiple convolutional networks, and a fusion network. The first feature map is input to the residual network. The multiple convolutional networks are cascaded. The first convolutional network of the multiple convolutional networks is connected to the residual network. Each of the multiple convolutional networks includes a feature pyramid network. All multiple convolutional networks are connected to the fusion network. The training module 1030 is configured to train the image classification network based on the second feature map.

[0146] In some possible implementations, the training module 1030 may be configured to: employ a multiple loss function to determine a loss based on a second feature map and a fifth feature map, wherein the fifth feature map is the ground truth feature map corresponding to the sample image, and the multiple loss function is a combination of the Dice loss function and a distance-based loss function; and train the image classification network based on the determined loss.

[0147] In some possible implementations, the multiple loss function is expressed as:

[0148]

[0149] Here, p represents the true value, and q represents the predicted value; P Oi P represents the probability that the i-th pixel is a tumor. 1i q represents the probability that the i-th pixel is not a tumor; Oi A value of 1 indicates that the i-th pixel is a lesion voxel, and a value of 0 indicates that the i-th pixel is a non-lesion voxel; q 1i A value of 0 indicates that the i-th pixel is a diseased voxel, and a value of 1 indicates that the i-th pixel is a non-diseased voxel; α and β are coefficients; Indicates the actual distance. This indicates the predicted distance.

[0150] Based on the same inventive concept, embodiments of this application provide an electronic device. For example... Figure 11 As shown, the electronic device 1100 includes a processor 1110 and a memory 1120. The memory 1120 is connected to the processor 1110 and configured to store executable instructions. The processor 1110 is configured to, when executing the executable instructions, implement the image classification method or the image classification network training method of the embodiments of this application.

[0151] Based on the same inventive concept, embodiments of this application provide a computer-readable storage medium. The storage medium stores executable instructions. When executed by a processor, the executable instructions implement the image classification method or the image classification network training method of embodiments of this application.

[0152] Based on the same inventive concept, embodiments of this application provide a computer program product. The computer program product includes executable instructions. When executed by a processor, the executable instructions implement the image classification method or the image classification network training method of embodiments of this application.

[0153] In this embodiment, multiple convolutional networks are constructed within the image classification network. Each convolutional network includes a feature pyramid network, and these networks are cascaded. The outputs of these networks are then fused through a fusion network. The feature pyramid network within the convolutional networks generates feature maps at different scales, enhancing information flow between networks and expanding the receptive field of view, thereby achieving the fusion of information at different depths. Simultaneously, the cascading of multiple convolutional networks increases the number of layers in the neural network, ensuring a high degree of abstraction in the feature maps. Therefore, when applying the image classification network of this application to process abdominal CT images, it is possible to extract features of hydrops pancreatic fluid more accurately and quickly, thereby improving the efficiency of diagnosing hydrops pancreatic fluid.

[0154] Furthermore, convolutional networks consist of discrete convolutional processes that require only a small amount of computation, thus requiring significantly less computation than traditional pyramid structures. In some cases, the computational cost of the convolutional networks in this embodiment can be reduced to one-fifth or even less of that of traditional pyramid structures.

[0155] Furthermore, the distribution of pixels in effusion is non-uniform in CT images. The acquired CT images include three or more consecutive images. These CT images are spatially continuous slices. Thus, by using three or more CT images, spatial information can be effectively increased, which is more conducive to the identification of features of pancreatic hydrops.

[0156] Furthermore, the ratio of the number of pixels with water accumulation to the number of pixels without water accumulation in the original image can follow a probability distribution of 3:1. This allows the original image to include pancreatic fluid accumulation in edge regions, thereby improving image classification performance.

[0157] Furthermore, in this embodiment, a corresponding gray-level co-occurrence matrix is ​​obtained based on the original image, and then the original image and the gray-level co-occurrence matrix are fused and processed. Here, the gray-level co-occurrence matrix can extract the texture information of the original image, thereby helping to overcome the problem of difficulty in identifying hydrops pancreatitis.

[0158] Additionally, a preprocessing network is placed before the residual network and is used to preprocess the fused original image and the gray-level co-occurrence matrix. The first feature map after preprocessing represents the location information of the hydrops pancreatic fluid, thus initially obtaining the location of the hydrops pancreatic fluid.

[0159] Finally, the image classification network in this application employs a multiple loss function to calculate the loss during training. Since the multiple loss function combines a distance-based loss function and a Dice loss function, it can accelerate the convergence speed of the image classification network and improve its overall performance and robustness.

[0160] Those skilled in the art will appreciate that the functionality described in conjunction with the various illustrative logic blocks, modules, and algorithmic steps disclosed herein can be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functionality described by the various illustrative logic blocks, modules, and steps can be stored or transmitted as one or more instructions or codes on a computer-readable medium and executed by a hardware-based processing unit. The computer-readable medium may comprise a computer-readable storage medium, which corresponds to a tangible medium, such as a data storage medium, or a communication medium that includes any medium facilitating the transfer of a computer program from one place to another (e.g., according to a communication protocol). In this way, a computer-readable medium may substantially correspond to (1) a non-transitory tangible computer-readable storage medium, or (2) a communication medium, such as a signal or carrier wave. The data storage medium may be any available medium accessible by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for implementing the techniques described in this application. A computer program product may comprise a computer-readable medium.

[0161] By way of example and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage devices, magnetic disk storage devices or other magnetic storage devices, flash memory, or any other medium that may be used to store desired program code in the form of instructions or data structures and is accessible by a computer. Furthermore, any connection is properly referred to as a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other temporary media, but are specifically addressed to non-temporary tangible storage media. As used herein, disks and optical discs include compact optical discs (CDs), laser optical discs, optical discs, digital versatile optical discs (DVDs), and Blu-ray discs, where disks typically reproduce data magnetically, while optical discs reproduce data optically using lasers. The combination of the above items should also be included within the scope of computer-readable media.

[0162] Instructions can be executed by one or more processors, such as digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Therefore, the term "processor" as used herein can refer to any of the foregoing structures or any other structures suitable for implementing the techniques described herein. Furthermore, in some aspects, the functionality described in the various illustrative logic blocks, modules, and steps described herein can be provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated into combined codecs. Moreover, the techniques can be fully implemented within one or more circuit or logic elements.

[0163] The technology of this application can be implemented in a wide variety of devices or apparatuses, including wireless handheld devices, integrated circuits (ICs), or a set of ICs (e.g., chipsets). The various components, modules, or units described in this application are intended to emphasize functional aspects of the apparatuses used to perform the disclosed technology, but do not necessarily need to be implemented by different hardware units. In fact, as described above, the various units can be combined with suitable software and / or firmware within a codec hardware unit, or provided via interoperable hardware units (comprising one or more processors as described above).

[0164] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0165] The above description is merely an exemplary embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image classification method, characterized in that, The method includes: A first feature map of the original image is obtained, wherein the original image is a computed tomography image of the abdomen; the first feature map is determined by fusing the original image and the gray-level co-occurrence matrix. A second feature map is determined based on the first feature map using an image classification network. The image classification network includes a residual network, multiple convolutional networks, and a fusion network. The first feature map is input to the residual network. The multiple convolutional networks are cascaded, with the first convolutional network connected to the residual network. Each convolutional network includes a feature pyramid network, and all convolutional networks are connected to the fusion network. The feature pyramid network includes a first bottom-up path, a first top-down path, a second bottom-up path, and a second top-down path, all of which are cascaded. Based on the second feature map, the original image is classified, wherein the image classification distinguishes between water accumulation pixels and organ pixels in the original image.

2. The method according to claim 1, characterized in that, The step of determining the second feature map based on the first feature map using an image classification network includes: The third feature map is determined based on the first feature map using the residual network. For each of the convolutional networks, a fourth feature map is determined based on the input feature map, wherein the input feature map of the first convolutional network is the third feature map, and the input feature map of any convolutional network other than the first convolutional network is the fourth feature map determined by the preceding convolutional network. The second feature map is obtained by fusing multiple fourth feature maps from the multiple convolutional networks through the fusion network.

3. The method according to claim 2, characterized in that, The first bottom-up path and the first top-down path are laterally connected, the first top-down path and the second bottom-up path are laterally connected, the second bottom-up path and the second top-down path are laterally connected, and the second top-down path and the first bottom-up path are laterally connected.

4. The method according to claim 2, characterized in that, The image classification network further includes a preprocessing network, which includes at least one residual structure. The step of determining the second feature map based on the first feature map using an image classification network further includes: The first feature map is preprocessed using a preprocessing network, wherein the preprocessed first feature map represents the location information of hydrops pancreatis.

5. The method according to claim 1, characterized in that, The original images comprise at least three consecutive computed tomography (CT) images.

6. The method according to claim 5, characterized in that, In the original image, the ratio between the number of water-filled pixels and the number of non-water-filled pixels is 3:

1.

7. The method according to claim 1, characterized in that, The process of obtaining the first feature map of the original image includes: Obtain the original image; Based on the original image, the gray-level co-occurrence matrix is ​​determined; The original image and the gray-level co-occurrence matrix are fused to obtain a first feature map.

8. A training method for an image classification network, characterized in that, The method includes: A first feature map of a sample image is obtained, wherein the sample image is a computed tomography image of the abdomen; the first feature map is determined by fusing the sample image and the gray-level co-occurrence matrix. A second feature map is determined based on the first feature map using an image classification network. The image classification network includes a residual network, multiple convolutional networks, and a fusion network. The first feature map is input to the residual network. The multiple convolutional networks are cascaded, with the first convolutional network connected to the residual network. Each convolutional network includes a feature pyramid network, and all convolutional networks are connected to the fusion network. The feature pyramid network includes a first bottom-up path, a first top-down path, a second bottom-up path, and a second top-down path, all of which are cascaded. The image classification network is trained based on the second feature map.

9. The method according to claim 8, characterized in that, The step of training the image classification network based on the second feature map includes: A multiple loss function is used to determine the loss based on the second feature map and the fifth feature map, wherein the fifth feature map is the real feature map corresponding to the sample image, and the multiple loss function is a combination of the Dice loss function and a distance-based loss function. The image classification network is trained based on the determined loss.

10. The method according to claim 9, characterized in that, The multiple loss function is expressed as follows: ； in, p Represents the actual value. q Indicates the predicted value; This represents the probability that the i-th pixel is a tumor. This represents the probability that the i-th pixel is not a tumor. A value of 1 indicates that the i-th pixel is a diseased voxel, and a value of 0 indicates that the i-th pixel is a non-disease voxel; A value of 0 indicates that the i-th pixel is a diseased voxel, and a value of 1 indicates that the i-th pixel is a non-diseased voxel; α and β are coefficients; Indicates the actual distance. This indicates the predicted distance.

11. An image classification device, characterized in that, The device includes: The first acquisition module is configured to acquire a first feature map of an original image, wherein the original image is a computed tomography scan image of the abdomen; the first feature map is determined by fusing the original image and a gray-level co-occurrence matrix. A first feature module is configured to determine a second feature map based on the first feature map using an image classification network. The image classification network includes a residual network, multiple convolutional networks, and a fusion network. The first feature map is input to the residual network. The multiple convolutional networks are cascaded, with the first convolutional network connected to the residual network. Each convolutional network includes a feature pyramid network, and all convolutional networks are connected to the fusion network. The feature pyramid network includes a first bottom-up path, a first top-down path, a second bottom-up path, and a second top-down path, all cascaded together. The classification module is configured to perform image classification on the original image based on the second feature map, wherein the image classification is the distinction between water accumulation pixels and organ pixels in the original image.

12. A training device for an image classification network, characterized in that, The device includes: The second acquisition module is configured to acquire a first feature map of a sample image, wherein the sample image is a computed tomography scan image of the abdomen; the first feature map is determined by fusing the sample image and a gray-level co-occurrence matrix. The second feature module is configured to determine a second feature map based on the first feature map using an image classification network. The image classification network includes a residual network, multiple convolutional networks, and a fusion network. The first feature map is input to the residual network. The multiple convolutional networks are cascaded, with the first convolutional network connected to the residual network. Each convolutional network includes a feature pyramid network, and all convolutional networks are connected to the fusion network. The feature pyramid network includes a first bottom-up path, a first top-down path, a second bottom-up path, and a second top-down path, all cascaded together. The training module is configured to train the image classification network based on the second feature map.

13. An electronic device, characterized in that, include: processor; A memory, connected to the processor and configured to store executable instructions; The processor is configured to, when executing the executable instructions, implement the image classification method of any one of claims 1 to 7, or the training method of the image classification network of any one of claims 8 to 10.

14. A computer-readable storage medium, characterized in that, The storage medium stores executable instructions, which, when executed by a processor, implement the image classification method of any one of claims 1 to 7, or the training method of the image classification network of any one of claims 8 to 10.

Images