Image classification method, device and storage medium

Abstract

The present application provides an image classification method, device and storage medium. The method extracts features of a target object in an image by calling a first module in an image classification model to obtain a first feature of the target object; and calls a second module in the image classification model to classify the first feature to obtain the category of the image; wherein the image classification model is obtained by training with multiple training samples, and each training sample includes a local sample sub-image corresponding to each target sample object in the sample image, and a global category label corresponding to each sample image. The present application solves the problem of low accuracy of image classification results caused by the inaccuracy of directly using multiple local level labels.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image classification method, device and storage medium. Background Art

[0002] Early organ lesions usually exist in the form of organ nodules, such as lung nodules, thyroid nodules, breast nodules, etc. Organ nodules have specific properties, which can include primary nodules, benign nodules, and metastatic nodules. The judgment or identification of the properties of organ nodules determines the subsequent treatment of nodules and is also related to the patient's life. Therefore, accurate judgment or identification of the properties of organ nodules is crucial.

[0003] In the related art, traditional machine learning methods and deep learning methods are mainly used for the identification of metastatic nodules. The former uses radiomics methods to extract features, and then combines classifiers to identify metastatic nodules. However, the features extracted by radiomics methods are fixed features with poor flexibility, which leads to low recognition accuracy of metastatic nodules. The latter automatically learns the features required for classification tasks through classification networks. Although it has high flexibility, deep learning has poor interpretability and is difficult to integrate into doctors' existing prior knowledge. In addition, in order to train the classification network, a lot of manpower is often required to label each nodule lesion accurately. At the same time, it is difficult to determine whether a nodule is a metastatic label, which also leads to low accuracy of nodule attribute recognition results. Summary of the invention

[0004] The present application provides an image classification method, device and storage medium to solve at least one technical problem in the prior art.

[0005] On the one hand, the present application provides an image classification method, comprising:

[0006] Acquire an image containing at least one target object;

[0007] Calling the first module in the image classification model to extract features of the target object to obtain a first feature of the target object;

[0008] Calling the second module in the image classification model to classify the features to obtain the category of the image;

[0009] The image classification model is obtained by training a plurality of training samples, and each training sample includes a local sample sub-image corresponding to each target sample object in the sample image, and a global category label corresponding to each sample image.

[0010] On the other hand, an image classification device is also provided, comprising:

[0011] An image acquisition module, used to acquire an image containing at least one target object;

[0012] A feature extraction module, used for calling the first module in the image classification model to extract features of each target object in the image to obtain a first feature of the target object;

[0013] A classification module, used for calling the second module in the image classification model to classify the first feature to obtain the category of the image;

[0014] The image classification model is obtained by training a plurality of training samples, and each training sample includes a local area sample image corresponding to each target sample object in the sample image, and a global category label corresponding to each sample image.

[0015] As an optional implementation manner, the image classification device further includes:

[0016] An attribute acquisition module, used to obtain attribute information of the target object;

[0017] A sampling module is used to perform sampling processing on the target object based on the attribute information to obtain a local sub-image of the target object, where the local sub-image at least includes the area where the target object is located.

[0018] As an optional implementation manner, the attribute information includes the center position and size of the object, and the sampling module includes:

[0019] A magnification determination submodule, used to determine the sampling magnification of each size dimension in the target object according to the size of the object;

[0020] A size determination submodule, used to determine the sampling size of the target object based on the size of the object and the corresponding sampling magnification;

[0021] A spacing determination submodule, used to obtain the number of sampling points of each size dimension in the target object, and determine the sampling spacing of the target object in the corresponding size dimension based on the number of sampling points and the sampling size of the target object;

[0022] The sampling submodule is used to perform sampling processing on the target object based on the sampling spacing, the number of sampling points and the center position of the target object to obtain a local sub-image of the target object.

[0023] As an optional implementation, the feature extraction module includes:

[0024] The feature extraction submodule is used to input the local sub-image corresponding to each target object in the image into the first module in the image classification model, and use the first module to extract features from each local sub-image to obtain the first feature corresponding to each target object.

[0025] As an optional implementation, the second module includes an attention unit, a fusion unit and a classification unit, and the classification unit includes:

[0026] A weight determination submodule, configured to call the attention unit in the second module, perform attention mapping on the first feature of the target object, and determine a target weight value of the target object; the size of the target weight value is proportional to the probability that the target object belongs to the target category;

[0027] A fusion submodule, used for calling the fusion unit in the second module to perform fusion processing on the first feature of the target object and the target weight value to determine the second feature of the image;

[0028] The classification submodule is used to call the classification unit in the second module to classify the second feature of the image and determine the category of the image.

[0029] As an optional implementation, the attention unit includes a first fully connected layer and an activation layer, and the weight determination submodule includes:

[0030] A weight determination unit, configured to input the first feature of each target object into the first fully connected layer to obtain an initial weight value of the target object;

[0031] A normalization processing unit is used to call the activation layer to perform normalization processing on the initial weight value to obtain a target weight value corresponding to the target object.

[0032] As an optional implementation, the fusion unit includes a processing layer and a fusion layer, and the fusion submodule includes:

[0033] A weighting unit, configured to call the processing layer to perform weighted processing on the first feature of the target object and the target weight value to obtain a third feature of the target object;

[0034] The fusion unit is used to call the fusion layer to perform fusion processing on each third feature to obtain the second feature of the image.

[0035] As an optional implementation, the classification unit includes a second fully connected layer, and the classification submodule includes:

[0036] An input unit, used for inputting the second feature into the second fully connected layer;

[0037] A classification processing unit is used to classify the second feature using the second fully connected layer to determine the category of the image.

[0038] As an optional implementation, the weight parameters in the attention unit and the classification unit are the same.

[0039] On the other hand, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements any of the above-mentioned image classification methods when executing the computer program.

[0040] On the other hand, a computer storage medium is also provided, wherein at least one instruction or at least one program is stored in the computer storage medium, and the at least one instruction or at least one program is loaded and executed by a processor to implement any of the above-mentioned image classification methods.

[0041] On the other hand, a computer program product or a computer program is also provided, wherein the computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs any of the above-mentioned image classification methods.

[0042] This application has at least the following technical effects:

[0043] The embodiment of the present application extracts features of the target object in the image by calling the first module in the image classification model to obtain the first feature corresponding to the target object; and calls the second module in the image classification model to classify the first feature to obtain the category of the image; wherein the image classification model is obtained by training with multiple training samples, and each training sample includes a local sample sub-image corresponding to each target sample object in the sample image, and a global category label corresponding to each sample image. Since the image classification model is obtained by training with multiple local sample sub-images and corresponding global category labels, and the global category label is relatively easy to determine, there is no need to obtain the local attribute label corresponding to each target sample object. In this way, multi-instance learning is performed through multi-sample training, and only global-level labels are needed to train and identify local-level category attributes, which not only solves the problem of low accuracy of image classification results caused by the inaccuracy of directly using multiple local-level labels, but also simplifies the manual input of multiple local-level labels. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0045] Figure 1 It is a flowchart of an image classification method provided in an embodiment of the present application;

[0046] Figure 2 It is a partial flow chart of another image classification method provided in an embodiment of the present application;

[0047] Figure 3 is a schematic diagram of a sampling process provided in an embodiment of the present application;

[0048] Figure 4 It is a flow chart of the classification processing steps provided in the embodiment of the present application;

[0049] Figure 5 It is a process schematic diagram of an image classification method provided in an embodiment of the present application;

[0050] Figure 6 It is a flowchart of an image classification model training method provided in an embodiment of the present application;

[0051] Figure 7 is a structural block diagram of an image classification device provided in an embodiment of the present application;

[0052] Figure 8 It is a hardware structure diagram of a computer device for implementing the method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0054] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0055] The following describes a specific embodiment of an image classification method of the present application. The method can be applied to an image classification device, which can be implemented in software and / or hardware, and can be integrated in an electronic device with image classification function, such as a server, a terminal, a medical detection device, etc.

[0056] The server may include but is not limited to at least one of an independent server, a distributed server, a cloud server, and a server cluster. The distributed server may be a blockchain structure, and any node in the blockchain structure may execute or participate in executing the image classification method. The terminal may include but is not limited to software running in a physical device, such as an application or client installed on the device, and may also include but is not limited to at least one of a desktop computer, a tablet computer, a laptop computer, a smart phone, and other types of physical devices installed with an application.

[0057] Figure 1 It is a flowchart of an image classification method provided in an embodiment of the present application. The present application provides method operation steps as described in the embodiment or flowchart, but may include more or fewer operation steps based on conventional or non-creative labor. The order of steps listed in the embodiment is only one way of executing the steps among many orders, and does not represent the only order of execution. When the actual system or product is executed, it can be executed in the order of the method shown in the embodiment or the drawings or in parallel (for example, in a parallel processor or multi-threaded processing environment). Specifically, Figure 1 As shown, this embodiment mainly takes the method applied to a server as an example for description, and the method may include:

[0058] S101, acquiring an image containing at least one target object.

[0059] The image may be an image corresponding to at least one imaging modality and requiring classification processing. The imaging modality may be a single modality such as magnetic resonance imaging (MRI), computed tomography (CT), positron emission tomography (PET), or a multi-modality such as positron emission computed tomography-computer tomography (PET-CT), positron emission computed tomography-magnetic resonance scanning (PET-MR).

[0060] The target object is the object that needs to be identified or judged in the image. Taking the medical field as an example, the target object can be lesions such as nodules, masses, cysts, etc. corresponding to different organs. Each medical image contains one or more target objects.

[0061] In one embodiment, the server may obtain an image containing at least one target object from a database or cloud according to the received image classification processing request instruction. The server may also establish communication with a medical instrument, and after receiving the image classification processing request instruction, obtain the scanned image data to be processed from the corresponding medical instrument, and then map it to the image domain to obtain an image containing at least one target object.

[0062] S103, calling the first module in the image classification model to extract features of the target object to obtain a first feature of the target object.

[0063] The image classification model is obtained by training a plurality of training samples, and each training sample includes a local sample sub-image corresponding to each target sample object in the sample image, and a global category label corresponding to each sample image.

[0064] The local sample sub-image can be obtained by sampling each target sample object in the sample image. The global category label is used to characterize the category attribute to which the corresponding sample image belongs. Taking the target object as a lung nodule as an example, the global category label is the patient-level category label corresponding to the lung nodule in the sample image. Exemplarily, the global category label can be a lung metastasis case and a non-lung metastasis case (such as a lung primary lesion case, a lung benign case, etc.). Among them, a lung metastasis case refers to a case formed by some malignant tumors (such as breast cancer, lymphoma, kidney cancer, and gastric cancer, etc.) metastasized to the lungs. Taking the target object as a lung nodule as an example, the patient-level category label can be comprehensively determined by the image data of the organ where the target object is located, the image data of other organs, or the patient's pathological data.

[0065] The image classification model may include a first module, and the server uses the first module to extract features of each target object in the image to obtain a first feature corresponding to each target object. The first feature is used to characterize the descriptive features of the regional sub-image containing the corresponding target object.

[0066] S105, calling the second module in the image classification model to classify the first feature to obtain the category of the image.

[0067] The image classification model may also include a second module, and the server uses the second module to classify the first feature to obtain the category corresponding to the image. The category is used to characterize the category attribute to which the image containing the target object belongs. Taking the target object as a lung nodule as an example, the target category may include a binary category attribute of lung metastasis cases and non-lung metastasis cases, and may also include multiple category attributes such as lung metastasis cases, lung primary lesion cases, and lung benign cases.

[0068] In actual applications, the server obtains an image containing at least one target object, and then detects at least one target object in the image. Then, the first module in the image classification model is used to extract features of the detected target object to obtain a first feature corresponding to each target object; then, the second module in the image classification model can be used to classify the first feature to obtain the category corresponding to the image.

[0069] The detection of the target object can be realized by using an object detection algorithm for the target object, or by using an object detection module for the target object. It should be understood that the object detection module can be independent of the first module, or integrated into the first module to realize the combined processing of target object detection and feature extraction. The object detection module can be independent of the second module, or integrated together.

[0070] This embodiment extracts features of the target objects in the image by calling the first module in the image classification model to obtain the first features corresponding to each target object; and calls the second module in the image classification model to classify the first features to obtain the target category corresponding to the image; wherein the image classification model is obtained by training with multiple training samples, and each training sample includes a local sample sub-image corresponding to each target sample object in the sample image, and a global category label corresponding to each sample image. Since the image classification model is obtained by training with multiple local sample sub-images and corresponding global category labels, and the global category label is relatively easy to determine, there is no need to obtain the local attribute label corresponding to each target sample object. In this way, multi-instance learning is performed through multi-sample training, and only global-level labels are needed to train and identify local-level category attributes, which not only solves the problem of low accuracy of image classification results caused by the inaccuracy of directly using multiple local-level labels, but also simplifies the manual input of multiple local-level labels.

[0071] In an alternative embodiment, if Figure 2 As shown, in the above step S103, before the step of calling the first module in the image classification model to extract features of each target object in the image and obtaining the first feature corresponding to each target object, the image classification method may further include:

[0072] S201: Obtain attribute information of the target object.

[0073] The attribute information may include the center position and size of the object. The center position of the object may include the coordinates of the center point of the target object. For example, the center position of the object may be represented by the coordinates (x, y, z) in the world coordinate system. The size of the object may include the size of the outline of the target object. For example, the width, height, and depth of the outline of the target object may be represented by (w, h, d).

[0074] In practical applications, taking lung nodules as the target object, each target object in the image can be detected by a lung nodule detection algorithm or a lung nodule detection module to obtain object attribute information corresponding to each target object in the image. It should be noted that for technicians in the relevant field, lung nodule detection is a relatively mature technology, so the principle of the lung nodule detection algorithm or the lung nodule detection module is not described in detail here, and the lung nodule detection algorithm or the lung nodule detection module used is not specifically limited.

[0075] S203: Perform sampling processing on the target object based on the attribute information to obtain a local sub-image of the target object.

[0076] The local sub-image at least includes the area where the target object is located. That is, each local sub-image includes the contour frame area where the target object is located and part of the background area near the contour frame of the target object.

[0077] This embodiment performs sampling processing on the target object through the attribute information corresponding to the target object. Since the attribute information includes the center position and size of the object, adaptive size sampling processing is implemented, thereby avoiding the error caused by manually setting the sampling area, avoiding the problem of the target object being too small and the sampling area being too large; and the problem of the target object being too large and the sampling area being too small, improving the rationality of the local sub-image of the target object, and being conducive to improving the accuracy of image classification.

[0078] In an optional implementation manner, the above step S203, performing sampling processing on the target object based on the attribute information to obtain a local sub-image of the target object may include:

[0079] S2031: Determine a sampling ratio of each size dimension in the target object according to the size of the object.

[0080] Optionally, the size of the object may include size information of the outline of the target object in three dimensions: width, height, and depth. According to the size of the object, the size information of the outline of the target object in these three dimensions may be obtained, and then the sampling magnification corresponding to each target object in these three dimensions may be determined according to the sizes obtained in these three dimensions. The sampling magnification is a value greater than or equal to 1.

[0081] In practical applications, the sampling ratio can be based on the percentage of the target object in the local sub-image (e.g., 70%). As an example only, the sampling ratios corresponding to the three size dimensions can be expressed as k1, k2, and k3, respectively, where the values ​​of k1, k2, and k3 can be the same or different. In addition, the sampling ratios of different target objects can be the same or different.

[0082] S2033: Determine a sampling size of the target object based on the size of the object and the corresponding sampling magnification.

[0083] Optionally, after determining the sampling magnifications corresponding to the various size dimensions in the target object, the sampling size of the target object is re-determined according to the size of the object and the corresponding sampling magnifications.

[0084] Just as an example, Figure 3 As shown, for a target object, its size is (w, h, d), and the sampling magnifications are k1, k2, and k3, then the sampling sizes of the target object are re-determined in these three dimensions as k1×w, k2×h, and k3×d, respectively, where the symbol "×" represents a multiplication operation. The sampling size may be the size in the world coordinate system.

[0085] S2035, obtaining the number of sampling points of each size dimension in the target object, and determining the sampling spacing of the target object in the corresponding size dimension based on the number of sampling points and the sampling size of the target object.

[0086] Optionally, the number of sampling points in each size dimension of the target object is obtained, and the number of sampling points is used to reflect the number of sampling pixel points or the number of sampling voxel points. Taking the sampling voxel points as an example, the number of sampling points corresponding to each size dimension in the target object can be expressed as crop_w, crop_h and crop_d. Then, according to the number of sampling points and the sampling size of the target object, the sampling spacing of the target object in the corresponding size dimension is determined. For a certain target object, its sampling spacing in the corresponding size dimension is (k1×w) / crop_w, (k2×h) / crop_h and (k3×d) / crop_d, where the symbol " / " represents a division operation.

[0087] S2037: Perform sampling processing on the target object based on the sampling interval, the number of sampling points and the center position of the target object to obtain a local sub-image of the target object.

[0088] Optionally, after obtaining the three information of sampling spacing, number of sampling points and center position of each target object, the sampling parameter information of each target object can be determined. Then, sampling processing is performed on each target object in the image according to the sampling parameter information to obtain a local sub-image corresponding to each target object.

[0089] This embodiment performs sampling processing on each target object in the image based on the number of sampling points and sampling size of each target object, the determined sampling spacing of each target object under the corresponding size dimension, and the number of sampling points and the center position, to obtain a local sub-image corresponding to each target object. In this way, the sampling spacing is automatically adjusted through the size information of the target object, thereby achieving sampling of adaptive size. In addition, since the sampling spacing is determined based on the sampling magnification, and the sampling magnification is greater than or equal to 1, a certain proportion of the background area can be obtained for each target object during the sampling process. Since the background area near the target object also contains some useful information, the accuracy of the image classification result is improved. In addition, since the local sub-image corresponding to each target object is determined based on the center position of the object, the center position ratio of the obtained local sub-image is fixed, which is convenient for guiding the image classification model to pay more attention to the area where the target object is located, and further improves the accuracy of the image classification result.

[0090] In another embodiment, before the step of performing sampling processing on the target object based on the sampling spacing, the number of sampling points and the center position of the target object to obtain a local sub-image of the target object, the method may further include:

[0091] S2036: Obtain a sampling offset of the target object.

[0092] Optionally, the sampling offset refers to an offset distance from the center position of the target object during the sampling process. The sampling magnifications of different target objects may be the same or different.

[0093] At this time, performing sampling processing on the target object based on the sampling spacing, the number of sampling points and the center position of the target object to obtain a local sub-image of the target object may include:

[0094] S2037', based on the sampling interval, sampling offset, number of sampling points and center position of the target object, perform sampling processing on the target object to obtain a local sub-image of the target object.

[0095] Optionally, after acquiring the four information of sampling spacing, sampling offset, number of sampling points and center position of each target object, the sampling parameter information of each target object can be determined. Then, sampling processing is performed on each target object in the image according to the sampling parameter information to obtain a local sub-image corresponding to each target object.

[0096] This embodiment obtains the sampling offset of the target object and performs sampling processing on the target object in combination with the sampling parameter information of the sampling spacing, number of sampling points and center position of the target object to obtain a local sub-image corresponding to the target object. This is conducive to improving the generalization performance of the image classification model.

[0097] Furthermore, in the above step S103, calling the first module in the image classification model to extract features of the target object to obtain the first feature of the target object may include:

[0098] S205: Input the local sub-image of the target object into the first module in the image classification model, and use the first module to extract features from the local sub-image to obtain a first feature of the target object.

[0099] In practical applications, such as Figure 5 As shown, the first module may include at least a convolutional layer (CNN) and a third fully connected layer (FC3). By sequentially inputting the local sub-image of the target object into the two convolutional layers and the third fully connected layer in the first module, convolution operations and fully connected operations are performed respectively to extract features of each local image and obtain the first feature of each target object. Each node in the fully connected layer is connected to all nodes in the convolutional layer, so as to integrate the features extracted by the convolutional layer to integrate local information with differences.

[0100] In an alternative embodiment, if Figure 4 As shown, the second module includes an attention unit, a fusion unit and a classification unit. At this time, in the above step S105, the calling of the second module in the image classification model to classify the first feature and obtain the category of the image may include:

[0101] S401, calling the attention unit in the second module to perform attention mapping on the first feature of the target object and determine the target weight value of the target object.

[0102] The size of the target weight value is proportional to the probability that the target object belongs to the target category.

[0103] Optionally, the attention unit includes a first fully connected layer and an activation layer. At this time, calling the attention unit in the second module, performing attention mapping on the first feature of the target object, and determining the target weight value of the target object may include:

[0104] S4011, inputting the first feature of the target object into the first fully connected layer to obtain an initial weight value of the target object;

[0105] S4013, calling the activation layer to normalize the initial weight value to obtain the target weight value of the target object.

[0106] In practical applications, continue as Figure 5 As shown, the first feature of the target object is input into the first fully connected layer (FC1) in the attention unit, and the first fully connected layer is used to perform fully connected processing on the first features of all target objects to obtain the initial weight value of the target object. Then, the initial weight value of the target object is normalized using the activation layer (such as the softmax activation function) in the attention unit to obtain the target weight value of the target object. The values ​​of the multiple target weight values ​​obtained by normalization are between 0 and 1, and the sum of these multiple target weight values ​​can be 1. The target weight value is used to characterize the probability of each target object to the target category to which the image belongs. The larger the target weight value, the higher the probability of the corresponding target object to the target category to which the image belongs, and vice versa.

[0107] S403, calling the fusion unit in the second module to fuse the first feature of the target object and the target weight value to determine the second feature of the image.

[0108] Optionally, the fusion unit includes a processing layer and a fusion layer. The second feature of the image is used to characterize the descriptive feature information of the image. At this time, calling the fusion unit in the second module to fuse the first feature of the target object and the corresponding target weight value to determine the second feature of the image may include:

[0109] S4031, calling the processing layer to perform weighted processing on the first feature of the target object and the target weight value to obtain a third feature of the target object;

[0110] S4033: Call the fusion layer to perform fusion processing on the third feature to obtain the second feature of the image.

[0111] In practical applications, continue as Figure 5As shown, the first feature of each target object and the corresponding target weight value are input into the processing layer in the fusion unit, and the first feature of each target object and the corresponding target weight value are weighted by the processing layer to obtain the third feature of each target object. Then, the third feature of each target object is fused by the fusion layer in the fusion unit to obtain the second feature of the image. The fusion process includes but is not limited to at least one of summation process, vector concatenation process, merging process, etc.

[0112] As an example only, the first features of n target objects are represented as A1, A2, ... An, and the corresponding target weight values ​​are represented as w1, w2, ... wn, respectively. Then, the third features of each target object obtained after weighted processing are represented as w1A1, w2A2, ... wnAn. The second feature E of the image obtained by fusion processing can be represented as E = w1A1 + w2A2, ... + wnAn.

[0113] S405, calling the classification unit in the second module to classify the second feature of the image and determine the category of the image.

[0114] Optionally, the classification unit includes a second fully connected layer (FC2). At this time, calling the classification unit in the second module to classify the second feature of the image and determine the category of the image includes:

[0115] S4051, inputting the second feature of the image into the second fully connected layer;

[0116] S4053: Use the second fully connected layer to classify the second feature of the image to determine the category of the image.

[0117] In practical applications, continue as Figure 5 As shown, the second feature of the image is input into the second fully connected layer (FC2), and the second fully connected layer is used to classify the second feature to determine the category to which the image belongs. Taking the target object as a lung nodule as an example, the category may include a binary classification attribute of a lung metastasis case and a non-lung metastasis case, and may also include a multi-classification category attribute such as a lung metastasis case, a lung primary lesion case, and a lung benign case.

[0118] In this embodiment, the attention unit in the second module performs attention mapping on the first feature of each target object to determine the target weight value of each target object. The fusion unit in the second module performs fusion processing on the first feature of each target object and the corresponding target weight value to determine the second feature of the image. The classification unit in the second module performs classification processing on the second feature of the image to determine the category of the image. In this way, by introducing the attention mechanism to perform weighted processing on the first feature of each target object, the descriptive features of the image are obtained, the proportion of useful features in the second feature of the image is increased, and this is conducive to improving the accuracy of the category to which the image belongs determined based on the second feature.

[0119] like Figure 6 As shown, in one embodiment, a method for training an image classification model is provided. This embodiment is mainly described by taking the method applied to a server as an example. The image classification model training method may include:

[0120] S601, obtaining a training sample set, wherein the training sample set includes a plurality of training samples, each training sample includes a local sample sub-image of a target sample object in a sample image, and a global category label corresponding to each sample image.

[0121] The sample image may be an image corresponding to at least one imaging modality and requiring classification processing. The target sample object is an object in the image that needs to be identified or judged. Taking the medical field as an example, the target sample object may be a nodule, a mass, a cyst or other lesion corresponding to different organs, and the number of target sample objects contained in each sample image is one or more.

[0122] The local sample sub-image can be obtained by intercepting the target sample object of the sample image. Specifically, it can be obtained by intercepting the outline frame of the target sample object. Alternatively, the target sample object is sampled based on the acquired attribute information of the target sample object to obtain a local sample sub-image of the target sample object, and the local sample sub-image is an image that at least includes the area where the target sample object is located.

[0123] The global category label is used to characterize the category attribute to which the sample image belongs. Taking the target object as a lung nodule as an example, the global category label is the patient-level category label corresponding to the lung nodule in the sample image. Exemplarily, the global category label can be a lung metastasis case and a non-lung metastasis case (such as a lung primary lesion case, a lung benign case, etc.). Among them, a lung metastasis case refers to a case formed by some malignant tumors (such as breast cancer, lymphoma, kidney cancer, and gastric cancer, etc.) metastasized to the lungs. Taking the target object as a lung nodule as an example, the patient-level category label can be comprehensively determined by the image data of the organ where the target object is located, the image data of other organs, or the patient's pathological data.

[0124] S603: Process the target sample object of the sample image through a neural network model to obtain a sample category of the sample image.

[0125] The neural network model includes a first module and a second module. The first module is used to extract features of a target sample object of a sample image to obtain a first sample feature of the target sample object. The second module is used to classify the first sample feature of the target sample object to obtain a sample category of the sample image.

[0126] In an optional embodiment, the first module may include at least a convolutional layer and a third fully connected layer.

[0127] In an optional embodiment, the second module may include an attention unit, a fusion unit, and a classification unit. The attention unit performs attention mapping on the first sample feature of the target sample object to determine the target weight value corresponding to the target sample; the size of the target weight value is proportional to the probability that the target sample object belongs to the target sample category. Then, the fusion unit performs fusion processing on the first sample feature of the target sample object and the corresponding target weight value to determine the second sample feature of the sample image. Then, the classification unit performs classification processing on the second sample feature of the sample image to determine the sample category corresponding to the sample image.

[0128] In an optional embodiment, the weight parameters in the attention unit and the classification unit are the same.

[0129] Alternatively, if Figure 5 As shown, the weight parameters of the first fully connected layer (FC1) in the attention unit and the second fully connected layer (FC2) in the classification unit can be configured to be the same, that is, the weights of the two fully connected layers in the image classification model are shared.

[0130] This embodiment realizes weight sharing by configuring the same weight parameters in the attention unit and the classification unit, which can reduce the model training parameters of the image classification model, thereby reducing the time consumption of model training and improving the efficiency of model training. In addition, since each node in the fully connected layer is connected to all nodes in the previous layer, based on its fully connected characteristics, the parameters that need to be trained in the fully connected layer are also the most. Therefore, by setting the weight sharing of the two fully connected layers in the image classification model, the training parameters of the image classification model can be effectively reduced, and the model training efficiency can be further improved.

[0131] S605 , determining a loss function corresponding to the sample image based on the target sample category of the sample image and the global category label of the sample image.

[0132] S607: Determine an image classification model based on the loss function and the neural network model.

[0133] Optionally, the neural network model when the loss function value meets the preset conditions can be determined as the optimal neural network model, that is, the optimal neural network model is used as the trained image classification model. In other words, the network parameters corresponding to the neural network model when the loss function value meets the preset conditions are used as the network parameters of the image classification model. When the loss function value does not meet the preset conditions, the network parameters are iteratively updated and the neural network model is continuously trained until the training end condition is reached.

[0134] It should be understood that the specific contents of the above steps S601 to S607 can be found in the above embodiments and will not be repeated here.

[0135] In this embodiment, since the image classification model is trained through multiple local sample sub-images and corresponding global category labels, and the global category labels are relatively easy to determine, there is no need to obtain additional local attribute labels of the target sample objects. In this way, multi-instance learning is performed through multi-sample training, and only global-level labels are needed to train and identify local-level category attributes. This not only solves the problem of low accuracy of image classification results caused by the inaccuracy of directly using multiple local-level labels, but also simplifies the manual input of multiple local-level labels. In addition, the trained image classification model can not only perform attribute recognition of each target object, but also perform attribute recognition of images containing at least one target object, realizing local-level recognition of a single target object and global-level recognition of the image, simplifying the training process of the image classification model, and improving the application scope of the image classification model.

[0136] The following are device embodiments of the present application, which can be used to execute the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

[0137] Please refer to Figure 7 , which shows a structural block diagram of an image classification device provided by an embodiment of the present application. The device has the function of implementing the device in the above method example, and the function can be implemented by hardware or by hardware executing corresponding software. The image classification device may include:

[0138] An image acquisition module 710 is used to acquire an image containing at least one target object;

[0139] A feature extraction module 720 is used to call the first module in the image classification model to extract features of each target object in the image to obtain a first feature corresponding to each target object;

[0140] A classification module 730 is used to call the second module in the image classification model to classify the first feature to obtain the category of the image;

[0141] The image classification model is obtained by training a plurality of training samples, and each training sample includes a local area sample image corresponding to each target sample object in the sample image, and a global category label corresponding to each sample image.

[0142] As an optional implementation manner, the image classification device further includes:

[0143] An attribute acquisition module, used to obtain attribute information of the target object;

[0144] A sampling module is used to perform sampling processing on the target object based on the attribute information to obtain a local sub-image of the target object, where the local sub-image at least includes the area where the target object is located.

[0145] As an optional implementation manner, the attribute information includes the center position and size of the object, and the sampling module includes:

[0146] A magnification determination submodule, used to determine the sampling magnification of each size dimension in the target object according to the size of the object;

[0147] A size determination submodule, used to determine the sampling size of the target object based on the size of the object and the corresponding sampling magnification;

[0148] A spacing determination submodule, used to obtain the number of sampling points of each size dimension in the target object, and determine the sampling spacing of the target object in the corresponding size dimension based on the number of sampling points and the sampling size of the target object;

[0149] The sampling submodule is used to perform sampling processing on the target object based on the sampling spacing, the number of sampling points and the center position of the target object to obtain a local sub-image of the target object.

[0150] As an optional implementation, the feature extraction module includes:

[0151] The feature extraction submodule is used to input the local sub-image corresponding to each target object in the image into the first module in the image classification model, and use the first module to extract features from each local sub-image to obtain the first feature corresponding to each target object.

[0152] As an optional implementation, the second module includes an attention unit, a fusion unit and a classification unit, and the classification unit includes:

[0153] A weight determination submodule, configured to call the attention unit in the second module, perform attention mapping on the first feature of the target object, and determine a target weight value of the target object; the size of the target weight value is proportional to the probability that the target object belongs to the target category;

[0154] A fusion submodule, used for calling the fusion unit in the second module to perform fusion processing on the first feature of the target object and the target weight value to determine the second feature of the image;

[0155] The classification submodule is used to call the classification unit in the second module to classify the second feature of the image and determine the category of the image.

[0156] As an optional implementation, the attention unit includes a first fully connected layer and an activation layer, and the weight determination submodule includes:

[0157] A weight determination unit, configured to input the first feature of each target object into the first fully connected layer to obtain an initial weight value of the target object;

[0158] A normalization processing unit is used to call the activation layer to perform normalization processing on the initial weight value to obtain a target weight value corresponding to the target object.

[0159] As an optional implementation, the fusion unit includes a processing layer and a fusion layer, and the fusion submodule includes:

[0160] A weighting unit, configured to call the processing layer to perform weighted processing on the first feature of the target object and the target weight value to obtain a third feature of the target object;

[0161] The fusion unit is used to call the fusion layer to perform fusion processing on each third feature to obtain the second feature of the image.

[0162] As an optional implementation, the classification unit includes a second fully connected layer, and the classification submodule includes:

[0163] An input unit, used for inputting the second feature into the second fully connected layer;

[0164] A classification processing unit is used to classify the second feature using the second fully connected layer to determine the category of the image.

[0165] As an optional implementation, the weight parameters in the attention unit and the classification unit are the same.

[0166] It should be noted that the specific contents of the above-mentioned device embodiments of the present application can be referenced to the above-mentioned method embodiments, which will not be repeated here.

[0167] An embodiment of the present application also provides a computer storage medium, in which at least one instruction or at least one program is stored, and the at least one instruction or at least one program is loaded and executed by a processor to implement the image classification method as described in any of the above embodiments.

[0168] The computer storage medium may be located in at least one of the multiple network devices of the computer network. Optionally, the storage medium may include, but is not limited to, various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0169] The embodiment of the present application also provides a computer program product or a computer program, wherein the computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the image classification method described in any of the above embodiments.

[0170] An embodiment of the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, any of the above-mentioned image classification methods is implemented.

[0171] Further, Figure 8 A hardware structure diagram of a computer device for implementing the method provided in the embodiment of the present application is shown. The computer device may be a server, a computer terminal, a mobile terminal or other device. The computer device may also participate in or include the apparatus provided in the embodiment of the present application. Figure 8 As shown, the computer device 8 may include one or more (82a, 82b, ..., 82n are used to illustrate) processors 82 (the processor 82 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 84 for storing data, and a transmission device 86 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 8 The structure shown is for illustration only and does not limit the structure of the above electronic device. Figure 8 More or fewer components as shown, or with Figure 8 Different configurations are shown.

[0172] It should be noted that the one or more processors 82 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components of the computer device 8. As involved in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0173] The memory 84 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the methods described in the embodiments of the present application. The processor 82 executes various functional applications and data processing by running the software programs and modules stored in the memory 84, that is, to implement the above-mentioned neural network processing method. The memory 84 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 84 may further include a memory remotely disposed relative to the processor 82, and these remote memories may be connected to the computer device 8 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0174] The transmission device 86 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer device 8. In one example, the transmission device 86 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 86 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0175] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer device 8 .

[0176] It should be noted that the above-mentioned sequence of the embodiments of the present application is for description only and does not represent the advantages and disadvantages of the embodiments. The above-mentioned specific embodiments of the present application are described. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0177] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and server embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0178] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0179] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. An image classification method, characterized in that: include: Acquire an image containing at least one target object; Calling the first module in the image classification model to extract features of the target object to obtain the first features of the target object; including: calling the first module in the image classification model to extract features of the local sub-images of each target object to obtain the first features of each target object; the local sub-images are obtained by sampling the target object, specifically including: determining the sampling magnification of each size dimension in the target object according to the size of the target object; determining the sampling size of the target object based on the size of the object and the corresponding sampling magnification; obtaining the number of sampling points in each size dimension of the target object, and determining the sampling spacing of the target object in the corresponding size dimension based on the number of sampling points and the sampling size of the target object; wherein the sampling magnification is greater than or equal to 1; Calling the second module in the image classification model to classify the first feature to obtain the category of the image; The image classification model is obtained by training a plurality of training samples, and each training sample includes a local sample sub-image of each target sample object in the sample image, and a global category label corresponding to each sample image.

2. The image classification method according to claim 1, characterized in that: Before the step of calling the first module in the image classification model to extract features of the target object and obtain the first feature of the target object, the method further includes: Obtaining attribute information of the target object; The target object is sampled based on the attribute information to obtain a local sub-image of the target object, where the local sub-image at least includes the area where the target object is located.

3. The image classification method according to claim 2, characterized in that: The attribute information includes the center position and size of the object, and the sampling process of the target object based on the attribute information to obtain a local sub-image of the target object includes: Determining the sampling ratio of each size dimension in the target object according to the size of the object; Determining a sampling size of the target object based on the size of the object and the corresponding sampling magnification; Acquire the number of sampling points of each size dimension in the target object, and determine the sampling spacing of the target object in the corresponding size dimension based on the number of sampling points and the sampling size of the target object; Based on the sampling interval, the number of sampling points and the center position of the target object, the target object is sampled to obtain a local sub-image of the target object.

4. The image classification method according to claim 3, characterized in that: Calling the first module in the image classification model to extract features of the target object to obtain the first feature of the target object includes: The local sub-image of the target object is input into the first module in the image classification model, and the first module is used to extract features from the local sub-image to obtain a first feature of the target object.

5. The image classification method according to any one of claims 1 to 4, characterized in that: The second module includes an attention unit, a fusion unit and a classification unit. The calling of the second module in the image classification model to classify the first feature to obtain the category of the image includes: Calling the attention unit in the second module to perform attention mapping on the first feature of the target object and determine a target weight value of the target object; the size of the target weight value is proportional to the probability that the target object belongs to the target category; calling a fusion unit in the second module to fuse the first feature of the target object and the target weight value to determine the second feature of the image; The classification unit in the second module is called to classify the second feature of the image to determine the category of the image.

6. The image classification method according to claim 5, characterized in that: The attention unit includes a first fully connected layer and an activation layer, and calling the attention unit in the second module to perform attention mapping on the first feature of the target object and determine the target weight value of the target object includes: Inputting the first feature of the target object into the first fully connected layer to obtain an initial weight value of the target object; The activation layer is called to normalize the initial weight value to obtain a target weight value of the target object.

7. The image classification method according to claim 5, characterized in that: The fusion unit includes a processing layer and a fusion layer, and calling the fusion unit in the second module to perform fusion processing on the first feature of the target object and the target weight value to determine the second feature of the image includes: Calling the processing layer to perform weighted processing on the first feature of the target object and the target weight value to obtain a third feature of the target object; The fusion layer is called to perform fusion processing on the third feature to obtain the second feature of the image.

8. The image classification method according to claim 5, characterized in that: The classification unit includes a second fully connected layer, and calling the classification unit in the second module to classify the second feature of the image to determine the category of the image includes: Inputting the second feature into the second fully connected layer; The second feature is classified by using the second fully connected layer to determine the category of the image.

9. An image classification device, characterized in that: include: An image acquisition module, used to acquire an image containing at least one target object; A feature extraction module, used for calling the first module in the image classification model to perform feature extraction on the target object and obtain the first feature of the target object; including: calling the first module in the image classification model to perform feature extraction on the local sub-image of each target object and obtain the first feature of each target object; the local sub-image is obtained by sampling the target object, specifically including: determining the sampling magnification of each size dimension in the target object according to the size of the target object; determining the sampling size of the target object based on the size of the object and the corresponding sampling magnification; obtaining the number of sampling points in each size dimension of the target object, and determining the sampling spacing of the target object in the corresponding size dimension based on the number of sampling points and the sampling size of the target object; wherein the sampling magnification is greater than or equal to 1; A classification module, used for calling the second module in the image classification model to classify the first feature to obtain the category of the image; The image classification model is obtained by training a plurality of training samples, and each training sample includes a local sample sub-image of each target sample object in the sample image and a global category label of each sample image.

10. A computer storage medium, characterized in that: The computer storage medium stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the processor to implement the image classification method as described in any one of claims 1-8.

11. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the image classification method according to any one of claims 1 to 8 when executing the computer program.

12. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes to implement the image classification method according to any one of claims 1 to 8.

Images