Image classification and model training method, device and storage medium

Abstract

Embodiments of the present application provide an image classification and model training method, device and storage medium. In the method, a skull image and attribute information are obtained, the attribute information including: class representation information and growth period representation information; a skull image classification model is used to encode the position of a feature point in a skull region of the skull image on the skull region to obtain position encoding information; the class representation information and the growth period representation information in the attribute information are encoded to obtain attribute encoding information; a multi-dimensional image feature vector is extracted from the skull image, and the multi-dimensional image feature vector, the position encoding information and the attribute encoding information are spliced; and the skull image is classified according to the fused feature vector obtained by splicing. In this way, a fused feature vector can be obtained based on the class representation information and the growth period representation information and the skull image, and the lesion features in the skull image can be accurately identified according to the fused feature vector to classify the skull image.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image classification and model training method, device and storage medium. Background Technology

[0002] Skull CT images (Computed Tomography, or CT for short) are one of the most important and commonly used examinations in clinical practice, and they play a significant role in disease diagnosis.

[0003] In existing technologies, deep learning techniques can be used to learn skull features from skull CT images. The presence of lesions in a patient can be determined by judging whether skull features that match the characteristics of a lesion are learned. However, existing methods for learning lesion features from skull CT images are more suitable for adults. For children or adolescents who are still growing, the learning effect on lesion features is not ideal and may not be able to accurately identify lesion features. Summary of the Invention

[0004] This application provides image classification and model training methods, devices, and storage media in several aspects to more accurately identify lesion features.

[0005] This application provides an image classification method, comprising: acquiring a user's skull image and the user's attribute information, the attribute information including at least: category representation information and growth period representation information; inputting the skull image and the attribute information into an information encoding network in a pre-trained skull image classification model, the information encoding network including a first information encoding subnetwork and a second information encoding subnetwork; using the first information encoding subnetwork to encode the position of each feature point in the skull region of the skull image to obtain position encoding information for each feature point; and using the second information encoding subnetwork to encode the category representation information and growth period representation information in the attribute information to obtain attribute encoding information. The location encoding information and attribute encoding information of each feature point are input into the encoding information fusion network in the skull image classification model. The encoding information fusion network includes a feature extraction subnetwork and an information fusion subnetwork. The feature extraction subnetwork extracts a multidimensional image feature vector for identifying lesion features from the skull image, and the information fusion subnetwork concatenates the multidimensional image feature vector, the location encoding information of each feature point, and the attribute encoding information to obtain a fused feature vector. The fused feature vector is input into the first classification network in the skull image classification model, and the skull image is classified into an abnormal skull image with lesion features or a normal skull image without lesion features based on the fused feature vector.

[0006] Optionally, the skull image classification model further includes a second classification network and / or a third classification network; the method further includes: when the skull image is classified as an abnormal skull image with lesion features, inputting the fused feature vector into the second classification network, identifying the category of the lesion feature based on the fused feature vector, and outputting the identified category of the lesion feature; and / or, inputting the fused feature vector into the third classification network, identifying the anatomical structure category of the lesion feature based on the fused feature vector, and outputting the identified anatomical structure category of the lesion feature.

[0007] Optionally, the first information encoding sub-network is used to encode the position of each feature point in the skull region of the skull image to obtain the position encoding information of each feature point. This includes: establishing a spherical coordinate system with the center point of the skull image as the origin in the first information encoding sub-network to obtain the spherical coordinates of each feature point in the skull region; and encoding the two angle information in the spherical coordinates of each feature point using a first trigonometric function to obtain the position encoding information of each feature point.

[0008] Optionally, the position encoding information of each feature point is obtained by encoding two angles in the spherical coordinates of each feature point using a first trigonometric function, including: using... The two angle information in the spherical coordinates of each feature point are encoded to obtain the position encoding information of each feature point; where θ is the polar angle of each feature point and φ is the azimuth angle of each feature point.

[0009] Optionally, the second information encoding subnetwork is used to encode the category representation information and growth period representation information in the attribute information to obtain attribute encoding information. This includes: in the second encoding subnetwork, using a second trigonometric function to encode the growth period representation information to obtain growth period encoding information; encoding the category representation information into a target encoding value according to the category represented by the category representation information, wherein the target encoding value is a preset first encoding value or a second encoding value, the first encoding value corresponding to a first category and the second encoding value corresponding to a second category; and generating the attribute encoding information based on the growth period encoding information and the target encoding value.

[0010] Optionally, the growth period characterization information is encoded using a second trigonometric function to obtain growth period encoded information, including: using... The growth period characterization information is encoded to obtain growth period coded information; wherein, A is the growth period characterization information.

[0011] Optionally, the feature extraction subnetwork is used to extract a multidimensional image feature vector for identifying lesion features from the skull image, including: starting from the upper left corner of the skull image, sequentially moving sampled image blocks of size M to the lower right corner, with each movement distance being K pixels; for each sampled image block, if it appears in the skull region, then extract the multidimensional image feature vector for identifying lesion features from the image block; if it does not appear in the skull region, then skip the image block.

[0012] This application embodiment also provides a method for training a skull image classification model, including: acquiring a user's skull image sample, the user's attribute information sample, and lesion feature labels corresponding to the skull image sample; the attribute information sample includes at least: category representation information sample and growth period representation information sample; inputting the skull image sample, the user's attribute information sample, and the lesion feature labels corresponding to the skull image sample into a preset skull image classification network; the skull image classification network includes: an information encoding network, an encoding information fusion network, and a first classification network; the information encoding network includes a first information encoding subnetwork and a second information encoding subnetwork; the encoding information fusion network includes a feature extraction subnetwork and an information fusion subnetwork; in any training round, the skull image sample and the attribute information sample are input into the information encoding network, and the first information encoding subnetwork is used to encode the position of each feature point in the skull region of the skull image sample on the skull region to obtain the position encoding information of each feature point; the second information encoding subnetwork is used to encode the position of each feature point in the skull region of the skull image sample; the second information encoding subnetwork is used to encode the position of each feature point in the skull region of the skull image sample. The code subnetwork encodes the category representation information samples and growth period representation information samples in the attribute information samples to obtain attribute encoding information. The location encoding information of each feature point and the attribute encoding information are input into the encoding information fusion network. The feature extraction subnetwork extracts multi-dimensional image feature vectors for identifying lesion features from the skull image samples. The information fusion subnetwork concatenates the multi-dimensional image feature vectors, the location encoding information of each feature point, and the attribute encoding information to obtain a fused feature vector. The fused feature vector is input into the first classification network. Based on the fused feature vector, the skull image samples are classified as abnormal skull images with lesion features or normal skull images without lesion features. A first prediction residual is calculated based on the classification result and the lesion feature label. The parameters of the skull image classification network are updated with the goal of the first prediction residual converging to a specified range. Training stops when the first prediction residual converges to the specified range, and a skull image classification model is output.

[0013] This application also provides an electronic device, including: a memory and a processor; wherein the memory is used to: store one or more computer instructions; the processor is used to execute the one or more computer instructions to: perform steps in the image classification method or the skull image classification model training method.

[0014] This application also provides a computer-readable storage medium that, when executed by a processor, enables the processor to implement the steps in the image classification method or the skull image classification model training method.

[0015] In this embodiment, skull images and attribute information can be acquired. The attribute information includes category representation information and growth period representation information. Using a skull image classification model, the positions of feature points in the skull region of the skull image are encoded to obtain positional encoding information. The category representation information and growth period representation information in the attribute information are encoded to obtain attribute encoding information. A multidimensional image feature vector is extracted from the skull image, and the multidimensional image feature vector, positional encoding information, and attribute encoding information are concatenated. The skull image is classified based on the fused feature vector obtained from the concatenation. In this way, a fused feature vector can be obtained based on the category representation information, growth period representation information, and skull image, and the lesion features in the skull image can be identified more accurately based on the fused feature vector, thus classifying the skull image. Attached Figure Description

[0016] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0017] Figure 1 A flowchart of an image classification method provided as an exemplary embodiment of this application;

[0018] Figure 2 A schematic diagram of the structure of a skull image classification model provided in an exemplary embodiment of this application;

[0019] Figure 3 A flowchart of a method for training a skull image classification model provided in an exemplary embodiment of this application;

[0020] Figure 4 A schematic diagram of the structure of an electronic device provided for an exemplary embodiment of this application. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] In existing technologies, deep learning techniques can be used to learn skull features from skull CT images. The presence of lesions in a patient can be determined by judging whether skull features that match the characteristics of a lesion are learned. However, existing methods for learning lesion features from skull CT images are more suitable for adults. For children or adolescents who are still growing, the learning effect on lesion features is not ideal and may not be able to accurately identify lesion features.

[0023] To address the aforementioned technical problems, a solution is provided in some embodiments of this application. The technical solutions provided by each embodiment of this application will be described in detail below with reference to the accompanying drawings.

[0024] Figure 1 A flowchart of an image classification method provided for an exemplary embodiment of this application is shown below. Figure 1 As shown, image classification methods may include:

[0025] Step 11: Obtain the user's skull image and the user's attribute information, which includes at least: category representation information and growth period representation information.

[0026] Step 12: Input the skull image and attribute information into the information encoding network of the pre-trained skull image classification model. The information encoding network includes a first information encoding subnetwork and a second information encoding subnetwork.

[0027] Step 13: Using the first information coding sub-network, the position of each feature point in the skull region of the skull image is encoded to obtain the position coding information of each feature point.

[0028] Step 14: Using the second information coding sub-network, the category representation information and growth period representation information in the attribute information are encoded to obtain the attribute coding information.

[0029] Step 15: Input the location encoding information and attribute encoding information of each feature point into the encoding information fusion network in the skull image classification model. The encoding information fusion network includes a feature extraction subnetwork and an information fusion subnetwork.

[0030] Step 16: Use the feature extraction subnetwork to extract multidimensional image feature vectors for identifying lesion features from the skull image, and use the information fusion subnetwork to concatenate the multidimensional image feature vectors, the position encoding information and attribute encoding information of each feature point to obtain the fused feature vector.

[0031] Step 17: Input the fused feature vector into the first classification network of the skull image classification model, and classify the skull image into an abnormal skull image with lesion features or a normal skull image without lesion features based on the fused feature vector.

[0032] This embodiment can be executed by an electronic device, which can be a computer, mobile phone, tablet computer, server, or medical terminal, etc., and this embodiment is not limited thereto. The electronic device may be equipped with a skull image classification model, which can be used to classify skull images into abnormal skull images with lesion features or normal skull images without lesion features. For example... Figure 2 As shown, the skull image classification model may include at least: an information encoding network, an encoding information fusion network, and a first classification network. The information encoding network may consist of a first information encoding subnetwork and a second information encoding subnetwork, and the encoding information fusion network may consist of a feature extraction subnetwork and an information fusion subnetwork.

[0033] In this embodiment, the electronic device can acquire a user's skull image and the user's attribute information. The skull image can be either a three-dimensional or two-dimensional image; this embodiment is not limited to either. The attribute information describes the user's personal attributes and includes at least: category representation information and growth period representation information.

[0034] Among them, the category representation information is used to represent the category to which the user belongs. This category representation information can be implemented as gender information, and the growth period representation information can be implemented as age information or bone age information, etc. This embodiment does not impose any restrictions.

[0035] Optionally, the original skull image acquired by the electronic device can be preprocessed to transform the pixel value of each voxel in the original skull image to a grayscale image range of 0-255 to obtain the skull image.

[0036] An electronic device can input skull images and attribute information into the information encoding network of a pre-trained skull image classification model. Using a first information encoding sub-network, the position of each feature point within the skull region of the skull image is encoded to obtain the positional encoding information of each feature point. The positional encoding information of any feature point represents its relative position within the skull region. In this way, the position of each feature point within the skull region can be encoded into positional encoding information that is recognizable by the skull image classification model.

[0037] The electronic device can utilize a second information encoding subnetwork to encode the category representation information and growth period representation information in the attribute information to obtain attribute encoded information. Specifically, the electronic device can use a preset encoding method to encode the category representation information and growth period representation information in the attribute information respectively, and obtain the attribute encoded information based on the results of the information encoding.

[0038] After the information encoding process of the aforementioned information encoding network, the electronic device can input the location encoding information and attribute encoding information of each feature point into the encoding information fusion network of the skull image classification model, so as to extract multidimensional image feature vectors from the skull image using the feature extraction subnetwork. These multidimensional image feature vectors can be used to identify lesion features; that is, they can characterize high-level semantic features in the skull image.

[0039] After extracting multidimensional image feature vectors using the feature extraction subnetwork, electronic devices can use the information fusion subnetwork to concatenate the multidimensional image feature vectors, the location encoding information of each feature point, and the attribute encoding information to obtain a fused feature vector.

[0040] Since the multidimensional image feature vector is a high-dimensional tensor, the electronic device can first use the Flatten (compression) function to stretch / flatten the multidimensional image feature vector into a one-dimensional vector, and then concatenate this one-dimensional vector with the position encoding information and attribute encoding information of each feature point to obtain the fused feature vector.

[0041] Based on the above steps, the electronic device can input the stitched fused feature vector into the first classification network of the skull image classification model, and classify the skull image into an abnormal skull image with lesion features or a normal skull image without lesion features according to the fused feature vector. Specifically, the electronic device can utilize the fully connected layer in the first classification network to perform binary classification of the skull image based on the fused feature vector.

[0042] In this embodiment, skull images and attribute information can be acquired. The attribute information includes category representation information and growth period representation information. Using a skull image classification model, the positions of feature points in the skull region of the skull image are encoded to obtain positional encoding information. The category representation information and growth period representation information in the attribute information are encoded to obtain attribute encoding information. A multidimensional image feature vector is extracted from the skull image, and the multidimensional image feature vector, positional encoding information, and attribute encoding information are concatenated. The skull image is classified based on the fused feature vector obtained from the concatenation. In this way, a fused feature vector can be obtained based on the category representation information, growth period representation information, and skull image, and the lesion features in the skull image can be identified more accurately based on the fused feature vector, thus classifying the skull image.

[0043] In some optional embodiments, the skull image classification model may further include at least one of the following networks: a second classification network and a third classification network. Based on this, the skull image classification model can not only classify skull images as abnormal or normal skull images, but also identify the categories of lesion features and anatomical structures in abnormal skull images. This will be further explained below in conjunction with Embodiment 1 and Embodiment 2.

[0044] Implementation Method 1: When a skull image is classified as an abnormal skull image with lesion features, the electronic device can input the fused feature vector into a second classification network, identify the category of the lesion features based on the fused feature vector, and output the identified category of the lesion features.

[0045] Taking fracture-related lesion features as an example, the lesion features can be categorized based on fracture morphology, including the following categories: linear fracture, open depressed fracture, closed depressed fracture, penetrating fracture, and comminuted fracture. Taking lung-related lesion features as an example, the lesion features can be pulmonary nodules or pulmonary masses, etc. It should be noted that the above are merely illustrative examples, and this embodiment does not limit the lesion features or their categories.

[0046] The electronic device can utilize the fully connected layer in the second classification network to perform multi-classification of the skull image based on the fused feature vector, so as to output the category of the identified lesion features.

[0047] Implementation Method 2: When a skull image is classified as an abnormal skull image with lesion features, the electronic device can input the fused feature vector into a third classification network, identify the anatomical structure category of the lesion features based on the fused feature vector, and output the identified anatomical structure category of the lesion features.

[0048] The anatomical structure category of a lesion feature is used to characterize the anatomical location involved in the lesion feature. Taking a lesion feature related to a fracture as an example, the human skeleton includes multiple bones, and the anatomical structure category of the lesion feature can be used to characterize the bone in the human skeleton involved in the lesion. In other words, the anatomical structure category of the lesion feature can be used to indicate which bone among multiple bones the lesion specifically involves.

[0049] It should be noted that the above implementation methods one and two can be executed individually or in combination, and this embodiment does not impose any restrictions. In this way, after classifying skull images into abnormal skull images or normal skull images, the skull image classification model can also identify the category of lesion features and anatomical structure category in abnormal skull images.

[0050] In some optional embodiments, step 13 in the foregoing embodiments, "using the first information encoding sub-network to encode the position of each feature point in the skull region of the skull image on the skull region to obtain the position encoding information of each feature point," can be implemented based on the following steps:

[0051] Step 131: In the first information coding sub-network, a spherical coordinate system is established with the center point of the skull image as the origin to obtain the spherical coordinates of each feature point on the skull region. After establishing the spherical coordinate system, each feature point on the skull region can be described by spherical coordinates (R, θ, φ), where R represents the Euclidean distance of each feature point from the origin, θ is the polar angle of each feature point relative to the origin, and φ is the azimuth angle of each feature point relative to the origin.

[0052] Step 132: Encode the two angle information in the spherical coordinates of each feature point using the first trigonometric function to obtain the positional encoding information of each feature point. The two angle information in the spherical coordinates of each feature point can be the polar angle θ of each feature point relative to the origin and the azimuth angle φ of each feature point relative to the origin, as mentioned earlier.

[0053] Specifically, electronic devices can utilize The two angles in the spherical coordinates of each feature point are encoded to obtain the positional encoding information of each feature point. Here, θ is the polar angle of each feature point mentioned earlier, and φ is the azimuth angle of each feature point mentioned earlier.

[0054] In this way, electronic devices can encode the two angles in the spherical coordinates of each feature point more accurately. The resulting positional encoding information of each feature point is more convenient for the skull image classification model to recognize, thereby improving the subsequent processing efficiency of the skull image classification model.

[0055] In some optional embodiments, step 14 in the foregoing embodiments, "using the second information coding sub-network to encode the category representation information and growth period representation information in the attribute information to obtain attribute coding information," can be implemented based on the following steps:

[0056] Step 141: In the second coding sub-network, the growth period representation information is encoded using the second trigonometric function to obtain the growth period coded information. Specifically, using... Information encoding is performed on the growth period representation information to obtain growth period coded information. Here, A represents the growth period representation information. Taking user age as an example, when the user is 5 years old, the growth period coded information obtained by encoding this growth period representation information is:

[0057] Step 142: Encode the category representation information into a target encoding value according to the category representation information. The target encoding value can be a preset first encoding value or a second encoding value, where the first encoding value corresponds to the first category and the second encoding value corresponds to the second category.

[0058] Taking category representation information as an example of representing a user's gender, if the gender represented by the category representation information is male, the electronic device can encode the category representation information as -1 (that is, the first encoding value corresponding to the first category); if the gender represented by the category representation information is female, the electronic device can encode the category representation information as 1 (that is, the second encoding value corresponding to the second category).

[0059] Step 143: Generate attribute coding information based on the growth period coding information and the target coding value. The electronic device can concatenate the growth period coding information and the target coding value to obtain the attribute coding information.

[0060] Through steps 141-143 above, the electronic device can accurately encode the category representation information and growth period representation information in the attribute information. The resulting attribute encoding information is more convenient for the skull image classification model to recognize, thereby improving the subsequent processing efficiency of the skull image classification model.

[0061] In some optional embodiments, step 16 in the foregoing embodiments, "extracting multidimensional image feature vectors from skull images for identifying lesion features using a feature extraction subnetwork," can be implemented based on the following steps:

[0062] Step 161: Starting from the upper left corner of the skull image, move the sampled image block of size M sequentially to the lower right corner, with each move being a distance of K pixels.

[0063] It should be noted that, since skull images may have high resolution, electronic devices can utilize a preset sliding window to progressively move and sample image blocks of size M from the upper left corner to the lower right corner of the skull image. This step aims to sample each image block of the skull image progressively, thereby achieving a more accurate sampling of the entire skull image.

[0064] Where M is not greater than half the size of the skull image, and M can be set according to actual design requirements, such as 32*32*32, 16*16*16, etc., this embodiment does not impose any restrictions. Where K is a natural number greater than 1, and K can be set according to actual design requirements, such as 8 or 16, etc., this embodiment does not impose any restrictions.

[0065] Step 162: For each sampled image patch, if it appears in the skull region, extract the multidimensional image feature vector for identifying lesion features from the image patch; if it does not appear in the skull region, skip the image patch. In this way, during the progressive sampling process, the electronic device can skip image patches in non-skull regions and extract multidimensional image feature vectors from image patches in the skull region.

[0066] Correspondingly, embodiments of this application also provide a method for training a skull image classification model, such as... Figure 3 As shown, methods for training skull image classification models may include:

[0067] Step 31: Obtain the user's skull image sample, the user's attribute information sample, and the lesion feature labels corresponding to the skull image sample. The attribute information sample includes at least: category representation information sample and growth period representation information sample.

[0068] Step 32: Input the skull image sample, the user's attribute information sample, and the lesion feature label corresponding to the skull image sample into the preset skull image classification network; the skull image classification network includes: an information encoding network, an encoding information fusion network, and a first classification network; the information encoding network includes a first information encoding subnetwork and a second information encoding subnetwork; the encoding information fusion network includes a feature extraction subnetwork and an information fusion subnetwork.

[0069] Step 33: In any training round, input the skull image sample and attribute information sample into the information encoding network. Use the first information encoding sub-network to encode the position of each feature point in the skull region of the skull image sample to obtain the position encoding information of each feature point. Use the second information encoding sub-network to encode the category representation information sample and growth period representation information sample in the attribute information sample to obtain the attribute encoding information.

[0070] Step 34: Input the location encoding information and attribute encoding information of each feature point into the encoding information fusion network. Use the feature extraction sub-network to extract the multi-dimensional image feature vector for identifying lesion features from the skull image sample. Then, use the information fusion sub-network to concatenate the multi-dimensional image feature vector, the location encoding information and attribute encoding information of each feature point to obtain the fused feature vector.

[0071] Step 35: Input the fused feature vector into the first classification network, classify the skull image samples into abnormal skull images with lesion features or normal skull images without lesion features according to the fused feature vector, and calculate the first prediction residual according to the classification result and lesion feature label.

[0072] Step 36: With the goal of the first prediction residual converging to the specified range, update the current parameters of the skull image classification network until the first prediction residual converges to the specified range, then stop training and output the skull image classification model.

[0073] This embodiment can be executed by an electronic device, which can be a computer, mobile phone, tablet computer, or medical terminal, etc. This embodiment is not limited to such devices.

[0074] The skull image sample can be implemented as a three-dimensional image or a two-dimensional image; this embodiment is not limited to either. The attribute information sample is used to describe the user's personal attributes, and this attribute information sample includes at least: a category representation information sample and a growth period representation information sample. The category representation information sample is used to represent the user's category, and this category representation information sample can be implemented as gender information, while the growth period representation information sample can be implemented as age information or bone age information, etc.; this embodiment is not limited to either.

[0075] The electronic device can acquire the raw skull image and respond to the operator's annotation operation for each cross section of the raw skull image. The lesion is segmented and annotated pixel by pixel on each cross section of the raw skull image, thereby obtaining the lesion feature label corresponding to the skull image sample.

[0076] Among them, multidimensional image feature vectors can be used to identify lesion features, that is, multidimensional image feature vectors can characterize high-level semantic features in skull images.

[0077] In step 35, the classification result is used to characterize whether the skull image sample is an abnormal skull image with lesion features. In practical scenarios, the classification result can be implemented in the form of a probability. For example, if the classification result is 60%, which is greater than the preset probability threshold of 50%, then the classification result can characterize the skull image sample as an abnormal skull image with lesion features. The electronic device can use a preset first loss function to calculate the first prediction residual based on the classification result and the lesion feature label. The first loss function can be used to evaluate the error between the classification result predicted by the skull image classification network in the current iteration and the lesion feature label. The first loss function can be implemented as a cross-entropy loss function. In step 36, the specified range can be set according to actual design requirements, such as 0.1 or 0.2, etc. This embodiment does not limit this.

[0078] In this embodiment, the electronic device can train the skull image classification network more accurately and efficiently based on the user's skull image samples, the user's attribute information samples, and the lesion feature labels corresponding to the skull image samples, so as to obtain a skull image classification model.

[0079] In some optional embodiments, the skull image classification network may further include at least one of the following classification networks: a second classification network and a third classification network. The skull image classification model training method also includes at least one of the following implementation methods:

[0080] In the third implementation method, when a skull image sample is classified as an abnormal skull image with lesion features, the electronic device can acquire the lesion category label corresponding to the skull image sample, input the fused feature vector into the second classification network, identify the category of the lesion feature based on the fused feature vector, and calculate the second prediction residual based on the identified lesion feature category and the lesion category label; with the goal of the second prediction residual converging to a specified range, the current parameters of the second classification network are updated until the second prediction residual converges to the specified range, at which point training stops and a new skull image classification model is output. The second classification network in the newly trained skull image classification model can have the ability to identify the category of lesion features.

[0081] Implementation Method 4: When a skull image sample is classified as an abnormal skull image with lesion features, the electronic device can acquire the anatomical structure category label corresponding to the skull image sample, input the fused feature vector into the second classification network, identify the anatomical structure category based on the fused feature vector, and calculate the third prediction residual based on the identified anatomical structure category and the anatomical structure category label. The parameters of the third classification network are updated with the goal of the third prediction residual converging to a specified range. Training stops when the third prediction residual converges to the specified range, and a new skull image classification model is output. The third classification network in the newly trained skull image classification model has the ability to identify anatomical structure categories.

[0082] In this way, the trained skull image classification model can be used to identify the categories of lesion features and anatomical structures in abnormal skull images.

[0083] Optionally, the first information coding sub-network is used to encode the position of each feature point in the skull region of the skull image to obtain the position coding information of each feature point. This includes: establishing a spherical coordinate system with the center point of the skull image sample as the origin in the first information coding sub-network to obtain the spherical coordinates of each feature point in the skull region; and encoding the two angle information in the spherical coordinates of each feature point using a first trigonometric function to obtain the position coding information of each feature point.

[0084] Optionally, the first trigonometric function is used to encode the two angles in the spherical coordinates of each feature point to obtain the position encoding information of each feature point, including: using... The two angles in the spherical coordinates of each feature point are encoded to obtain the position encoding information of each feature point; where θ is the polar angle of each feature point and φ is the azimuth angle of each feature point.

[0085] Optionally, the second information encoding subnetwork is used to encode the category representation information samples and growth period representation information samples in the attribute information samples to obtain attribute encoding information. This includes: in the second encoding subnetwork, using a second trigonometric function to encode the growth period representation information samples to obtain growth period encoding information; encoding the category representation information samples into target encoding values ​​according to the categories they represent, where the target encoding value is a preset first encoding value or a second encoding value, where the first encoding value corresponds to the first category and the second encoding value corresponds to the second category; and generating attribute encoding information based on the growth period encoding information and the target encoding value.

[0086] Optionally, the second trigonometric function is used to encode the growth stage representation information sample to obtain the growth stage encoded information, including: using... Information encoding is performed on the growth period characterization information sample to obtain growth period encoded information; where A is the growth period characterization information sample.

[0087] Optionally, a feature extraction subnetwork is used to extract a multidimensional image feature vector for identifying lesion features from the skull image sample, including: starting from the upper left corner of the skull image sample, gradually moving the sampled image blocks of size M to the lower right corner, with each movement distance being K pixels; for each sampled image block, if it appears in the skull region, then extract the multidimensional image feature vector for identifying lesion features from the image block; if it does not appear in the skull region, then skip the image block.

[0088] It should be noted that the execution subject of each step of the method provided in the above embodiments can be the same device, or the method can be executed by different devices. For example, the execution subject of steps 11 to 14 can be device A; or the execution subject of steps 11 and 12 can be device A, and the execution subject of steps 13 and 14 can be device B; and so on.

[0089] Furthermore, some processes described in the above embodiments and accompanying drawings include multiple operations that appear in a specific order. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or they may be executed in parallel. The operation numbers, such as 11, 12, etc., are merely used to distinguish different operations and do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel.

[0090] It should be noted that the terms "first" and "second" in this article are used to distinguish different messages, devices, modules, etc., and do not represent a chronological order, nor do they limit "first" and "second" to different types.

[0091] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an exemplary embodiment of this application, as shown below. Figure 4 As shown, the electronic device includes a memory 401 and a processor 402.

[0092] Memory 401 is used to store computer programs and can be configured to store various other data to support operation on the terminal device. Examples of this data include instructions for any application or method used to operate on the terminal device, contact data, phone book data, messages, pictures, videos, etc.

[0093] In some embodiments, the processor 402, coupled to the memory 401, is configured to execute a computer program in the memory 401 for: acquiring a user's skull image and the user's attribute information, the attribute information including at least: category representation information and growth period representation information; inputting the skull image and attribute information into an information encoding network in a pre-trained skull image classification model, the information encoding network including a first information encoding subnetwork and a second information encoding subnetwork; using the first information encoding subnetwork to encode the position of each feature point in the skull region of the skull image to obtain position encoding information for each feature point; and using the second information encoding subnetwork to encode the category representation information and growth period representation information in the attribute information. Information is encoded to obtain attribute encoding information. The location encoding information and attribute encoding information of each feature point are input into the encoding information fusion network in the skull image classification model. The encoding information fusion network includes a feature extraction subnetwork and an information fusion subnetwork. The feature extraction subnetwork is used to extract multi-dimensional image feature vectors for identifying lesion features from the skull image. The information fusion subnetwork is used to concatenate the multi-dimensional image feature vector, the location encoding information and attribute encoding information of each feature point to obtain a fused feature vector. The fused feature vector is input into the first classification network in the skull image classification model. Based on the fused feature vector, the skull image is classified into an abnormal skull image with lesion features or a normal skull image without lesion features.

[0094] Optionally, the skull image classification model further includes: a second classification network and / or a third classification network; the processor 402 is further configured to: when the skull image is classified as an abnormal skull image with lesion features, input the fused feature vector into the second classification network, identify the category of the lesion features based on the fused feature vector, and output the category of the identified lesion features; and / or, input the fused feature vector into the third classification network, identify the anatomical structure category of the lesion features based on the fused feature vector, and output the anatomical structure category of the identified lesion features.

[0095] Optionally, when the processor 402 uses the first information coding sub-network to encode the position of each feature point in the skull region of the skull image to obtain the position coding information of each feature point, it specifically performs the following: in the first information coding sub-network, a spherical coordinate system is established with the center point of the skull image as the origin to obtain the spherical coordinates of each feature point in the skull region; the two angle information in the spherical coordinates of each feature point are encoded using the first trigonometric function to obtain the position coding information of each feature point.

[0096] Optionally, the processor 402 uses the first trigonometric function to encode the two angle information in the spherical coordinates of each feature point, specifically for obtaining the position encoding information of each feature point: using The two angles in the spherical coordinates of each feature point are encoded to obtain the position encoding information of each feature point; where θ is the polar angle of each feature point and φ is the azimuth angle of each feature point.

[0097] Optionally, when the processor 402 uses the second information encoding sub-network to encode the category representation information and growth period representation information in the attribute information to obtain attribute encoded information, it specifically performs the following: in the second encoding sub-network, it uses the second trigonometric function to encode the growth period representation information to obtain growth period encoded information; according to the category represented by the category representation information, it encodes the category representation information into a target encoded value, the target encoded value being a preset first encoded value or a second encoded value, the first encoded value corresponding to the first category, and the second encoded value corresponding to the second category; and generates attribute encoded information based on the growth period encoded information and the target encoded value.

[0098] Optionally, the processor 402 uses the second trigonometric function to encode the growth period characterization information. When obtaining the growth period encoded information, it is specifically used to: utilize... Information encoding is performed on the growth period characterization information to obtain growth period coded information; where A is the growth period characterization information.

[0099] Optionally, when the processor 402 extracts a multidimensional image feature vector for identifying lesion features from the skull image using the feature extraction sub-network, it specifically performs the following: starting from the upper left corner of the skull image, it sequentially moves the sampled image blocks of size M to the lower right corner, with each move being a distance of K pixels; for each sampled image block, if it appears in the skull region, the multidimensional image feature vector for identifying lesion features is extracted from the image block; if it does not appear in the skull region, the image block is skipped.

[0100] In other embodiments, processor 402, coupled to memory 401, is used to execute a computer program in memory 401 for: acquiring a user's skull image sample, a user's attribute information sample, and lesion feature labels corresponding to the skull image sample; the attribute information sample includes at least: category representation information sample and growth period representation information sample; inputting the skull image sample, the user's attribute information sample, and the lesion feature labels corresponding to the skull image sample into a preset skull image classification network; the skull image classification network includes: an information encoding network, an encoding information fusion network, and a first classification network; the information encoding network includes a first information encoding subnetwork and a second information encoding subnetwork; the encoding information fusion network includes a feature extraction subnetwork and an information fusion subnetwork; in any training round, the skull image sample and attribute information sample are input into the information encoding network, and the first information encoding subnetwork is used to encode the position of each feature point in the skull region of the skull image sample to obtain each feature point. The system first encodes the location information; then, using a second information encoding subnetwork, it encodes the category representation information samples and growth period representation information samples in the attribute information samples to obtain attribute encoding information; the location encoding information and attribute encoding information of each feature point are input into the encoding information fusion network, and the feature extraction subnetwork extracts multi-dimensional image feature vectors for identifying lesion features from the skull image samples, and the information fusion subnetwork concatenates the multi-dimensional image feature vectors, the location encoding information and attribute encoding information of each feature point to obtain a fused feature vector; the fused feature vector is input into the first classification network, and the skull image samples are classified into abnormal skull images with lesion features or normal skull images without lesion features according to the fused feature vector, and the first prediction residual is calculated according to the classification result and the lesion feature label; the parameters of the skull image classification network are updated with the goal of the first prediction residual converging to a specified range, and training stops when the first prediction residual converges to the specified range, and the skull image classification model is output.

[0101] Furthermore, such as Figure 4 As shown, the electronic device also includes other components such as a communication component 403 and a display component 404. Figure 4 The diagram only shows some components and does not mean that the electronic device includes only these components. Figure 4 The components shown.

[0102] The above Figure 4 The memory in the memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0103] The above Figure 4 The communication component is configured to facilitate wired or wireless communication between the device containing the communication component and other devices. The device containing the communication component can access wireless networks based on communication standards, such as WiFi, 2G, 3G, 4G, or 5G, or combinations thereof. In one exemplary embodiment, the communication component receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the communication component may be implemented based on Near Field Communication (NFC), Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wideband (UWB), Bluetooth (BT), and other technologies.

[0104] In this embodiment, the electronic device can acquire skull images and attribute information, including category representation information and growth period representation information. Using a skull image classification model, the positions of feature points in the skull region of the skull image are encoded to obtain positional encoding information. The category representation information and growth period representation information in the attribute information are encoded to obtain attribute encoding information. A multidimensional image feature vector is extracted from the skull image, and the multidimensional image feature vector, positional encoding information, and attribute encoding information are concatenated. The skull image is classified based on the fused feature vector obtained from the concatenation. In this way, a fused feature vector can be obtained based on the category representation information, growth period representation information, and skull image, and the lesion features in the skull image can be identified more accurately based on the fused feature vector, thus classifying the skull image.

[0105] Accordingly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed, can perform the steps that can be executed by an electronic device in the above method embodiments.

[0106] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0107] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0108] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0109] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0110] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0111] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0112] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0113] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0114] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. An image classification method, characterized by, The method comprises the following steps: obtaining a skull image of a user and attribute information of the user, the attribute information at least comprising category representation information and growth period representation information; inputting the skull image and the attribute information into an information encoding network in a pre-trained skull image classification model, the information encoding network comprising a first information encoding subnetwork and a second information encoding subnetwork; in the first information encoding subnetwork, a spherical coordinate system is established with a center point of the skull image as the origin to obtain spherical coordinates of each feature point on the skull region; two angle information in the spherical coordinates of each feature point are encoded by using a first trigonometric function to obtain position encoding information of each feature point; in the second information encoding subnetwork, the growth period representation information is encoded by using a second trigonometric function to obtain growth period encoding information; the category representation information is encoded into a target encoding value according to the category represented by the category representation information, the target encoding value being a pre-set first encoding value or a second encoding value, the first encoding value corresponding to a first category and the second encoding value corresponding to a second category; the attribute encoding information is generated according to the growth period encoding information and the target encoding value; the position encoding information of each feature point and the attribute encoding information are input into an encoding information fusion network in the skull image classification model, the encoding information fusion network comprising a feature extraction subnetwork and an information fusion subnetwork; a multi-dimensional image feature vector for identifying lesion features is extracted from the skull image by using the feature extraction subnetwork, and the multi-dimensional image feature vector, the position encoding information of each feature point and the attribute encoding information are spliced by using the information fusion subnetwork to obtain a fusion feature vector; the fusion feature vector is input into a first classification network in the skull image classification model, and the skull image is classified as an abnormal skull image with lesion features or a normal skull image without lesion features according to the fusion feature vector.

2. The method of claim 1, wherein, The skull image classification model further comprises a second classification network and / or a third classification network; the method further comprises: in the case where the skull image is classified as an abnormal skull image with lesion features, the fusion feature vector is input into the second classification network to identify the category of the lesion features according to the fusion feature vector, and the identified category of the lesion features is output; and / or, the fusion feature vector is input into the third classification network to identify the anatomical structure category of the lesion features according to the fusion feature vector, and the identified anatomical structure category of the lesion features is output.

3. The method of claim 1, wherein, the two angle information in the spherical coordinates of each feature point are encoded by using a first trigonometric function to obtain the position encoding information of each feature point, comprising: Utilizing two angle information in the spherical coordinates of each feature point is encoded to obtain the position encoding information of each feature point; wherein, is the polar angle of said each feature point and φ is the azimuth angle of said each feature point.

4. The method of claim 1, wherein, the growth period representation information is encoded by using a second trigonometric function to obtain the growth period encoding information, comprising: Utilizing informationally encoding the growth period characterization information to obtain growth period encoded information; wherein A is the growth period characterization information.

5. The method of claim 1, wherein, the multi-dimensional image feature vector for identifying lesion features is extracted from the skull image by using the feature extraction subnetwork, comprising: Start from the upper left corner of the skull image and move step by step to the lower right corner, each time moving a distance of K pixels, to sample an image block of size M; For each sampled image block, if it appears in the skull region, extract a multi-dimensional image feature vector for identifying lesion features from the image block, if it does not appear in the skull region, skip the image block. 6.A method for training a skull image classification model, characterized in that, Comprise: Obtain a skull image sample of a user, an attribute information sample of the user, and a lesion feature label corresponding to the skull image sample; The attribute information sample at least includes: a category representation information sample and a growth period representation information sample; Input the skull image sample, the attribute information sample of the user, and the lesion feature label corresponding to the skull image sample into a preset skull image classification network; the skull image classification network comprises: an information encoding network, an encoded information fusion network, and a first classification network; the information encoding network comprises a first information encoding subnetwork and a second information encoding subnetwork; the encoded information fusion network comprises a feature extraction subnetwork and an information fusion subnetwork; In any training round, input the skull image sample and the attribute information sample into the information encoding network, establish a spherical coordinate system with the center point of the skull image as the origin in the first information encoding subnetwork to obtain the spherical coordinates of each feature point on the skull region; encode two angle information in the spherical coordinates of each feature point using a first trigonometric function to obtain the position encoding information of each feature point; in the second information encoding subnetwork, encode the growth period representation information using a second trigonometric function to obtain growth period encoding information; encode the category representation information into a target encoding value according to the category represented by the category representation information, the target encoding value being a preset first encoding value or a second encoding value, the first encoding value corresponding to a first category, and the second encoding value corresponding to a second category; generate the attribute encoding information according to the growth period encoding information and the target encoding value; Input the position encoding information of each feature point and the attribute encoding information into the encoded information fusion network, extract a multi-dimensional image feature vector for identifying lesion features from the skull image sample using the feature extraction subnetwork, and splice the multi-dimensional image feature vector, the position encoding information of each feature point, and the attribute encoding information using the information fusion subnetwork to obtain a fusion feature vector; Input the fusion feature vector into the first classification network, classify the skull image sample into an abnormal skull image with lesion features or a normal skull image without lesion features according to the fusion feature vector, and calculate a first prediction residual according to the classification result and the lesion feature label; Update the current parameters of the skull image classification network until the first prediction residual converges to a specified range, stop training when the first prediction residual converges to the specified range, and output a skull image classification model.

7. An electronic device, comprising: Comprise: a memory and a processor; wherein the memory is configured to store one or more computer instructions; and the processor is configured to execute the one or more computer instructions to perform the steps in the method of any one of claims 1-5 or claim 6.

8. A computer-readable storage medium, characterized in that, The computer program, when executed by a processor, causes the processor to be capable of performing the steps in the method of any one of claims 1-5 or claim 6.

Images