Image recognition method, device, electronic device and storage medium

Abstract

The present disclosure provides an image recognition method, device, electronic device and storage medium, the method comprising: acquiring at least one three-dimensional image of a target object; compressing each of the three-dimensional images to obtain a first target image and a second target image related to each of the three-dimensional images; determining a target category of the three-dimensional image based on the first target image and the second target image; wherein the target category comprises a first category for indicating a phase to which the three-dimensional image belongs, and / or a second category for indicating an imaging method of the three-dimensional image.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to an image recognition method, device, electronic device and storage medium. Background Art

[0002] Dynamic Contrast Enhanced (DCE) computed tomography (CT) and dynamic contrast enhanced magnetic resonance imaging (MRI) are commonly used imaging technologies in clinical tumor diagnosis today. They can not only improve the sensitivity of tumor detection, but also the tumor hemodynamic information they reflect is of great significance for improving the accuracy of tumor qualitative diagnosis (for example, the diagnosis results can be benign, malignant, primary or secondary, etc.).

[0003] Generally, in order to ensure the accuracy of diagnosis, it is necessary to accurately identify the different phases of dynamic enhanced CT or dynamic enhanced MRI. Summary of the invention

[0004] In view of this, the present disclosure at least provides an image recognition method, device, electronic device and storage medium.

[0005] In a first aspect, the present disclosure provides an image recognition method, comprising:

[0006] acquiring at least one three-dimensional image of a target object;

[0007] Performing compression processing on each of the three-dimensional images to obtain a first target image and a second target image related to each of the three-dimensional images;

[0008] Determining an object category of the three-dimensional image based on the first object image and the second object image;

[0009] The target category includes a first category for indicating the phase to which the three-dimensional image belongs, and / or a second category for indicating the imaging mode of the three-dimensional image.

[0010] In the above method, by compressing the three-dimensional image, a first target image and a second target image related to each three-dimensional image are obtained, and the three-dimensional image is compressed into a two-dimensional image, which reduces the data volume of the image and reduces the redundant information in the three-dimensional image. Then, based on the first target image and the second target image related to the three-dimensional image, when determining the target category corresponding to the three-dimensional image, the efficiency of image recognition can be improved.

[0011] In a possible implementation manner, the compressing each of the three-dimensional images to obtain a first target image and a second target image related to each of the three-dimensional images includes:

[0012] For each of the three-dimensional images, dividing the three-dimensional image into a plurality of three-dimensional image blocks according to a set cross-sectional direction;

[0013] Each of the three-dimensional image blocks is compressed respectively to obtain a first target image and a second target image corresponding to each of the three-dimensional image blocks; wherein the first target image is obtained by taking the average value of pixel information of each pixel projected to the same pixel position after projecting each pixel in the three-dimensional image block according to the cross-sectional direction; and the second target image is obtained by taking the maximum value of pixel information of each pixel projected to the same pixel position;

[0014] The first target image and the second target image corresponding to each three-dimensional image block are determined as the first target image and the second target image related to the three-dimensional image.

[0015] In the disclosed embodiment, each three-dimensional image is divided into a plurality of three-dimensional image blocks according to the set cross-sectional direction, and each three-dimensional image block is compressed to obtain a first target image and a second target image corresponding to the three-dimensional image block; wherein the pixel information of each pixel point in the first target image and the second target image is determined according to the pixel information of each pixel point in the three-dimensional image block, that is, the first target image and the second target image retain the image features used for image recognition in the three-dimensional image block. Then, when the target category corresponding to the three-dimensional image is determined based on the first target image and the second target image related to the three-dimensional image, the recognition process takes into account both accuracy and efficiency, and the image recognition effect is better.

[0016] In a possible implementation manner, determining the target category of the three-dimensional image based on the first target image and the second target image includes:

[0017] Performing feature processing on the first target image and the second target image corresponding to the at least one three-dimensional image to obtain target feature data corresponding to each of the three-dimensional images;

[0018] Extracting features from the target feature data corresponding to each three-dimensional image to generate local feature data corresponding to each three-dimensional image;

[0019] Based on the local feature data corresponding to the three-dimensional image, an object category corresponding to the three-dimensional image is determined.

[0020] In a possible implementation manner, determining the target category corresponding to the three-dimensional image based on the local feature data corresponding to the three-dimensional image includes:

[0021] Performing global feature extraction on the local feature data corresponding to the three-dimensional image to obtain global feature data corresponding to the three-dimensional image;

[0022] Based on the local feature data and the global feature data corresponding to the three-dimensional image, obtaining fused feature data corresponding to the three-dimensional image;

[0023] Based on the fused feature data, the target category corresponding to the three-dimensional image is determined.

[0024] Taking into account the differences in features corresponding to different target categories, and the possible differences in features corresponding to different target users under the same target category, local feature data and global feature data of the three-dimensional image can be extracted here to enrich the features of the extracted three-dimensional image, and then based on the fusion feature data with more comprehensive feature information, the target category of the three-dimensional image can be determined more accurately.

[0025] In a possible implementation manner, performing global feature extraction on the local feature data corresponding to the three-dimensional image to obtain the global feature data corresponding to the three-dimensional image includes:

[0026] Using a plurality of global feature extraction methods, respectively extracting global features from the local feature data to obtain a plurality of global feature data corresponding to the three-dimensional image;

[0027] The obtaining, based on the local feature data and the global feature data corresponding to the three-dimensional image, fused feature data corresponding to the three-dimensional image includes:

[0028] Determining residual feature data between the local feature data and each of the global feature data;

[0029] The local feature data corresponding to the three-dimensional image and a plurality of the residual feature data are concatenated to obtain fused feature data corresponding to the three-dimensional image.

[0030] Here, setting a variety of global feature extraction methods can enrich the feature information of the global feature data and reduce the omission of feature information.

[0031] In a possible implementation manner, performing feature processing on the first target image and the second target image corresponding to the at least one three-dimensional image to obtain target feature data corresponding to each of the three-dimensional images includes:

[0032] Performing feature extraction on the first target image and the second target image corresponding to each of the three-dimensional images to obtain intermediate feature data corresponding to the three-dimensional image;

[0033] Based on the intermediate feature data, determining region boundary information; wherein the region boundary information is used to characterize a target region where the target object is located in the three-dimensional image;

[0034] The intermediate feature data is intercepted according to the region boundary information to obtain target feature data corresponding to the three-dimensional image.

[0035] In the above method, the region boundary information can be determined based on the intermediate feature data, and the region boundary information is used to characterize the target region where the target object is located in the three-dimensional image, that is, the feature data corresponding to the target region where the target object is located in the intermediate feature data is the more important feature data, and the feature data other than the feature data corresponding to the target region in the intermediate feature data is the unimportant feature data. Therefore, the intermediate feature data can be intercepted according to the region boundary information, and the other unimportant feature data can be screened out to obtain the target feature data corresponding to the three-dimensional image, so that the target category of the three-dimensional image can be determined relatively quickly based on the target feature data. At the same time, screening out other unimportant feature data can alleviate the interference of other feature data on target category recognition, thereby improving the accuracy of target category recognition.

[0036] In a possible implementation manner, after obtaining the intermediate feature data corresponding to the three-dimensional image, the method further includes:

[0037] Based on the intermediate feature data, determining scale information representing a degree of completeness of the target object in the three-dimensional image;

[0038] The step of intercepting the intermediate feature data according to the region boundary information to obtain target feature data corresponding to the three-dimensional image includes:

[0039] When the ratio information is greater than or equal to a set ratio threshold, the intermediate feature data is intercepted according to the region boundary information to obtain target feature data corresponding to the three-dimensional image.

[0040] In the above method, the proportion information representing the completeness of the target object in the three-dimensional image can be determined. When the proportion information is greater than or equal to the set proportion threshold, it represents that the detected target object is relatively complete. When the completeness of the target object is large, it represents that the image quality of the three-dimensional image is good. It can then be used to extract target feature data and identify the target category.

[0041] In a possible implementation manner, the method further includes:

[0042] When the ratio information is smaller than the set ratio threshold, first prompt information is generated to prompt that the target object of the three-dimensional image is incomplete.

[0043] In the above method, when the target object is less complete, the information of the target object in the three-dimensional image is seriously missing, and the image quality of the three-dimensional image is poor. At this time, there is no need to identify the target category, and a first prompt information can be generated to indicate that the detected target object is incomplete so that the detection personnel can re-detect the target object.

[0044] In a possible implementation manner, intercepting the intermediate feature data according to the region boundary information to obtain target feature data corresponding to the three-dimensional image includes:

[0045] According to the region boundary information, the intermediate feature data is intercepted to obtain feature data to be processed corresponding to the three-dimensional image;

[0046] The feature data to be processed is resized to obtain target feature data corresponding to the three-dimensional image.

[0047] In a possible implementation manner, the target category corresponding to the three-dimensional image is determined based on a target neural network; the target neural network is trained by the following steps:

[0048] Acquire a training sample set, wherein the training sample set includes sample image sets corresponding to a plurality of sample objects respectively, each sample image set includes at least one sample image group corresponding to the sample object, and the sample image group includes a first sample image and a second sample image;

[0049] Using the sample image set corresponding to the sample object, training a neural network to obtain a trained neural network;

[0050] If the neural network after this training does not meet the set cutoff condition, return to the step of using the sample image set corresponding to the sample object to train the neural network to obtain the neural network after this training;

[0051] When the neural network after this training meets the set cutoff condition, the neural network after this training is used as the target neural network.

[0052] In a possible implementation manner, the using the sample image set corresponding to the sample object to train the neural network to obtain the trained neural network includes:

[0053] Select a first sample object from the training sample set, input a first sample image set corresponding to the first sample object into the neural network, train the neural network once, and obtain a neural network after the training; or

[0054] A first sample object is selected from the training sample set, and a target sample image group is selected from the sample image group included in the first sample image set corresponding to the first sample object, the target sample image group is input into the neural network, and the neural network is trained once to obtain the neural network after this training.

[0055] In order to enable the target neural network to more accurately determine the target category of the target object, the first sample image set corresponding to the first sample object can be input into the neural network, and the neural network can be trained once to obtain the trained neural network. At the same time, in order to improve the robustness of the neural network when the number of sample image groups corresponding to the sample objects is small, here, a target sample image group can be selected, and the neural network can be trained once using the selected target sample image group to obtain the trained neural network.

[0056] In a possible implementation manner, after determining the target category of the three-dimensional image, the method further includes:

[0057] Based on the determined target category corresponding to each of the three-dimensional images, determining whether a target category set corresponding to the three-dimensional image matches a set standard target category set;

[0058] If there is no match, a second prompt message for the target object is generated to prompt that the three-dimensional image corresponding to the target object is incorrect.

[0059] Here, after obtaining the target category of the three-dimensional image, it is possible to determine whether the target category set corresponding to the three-dimensional image matches the set standard target category set based on the target category corresponding to each three-dimensional image. If there is a mismatch, a second prompt message can be generated to prompt that the three-dimensional image corresponding to the target object is incorrect, so that the detection personnel can re-detect the target object.

[0060] In a possible implementation manner, after determining the target category of the three-dimensional image, the method further includes:

[0061] The display screen is controlled to display the three-dimensional image matching the target category in the target display area corresponding to each target category.

[0062] Here, the display screen can be controlled to display the three-dimensional image matching the target category in the target display area corresponding to each target category, without manually controlling the target display area of ​​the display screen to display the corresponding three-dimensional image, thereby improving the display efficiency of the three-dimensional image.

[0063] The effects of the following devices, electronic devices, etc. are described in the description of the above method and will not be repeated here.

[0064] In a second aspect, the present disclosure provides an image recognition device, comprising:

[0065] An acquisition module, used to acquire at least one three-dimensional image of a target object;

[0066] A compression module, used for compressing each of the three-dimensional images to obtain a first target image and a second target image related to each of the three-dimensional images;

[0067] a determination module, configured to determine a target category of the three-dimensional image based on the first target image and the second target image;

[0068] The target category includes a first category for indicating the phase to which the three-dimensional image belongs, and / or a second category for indicating the imaging mode of the three-dimensional image.

[0069] In a possible implementation manner, the compression module, when compressing each of the three-dimensional images to obtain the first target image and the second target image related to each of the three-dimensional images, is used to:

[0070] For each of the three-dimensional images, dividing the three-dimensional image into a plurality of three-dimensional image blocks according to a set cross-sectional direction;

[0071] Each of the three-dimensional image blocks is compressed respectively to obtain a first target image and a second target image corresponding to each of the three-dimensional image blocks; wherein the first target image is obtained by taking the average value of pixel information of each pixel projected to the same pixel position after projecting each pixel in the three-dimensional image block according to the cross-sectional direction; and the second target image is obtained by taking the maximum value of pixel information of each pixel projected to the same pixel position;

[0072] The first target image and the second target image corresponding to each three-dimensional image block are determined as the first target image and the second target image related to the three-dimensional image.

[0073] In a possible implementation manner, when determining the target category of the three-dimensional image based on the first target image and the second target image, the determination module is configured to:

[0074] Performing feature processing on the first target image and the second target image corresponding to the at least one three-dimensional image to obtain target feature data corresponding to each of the three-dimensional images;

[0075] Extracting features from the target feature data corresponding to each three-dimensional image to generate local feature data corresponding to each three-dimensional image;

[0076] Based on the local feature data corresponding to the three-dimensional image, an object category corresponding to the three-dimensional image is determined.

[0077] In a possible implementation manner, the determination module, when determining the target category corresponding to the three-dimensional image based on the local feature data corresponding to the three-dimensional image, is configured to:

[0078] Performing global feature extraction on the local feature data corresponding to the three-dimensional image to obtain global feature data corresponding to the three-dimensional image;

[0079] Based on the local feature data and the global feature data corresponding to the three-dimensional image, obtaining fused feature data corresponding to the three-dimensional image;

[0080] Based on the fused feature data, the target category corresponding to the three-dimensional image is determined.

[0081] In a possible implementation manner, the determination module, when performing global feature extraction on the local feature data corresponding to the three-dimensional image to obtain the global feature data corresponding to the three-dimensional image, is used to:

[0082] Using a plurality of global feature extraction methods, respectively extracting global features from the fused feature data to obtain a plurality of global feature data corresponding to the three-dimensional image;

[0083] The determining module, when obtaining the fused feature data corresponding to the three-dimensional image based on the local feature data and the global feature data corresponding to the three-dimensional image, is used to:

[0084] Determining residual feature data between the local feature data and each of the global feature data;

[0085] The local feature data corresponding to the three-dimensional image and a plurality of the residual feature data are concatenated to obtain fused feature data corresponding to the three-dimensional image.

[0086] In a possible implementation manner, the determination module, when performing feature processing on the first target image and the second target image corresponding to the at least one three-dimensional image to obtain target feature data corresponding to each of the three-dimensional images, is used to:

[0087] Performing feature extraction on the first target image and the second target image corresponding to each of the three-dimensional images to obtain intermediate feature data corresponding to the three-dimensional image;

[0088] Based on the intermediate feature data, determining region boundary information; wherein the region boundary information is used to characterize a target region where the target object is located in the three-dimensional image;

[0089] The intermediate feature data is intercepted according to the region boundary information to obtain target feature data corresponding to the three-dimensional image.

[0090] In a possible implementation manner, after obtaining the intermediate feature data corresponding to the three-dimensional image, the determination module is further configured to:

[0091] Based on the intermediate feature data, determining scale information representing a degree of completeness of the target object in the three-dimensional image;

[0092] The determination module, when intercepting the intermediate feature data according to the region boundary information to obtain the target feature data corresponding to the three-dimensional image, is used to:

[0093] When the ratio information is greater than or equal to a set ratio threshold, the intermediate feature data is intercepted according to the region boundary information to obtain target feature data corresponding to the three-dimensional image.

[0094] In a possible implementation manner, the device further includes: a first generating module, configured to:

[0095] When the ratio information is smaller than the set ratio threshold, first prompt information is generated to prompt that the target object of the three-dimensional image is incomplete.

[0096] In a possible implementation manner, the determination module, when intercepting the intermediate feature data according to the region boundary information to obtain the target feature data corresponding to the three-dimensional image, is used to:

[0097] According to the region boundary information, the intermediate feature data is intercepted to obtain feature data to be processed corresponding to the three-dimensional image;

[0098] The feature data to be processed is resized to obtain target feature data corresponding to the three-dimensional image.

[0099] In a possible implementation manner, the target category corresponding to the three-dimensional image is determined based on a target neural network; the device further includes a training module for training the target neural network through the following steps:

[0100] Acquire a training sample set, wherein the training sample set includes sample image sets corresponding to a plurality of sample objects respectively, each sample image set includes at least one sample image group corresponding to the sample object, and the sample image group includes a first sample image and a second sample image;

[0101] Using the sample image set corresponding to the sample object, training a neural network to obtain a trained neural network;

[0102] If the neural network after this training does not meet the set cutoff condition, return to the step of using the sample image set corresponding to the sample object to train the neural network to obtain the neural network after this training;

[0103] When the neural network after this training meets the set cutoff condition, the neural network after this training is used as the target neural network.

[0104] In a possible implementation manner, the training module, when training a neural network using the sample image set corresponding to the sample object to obtain the trained neural network, is used to:

[0105] Select a first sample object from the training sample set, input a first sample image set corresponding to the first sample object into the neural network, train the neural network once, and obtain a neural network after the training; or

[0106] A first sample object is selected from the training sample set, and a target sample image group is selected from the sample image group included in the first sample image set corresponding to the first sample object, the target sample image group is input into the neural network, and the neural network is trained once to obtain the neural network after this training.

[0107] In a possible implementation manner, after determining the target category corresponding to the three-dimensional image, the method further includes: a second generating module configured to:

[0108] Based on the determined target category corresponding to each of the three-dimensional images, determining whether a target category set corresponding to the three-dimensional image matches a set standard target category set;

[0109] If there is no match, second prompt information for the target object is generated to prompt that the three-dimensional image corresponding to the target object is incorrect.

[0110] In a possible implementation manner, after determining the target category corresponding to the three-dimensional image, the method further includes: a control module configured to:

[0111] The display screen is controlled to display the three-dimensional image matching the target category in the target display area corresponding to each target category.

[0112] In a third aspect, the present disclosure provides an electronic device comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the image recognition method described in the first aspect or any one of the embodiments are performed.

[0113] In a fourth aspect, the present disclosure provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the image recognition method as described in the first aspect or any one of the embodiments above are executed.

[0114] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0115] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following is a brief introduction to the drawings required for use in the embodiments. The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used together with the specification to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can also be obtained based on these drawings without creative work.

[0116] Figure 1 A schematic diagram showing a flow chart of an image recognition method provided by an embodiment of the present disclosure;

[0117] Figure 2 A schematic diagram of a three-dimensional image in an image recognition method provided by an embodiment of the present disclosure is shown;

[0118] Figure 3 A schematic diagram showing a region boundary on intermediate feature data in an image recognition method provided by an embodiment of the present disclosure;

[0119] Figure 4 A schematic diagram showing a flow chart of another image recognition method provided by an embodiment of the present disclosure;

[0120] Figure 5 A schematic diagram of the structure of a target neural network in an image recognition method provided by an embodiment of the present disclosure is shown;

[0121] Figure 6 A schematic diagram of the structure of a target neural network mid-phase recognition module provided by an embodiment of the present disclosure is shown;

[0122] Figure 7 A schematic diagram of the architecture of an image recognition device provided by an embodiment of the present disclosure is shown;

[0123] Figure 8 A schematic structural diagram of an electronic device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0124] In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The components of the embodiments of the present disclosure generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the present disclosure for protection, but merely represents the selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present disclosure.

[0125] Dynamic Contrast Enhanced (DCE) computed tomography (CT) and dynamic contrast enhanced magnetic resonance imaging (MRI) are commonly used imaging technologies in clinical tumor diagnosis today. They can not only improve the sensitivity of tumor detection, but also reflect the tumor hemodynamic information, which is of great significance for improving the accuracy of tumor qualitative diagnosis (such as the diagnosis result can be benign, malignant, primary or secondary, etc.). In order to ensure the accuracy of diagnosis, it is necessary to accurately identify the different phases of dynamic contrast enhanced CT or dynamic contrast enhanced MRI.

[0126] In one method, the detected dynamic enhanced CT image (or MRI image) can be input into the trained neural network to identify the phases corresponding to different dynamic enhanced CT images. For example, the phases of the dynamic enhanced CT image can include the plain scan phase, arterial phase, portal phase, delayed phase, etc. However, since the dynamic enhanced CT image is a three-dimensional image, the information of the three-dimensional image is relatively complex, so when the neural network directly recognizes the dynamic enhanced CT image, the network has a large amount of calculation and low recognition efficiency. In addition, there is a large amount of redundant information in the dynamic enhanced CT image, which will cause a waste of network computing resources.

[0127] In order to alleviate the above problems, the embodiments of the present disclosure provide an image recognition method, device, electronic device and storage medium.

[0128] The defects existing in the above solutions are the results obtained by the inventor after practice and careful research. Therefore, the discovery process of the above problems and the solutions proposed by the present disclosure for the above problems below should be the contributions made by the inventor to the present disclosure during the disclosure process.

[0129] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0130] It should be noted that: if the technical solution of this application involves personal information, the product using the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing personal information. If the technical solution of this application involves sensitive personal information, the product using the technical solution of this application has obtained the individual's separate consent before processing sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, clear and prominent signs are set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that he or she agrees to collect his or her personal information; or on the device for processing personal information, when the personal information processing rules are notified by obvious signs / information, the individual's authorization is obtained through pop-up information or by asking the individual to upload his or her personal information; among them, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

[0131] To facilitate understanding of the embodiments of the present disclosure, an image recognition method disclosed in the embodiments of the present disclosure is first introduced in detail. The execution subject of the image recognition method provided in the embodiments of the present disclosure may be a server, a terminal device, etc., wherein the server may include a local server, a cloud server, etc., and the terminal device may include a computer, a tablet, a mobile device, etc.

[0132] See also Figure 1 FIG. 1 is a flow chart of an image recognition method provided by an embodiment of the present disclosure, the method comprising S101-S103, wherein:

[0133] S101, acquiring at least one three-dimensional image of a target object;

[0134] S102, performing compression processing on each of the three-dimensional images to obtain a first target image and a second target image related to each of the three-dimensional images;

[0135] S103, determining a target category of the three-dimensional image based on the first target image and the second target image, wherein the target category includes a first category for indicating a phase to which the three-dimensional image belongs, and / or a second category for indicating an imaging method of the three-dimensional image.

[0136] In the above method, by compressing the three-dimensional image, a first target image and a second target image related to each three-dimensional image are obtained, and the three-dimensional image is compressed into a two-dimensional image, which reduces the data volume of the image and reduces the redundant information in the three-dimensional image. Then, based on the first target image and the second target image related to the three-dimensional image, when determining the target category corresponding to the three-dimensional image, the efficiency of image recognition can be improved.

[0137] S101 to S103 are described in detail below.

[0138] For S101:

[0139] The target object can be an organ to be detected of any target user. The target user can be any patient undergoing image scanning and detection, and the organs to be detected can include, for example, the head, liver, pancreas, etc., wherein the image scanning and detection can include contrast computed tomography (CT), contrast magnetic resonance imaging (MRI), dynamic enhanced CT, dynamic enhanced MRI, etc.

[0140] After performing image scanning and detection on the target object, one or more three-dimensional images corresponding to the target object may be generated. The three-dimensional image may be a dynamic enhanced CT image, or an MRI image, and the like.

[0141] During implementation, the target object can be subjected to CT detection at different time points to obtain a three-dimensional image corresponding to the target object at that time point, and then multiple three-dimensional images corresponding to the target object can be generated. For example, the target object can be subjected to image scanning detection before the contrast agent is injected into the target object to obtain the first three-dimensional image; after the contrast agent is injected into the target object, the target object can be subjected to image scanning detection at an interval of 5 seconds from the time of injection of the contrast agent to obtain the second three-dimensional image; the target object can be subjected to image scanning detection at an interval of 1 minute from the time of injection of the contrast agent to obtain the third three-dimensional image, etc.

[0142] For S102:

[0143] During implementation, for each three-dimensional image, the three-dimensional image may be compressed according to the set cross-sectional direction to obtain a first target image and a second target image related to the three-dimensional image. For example, each pixel point on the three-dimensional image may be projected along the set cross-sectional direction, and the pixel information of each pixel point projected to the same pixel point position may be averaged to obtain the first target image related to the three-dimensional image, and the pixel information of each pixel point projected to the same pixel point position may be maximized to obtain the second target image related to the three-dimensional image.

[0144] Alternatively, for each three-dimensional image, multiple frames of two-dimensional cross-sectional images can be selected from the three-dimensional image according to the set cross-sectional direction, and the pixel information at the same pixel position in the selected multiple frames of two-dimensional cross-sectional images can be averaged to obtain the first target image, and the pixel information at the same pixel position can be maximized to obtain the second target image.

[0145] In an optional implementation manner, in S102, the compressing each of the three-dimensional images to obtain the first target image and the second target image related to each of the three-dimensional images may include:

[0146] Step 1: for each of the three-dimensional images, divide the three-dimensional image into a plurality of three-dimensional image blocks according to a set cross-sectional direction;

[0147] Step 2: compress each of the three-dimensional image blocks to obtain a first target image and a second target image corresponding to each of the three-dimensional image blocks; wherein the first target image is obtained by taking the average value of pixel information of each pixel projected to the same pixel position after projecting each pixel in the three-dimensional image block according to the cross-sectional direction; and the second target image is obtained by taking the maximum value of pixel information of each pixel projected to the same pixel position;

[0148] Step three: determine the first target image and the second target image corresponding to each three-dimensional image block as the first target image and the second target image related to the three-dimensional image.

[0149] In the disclosed embodiment, each three-dimensional image is divided into a plurality of three-dimensional image blocks according to the set cross-sectional direction, and each three-dimensional image block is compressed to obtain a first target image and a second target image corresponding to the three-dimensional image block; wherein the pixel information of each pixel point in the first target image and the second target image is determined according to the pixel information of each pixel point in the three-dimensional image block, that is, the first target image and the second target image retain the image features used for image recognition in the three-dimensional image block. Then, when the target category corresponding to the three-dimensional image is determined based on the first target image and the second target image related to the three-dimensional image, the recognition process takes into account both accuracy and efficiency, and the image recognition effect is better.

[0150] During implementation, the three-dimensional image can be divided into a plurality of three-dimensional image blocks according to the set cross-sectional direction. The number of three-dimensional image blocks can be set according to actual needs, for example, the number of three-dimensional image blocks can be 3, 5, etc. Preferably, the three-dimensional image can be divided into 5 three-dimensional image blocks. Among them, the cross-sectional direction can be a direction matching the coronal plane. Figure 2 As shown, the Figure 2 The cross section 21 is included, that is, the cross section 21 can be a plane formed by the x-axis and the z-axis, and the plane direction where the cross section is located can be the cross section direction.

[0151] During implementation, for each 3D image block, the 3D image can be projected in the cross-sectional direction to obtain a first target image and a second target image. The first target image is obtained by taking the average value of the pixel information of each pixel point projected to the same pixel point position in the 3D image block, and the second target image is obtained by taking the maximum value of the pixel information of each pixel point projected to the same pixel point position in the 3D image block. Figure 2 Taking the cross-sectional direction in as an example, each pixel point in the three-dimensional image block can be projected along the y-axis direction. In the projection process, the average value of each pixel point at the same pixel position is taken to obtain the first target image, and the maximum value of each pixel point at the same pixel position is taken to obtain the second target image. The same pixel position means that the pixel position of each projection point on the projection plane (i.e., the plane formed by the x-axis and the z-axis) is the same.

[0152] For S103:

[0153] For each three-dimensional image, based on the first target image and the second target image corresponding to the three-dimensional image, the target category corresponding to the three-dimensional image is determined, wherein the target category includes a first category for indicating the phase to which the three-dimensional image belongs, and / or a second category for indicating the imaging mode of the three-dimensional image.

[0154] The types and quantities of the first category and the second category can be set according to actual conditions. For example, when the three-dimensional image is a dynamic enhanced CT image, the first category may include: plain scan phase, arterial phase, portal phase, delayed phase; when the three-dimensional image is an MRI image, the first category of the phase may include: early arterial phase, late arterial phase, portal phase, delayed phase, equilibrium phase, hepatobiliary phase, etc.; the second category may include: T1W, T2W, diffusion weighted imaging, surface diffusion coefficient imaging, etc.

[0155] During implementation, the region boundary information of the region where the target object is located in the first target image and the second target image can be determined, and the first target image and the second target image can be intercepted using the determined region boundary information to obtain the intercepted first target image and the intercepted second target image, and then the intercepted first target image and the intercepted second target image can be used to determine the target category corresponding to the three-dimensional image. Among them, the region boundary information of the region where the target object is located in the first target image and the second target image can be determined using the first neural network for information extraction.

[0156] Exemplarily, the target neural network can be used to determine the target category corresponding to the three-dimensional image based on the first target image and the second target image. Alternatively, the first preset target image and the second preset target image corresponding to each preset category can be predetermined; then the first target image is compared with the first preset target images corresponding to each preset category to obtain the first similarity between the first target image and each first preset target image, and the second target image is compared with the second preset target images corresponding to each preset category to obtain the second similarity between the second target image and each second preset target image; finally, based on the first similarity and the second similarity, the target category corresponding to the three-dimensional image is determined; for example, a judgment condition can be set: the first similarity is greater than the first threshold value and the second similarity is greater than the second threshold value, and the preset category that meets the above judgment condition is determined as the target category corresponding to the three-dimensional image.

[0157] In an optional implementation, determining the target category corresponding to the three-dimensional image based on the first target image and the second target image includes steps A1 to A3 (not shown), wherein:

[0158] Step A1, performing feature processing on the first target image and the second target image corresponding to the at least one three-dimensional image to obtain target feature data corresponding to each of the three-dimensional images;

[0159] Step A2, extracting features from the target feature data corresponding to each three-dimensional image to generate local feature data corresponding to each three-dimensional image;

[0160] Step A3: determining the target category corresponding to the three-dimensional image based on the local feature data corresponding to the three-dimensional image.

[0161] In step A1, in one implementation, feature extraction may be performed on the first target image and the second target image corresponding to the three-dimensional image to obtain intermediate feature data corresponding to each three-dimensional image, and the obtained intermediate feature data may then be used as target feature data corresponding to the three-dimensional image.

[0162] Exemplarily, the first target image and the second target image can be cascaded in the channel direction as a multi-channel target image. For example, if the first target image is a 3-channel image and the second target image is a 3-channel image, the first target image and the second target image can be cascaded to obtain a 6-channel target image, and then feature processing is performed on the 6-channel target image to obtain target feature data corresponding to each three-dimensional image.

[0163] In implementation, if the three-dimensional image is divided to obtain n three-dimensional image blocks, then after compression, n first target images and n second target images corresponding to the three-dimensional image can be obtained, and the n first target images and n second target images can be cascaded to obtain 2n target images, and features of the 2n target images are extracted to obtain target feature data corresponding to the three-dimensional image. Where n is a positive integer greater than 1.

[0164] In another embodiment, in step A1, the feature processing of the first target image and the second target image corresponding to the at least one three-dimensional image to obtain target feature data corresponding to each of the three-dimensional images may include steps A11 to A13 (not shown), wherein:

[0165] Step A11, performing feature extraction on the first target image and the second target image corresponding to each of the three-dimensional images to obtain intermediate feature data corresponding to the three-dimensional image;

[0166] Step A12, determining region boundary information based on the intermediate feature data; wherein the region boundary information is used to characterize the target region where the target object is located in the three-dimensional image;

[0167] Step A13: intercepting the intermediate feature data according to the region boundary information to obtain target feature data corresponding to the three-dimensional image.

[0168] In the above method, the region boundary information can be determined based on the intermediate feature data, and the region boundary information is used to characterize the target region where the target object is located in the three-dimensional image, that is, the feature data corresponding to the target region where the target object is located in the intermediate feature data is the more important feature data, and the feature data other than the feature data corresponding to the target region in the intermediate feature data is the unimportant feature data. Therefore, the intermediate feature data can be intercepted according to the region boundary information, and the other unimportant feature data can be screened out to obtain the target feature data corresponding to the three-dimensional image, so that the target category of the three-dimensional image can be determined relatively quickly based on the target feature data. At the same time, screening out other unimportant feature data can alleviate the interference of other feature data on target category recognition, thereby improving the accuracy of target category recognition.

[0169] In step A11, feature extraction can be performed on the first target image and the second target image corresponding to the three-dimensional image to obtain intermediate feature data corresponding to the three-dimensional image. When using the target neural network to determine the target category corresponding to the three-dimensional image, the backbone network in the target neural network can be used to extract features from the first target image and the second target image to obtain intermediate feature data corresponding to the three-dimensional image. For example, the network structure of the backbone network can be an EfficientNet network structure. Among them, since the number of three-dimensional images obtained by scanning each sample object image is small, the BatchNormalization layer in the EfficientNet network structure can be replaced with a GroupNormalization layer to improve the robustness of the target neural network.

[0170] In step A12, when implemented, the region boundary information can be regressed based on the intermediate feature data, and the region boundary information is used to characterize the target region where the target object is located in the three-dimensional image. For example, the region boundary information may include upper boundary information corresponding to the first object and lower boundary information corresponding to the second object. The first object and the second object can be set according to actual conditions.

[0171] Exemplarily, when the target object is the liver, the upper boundary corresponding to the first object may include the upper boundary corresponding to the liver apex and / or the upper boundary corresponding to the spleen, etc. For example, the higher boundary of the upper boundary corresponding to the liver apex and the upper boundary corresponding to the spleen may be selected as the upper boundary corresponding to the first object, that is, when the upper boundary corresponding to the liver apex is higher than the upper boundary corresponding to the spleen, the upper boundary corresponding to the liver apex is used as the upper boundary of the first object. The lower boundary corresponding to the second object may include the lower boundary corresponding to the kidney and / or the lower boundary corresponding to the liver, etc. For example, the lower boundary of the lower boundary corresponding to the kidney and the lower boundary corresponding to the liver may be selected as the lower boundary corresponding to the second object, that is, when the lower boundary corresponding to the kidney is lower than the lower boundary corresponding to the liver, the lower boundary corresponding to the kidney is used as the lower boundary of the second object.

[0172] The upper boundary information may be the first ordinate value of the upper boundary in the coordinate system corresponding to the intermediate feature data, and the lower boundary information may be the second ordinate value of the lower boundary in the coordinate system corresponding to the intermediate feature data. Figure 3 As shown, the figure shows the regional boundary information on the intermediate feature data, and the regional boundary information includes an upper boundary line 31 corresponding to the upper boundary information and a lower boundary line 32 corresponding to the lower boundary information. That is, the upper boundary information can be the first ordinate value on the z-axis, and the lower boundary information can be the second ordinate value on the z-axis. The feature data between the upper boundary line 31 and the lower boundary line 32 can be the intercepted feature data to be processed.

[0173] In step A13, in an optional implementation, the intercepting the intermediate feature data according to the area boundary information to obtain the target feature data corresponding to the three-dimensional image may include: intercepting the intermediate feature data according to the area boundary information to obtain the feature data to be processed corresponding to the three-dimensional image; and resizing the feature data to be processed to obtain the target feature data corresponding to the three-dimensional image.

[0174] Here, the intermediate feature data may be intercepted according to the region boundary information to obtain the feature data to be processed corresponding to the three-dimensional image. The feature data to be processed may be feature data in the intermediate feature data located between the upper boundary and the lower boundary indicated by the region boundary information.

[0175] The feature data to be processed can then be resized to obtain target feature data at a target size, where the target size can be set as needed. During implementation, a ROI Pooling processing layer can be used to resize the feature data to be processed to obtain target feature data at a target size.

[0176] In an optional implementation, after obtaining the intermediate feature data corresponding to the three-dimensional image, the method further includes: determining, based on the intermediate feature data, scale information representing a degree of completeness of the target object in the three-dimensional image;

[0177] The intercepting the intermediate feature data according to the region boundary information to obtain the target feature data corresponding to the three-dimensional image may include: when the ratio information is greater than or equal to a set ratio threshold, intercepting the intermediate feature data according to the region boundary information to obtain the target feature data corresponding to the three-dimensional image.

[0178] In the above method, the proportion information representing the completeness of the target object in the three-dimensional image can be determined. When the proportion information is greater than or equal to the set proportion threshold, it represents that the detected target object is relatively complete. When the completeness of the target object is large, it represents that the image quality of the three-dimensional image is good. It can then be used to extract target feature data and identify the target category.

[0179] After obtaining the intermediate feature data, the ratio information representing the completeness of the target object in the three-dimensional image can also be determined based on the intermediate feature data. If the ratio information is 0.8, the completeness of the target object in the three-dimensional image is 0.8; if the ratio information is 0.5, the completeness of the target object in the three-dimensional image is 0.5.

[0180] The ratio threshold can be set as needed, for example, the target threshold can be 0.7, 0.8, etc. When the ratio information is greater than or equal to the ratio threshold, the intermediate feature data is intercepted according to the region boundary information to obtain the target feature data corresponding to the three-dimensional image; when the ratio information is less than the ratio threshold, the intermediate feature data can be screened out and the subsequent process of determining the target category is not performed.

[0181] For example, if the target category of the three-dimensional image is determined by the trained target neural network, in the training process of the target neural network, adding a proportional regression branch corresponding to the determined proportional information can be used as an auxiliary task, and through the gradient backpropagation constraint, the target neural network can better learn the target object as a whole; in addition, the proportional regression branch can also ensure the integrity of the information input into the phase recognition module in the target neural network, and improve the stability of the model training. In the inference stage of the target neural network, the proportional regression branch can effectively eliminate the sample three-dimensional images of the target object that are not fully scanned, and play a role in image quality control access.

[0182] In one embodiment, the method further includes: when the ratio information is less than the set ratio threshold, generating first prompt information to prompt that the target object of the three-dimensional image is incomplete.

[0183] When the ratio information is less than the set ratio threshold, a first prompt message for the target object can be generated. For example, the first prompt message may be "The detected liver is incomplete, please re-scan and test". The first prompt message is used to remind medical staff performing the scanning and testing that the detected target object is incomplete when performing an image scan on the target object, so that the medical staff can re-test the target object to avoid the situation where the user to whom the target object belongs has to register twice.

[0184] In the above method, when the target object is less complete, the information of the target object in the three-dimensional image is seriously missing, and the image quality of the three-dimensional image is poor. At this time, there is no need to identify the target category, and a first prompt information can be generated to indicate that the detected target object is incomplete so that the detection personnel can re-detect the target object.

[0185] In step A2, feature extraction may be performed on target feature data corresponding to the plurality of three-dimensional images to generate local feature data corresponding to each three-dimensional image. For example, the target feature data may be feature extracted using a Reshape layer and a fully connected layer (FCLayers) to generate local feature data corresponding to the three-dimensional image.

[0186] In step A3, in an optional implementation manner, determining the target category corresponding to the three-dimensional image based on the local feature data corresponding to the three-dimensional image includes steps A31 to A33 (not shown), wherein:

[0187] Step A31, performing global feature extraction on the local feature data corresponding to the three-dimensional image to obtain global feature data corresponding to the three-dimensional image;

[0188] Step A32, obtaining fused feature data corresponding to the three-dimensional image based on the local feature data and the global feature data corresponding to the three-dimensional image;

[0189] Step A33: determining the target category corresponding to the three-dimensional image based on the fused feature data.

[0190] Taking into account the differences in features corresponding to different target categories, and the possible differences in features corresponding to different target users under the same target category, local feature data and global feature data of the three-dimensional image can be extracted here to enrich the features of the extracted three-dimensional image, and then based on the fusion feature data with more comprehensive feature information, the target category of the three-dimensional image can be determined more accurately.

[0191] In an optional implementation, in step A31, performing global feature extraction on local feature data corresponding to the three-dimensional image to obtain global feature data corresponding to the three-dimensional image may include: using a plurality of set global feature extraction methods to perform global feature extraction on the local feature data respectively to obtain multiple global feature data corresponding to the three-dimensional image.

[0192] For example, the global maximum pooling layer (Global Max Pooling, GMP) and the global average pooling layer (Global average Pooling, GAP) can be used to perform global feature extraction on local feature data respectively to obtain multiple global feature data corresponding to the three-dimensional image.

[0193] Here, setting a variety of global feature extraction methods can enrich the feature information of the global feature data and reduce the omission of feature information.

[0194] Step A32, based on the local feature data and the global feature data corresponding to the three-dimensional image, obtaining the fused feature data corresponding to the three-dimensional image, may include: determining the residual feature data between the local feature data and each global feature data; splicing the local feature data corresponding to the three-dimensional image and multiple residual feature data to obtain the fused feature data corresponding to the three-dimensional image.

[0195] During implementation, each global feature data can be subtracted from the local feature data to obtain residual feature data between each global feature data and the local feature data; the residual feature data and the local feature data are then cascaded to obtain fused feature data corresponding to the three-dimensional image; in A33, the target category corresponding to the three-dimensional image can be determined based on the fused feature data.

[0196] In an optional implementation, the target category corresponding to the three-dimensional image is determined based on a target neural network; the target neural network is trained by the following steps C1-C3 (not shown):

[0197] Step C1, obtaining a training sample set, wherein the training sample set includes a plurality of sample image sets corresponding to sample objects, each sample image set includes at least one sample image group corresponding to the sample object, and the sample image group includes a first sample image and a second sample image;

[0198] Step C2, training a neural network using the sample image set corresponding to the sample object to obtain a neural network after the training;

[0199] Step C3, when the neural network after this training does not meet the set cutoff condition, return to the step of using the sample image set corresponding to the sample object to train the neural network to obtain the neural network after this training; when the neural network after this training meets the set cutoff condition, use the neural network after this training as the target neural network.

[0200] The training sample set includes sample image sets corresponding to multiple sample objects respectively, and the sample image set includes at least one sample image group corresponding to the sample object. Each sample image group includes a first sample image and a second sample image obtained by compressing a sample three-dimensional image of the sample object.

[0201] During implementation, each sample 3D image corresponding to the sample object can be divided to obtain n 3D image blocks, and then the n 3D image blocks are compressed to obtain n first sample images and n second sample images corresponding to the sample 3D image. The n first sample images and n second sample images constitute a sample image group corresponding to the sample 3D image of the sample object, and then sample image groups corresponding to multiple sample 3D images under the sample object can be obtained. The sample image groups corresponding to multiple sample objects constitute a training sample set.

[0202] Then, the sample image set corresponding to the sample object can be input into the neural network to train the neural network to obtain the trained neural network.

[0203] In an optional implementation manner, in step C2, the step of training a neural network using the sample image set corresponding to the sample object to obtain the trained neural network may include:

[0204] Method 1: selecting a first sample object from the training sample set, inputting a first sample image set corresponding to the first sample object into the neural network, training the neural network once, and obtaining a neural network after the training;

[0205] Method 2: Select a first sample object from the training sample set, and select a target sample image group from the sample image group included in the first sample image set corresponding to the first sample object, input the target sample image group into the neural network, train the neural network once, and obtain the neural network after this training.

[0206] In method 1, the sample image set corresponding to each sample object in the training sample set can be used as a mini-batch data of the neural network. For example, a sample object can be randomly selected from the training sample set as the first sample object, and the first sample image set corresponding to the first sample object can be used as the mini-batch data used in this training. The mini-batch data used in this training is input into the neural network, and the neural network is trained once to obtain the neural network after this training.

[0207] In the second method, it is considered that in actual scenarios, the three-dimensional image of the target object obtained by scanning may be incomplete. For example, in a CT image, when performing image scanning and detection on the target object, it is necessary to obtain a three-dimensional image of the target object in the plain scan period, a three-dimensional image of the arterial period, a three-dimensional image of the portal venous period, and a three-dimensional image of the delayed period. If the target object only obtains part of the three-dimensional images of the above four phases, it is determined that the three-dimensional image of the target object obtained by scanning is incomplete.

[0208] In order to enable the trained target neural network to more accurately identify the target category of the three-dimensional image when the three-dimensional image is incomplete and to improve the robustness of the target neural network, some sample image groups can be selected from the sample image groups included in the first sample image set corresponding to the first sample object as target sample image groups (the target sample image groups constitute the mini-batch data used in one training of the neural network), the target sample image groups can be input into the neural network, the neural network is trained once, and the neural network after this training is obtained.

[0209] In the case that the neural network after this training does not meet the set cutoff conditions, the network parameters of the target neural network can be updated by using the gradient back propagation method, and the neural network after the network parameters are updated can be trained again using the sample image set corresponding to the sample object until the trained neural network meets the cutoff conditions; in the case that the neural network after this training meets the set cutoff conditions, the neural network after this training is used as the target neural network.

[0210] Among them, the cutoff condition can be the convergence of the trained neural network. For example, when the recognition accuracy of the neural network after this training does not increase significantly, or when the loss value of the neural network after this training does not decrease significantly, it is determined that the neural network after this training meets the set cutoff condition.

[0211] In order to enable the target neural network to more accurately determine the target category of the target object, the first sample image set corresponding to the first sample object can be input into the neural network, and the neural network can be trained once to obtain the trained neural network. At the same time, in order to improve the robustness of the neural network when the number of sample image groups corresponding to the sample objects is small, here, a target sample image group can be selected, and the neural network can be trained once using the selected target sample image group to obtain the trained neural network.

[0212] In an optional implementation manner, after determining the target category of the three-dimensional image, the method further includes:

[0213] Based on the determined target category corresponding to each of the three-dimensional images, determining whether a target category set corresponding to the three-dimensional image matches a set standard target category set;

[0214] If there is no match, second prompt information for the target object is generated to prompt that the three-dimensional image corresponding to the target object is incorrect.

[0215] During implementation, the target categories corresponding to the determined three-dimensional images are combined to obtain a target category set corresponding to the three-dimensional image. Exemplarily, if the three-dimensional image is a dynamic enhanced CT image, the set standard target category set includes the plain scan period, arterial phase, portal phase, and delayed phase, and if the target category set corresponding to the three-dimensional image is determined to include the plain scan period, arterial phase, portal phase, and delayed phase, it is determined to match the set standard target category set; if the target category set corresponding to the three-dimensional image is determined to include some phases among the plain scan period, arterial phase, portal phase, and delayed phase, it is determined to not match the set standard target category set.

[0216] Here, after obtaining the target category of the three-dimensional image, it is possible to determine whether the target category set corresponding to the three-dimensional image matches the set standard target category set based on the target category corresponding to each three-dimensional image. If there is no match, a second prompt message can be generated to prompt that the three-dimensional image corresponding to the target object is incorrect. For example, it can be prompted that there is a missing sequence after the image scan. The second prompt message can be, for example, "missing three-dimensional image of the portal vein", so that the detection personnel can re-detect the target object.

[0217] In an optional implementation, after determining the target category of the three-dimensional image, the method further includes: controlling the display screen to display the three-dimensional image matching the target category in the target display areas corresponding to the respective target categories.

[0218] For example, when the three-dimensional image is a dynamically enhanced CT image, the upper left area on the display screen can be set to display the three-dimensional image of the plain scan period, the upper right area can be set to display the three-dimensional image of the arterial period, the lower left area can be set to display the three-dimensional image of the portal venous period, and the lower right area can be set to display the three-dimensional image of the delayed period; and after determining the target category corresponding to the three-dimensional image, the display screen can be controlled to display the three-dimensional image matching the target category in the target display area corresponding to each target category.

[0219] Here, the display screen can be controlled to display the three-dimensional image matching the target category in the target display area corresponding to each target category, without manually controlling the target display area of ​​the display screen to display the corresponding three-dimensional image, thereby improving the display efficiency of the three-dimensional image.

[0220] See also Figure 4 As shown, combined Figure 4 The image recognition method proposed in the present disclosure is described, and the method comprises the following steps:

[0221] Step 1: Acquire multiple three-dimensional images corresponding to the target object.

[0222] Right now Figure 4 The three-dimensional image 1, the three-dimensional image 2, the three-dimensional image 3, and the three-dimensional image 4 are multiple three-dimensional images corresponding to the acquired target object.

[0223] Step 2: Divide each 3D image into multiple 3D image blocks.

[0224] For example, the 3D image 1 may be divided into five 3D image blocks 1 , the 3D image 2 may be divided into five 3D image blocks 2 , the 3D image 3 may be divided into five 3D image blocks 3 , and the 3D image 4 may be divided into five 3D image blocks 4 .

[0225] Step 3: compress each three-dimensional image block to obtain a first target image and a second target image corresponding to each three-dimensional image block.

[0226] Figure 4 In the example, for the five three-dimensional image blocks 1 corresponding to the three-dimensional image 1, each three-dimensional image block 1 is compressed according to the first compression method to obtain a first target image corresponding to each three-dimensional image block 1. Five first target images corresponding to the three-dimensional image 1 (i.e. Figure 4 Based on the same method, the five first target images corresponding to the three-dimensional image 2 can be obtained (i.e. Figure 4 ), the five first target images corresponding to the three-dimensional image 3 (i.e. Figure 4 ), the five first target images corresponding to the three-dimensional image 4 (i.e. Figure 45 first target images in row 41 of ).

[0227] For the five three-dimensional image blocks 1 corresponding to the three-dimensional image 1, each three-dimensional image block 1 is compressed according to the second compression method to obtain a second target image corresponding to each three-dimensional image block 1. Five second target images corresponding to the three-dimensional image 1 (i.e. Figure 4 Based on the same method, we can obtain the five second target images corresponding to the three-dimensional image 2 (i.e. Figure 4 ), the five second target images corresponding to the three-dimensional image 3 (i.e. Figure 4 ), the five second target images corresponding to the three-dimensional image 4 (i.e. Figure 4 5 second target images in row 42 of ).

[0228] Among them, the first compression method can be to take the average value of the pixel information of each pixel point projected to the same pixel point position in the three-dimensional image block; the second compression method can be to take the maximum value of the pixel information of each pixel point projected to the same pixel point position in the three-dimensional image block.

[0229] Step 4: Input the first target image and the second target image corresponding to the multiple three-dimensional images into the target neural network for recognition to obtain the target category corresponding to each three-dimensional image.

[0230] See also Figure 5 The schematic diagram of the target neural network structure shown in Figure 1 is combined with Figure 5 Step 4 is described in detail. Step 4 may include the following steps:

[0231] Step 41: Use the backbone network Backbone in the target neural network to perform feature extraction on the first target image and the second target image corresponding to the multiple three-dimensional images to obtain intermediate feature data corresponding to each three-dimensional image. For example, the backbone network can be an EfficientNet network.

[0232] Step 42: using the boundary regression branch in the target neural network to determine the region boundary information based on the intermediate feature data, wherein the region boundary information is used to characterize the target region where the target object is located in the three-dimensional image.

[0233] Exemplarily, the method may further include step 43, determining the proportion information representing the completeness of the target object in the three-dimensional image based on the intermediate feature data. A proportion regression branch may be set in the target neural network, and the proportion regression branch and the intermediate feature data may be used to determine the proportion information representing the completeness of the target object in the three-dimensional image.

[0234] When the ratio information is greater than or equal to the set ratio threshold, the target feature data of the three-dimensional image is obtained. When the ratio information is less than the set ratio threshold, the first prompt information corresponding to the target object is generated to prompt that when the target object is image scanned, the target object detected is incomplete.

[0235] Step 44: intercept the intermediate feature data according to the region boundary information to obtain feature data to be processed corresponding to the three-dimensional image; use the regional pooling ROI Pooling layer to resize the feature data to be processed to obtain target feature data corresponding to the three-dimensional image.

[0236] Step 45: The target feature data corresponding to each three-dimensional image can be input into the phase recognition module included in the target neural network to determine the target category corresponding to each three-dimensional image, for example, including the plain scan phase, arterial phase, portal phase, and delayed phase.

[0237] See also Figure 6 As shown, combined Figure 6 The step 45 is specifically described as shown, and the specific step 45 may include the following steps:

[0238] Step 451: Use the Reshape layer and the fully connected layer (FC Layers) to extract the target feature data corresponding to the multiple three-dimensional images, and generate the local feature data corresponding to each three-dimensional image. Figure 6 The size of the local feature data shown is [B, C], where B is the number of three-dimensional images and C is the dimension of the local feature data corresponding to each three-dimensional image.

[0239] Step 452: Use the global maximum pooling layer GMP and the global average pooling layer GAP to perform global feature extraction on the local feature data to obtain first global feature data and second global feature data corresponding to the three-dimensional image.

[0240] Step 453: Determine first residual feature data between the first global feature data and the local feature data, and second residual feature data between the second global feature data and the local feature data using a network processing layer for performing residual operations. Figure 6 As shown, the dimension corresponding to the first residual feature data and the second residual feature data is C.

[0241] Step 454: Concatenate the first residual feature data and the second residual feature data with the local feature data to obtain fused feature data. The dimension of the fused feature data is 3×C.

[0242] Step 455: Use the fully connected layer and the classification softmax layer to process the fused feature data to determine the target category corresponding to each three-dimensional image.

[0243] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0244] Based on the same concept, the present disclosure also provides an image recognition device, see Figure 7 As shown, it is a schematic diagram of the architecture of the image recognition device provided by the embodiment of the present disclosure, including an acquisition module 701, a compression module 702, and a determination module 703. Specifically:

[0245] An acquisition module 701 is used to acquire at least one three-dimensional image of a target object;

[0246] A compression module 702, configured to perform compression processing on each of the three-dimensional images to obtain a first target image and a second target image related to each of the three-dimensional images;

[0247] A determination module 703, configured to determine an object category of the three-dimensional image based on the first object image and the second object image;

[0248] The target category includes a first category for indicating the phase to which the three-dimensional image belongs, and / or a second category for indicating the imaging mode of the three-dimensional image.

[0249] In a possible implementation manner, the compression module 702, when performing compression processing on each of the three-dimensional images to obtain the first target image and the second target image related to each of the three-dimensional images, is configured to:

[0250] For each of the three-dimensional images, dividing the three-dimensional image into a plurality of three-dimensional image blocks according to a set cross-sectional direction;

[0251] Each of the three-dimensional image blocks is compressed respectively to obtain a first target image and a second target image corresponding to each of the three-dimensional image blocks; wherein the first target image is obtained by taking the average value of pixel information of each pixel projected to the same pixel position after projecting each pixel in the three-dimensional image block according to the cross-sectional direction; and the second target image is obtained by taking the maximum value of pixel information of each pixel projected to the same pixel position;

[0252] The first target image and the second target image corresponding to each three-dimensional image block are determined as the first target image and the second target image related to the three-dimensional image.

[0253] In a possible implementation manner, the determination module 703, when determining the target category corresponding to the three-dimensional image based on the first target image and the second target image, is configured to:

[0254] Performing feature processing on the first target image and the second target image corresponding to the at least one three-dimensional image to obtain target feature data corresponding to each of the three-dimensional images;

[0255] Extracting features from the target feature data corresponding to each three-dimensional image to generate local feature data corresponding to each three-dimensional image;

[0256] Based on the local feature data corresponding to the three-dimensional image, an object category corresponding to the three-dimensional image is determined.

[0257] In a possible implementation manner, the determination module 703, when determining the target category corresponding to the three-dimensional image based on the local feature data corresponding to the three-dimensional image, is configured to:

[0258] Performing global feature extraction on the local feature data corresponding to the three-dimensional image to obtain global feature data corresponding to the three-dimensional image;

[0259] Based on the local feature data and the global feature data corresponding to the three-dimensional image, obtaining fused feature data corresponding to the three-dimensional image;

[0260] Based on the fused feature data, the target category corresponding to the three-dimensional image is determined.

[0261] In a possible implementation manner, the determination module 703, when performing global feature extraction on the local feature data corresponding to the three-dimensional image to obtain the global feature data corresponding to the three-dimensional image, is used to:

[0262] Using a plurality of global feature extraction methods, respectively extracting global features from the local feature data to obtain a plurality of global feature data corresponding to the three-dimensional image;

[0263] The determining module 703, when obtaining the fused feature data corresponding to the three-dimensional image based on the local feature data and the global feature data corresponding to the three-dimensional image, is used to:

[0264] Determining residual feature data between the local feature data and each of the global feature data;

[0265] The local feature data corresponding to the three-dimensional image and a plurality of the residual feature data are concatenated to obtain fused feature data corresponding to the three-dimensional image.

[0266] In a possible implementation manner, the determination module 703, when performing feature processing on the first target image and the second target image corresponding to the at least one three-dimensional image to obtain target feature data corresponding to each of the three-dimensional images, is configured to:

[0267] Performing feature extraction on the first target image and the second target image corresponding to each of the three-dimensional images to obtain intermediate feature data corresponding to the three-dimensional image;

[0268] Based on the intermediate feature data, determining region boundary information; wherein the region boundary information is used to characterize a target region where the target object is located in the three-dimensional image;

[0269] The intermediate feature data is intercepted according to the region boundary information to obtain target feature data corresponding to the three-dimensional image.

[0270] In a possible implementation manner, after obtaining the intermediate feature data corresponding to the three-dimensional image, the determination module 703 is further configured to:

[0271] Based on the intermediate feature data, determining scale information representing a degree of completeness of the target object in the three-dimensional image;

[0272] The determination module, when intercepting the intermediate feature data according to the region boundary information to obtain the target feature data corresponding to the three-dimensional image, is used to:

[0273] When the ratio information is greater than or equal to a set ratio threshold, the intermediate feature data is intercepted according to the region boundary information to obtain target feature data corresponding to the three-dimensional image.

[0274] In a possible implementation manner, the apparatus further includes: a first generating module 704, configured to:

[0275] When the ratio information is smaller than the set ratio threshold, first prompt information is generated to prompt that the target object of the three-dimensional image is incomplete.

[0276] In a possible implementation manner, the determination module 703, when intercepting the intermediate feature data according to the region boundary information to obtain the target feature data corresponding to the three-dimensional image, is configured to:

[0277] According to the region boundary information, the intermediate feature data is intercepted to obtain feature data to be processed corresponding to the three-dimensional image;

[0278] The feature data to be processed is resized to obtain target feature data corresponding to the three-dimensional image.

[0279] In a possible implementation manner, the target category corresponding to the three-dimensional image is determined based on a target neural network; the device further includes a training module 705, which is used to train the target neural network through the following steps:

[0280] Acquire a training sample set, wherein the training sample set includes sample image sets corresponding to a plurality of sample objects respectively, each sample image set includes at least one sample image group corresponding to the sample object, and the sample image group includes a first sample image and a second sample image;

[0281] Using the sample image set corresponding to the sample object, training a neural network to obtain a trained neural network;

[0282] If the neural network after this training does not meet the set cutoff condition, return to the step of using the sample image set corresponding to the sample object to train the neural network to obtain the neural network after this training;

[0283] When the neural network after this training meets the set cutoff condition, the neural network after this training is used as the target neural network.

[0284] In a possible implementation manner, the training module 705, when training a neural network using the sample image set corresponding to the sample object to obtain the trained neural network, is used to:

[0285] Select a first sample object from the training sample set, input a first sample image set corresponding to the first sample object into the neural network, train the neural network once, and obtain a neural network after the training; or

[0286] A first sample object is selected from the training sample set, and a target sample image group is selected from the sample image group included in the first sample image set corresponding to the first sample object, the target sample image group is input into the neural network, and the neural network is trained once to obtain the neural network after this training.

[0287] In a possible implementation manner, after determining the target category corresponding to the three-dimensional image, the method further includes: a second generating module 706, configured to:

[0288] Based on the determined target category corresponding to each of the three-dimensional images, determining whether a target category set corresponding to the three-dimensional image matches a set standard target category set;

[0289] If there is no match, a second prompt message for the target object is generated to prompt that the three-dimensional image corresponding to the target object is incorrect.

[0290] In a possible implementation manner, after determining the target category corresponding to the three-dimensional image, the method further includes: a control module 707 configured to:

[0291] The display screen is controlled to display the three-dimensional image matching the target category in the target display area corresponding to each target category.

[0292] In some embodiments, the functions or templates contained in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0293] Based on the same technical concept, the embodiment of the present disclosure also provides an electronic device. Figure 8 As shown, it is a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure, including a processor 801, a memory 802, and a bus 803. Among them, the memory 802 is used to store execution instructions, including a memory 8021 and an external memory 8022; the memory 8021 here is also called an internal memory, which is used to temporarily store the operation data in the processor 801, and the data exchanged with the external memory 8022 such as a hard disk. The processor 801 exchanges data with the external memory 8022 through the memory 8021. When the electronic device 800 is running, the processor 801 communicates with the memory 802 through the bus 803, so that the processor 801 executes the following instructions:

[0294] acquiring at least one three-dimensional image of a target object;

[0295] Performing compression processing on each of the three-dimensional images to obtain a first target image and a second target image related to each of the three-dimensional images;

[0296] Based on the first target image and the second target image, the target category of the three-dimensional image is determined; wherein the target category includes a first category for indicating the phase to which the three-dimensional image belongs, and / or a second category for indicating the imaging method of the three-dimensional image.

[0297] The specific processing flow of the processor 801 may refer to the description of the above method embodiment, which will not be repeated here.

[0298] In addition, the embodiment of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the image recognition method described in the above method embodiment are executed. The storage medium can be a volatile or non-volatile computer-readable storage medium.

[0299] The present disclosure also provides a computer program product that carries a program code. The program code includes instructions that can be used to execute the steps of the image recognition method described in the above method embodiment. For details, please refer to the above method embodiment, which will not be repeated here.

[0300] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is implemented as a computer storage medium. In another optional embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).

[0301] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the device or unit can be electrical, mechanical or other forms.

[0302] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0303] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0304] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0305] The above are only specific embodiments of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present disclosure, which should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. An image recognition method, characterized in that: include: acquiring at least one three-dimensional image of a target object; Performing compression processing on each of the three-dimensional images to obtain a first target image and a second target image related to each of the three-dimensional images, wherein the first target image and the second target image are two-dimensional images, and pixel information of each pixel point in the first target image and the second target image is determined according to pixel information of a corresponding pixel point in the three-dimensional image; Determining an object category of the three-dimensional image based on the first object image and the second object image; The target category includes a first category for indicating the phase to which the three-dimensional image belongs, and / or a second category for indicating the imaging mode of the three-dimensional image.

2. The method according to claim 1, characterized in that: The compressing each of the three-dimensional images to obtain a first target image and a second target image related to each of the three-dimensional images includes: For each of the three-dimensional images, dividing the three-dimensional image into a plurality of three-dimensional image blocks according to a set cross-sectional direction; Each of the three-dimensional image blocks is compressed respectively to obtain a first target image and a second target image corresponding to each of the three-dimensional image blocks; wherein the first target image is obtained by taking the average value of pixel information of each pixel projected to the same pixel position after projecting each pixel in the three-dimensional image block according to the cross-sectional direction; and the second target image is obtained by taking the maximum value of pixel information of each pixel projected to the same pixel position; The first target image and the second target image corresponding to each three-dimensional image block are determined as the first target image and the second target image related to the three-dimensional image.

3. The method according to claim 1, characterized in that The determining the target category of the three-dimensional image based on the first target image and the second target image includes: Performing feature processing on the first target image and the second target image corresponding to the at least one three-dimensional image to obtain target feature data corresponding to each of the three-dimensional images; Extracting features from the target feature data corresponding to each three-dimensional image to generate local feature data corresponding to each three-dimensional image; Based on the local feature data corresponding to the three-dimensional image, an object category corresponding to the three-dimensional image is determined.

4. The method according to claim 3, characterized in that The determining, based on the local feature data corresponding to the three-dimensional image, a target category corresponding to the three-dimensional image includes: Performing global feature extraction on the local feature data corresponding to the three-dimensional image to obtain global feature data corresponding to the three-dimensional image; Based on the local feature data and the global feature data corresponding to the three-dimensional image, obtaining fused feature data corresponding to the three-dimensional image; Based on the fused feature data, the target category corresponding to the three-dimensional image is determined.

5. The method according to claim 4, characterized in that The extracting global features from the local feature data corresponding to the three-dimensional image to obtain global feature data corresponding to the three-dimensional image includes: Using a plurality of global feature extraction methods, respectively extracting global features from the local feature data to obtain a plurality of global feature data corresponding to the three-dimensional image; The obtaining, based on the local feature data and the global feature data corresponding to the three-dimensional image, fused feature data corresponding to the three-dimensional image includes: Determining residual feature data between the local feature data and each of the global feature data; The local feature data corresponding to the three-dimensional image and a plurality of the residual feature data are concatenated to obtain fused feature data corresponding to the three-dimensional image.

6. The method according to any one of claims 3 to 5, characterized in that: The performing feature processing on the first target image and the second target image corresponding to the at least one three-dimensional image to obtain target feature data corresponding to each of the three-dimensional images includes: Performing feature extraction on the first target image and the second target image corresponding to each of the three-dimensional images to obtain intermediate feature data corresponding to the three-dimensional image; Based on the intermediate feature data, determining region boundary information; wherein the region boundary information is used to characterize a target region where the target object is located in the three-dimensional image; The intermediate feature data is intercepted according to the region boundary information to obtain target feature data corresponding to the three-dimensional image.

7. The method according to claim 6, characterized in that After obtaining the intermediate feature data corresponding to the three-dimensional image, the method further includes: Based on the intermediate feature data, determining scale information representing a degree of completeness of the target object in the three-dimensional image; The step of intercepting the intermediate feature data according to the region boundary information to obtain target feature data corresponding to the three-dimensional image includes: When the ratio information is greater than or equal to a set ratio threshold, the intermediate feature data is intercepted according to the region boundary information to obtain target feature data corresponding to the three-dimensional image.

8. The method according to claim 7, characterized in that The method further comprises: When the ratio information is smaller than the set ratio threshold, first prompt information is generated to prompt that the target object of the three-dimensional image is incomplete.

9. The method according to claim 6, characterized in that The step of intercepting the intermediate feature data according to the region boundary information to obtain target feature data corresponding to the three-dimensional image includes: According to the region boundary information, the intermediate feature data is intercepted to obtain feature data to be processed corresponding to the three-dimensional image; The feature data to be processed is resized to obtain target feature data corresponding to the three-dimensional image.

10. The method according to any one of claims 1 to 5, characterized in that: The target category corresponding to the three-dimensional image is determined based on a target neural network; the target neural network is trained by the following steps: Acquire a training sample set, wherein the training sample set includes sample image sets corresponding to a plurality of sample objects respectively, each sample image set includes at least one sample image group corresponding to the sample object, and the sample image group includes a first sample image and a second sample image; Using the sample image set corresponding to the sample object, training a neural network to obtain a trained neural network; If the neural network after this training does not meet the set cutoff condition, return to the step of using the sample image set corresponding to the sample object to train the neural network to obtain the neural network after this training; When the neural network after this training meets the set cutoff condition, the neural network after this training is used as the target neural network.

11. The method according to claim 10, characterized in that The step of training a neural network by using the sample image set corresponding to the sample object to obtain the trained neural network includes: Select a first sample object from the training sample set, input a first sample image set corresponding to the first sample object into the neural network, train the neural network once, and obtain a neural network after the training; or A first sample object is selected from the training sample set, and a target sample image group is selected from the sample image group included in the first sample image set corresponding to the first sample object, the target sample image group is input into the neural network, and the neural network is trained once to obtain the neural network after this training.

12. The method according to any one of claims 1 to 5, characterized in that: After determining the target category of the three-dimensional image, the method further includes: Based on the determined target category corresponding to each of the three-dimensional images, determining whether a target category set corresponding to the three-dimensional image matches a set standard target category set; If there is no match, second prompt information for the target object is generated to prompt that the three-dimensional image corresponding to the target object is incorrect.

13. The method according to any one of claims 1 to 5, characterized in that: After determining the target category of the three-dimensional image, the method further includes: The display screen is controlled to display the three-dimensional image matching the target category in the target display area corresponding to each target category.

14. An image recognition device, characterized in that: include: An acquisition module, used to acquire at least one three-dimensional image of a target object; a compression module, configured to perform compression processing on each of the three-dimensional images to obtain a first target image and a second target image related to each of the three-dimensional images, wherein the first target image and the second target image are two-dimensional images, and pixel information of each pixel point in the first target image and the second target image is determined according to pixel information of a corresponding pixel point in the three-dimensional image; a determination module, configured to determine a target category of the three-dimensional image based on the first target image and the second target image; The target category includes a first category for indicating the phase to which the three-dimensional image belongs, and / or a second category for indicating the imaging mode of the three-dimensional image.

15. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the image recognition method as described in any one of claims 1 to 13 are performed.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the image recognition method according to any one of claims 1 to 13 are executed.

Images