Medical information retrieval method and related equipment

Abstract

The embodiment of the invention provides a medical information retrieval method and related equipment. The medical information retrieval method comprises the steps that a medical image is input into a three-dimensional model, a target three-dimensional image is obtained, and the target three-dimensional image is a three-dimensional image containing predetermined features; inputting the target three-dimensional image into an encoder to obtain shape information and classification information; obtaining a low-dimensional vector according to the shape information and the classification information; and performing comparison in a database according to the low-dimensional vectors to obtain a predetermined number of similar three-dimensional images. According to the technical scheme provided by the embodiment of the invention, the accuracy of similar three-dimensional image retrieval can be improved, so that the problem that the retrieved medical image is not accurate enough due to the fact that a depth model at the present stage lacks utilization of key clues is solved.

Description

Technical Field

[0001] The present application relates to the field of computer and communication technology, and in particular, to a medical information retrieval method and related equipment. Background Art

[0002] With the explosive growth of medical imaging, a large number of doctors are needed to complete medical research, medical analysis and medical diagnosis based on medical imaging.

[0003] However, doctors currently have limited energy, and reading medical images takes a certain amount of time and is highly dependent on the doctor's personal experience and vision, which makes the reading results easily interfered by subjective factors. Although there are solutions to use deep models for medical image retrieval, the current deep models lack the use of key clues, resulting in inaccurate retrieved medical images, which affects medical research, medical analysis, and medical diagnosis. Summary of the invention

[0004] The embodiments of the present application provide a medical information retrieval method and related equipment, which can at least to some extent overcome the problem that the current deep model lacks the use of key clues, resulting in inaccurate retrieved medical images.

[0005] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by the practice of the present application.

[0006] According to one aspect of an embodiment of the present application, a medical information retrieval method is provided, comprising:

[0007] A medical image is input into a three-dimensional model to obtain a target three-dimensional image, wherein the target three-dimensional image is a three-dimensional image containing predetermined features; the target three-dimensional image is input into an encoder to obtain shape information and classification information; a low-dimensional vector is obtained based on the shape information and the classification information; and a comparison is performed in a database based on the low-dimensional vector to obtain a predetermined number of similar three-dimensional images.

[0008] In some embodiments of the present application, the encoder includes a spatial encoding module, a shape encoding module, and a classification encoding module, and the inputting the target three-dimensional image into the encoder to obtain shape information and classification information specifically includes: inputting the target three-dimensional image and the medical image into the spatial encoding module to obtain spatial information; inputting the target three-dimensional image and the spatial information into the shape encoding module to obtain shape information; inputting the target three-dimensional image into the classification encoding module to obtain classification information.

[0009] In some embodiments of the present application, the spatial information includes target relative coordinates and base point relative coordinates, and the inputting the target three-dimensional image and the medical image into the spatial encoding module to obtain the spatial information specifically includes: inputting the target three-dimensional image and the medical image into the spatial encoding module to obtain base point relative coordinates, the base point relative coordinates being the relative position coordinates of the base point in the target three-dimensional image in the medical image; and extracting target relative coordinates of multiple target three-dimensional images based on the base point relative coordinates, the target relative coordinates being the relative position coordinates of each target point in the target three-dimensional image in the medical image.

[0010] In some embodiments of the present application, extracting the spatial point coordinates of multiple target three-dimensional images based on the target position coordinates specifically includes: taking the base point as a reference point, extracting the corresponding target point according to a predetermined image size; and determining the target relative coordinates corresponding to the target point.

[0011] In some embodiments of the present application, the shape information includes a target feature image, and the shape encoding module of the target three-dimensional image and the spatial information obtains the shape information, specifically comprising: obtaining an intermediate feature image by subjecting the target three-dimensional image to multiple layers of convolution and pooling; and splicing and fusing the intermediate feature image and the spatial information to obtain a target feature image.

[0012] In some embodiments of the present application, the intermediate feature image includes a minimum intermediate feature image and an ordinary intermediate feature image, the target feature image includes a minimum target feature image and an ordinary target feature image, and the stitching and fusing of the intermediate feature image and the spatial information to obtain the target feature image specifically includes: stitching and fusing the minimum intermediate feature image and the corresponding spatial information to obtain the minimum target feature image; fusing the ordinary intermediate feature image, the corresponding spatial information and the minimum intermediate feature image to obtain the ordinary target feature image.

[0013] In some embodiments of the present application, the common intermediate feature image, the corresponding spatial information and the minimum intermediate feature image are fused to obtain a common target feature image, specifically including: upsampling the minimum intermediate feature image to obtain an upsampled image; fusing the common intermediate feature image, the corresponding spatial information and the upsampled image to obtain a common target feature image.

[0014] In some embodiments of the present application, the ordinary intermediate feature image, the corresponding spatial information and the upsampled image are fused to obtain an ordinary target feature image, specifically including: performing point-adding fusion processing on the ordinary intermediate feature image and the upsampled image to obtain a point-adding image; performing splicing fusion processing on the point-adding image and the corresponding spatial information to obtain an ordinary target feature image.

[0015] In some embodiments of the present application, the target three-dimensional image is subjected to multiple layers of convolution and pooling to obtain an intermediate feature image, which specifically includes: inputting the target three-dimensional image into a shape coding module, and subjecting the target three-dimensional image to multiple layers of convolution and pooling to obtain a common intermediate feature image; and subjecting the common intermediate feature image to convolution and pooling to obtain a minimum intermediate feature image.

[0016] In some embodiments of the present application, the classification information includes a residual image and a classification result, and the step of inputting the target three-dimensional image into the classification encoding module to obtain the classification information specifically includes: inputting the target three-dimensional image into a residual network to obtain a residual image; and obtaining a classification result based on the residual image.

[0017] In some embodiments of the present application, obtaining a classification result based on the residual image specifically includes: passing the residual image through an adaptive average pooling layer to obtain a pooling result; and passing the pooling result through a fully connected layer to obtain a corresponding classification result.

[0018] In some embodiments of the present application, the training method of the classification encoding module includes:

[0019] A target three-dimensional image sample set is obtained, wherein the target three-dimensional image sample set includes multiple target three-dimensional image samples, each of which has a corresponding classification label; the target three-dimensional image samples are input into the classification coding module one by one in batches for multiple times to obtain corresponding classification results; and according to the classification labels and the classification results, parameters of the classification coding module are updated until a first predetermined end condition is reached, and the training is terminated to obtain a trained classification coding module.

[0020] In some embodiments of the present application, the parameters of the classification coding module are updated according to the classification label and the classification result until a first predetermined end condition is reached, the training is terminated, and a trained classification coding module is obtained, specifically including: if the classification results of only no more than a first predetermined number of target three-dimensional image samples in the sample set are consistent with their classification labels, the parameters of the classification coding module are updated; if the classification results of more than a first predetermined number of target three-dimensional image samples in the sample set are consistent with their classification labels, the first predetermined end condition is reached, the training is terminated, and a trained classification coding module is obtained.

[0021] In some embodiments of the present application, the parameters of the classification coding module are updated according to the classification label and the classification result until a first predetermined end condition is reached, the training is terminated, and a trained classification coding module is obtained, specifically including: determining a classification loss function according to the classification label and the classification result; according to the classification loss function, the parameters of the classification coding module are updated until the first predetermined end condition is reached, the training is terminated, and a trained classification coding module is obtained.

[0022] In some embodiments of the present application, obtaining a low-dimensional vector based on the shape information and the classification information specifically includes: splicing the shape information and the classification information to obtain splicing information; and obtaining the low-dimensional vector after convolution, pooling and compression of the splicing information.

[0023] In some embodiments of the present application, the training method of the encoder includes: obtaining a target three-dimensional image sample set, the target three-dimensional image sample set including multiple target three-dimensional image samples, each of the target three-dimensional image samples having a corresponding classification label; inputting the target three-dimensional image samples into the encoder one by one in batches for multiple times to obtain corresponding low-dimensional vectors; sampling the target three-dimensional image sample set according to the classification labels to obtain triplets, each of the triplets including three target three-dimensional image samples, and only two of the three target three-dimensional image samples have the same classification label; and updating the parameters of the encoder according to the triplets until a second predetermined end condition is reached, thereby terminating the training and obtaining a trained encoder.

[0024] In some embodiments of the present application, updating the parameters of the encoder according to the triplet until a second predetermined end condition is reached, terminating the training, and obtaining a trained encoder specifically includes: determining an intra-class distance and an inter-class distance of target three-dimensional image samples in the triplet, wherein the intra-class distance is a distance between two target three-dimensional image samples with the same classification labels, and the inter-class distance is a distance between two target three-dimensional image samples with different classification labels;

[0025] The encoder is parameter updated according to the intra-class distance and the inter-class distance until a second predetermined end condition is reached, and the training is terminated to obtain a trained encoder.

[0026] In some embodiments of the present application, the updating of parameters of the encoder according to the intra-class distance and the inter-class distance until a second predetermined end condition is reached, the training is terminated, and a trained encoder is obtained, specifically comprising: if only no more than a second predetermined number of triples in the sample set have an intra-class distance less than the inter-class distance, the updating of parameters of the encoder;

[0027] If there are more than a second predetermined number of triplets in the sample set whose intra-class distance is smaller than the inter-class distance, then a second predetermined end condition is met, the training is ended, and a trained encoder is obtained.

[0028] In some embodiments of the present application, the encoder parameters are updated according to the intra-class distance and the inter-class distance until a second predetermined end condition is reached, the training is terminated, and a trained encoder is obtained, specifically including: determining a distance loss function according to the intra-class distance and the inter-class distance; according to the distance loss function, the encoder parameters are updated until the second predetermined end condition is reached, the training is terminated, and a trained encoder is obtained.

[0029] According to one aspect of an embodiment of the present application, a medical information retrieval device is provided, comprising: an image cutting module, used to input a medical image into a three-dimensional model to obtain a target three-dimensional image, wherein the target three-dimensional image is a three-dimensional image containing predetermined features; an information acquisition module, used to input the target three-dimensional image into an encoder to obtain shape information and classification information; a low-dimensional vector module, used to obtain a low-dimensional vector based on the shape information and the classification information; and an image output module, used to compare in a database based on the low-dimensional vector to obtain a predetermined number of similar three-dimensional images.

[0030] In some embodiments of the present application, the encoder includes a spatial encoding module, a shape encoding module, and a classification encoding module, and the information acquisition module specifically includes: a spatial information submodule, used to input the target three-dimensional image and the medical image into the spatial encoding module to obtain spatial information; a shape information submodule, used to input the target three-dimensional image and the spatial information into the shape encoding module to obtain shape information; and a classification information submodule, used to input the target three-dimensional image into the classification encoding module to obtain classification information.

[0031] In some embodiments of the present application, the spatial information includes target relative coordinates and base point relative coordinates, and the spatial information submodule specifically includes: a base point coordinate unit, used to input the target three-dimensional image and the medical image into the spatial encoding module to obtain the base point relative coordinates, and the base point relative coordinates are the relative position coordinates of the base point in the target three-dimensional image in the medical image; a target coordinate unit, used to extract the target relative coordinates of multiple target three-dimensional images according to the base point relative coordinates, and the target relative coordinates are the relative position coordinates of each target point in the target three-dimensional image in the medical image.

[0032] In some embodiments of the present application, the target coordinate unit is specifically used to execute: taking the base point as a reference point, extracting the corresponding target point according to a predetermined image size; and determining the target relative coordinates corresponding to the target point.

[0033] In some embodiments of the present application, the shape information includes a target feature image, and the shape information submodule specifically includes: an intermediate image submodule, used to obtain an intermediate feature image from the target three-dimensional image through multiple layers of convolution and pooling; a target image submodule, used to splice and fuse the intermediate feature image and spatial information to obtain a target feature image.

[0034] In some embodiments of the present application, the intermediate feature image includes a minimum intermediate feature image and an ordinary intermediate feature image, the target feature image includes a minimum target feature image and an ordinary target feature image, and the target image submodule specifically includes: a minimum target image unit, used to splice and fuse the minimum intermediate feature image and the corresponding spatial information to obtain a minimum target feature image; and an ordinary target image unit, used to fuse the ordinary intermediate feature image, the corresponding spatial information and the minimum intermediate feature image to obtain an ordinary target feature image.

[0035] In some embodiments of the present application, the common target image unit specifically includes: an upsampling subunit, configured to perform upsampling processing on the minimum intermediate feature image to obtain an upsampled image;

[0036] The fusion subunit is used to fuse the common intermediate feature image, the corresponding spatial information and the upsampled image to obtain a common target feature image.

[0037] In some embodiments of the present application, the fusion subunit is specifically used to perform: performing point-adding fusion processing on the ordinary intermediate feature image and the upsampled image to obtain a point-added image; performing splicing and fusion processing on the point-added image and corresponding spatial information to obtain an ordinary target feature image.

[0038] In some embodiments of the present application, the intermediate image submodule specifically includes: a common intermediate image unit, used to input the target three-dimensional image into a shape coding module, and obtain a common intermediate feature image through multiple layers of convolution and pooling; a minimum intermediate image unit, used to obtain a minimum intermediate feature image through convolution and pooling of the common intermediate feature image.

[0039] In some embodiments of the present application, the classification information includes a residual image and a classification result, and the classification information submodule specifically includes: a residual image submodule, used to input the target three-dimensional image into the residual network to obtain a residual image; and a classification result submodule, used to obtain a classification result based on the residual image.

[0040] In some embodiments of the present application, the classification result submodule specifically includes: a pooling result unit, which is used to obtain a pooling result by passing the residual image through an adaptive average pooling layer; and a classification result unit, which is used to pass the pooling result through a fully connected layer to obtain a corresponding classification result.

[0041] In some embodiments of the present application, the medical information retrieval device also includes: a first sample acquisition module, used to acquire a target three-dimensional image sample set, the target three-dimensional image sample set includes multiple target three-dimensional image samples, and each of the target three-dimensional image samples has a corresponding classification label; a first sample input module, used to input the target three-dimensional image samples in batches and multiple times one by one into the classification coding module to obtain corresponding classification results; a first parameter updating module, used to update the parameters of the classification coding module according to the classification label and the classification result until a first predetermined end condition is reached, the training is ended, and a trained classification coding module is obtained.

[0042] In some embodiments of the present application, the first parameter updating module is specifically used to execute: if the classification results of only no more than a first predetermined number of target three-dimensional image samples in the sample set are consistent with their classification labels, then the parameters of the classification coding module are updated; if the classification results of more than a first predetermined number of target three-dimensional image samples in the sample set are consistent with their classification labels, then the first predetermined ending condition is met, the training is ended, and a trained classification coding module is obtained.

[0043] In some embodiments of the present application, the first parameter updating module is specifically used to execute: determining a classification loss function based on the classification label and the classification result; updating parameters of the classification encoding module based on the classification loss function until the first predetermined end condition is reached, thereby ending the training and obtaining a trained classification encoding module.

[0044] In some embodiments of the present application, the low-dimensional vector module specifically includes: a splicing information submodule, used to splice the shape information and the classification information to obtain splicing information; a low-dimensional vector submodule, used to convolve, pool and compress the splicing information to obtain the low-dimensional vector.

[0045] In some embodiments of the present application, the medical information retrieval device also includes: a second sample acquisition module, used to acquire a target three-dimensional image sample set, the target three-dimensional image sample set includes multiple target three-dimensional image samples, and each of the target three-dimensional image samples has a corresponding classification label; a second sample input module, used to input the target three-dimensional image samples into the encoder one by one in batches multiple times to obtain corresponding low-dimensional vectors; a triplet sampling generation module, used to sample in the target three-dimensional image sample set according to the classification label to obtain a triplet, each of the triplet contains three target three-dimensional image samples, and only two of the three target three-dimensional image samples have the same classification label; a second parameter updating module, used to update the parameters of the encoder according to the triplet until the second predetermined end condition is reached, the training is terminated, and a trained encoder is obtained.

[0046] In some embodiments of the present application, the second parameter updating module specifically includes: a distance calculation unit, used to determine the intra-class distance and inter-class distance of the target three-dimensional image samples in the triplet, the intra-class distance is the distance between two target three-dimensional image samples with the same classification label, and the inter-class distance is the distance between two target three-dimensional image samples with different classification labels; a parameter updating unit, used to update the parameters of the encoder according to the intra-class distance and the inter-class distance until a second predetermined end condition is reached, and the training is terminated to obtain a trained encoder.

[0047] In some embodiments of the present application, the parameter updating unit is specifically used to execute: if no more than a second predetermined number of triplets in the sample set have an intra-class distance less than the inter-class distance, then the parameters of the encoder are updated; if more than a second predetermined number of triplets in the sample set have an intra-class distance less than the inter-class distance, then the second predetermined end condition is met, the training is ended, and a trained encoder is obtained.

[0048] In some embodiments of the present application, the parameter updating unit is specifically used to perform: determining a distance loss function based on the intra-class distance and the inter-class distance; updating parameters of the encoder based on the distance loss function until the second predetermined end condition is reached, thereby ending the training and obtaining a trained encoder.

[0049] According to one aspect of an embodiment of the present application, a computer-readable medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the medical information retrieval method as described in the above embodiment is implemented.

[0050] According to one aspect of an embodiment of the present application, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the medical information retrieval method as described in the above embodiments.

[0051] In the technical solutions provided in some embodiments of the present application, a medical image is first input into a three-dimensional model for cutting to obtain multiple three-dimensional images, and a three-dimensional image containing a predetermined feature is screened out from the multiple three-dimensional images, which is a target three-dimensional image. The target three-dimensional image is then input into an encoder to obtain shape information and classification information, and the shape information and classification information are converted into a low-dimensional vector, and the shape information and classification information are both key clues that are strongly associated with the predetermined feature. Finally, based on the key clues such as the shape information and classification information contained in the low-dimensional vector, several similar three-dimensional images similar to the target three-dimensional image are retrieved, which can improve the accuracy of similar three-dimensional image retrieval, thereby solving the problem that the current deep model lacks the use of key clues, resulting in inaccurate retrieved medical images.

[0052] These similar 3D images can be used as references to judge the content of the target 3D image, providing doctors with reliable reference materials for medical research and medical diagnosis and treatment, and reducing the burden on relevant personnel.

[0053] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0055] Figure 1 A schematic diagram of an exemplary system architecture to which the technical solution of the embodiments of the present application can be applied is shown.

[0056] Figure 2 A flow chart of a medical information retrieval method provided in an embodiment of the present application is shown.

[0057] Figure 3 A schematic diagram of a medical image processing process provided in an embodiment of the present application is shown.

[0058] Figure 4 Shown according to Figure 2A specific implementation flow chart of step S200 in the medical information retrieval method shown in the corresponding embodiment.

[0059] Figure 5 A schematic diagram of another medical image processing process provided in an embodiment of the present application is shown.

[0060] Figure 6 Shown according to Figure 4 A specific implementation flowchart of step S210 in the medical information retrieval method shown in the corresponding embodiment.

[0061] Figure 7 Shown according to Figure 4 A specific implementation flowchart of step S220 in the medical information retrieval method shown in the corresponding embodiment.

[0062] Figure 8 Shown according to Figure 4 A specific implementation flowchart of step S230 in the medical information retrieval method shown in the corresponding embodiment.

[0063] Fig. 9 A schematic diagram of the structure of a medical information retrieval device provided in an embodiment of the present application.

[0064] Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0065] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this application will be more comprehensive and complete and fully convey the concept of example embodiments to those skilled in the art.

[0066] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present application. However, those skilled in the art will appreciate that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, known methods, devices, realizations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0067] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0068] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0069] Figure 1 A schematic diagram of an exemplary system architecture to which the technical solution of the embodiments of the present application can be applied is shown.

[0070] like Figure 1 As shown, the system architecture may include terminal devices (such as Figure 1 The embodiment of the present invention is a schematic diagram of a mobile phone 101, a tablet computer 102, a portable computer 103, a desktop computer, etc., a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the terminal device and the server 105. The network 104 may include various connection types, such as a wired communication link, a wireless communication link, etc.

[0071] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. According to the implementation requirements, there may be any number of terminal devices, networks and servers. For example, the server 105 may be a server cluster composed of multiple servers.

[0072] The user can use the terminal device to interact with the server 105 through the network 104 to receive or send messages, etc. The server 105 can be a server that provides various services. For example, the user uses the terminal device 103 (or the terminal device 101 or 102) to upload a medical image to the server 105, and the server 105 can input the medical image into the three-dimensional model to obtain a target three-dimensional image, which is a three-dimensional image containing predetermined features; input the target three-dimensional image into the encoder to obtain shape information and classification information; obtain a low-dimensional vector based on the shape information and the classification information; and compare the low-dimensional vector in the database to obtain a predetermined number of similar three-dimensional images.

[0073] It should be noted that the medical information retrieval method provided in the embodiment of the present application is generally executed by the server 105, and accordingly, the medical information retrieval device is generally set in the server 105. However, in other embodiments of the present application, the terminal device may also have similar functions as the server, so as to execute the medical information retrieval scheme provided in the embodiment of the present application.

[0074] The implementation details of the technical solution of the embodiment of the present application are described in detail below:

[0075] Figure 2 A flowchart of a medical information retrieval method according to an embodiment of the present application is shown. The medical information retrieval method can be executed by a server, which can be Figure 1 Refer to the server shown in . Figure 2 As shown, the medical information retrieval method at least includes:

[0076] Step S100: inputting a medical image into a three-dimensional model to obtain a target three-dimensional image, wherein the target three-dimensional image is a three-dimensional image containing predetermined features.

[0077] Step S200: input the target three-dimensional image into an encoder to obtain shape information and classification information.

[0078] Step S300: obtaining a low-dimensional vector according to the shape information and the classification information.

[0079] Step S400: performing comparison in a database based on the low-dimensional vector to obtain a predetermined number of similar three-dimensional images.

[0080] In the embodiments of the present application, reference can be made to Figure 3 First, the medical image is input into the 3D model for cutting to obtain multiple 3D images. The 3D images containing the predetermined features in the multiple 3D images are screened out, which is the target 3D image. Then the target 3D image is input into the encoder to obtain shape information and classification information. The shape information and classification information are converted into low-dimensional vectors. The shape information and classification information are both key clues that are strongly associated with the predetermined features. Finally, according to the key clues such as shape information and classification information contained in the low-dimensional vector, several similar 3D images similar to the target 3D image are retrieved, which can improve the accuracy of similar 3D image retrieval, thereby solving the problem that the current deep model lacks the use of key clues, resulting in inaccurate retrieved medical images.

[0081] These similar 3D images can be used as references to judge the content of the target 3D image, providing doctors with reliable reference materials for medical research and medical diagnosis and treatment, and reducing the burden on relevant personnel.

[0082] In step S100, the 3D model first cuts the medical image to obtain multiple 3D images, and then selects the target 3D image from the multiple 3D images. The target 3D image is a 3D image containing predetermined features, which may be site features, such as cardia, posterior basal segment of right lower lobe, ulna, etc., or abnormal features, such as image problems such as light coloring and blurring, and organ problems such as nodules, burrs, and lesions.

[0083] In step S200, a three-dimensional image containing predetermined features is input into an encoder, information is extracted, and a corresponding three-dimensional image is obtained.

[0084] Specifically, in some embodiments, the specific implementation of step S200 can be found in Figure 4 . Figure 4 is based on Figure 2 The detailed description of step S200 in the medical information retrieval method shown in the corresponding embodiment, in the medical information retrieval method, the encoder includes a spatial encoding module, a shape encoding module, and a classification encoding module, and step S200 may include the following steps:

[0085] Step S210, inputting the target three-dimensional image and the medical image into the spatial encoding module to obtain spatial information;

[0086] Step S220, the target three-dimensional image and the spatial information are encoded in the shape coding module to obtain shape information;

[0087] Step S230: input the target three-dimensional image into the classification coding module to obtain classification information.

[0088] In this embodiment, it can be combined with reference to Figure 5 , inputting the target 3D image into different encoding modules respectively, different information can be obtained. By reducing the dimensions of various information to form low-dimensional vectors, the association between the two images can be discovered more deeply, which helps to better judge the similarity between the two images and improve the accuracy of judgment.

[0089] In step S210, the target three-dimensional image and the medical image are input into the spatial encoding module together, so as to obtain the position of the target three-dimensional image in the medical image, that is, the spatial information.

[0090] Specifically, in some embodiments, the specific implementation of step S210 can be found in Figure 6 . Figure 6 is based on Figure 4 The detailed description of step S210 in the medical information retrieval method shown in the corresponding embodiment, in the medical information retrieval method, the spatial information includes the target relative coordinates and the base point relative coordinates, and step S210 may include the following steps:

[0091] Step S212, inputting the target three-dimensional image and the medical image into the space encoding module to obtain base point relative coordinates, where the base point relative coordinates are relative position coordinates of the base point in the target three-dimensional image in the medical image;

[0092] Step S214, extracting target relative coordinates of a plurality of the target three-dimensional images according to the base point relative coordinates, wherein the target relative coordinates are relative position coordinates of each target point in the target three-dimensional image in the medical image.

[0093] In this embodiment, it can be combined with reference to Figure 5 After the target 3D image is segmented in step S100, the coordinates of the target 3D image in the medical image are known. Specifically, the base point of the medical image (e.g., the upper left corner) is the origin coordinate (0,0,0), and the coordinates of its diagonal point (e.g., the diagonal point of the upper left corner is the lower right corner) are (X l ,Y l ,Z l ). Then the base point coordinates of the target 3D image are (X n ,Y n ,Z n ), the base point coordinates of the target three-dimensional image are the coordinates of the target three-dimensional image in the medical image. After the target three-dimensional image and the medical image are input into the spatial encoding module together, the relative position coordinates of the base point of the target three-dimensional image in the medical image (i.e., the base point relative coordinates) can be obtained.

[0094] According to the base point relative coordinates, the relative position coordinates of multiple target points in the medical image are extracted from the target three-dimensional image according to specific rules, that is, the target relative coordinates. The target points are some specific points belonging to the target three-dimensional image. The base points can be regarded as a special target point, which together with other target points can show the outline of the target three-dimensional image. In some embodiments, the target points include at least 8 vertices of the target three-dimensional image.

[0095] Specifically, in some embodiments, the specific implementation of step S210 can refer to the following embodiments. Figure 6 The detailed description of step S214 in the medical information retrieval method shown in the corresponding embodiment, in the medical information retrieval method, step S214 may include the following steps:

[0096] Taking the base point as a reference point, extracting the corresponding target point according to a predetermined image size;

[0097] Determine the target relative coordinates corresponding to the target point.

[0098] In this embodiment, the specific rule is to extract the corresponding target points according to different image sizes. The larger the image size, the more target points need to be extracted. When images of different sizes are generated, the base point of the target three-dimensional image is used as the reference point for generation. After the target point is extracted, the target relative coordinates corresponding to the target point are determined. The determination method can be to determine the absolute coordinates first, then determine the relative coordinates, or to directly determine the relative coordinates.

[0099] For example, in one embodiment, there are three sizes of images, namely 2×2×2, 4×4×4, and 8×8×8. For these three sizes of images, the method of extracting target points may be that for a 2×2×2 image, eight vertices of the image may be extracted as target points; for a 4×4×4 image, the midpoints of 12 prisms of the image may be additionally extracted on the basis of eight vertices; for an 8×8×8 image, all trisection points on the 12 prisms of the image and the midpoints of six faces may be additionally extracted on the basis of eight vertices as target points. This application does not limit this.

[0100] In step S220, the target three-dimensional image and the spatial information are transmitted to the shape encoding module to obtain an image with position information that can display shape features, that is, shape information.

[0101] Specifically, in some embodiments, the specific implementation of step S220 can be found in Figure 7 . Figure 7 is based on Figure 4 The detailed description of step S220 in the medical information retrieval method shown in the corresponding embodiment, in the medical information retrieval method, the shape information includes a target feature image, and step S220 may include the following steps:

[0102] Step S222, obtaining an intermediate feature image by performing multi-layer convolution and pooling on the target three-dimensional image;

[0103] Step S224, splicing and fusing the intermediate feature image and the spatial information to obtain a target feature image.

[0104] In the embodiments of the present application, reference can be made to Figure 5 After the target three-dimensional image passes through a convolution layer, it passes through multiple convolution pooling networks to obtain an intermediate feature image. Each convolution pooling network includes two convolution layers and a normalization layer and a pooling layer after the convolution layer. The convolution layer can be a central difference convolution (CDC) layer, the normalization layer can be a batch normalization (batch norm) layer, and the pooling layer can be a maximum pooling (max pooling) layer. Then, the intermediate feature image and the spatial information obtained in step S210 are spliced ​​and fused to obtain the largest image with position information that can display shape features, that is, the target feature image.

[0105] By aggregating the intensity and gradient information through the CDC operator, detailed information of the nodule edge context can be captured, making the determination in step S400 more accurate.

[0106] In step S222, the intermediate feature image is a feature image that has undergone multiple layers of convolution, normalization, and pooling. After multiple convolutions, normalizations, and pooling, its appearance features become more and more obvious, and it is more conducive to capturing corresponding information.

[0107] For example, when the predetermined feature is a lung nodule, from the nodule edge, some lung nodules are mostly non-lobed, while others are lobed or have burr signs (or spinous protrusions). From the nodule shape, some nodules have smooth boundaries, while others have irregular, lobed or burr-like boundaries. When the target 3D image undergoes multiple rounds of convolution and pooling, the intermediate feature image obtained is more likely to capture such information as nodules.

[0108] Specifically, in some embodiments, the specific implementation of step S222 can refer to the following embodiments. Figure 7 The detailed description of step S222 in the medical information retrieval method shown in the corresponding embodiment, in the medical information retrieval method, the intermediate feature image includes a minimum intermediate feature image and a common intermediate feature image, and step S222 may include the following steps:

[0109] Inputting the target three-dimensional image into a shape coding module, and performing multi-layer convolution and pooling to obtain a common intermediate feature image;

[0110] The common intermediate feature image is subjected to convolution pooling to obtain a minimum intermediate feature image.

[0111] In an embodiment of the present application, each time the intermediate feature image passes through a convolutional pooling network, the intermediate feature image obtained will become smaller. The image obtained after passing through multiple convolutional pooling networks is the minimum intermediate feature image, and the other intermediate feature images are ordinary intermediate feature images.

[0112] In step S224, the intermediate feature image and the corresponding coordinates based on the spatial relative position encoding are connected together as input for encoder metric learning optimization, so that the judgment of step S400 is more accurate.

[0113] Specifically, in some embodiments, the specific implementation of step S224 can refer to the following embodiments. Figure 7 Detailed description of step S224 in the medical information retrieval method shown in the corresponding embodiment, in the medical information retrieval method, the intermediate feature image includes a minimum intermediate feature image and a common intermediate feature image, and the target feature image includes a minimum target feature image and a common target feature image, and step S224 may include the following steps:

[0114] The minimum intermediate feature image and the corresponding spatial information are spliced ​​and fused to obtain a minimum target feature image;

[0115] The common intermediate feature image, the corresponding spatial information and the minimum intermediate feature image are fused to obtain a common target feature image.

[0116] In the embodiment of the present application, the intermediate feature image and the corresponding spatial information are spliced ​​together to form a target feature image having both shape features and coordinate information. The target feature image has rich multi-dimensional information, which can improve the accuracy of judgment when making judgments.

[0117] Among them, the minimum intermediate feature image is directly spliced ​​and fused with the corresponding spatial information to obtain the corresponding target feature image, that is, the minimum target feature image.

[0118] For the common intermediate feature image, it is necessary to first perform a pre-fusion process with the minimum intermediate feature image, and then splice and fuse it with the corresponding spatial information to obtain the corresponding target feature image, that is, the common target feature image.

[0119] Among them, in some specific embodiments, the above-mentioned fusion processing of the common intermediate feature image, the corresponding spatial information and the minimum intermediate feature image to obtain the common target feature image specifically includes:

[0120] Performing upsampling processing on the minimum intermediate feature image to obtain an upsampled image;

[0121] The common intermediate feature image, the corresponding spatial information and the up-sampled image are fused to obtain a common target feature image.

[0122] In this embodiment, for the common feature image, it is necessary to first perform a pre-fusion process with the upsampled image obtained by upsampling the minimum intermediate feature image at least once, and then splice and fuse it with the corresponding spatial information to obtain the corresponding target feature image, that is, the common target feature image.

[0123] Among them, in some specific embodiments, the above-mentioned fusion processing of the common intermediate feature image, the corresponding spatial information and the upsampled image to obtain the common target feature image specifically includes:

[0124] Performing point-adding fusion processing on the common intermediate feature image and the up-sampled image to obtain a point-added image;

[0125] The point-added image and the corresponding spatial information are spliced ​​and fused to obtain a common target feature image.

[0126] In this embodiment, for the ordinary feature image, it is necessary to first perform a pre-point fusion process with the upsampled image obtained by upsampling the minimum intermediate feature image at least once, and then splice and fuse it with the corresponding spatial information to obtain the corresponding target feature image, that is, the ordinary target feature image.

[0127] In a specific embodiment, there are three sizes of images, which are 2×2×2, 4×4×4, and 8×8×8. The spatial encoding module outputs the target relative coordinates of the above three images, which can be recorded as the first target relative coordinates, the second target relative coordinates, and the third target relative coordinates.

[0128] Correspondingly, the intermediate feature images also have three sizes, and their sizes are also 2×2×2, 4×4×4, and 8×8×8 respectively. They correspond to the first intermediate feature image, the second intermediate feature image, and the third intermediate feature image. Among them, the first intermediate feature image is the smallest intermediate feature image, and the others are ordinary intermediate feature images. The first intermediate feature image is obtained by passing the second intermediate feature image through the above-mentioned convolutional pooling network, and the second intermediate feature image is obtained by passing the third intermediate feature image through the above-mentioned convolutional pooling network.

[0129] The first intermediate feature image is directly spliced ​​and fused with the corresponding first target relative coordinates to obtain a first target feature image, which is the minimum target feature image.

[0130] For the second intermediate feature image, after passing through a convolution layer, it is point-added and fused with the first up-sampled image obtained by up-sampling the first intermediate feature image once, to obtain the first point-added image. Then, the first point-added image and the corresponding second target relative coordinates are spliced ​​and fused to obtain the second target feature image, which is a common target feature image.

[0131] For the third intermediate feature image, after passing through a convolution layer, it is point-added and fused with the second up-sampled image obtained by up-sampling the second intermediate feature image once (it can also be understood as the second up-sampled image obtained by up-sampling the first intermediate feature image twice) to obtain the second point-added image. Then, the second point-added image and the corresponding third target relative coordinates are spliced ​​and fused to obtain the third target feature image, which is a common target feature image.

[0132] In step S230, the target three-dimensional image is input into a classification coding module to obtain a classification result of a predetermined feature in the target three-dimensional image, that is, classification information.

[0133] Specifically, in some embodiments, the specific implementation of step S230 can be found in Figure 8 . Figure 8 is based on Figure 4 The detailed description of step S230 in the medical information retrieval method shown in the corresponding embodiment, in the medical information retrieval method, the classification information includes the residual image and the classification result, and step S230 may include the following steps:

[0134] Step S232, inputting the target three-dimensional image into a residual network to obtain a residual image;

[0135] Step S234, obtaining a classification result according to the residual image.

[0136] In the embodiments of the present application, reference can be made to Figure 5 First, the target image is input into the residual network to obtain the residual image, and then the corresponding classification result is obtained based on the residual image. The residual image contains rich semantic information related to the predetermined features. According to the residual image, the corresponding classification result can be obtained.

[0137] The classification result is the classification result of the predetermined feature in the target three-dimensional image. When the predetermined feature is the radius, the corresponding classification result can be the age group, gender, etc. corresponding to the radius. It can be judged by the density of the corresponding position of the radius or the corresponding size of the radius.

[0138] When the predetermined feature is an abnormal feature of the image, the classification result may be the degree of abnormality (such as general abnormality and severe abnormality), which may be determined by the density of the abnormal position of the image or by the area of ​​the abnormal position of the image.

[0139] When the predetermined feature is a pulmonary nodule, the corresponding classification results can be solid and subsolid. According to density, pulmonary nodules can be divided into three categories: solid nodules, partially solid nodules, and pure ground glass nodules. The latter two are collectively called subsolid nodules. Among pulmonary nodules, partially solid nodules have the highest malignancy rate, followed by pure ground glass nodules and solid nodules.

[0140] That is, in some embodiments of the present application, the classification feature may be a density feature.

[0141] In step S232, the residual network includes 4 residual blocks, each residual block includes at least 2 convolutional layers and one pooling layer, and a residual image is obtained after iteration of the 4 residual blocks. The residual image contains a large amount of semantic information related to the classification features.

[0142] In step S234, the residual image is processed by pooling and the like to be converted into a low-dimensional vector, and the classification result of the predetermined feature in the image is determined based on the low-dimensional vector.

[0143] Specifically, in some embodiments, the specific implementation of step S232 can refer to the following embodiments. Figure 8 The detailed description of step S234 in the medical information retrieval method shown in the corresponding embodiment, in the medical information retrieval method, step S234 may include the following steps:

[0144] The residual image is passed through an adaptive average pooling layer to obtain a pooling result;

[0145] The pooling result is passed through a fully connected layer to obtain the corresponding classification result.

[0146] In this embodiment, the residual image passes through a pooling layer and then a fully connected layer to obtain a corresponding classification result. The pooling layer is an adaptive average pooling layer, and the fully connected layer is used for classification.

[0147] The above classification encoding module can be understood as a classification model, and its training method can be optimized and learned according to the classification results of the samples and the label results of the samples.

[0148] In some embodiments, the training method of the classification encoding module includes:

[0149] Acquire a target three-dimensional image sample set, wherein the target three-dimensional image sample set includes a plurality of target three-dimensional image samples, each of the target three-dimensional image samples has a corresponding classification label;

[0150] Inputting the target three-dimensional image samples into the classification coding module one by one in batches for multiple times to obtain corresponding classification results;

[0151] According to the classification label and the classification result, the parameters of the classification coding module are updated until a first predetermined end condition is reached, and the training is terminated to obtain a trained classification coding module.

[0152] There are multiple first predetermined termination conditions, which may be that the classification results of more than a first predetermined number of target three-dimensional image samples in the sample set are consistent with their classification labels, or that the loss function converges or is less than a predetermined loss.

[0153] In one embodiment, the updating of parameters of the classification encoding module according to the classification label and the classification result until a first predetermined end condition is reached, the training is terminated, and a trained classification encoding module is obtained, specifically including:

[0154] If the classification results of only no more than a first predetermined number of target three-dimensional image samples in the sample set are consistent with their classification labels, updating parameters of the classification encoding module;

[0155] If the classification results of more than a first predetermined number of target three-dimensional image samples in the sample set are consistent with their classification labels, the first predetermined end condition is met, the training is ended, and a trained classification encoding module is obtained.

[0156] In another embodiment, the updating of parameters of the classification encoding module according to the classification label and the classification result until a first predetermined end condition is reached, the training is terminated, and a trained classification encoding module is obtained, specifically comprising:

[0157] Determining a classification loss function according to the classification label and the classification result;

[0158] According to the classification loss function, the parameters of the classification coding module are updated until the first predetermined end condition is reached, and the training is terminated to obtain a trained classification coding module.

[0159] The loss function reaches the first predetermined end condition, that is, the loss function converges or is less than the predetermined loss. Specifically, the binary cross entropy loss (BCE loss) function can be used to optimize the classification coding module.

[0160] In step S300, after obtaining the target feature image that integrates the spatial information and the shape information and the residual image that contains the classification information, they are concatenated and compressed to obtain a low-dimensional vector.

[0161] Specifically, in some embodiments, the specific implementation of step S300 can refer to the following embodiments. Figure 2 The detailed description of step S300 in the medical information retrieval method shown in the corresponding embodiment, in the medical information retrieval method, step S300 may include the following steps:

[0162] splicing the shape information and the classification information to obtain splicing information;

[0163] The concatenated information is convolved, pooled and compressed to obtain the low-dimensional vector.

[0164] In this embodiment, the shape information and the classification information are first concatenated together to obtain concatenated information, and the concatenated information is convolved, pooled, and compressed to obtain a low-dimensional vector.

[0165] Specifically, the description is still made by taking the example that the target feature image includes the first target feature image, the second target feature image, and the third target feature image.

[0166] First, the first target feature image and the second target feature image are upsampled respectively to form a feature image of the same size as the third target feature image. Then, the three feature images and the residual image are spliced ​​and fused together to obtain a spliced ​​feature image, that is, splicing information. The spliced ​​feature image is input into two convolution kernels, followed by the Relu activation function and the 3D adaptive average pooling layer, and finally mapped through the fully connected layer to obtain a low-dimensional vector of length 512.

[0167] In step S400, the low-dimensional vector is compared and screened in the database to determine the similarity between the low-dimensional vector of the feature image in the database and the low-dimensional vector of the target three-dimensional image, and the feature images in the database are sorted according to the similarity, and a predetermined number of feature images with the highest similarity are output as similar three-dimensional images. The predetermined number can be set in advance according to the needs, and can be 2, 3, 5, etc., which is not limited in the present disclosure.

[0168] In some embodiments, the encoder training method includes:

[0169] Acquire a target three-dimensional image sample set, wherein the target three-dimensional image sample set includes a plurality of target three-dimensional image samples, each of the target three-dimensional image samples has a corresponding classification label;

[0170] Inputting the target three-dimensional image samples into the encoder one by one in batches for multiple times to obtain corresponding low-dimensional vectors;

[0171] According to the classification label, sampling is performed in the target three-dimensional image sample set to obtain triplets, each of the triplets includes three target three-dimensional image samples, and only two of the three target three-dimensional image samples have the same classification label;

[0172] According to the triplet, parameters of the encoder are updated until a second predetermined end condition is reached, and the training is terminated to obtain a trained encoder.

[0173] In this embodiment, two target three-dimensional image samples with the same classification label and one target three-dimensional image sample with a different classification label are formed into a triplet, and the parameters of the encoder are updated and optimized by analyzing the similarities of the target three-dimensional images of the same classification and the differences of the target three-dimensional images of different classifications to obtain a trained encoder.

[0174] Specifically, the above-mentioned updating of parameters of the encoder according to the triplet until the second predetermined end condition is reached, ending the training, and obtaining the trained encoder specifically includes:

[0175] Determine an intra-class distance and an inter-class distance of the target three-dimensional image samples in the triplet, wherein the intra-class distance is a distance between two target three-dimensional image samples with the same classification label, and the inter-class distance is a distance between two target three-dimensional image samples with different classification labels;

[0176] The encoder is parameter updated according to the intra-class distance and the inter-class distance until a second predetermined end condition is reached, and the training is terminated to obtain a trained encoder.

[0177] There are multiple first predetermined termination conditions, which may be that there are more than a second predetermined number of triples in the sample set whose intra-class distance is smaller than the inter-class distance, or that the loss function converges or is smaller than a predetermined loss.

[0178] In one embodiment, the updating of the parameters of the encoder according to the intra-class distance and the inter-class distance until a second predetermined end condition is reached, and the training is terminated to obtain a trained encoder, specifically includes:

[0179] If only no more than a second predetermined number of triples in the sample set have intra-class distances smaller than inter-class distances, updating parameters of the encoder;

[0180] If there are more than a second predetermined number of triplets in the sample set whose intra-class distance is smaller than the inter-class distance, then a second predetermined end condition is met, the training is ended, and a trained encoder is obtained.

[0181] In another embodiment, the updating of parameters of the encoder according to the intra-class distance and the inter-class distance until a second predetermined end condition is reached, and the training is terminated to obtain a trained encoder, specifically includes:

[0182] Determine a distance loss function according to the intra-class distance and the inter-class distance;

[0183] According to the distance loss function, the parameters of the encoder are updated until the second predetermined end condition is reached, and the training is terminated to obtain a trained encoder.

[0184] Specifically, the distance loss function can be calculated using the following formula:

[0185] l triplet =[d ap -d an +m] +

[0186] Among them, l triplet is the distance loss function, d ap represents the intra-class distance, d an represents the distance between classes, and m is a predefined margin. []+ is the label operator. For the function y=[x] + , when x<0, y=0; when x≥0, y=x.

[0187] According to the above definition, the distance between classes must be greater than the distance within the class by at least m, otherwise a loss will occur.

[0188] The following describes an apparatus embodiment of the present application, which can be used to execute the medical information retrieval method in the above-mentioned embodiment of the present application. For details not disclosed in the apparatus embodiment of the present application, please refer to the above-mentioned embodiment of the medical information retrieval method of the present application.

[0189] Fig. 9 A block diagram of a medical information retrieval device according to an embodiment of the present application is shown.

[0190] Reference Fig. 9 As shown, according to an embodiment of the present application, a medical information retrieval device 900 includes: an image cutting module 910, which is used to input a medical image into a three-dimensional model to obtain a target three-dimensional image, wherein the target three-dimensional image is a three-dimensional image containing predetermined features; an information acquisition module 920, which is used to input the target three-dimensional image into an encoder to obtain shape information and classification information; a low-dimensional vector module 930, which is used to obtain a low-dimensional vector based on the shape information and the classification information; and an image output module 940, which is used to compare in a database based on the low-dimensional vector to obtain a predetermined number of similar three-dimensional images.

[0191] In some embodiments of the present application, the encoder includes a spatial encoding module, a shape encoding module, and a classification encoding module, and the information acquisition module specifically includes: a spatial information submodule, used to input the target three-dimensional image and the medical image into the spatial encoding module to obtain spatial information; a shape information submodule, used to input the target three-dimensional image and the spatial information into the shape encoding module to obtain shape information; and a classification information submodule, used to input the target three-dimensional image into the classification encoding module to obtain classification information.

[0192] In some embodiments of the present application, the spatial information includes target relative coordinates and base point relative coordinates, and the spatial information submodule specifically includes: a base point coordinate unit, used to input the target three-dimensional image and the medical image into the spatial encoding module to obtain the base point relative coordinates, and the base point relative coordinates are the relative position coordinates of the base point in the target three-dimensional image in the medical image; a target coordinate unit, used to extract the target relative coordinates of multiple target three-dimensional images according to the base point relative coordinates, and the target relative coordinates are the relative position coordinates of each target point in the target three-dimensional image in the medical image.

[0193] In some embodiments of the present application, the target coordinate unit is specifically used to execute: taking the base point as a reference point, extracting the corresponding target point according to a predetermined image size; and determining the target relative coordinates corresponding to the target point.

[0194] In some embodiments of the present application, the shape information includes a target feature image, and the shape information submodule specifically includes: an intermediate image submodule, used to obtain an intermediate feature image from the target three-dimensional image through multiple layers of convolution and pooling; a target image submodule, used to splice and fuse the intermediate feature image and spatial information to obtain a target feature image.

[0195] In some embodiments of the present application, the intermediate feature image includes a minimum intermediate feature image and an ordinary intermediate feature image, the target feature image includes a minimum target feature image and an ordinary target feature image, and the target image submodule specifically includes: a minimum target image unit, used to splice and fuse the minimum intermediate feature image and the corresponding spatial information to obtain a minimum target feature image; and an ordinary target image unit, used to fuse the ordinary intermediate feature image, the corresponding spatial information and the minimum intermediate feature image to obtain an ordinary target feature image.

[0196] In some embodiments of the present application, the common target image unit specifically includes: an upsampling subunit, configured to perform upsampling processing on the minimum intermediate feature image to obtain an upsampled image;

[0197] The fusion subunit is used to fuse the common intermediate feature image, the corresponding spatial information and the upsampled image to obtain a common target feature image.

[0198] In some embodiments of the present application, the fusion subunit is specifically used to perform: performing point-adding fusion processing on the ordinary intermediate feature image and the upsampled image to obtain a point-added image; performing splicing and fusion processing on the point-added image and corresponding spatial information to obtain an ordinary target feature image.

[0199] In some embodiments of the present application, the intermediate image submodule specifically includes: a common intermediate image unit, used to input the target three-dimensional image into a shape coding module, and obtain a common intermediate feature image through multiple layers of convolution and pooling; a minimum intermediate image unit, used to obtain a minimum intermediate feature image through convolution and pooling of the common intermediate feature image.

[0200] In some embodiments of the present application, the classification information includes a residual image and a classification result, and the classification information submodule specifically includes: a residual image submodule, used to input the target three-dimensional image into the residual network to obtain a residual image; and a classification result submodule, used to obtain a classification result based on the residual image.

[0201] In some embodiments of the present application, the classification result submodule specifically includes: a pooling result unit, which is used to obtain a pooling result by passing the residual image through an adaptive average pooling layer; and a classification result unit, which is used to pass the pooling result through a fully connected layer to obtain a corresponding classification result.

[0202] In some embodiments of the present application, the medical information retrieval device also includes: a first sample acquisition module, used to acquire a target three-dimensional image sample set, the target three-dimensional image sample set includes multiple target three-dimensional image samples, and each of the target three-dimensional image samples has a corresponding classification label; a first sample input module, used to input the target three-dimensional image samples in batches and multiple times one by one into the classification coding module to obtain corresponding classification results; a first parameter updating module, used to update the parameters of the classification coding module according to the classification label and the classification result until a first predetermined end condition is reached, the training is ended, and a trained classification coding module is obtained.

[0203] In some embodiments of the present application, the first parameter updating module is specifically used to execute: if the classification results of only no more than a first predetermined number of target three-dimensional image samples in the sample set are consistent with their classification labels, then the parameters of the classification coding module are updated; if the classification results of more than a first predetermined number of target three-dimensional image samples in the sample set are consistent with their classification labels, then the first predetermined ending condition is met, the training is ended, and a trained classification coding module is obtained.

[0204] In some embodiments of the present application, the first parameter updating module is specifically used to execute: determining a classification loss function based on the classification label and the classification result; updating parameters of the classification encoding module based on the classification loss function until the first predetermined end condition is reached, thereby ending the training and obtaining a trained classification encoding module.

[0205] In some embodiments of the present application, the low-dimensional vector module specifically includes: a splicing information submodule, used to splice the shape information and the classification information to obtain splicing information; a low-dimensional vector submodule, used to convolve, pool and compress the splicing information to obtain the low-dimensional vector.

[0206] In some embodiments of the present application, the medical information retrieval device also includes: a second sample acquisition module, used to acquire a target three-dimensional image sample set, the target three-dimensional image sample set includes multiple target three-dimensional image samples, and each of the target three-dimensional image samples has a corresponding classification label; a second sample input module, used to input the target three-dimensional image samples into the encoder one by one in batches multiple times to obtain corresponding low-dimensional vectors; a triplet sampling generation module, used to sample in the target three-dimensional image sample set according to the classification label to obtain a triplet, each of the triplet contains three target three-dimensional image samples, and only two of the three target three-dimensional image samples have the same classification label; a second parameter updating module, used to update the parameters of the encoder according to the triplet until the second predetermined end condition is reached, the training is terminated, and a trained encoder is obtained.

[0207] In some embodiments of the present application, the second parameter updating module specifically includes: a distance calculation unit, used to determine the intra-class distance and inter-class distance of the target three-dimensional image samples in the triplet, the intra-class distance is the distance between two target three-dimensional image samples with the same classification label, and the inter-class distance is the distance between two target three-dimensional image samples with different classification labels; a parameter updating unit, used to update the parameters of the encoder according to the intra-class distance and the inter-class distance until a second predetermined end condition is reached, and the training is terminated to obtain a trained encoder.

[0208] In some embodiments of the present application, the parameter updating unit is specifically used to execute: if no more than a second predetermined number of triplets in the sample set have an intra-class distance less than the inter-class distance, then the parameters of the encoder are updated; if more than a second predetermined number of triplets in the sample set have an intra-class distance less than the inter-class distance, then the second predetermined end condition is met, the training is ended, and a trained encoder is obtained.

[0209] In some embodiments of the present application, the parameter updating unit is specifically used to perform: determining a distance loss function based on the intra-class distance and the inter-class distance; updating parameters of the encoder based on the distance loss function until the second predetermined end condition is reached, thereby ending the training and obtaining a trained encoder.

[0210] In the technical solutions provided in some embodiments of the present application, a medical image is first input into a three-dimensional model for cutting to obtain multiple three-dimensional images, and a three-dimensional image containing a predetermined feature is screened out from the multiple three-dimensional images, which is a target three-dimensional image. The target three-dimensional image is then input into an encoder to obtain shape information and classification information, and the shape information and classification information are converted into a low-dimensional vector, and the shape information and classification information are both key clues that are strongly associated with the predetermined feature. Finally, based on the key clues such as the shape information and classification information contained in the low-dimensional vector, several similar three-dimensional images similar to the target three-dimensional image are retrieved, which can improve the accuracy of similar three-dimensional image retrieval, thereby solving the problem that the current deep model lacks the use of key clues, resulting in inaccurate retrieved medical images.

[0211] These similar 3D images can be used as references to judge the content of the target 3D image, providing doctors with reliable reference materials for medical research and medical diagnosis and treatment, and reducing the burden on relevant personnel.

[0212] Fig.10 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown.

[0213] It should be noted that Fig.10 The computer system of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0214] like Fig.10 As shown, the computer system includes a central processing unit (CPU) 1801, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 1802 or the program loaded from the storage part 1808 to the random access memory (RAM) 1803, such as executing the method described in the above embodiment. In RAM 1803, various programs and data required for system operation are also stored. CPU 1801, ROM 1802 and RAM 1803 are connected to each other through bus 1804. Input / output (I / O) interface 1805 is also connected to bus 1804.

[0215] The following components are connected to the I / O interface 1805: an input section 1806 including a keyboard, a mouse, etc.; an output section 1807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1808 including a hard disk, etc.; and a communication section 1809 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1809 performs communication processing via a network such as the Internet. A drive 1810 is also connected to the I / O interface 1805 as needed. A removable medium 1811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1810 as needed so that a computer program read therefrom is installed into the storage section 1808 as needed.

[0216] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 1809, and / or installed from a removable medium 1811. When the computer program is executed by a central processing unit (CPU) 1801, various functions defined in the system of the present application are executed.

[0217] It should be noted that the computer-readable medium shown in the embodiment of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, - but not limited to - an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by an instruction execution system, device or device or used in combination with it. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a computer-readable computer program is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0218] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0219] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0220] As another aspect, the present application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by an electronic device, the electronic device implements the method described in the above embodiment.

[0221] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0222] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the implementation methods of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the implementation methods of the present application.

[0223] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the embodiments disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application.

[0224] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A medical information retrieval method, characterized in that: The medical information retrieval method comprises: Inputting the medical image into the three-dimensional model to obtain a target three-dimensional image, wherein the target three-dimensional image is a three-dimensional image containing predetermined features; Inputting the target three-dimensional image into an encoder to obtain shape information and classification information; Obtaining a low-dimensional vector according to the shape information and the classification information; According to the low-dimensional vector, comparison is performed in a database to obtain a predetermined number of similar three-dimensional images.

2. The medical information retrieval method according to claim 1, characterized in that: The encoder includes a space encoding module, a shape encoding module, and a classification encoding module, and the inputting the target three-dimensional image into the encoder to obtain shape information and classification information specifically includes: Inputting the target three-dimensional image and the medical image into the spatial encoding module to obtain spatial information; The target three-dimensional image and the spatial information are encoded in a shape encoding module to obtain shape information; The target three-dimensional image is input into the classification coding module to obtain classification information.

3. The medical information retrieval method according to claim 2, characterized in that: The spatial information includes target relative coordinates and base point relative coordinates. The inputting the target three-dimensional image and the medical image into the spatial encoding module to obtain the spatial information specifically includes: Inputting the target three-dimensional image and the medical image into the spatial encoding module to obtain relative coordinates of base points, where the relative coordinates of base points are relative position coordinates of base points in the target three-dimensional image in the medical image; According to the base point relative coordinates, target relative coordinates of a plurality of the target three-dimensional images are extracted, and the target relative coordinates are the relative position coordinates of each target point in the target three-dimensional image in the medical image.

4. The medical information retrieval method according to claim 2, characterized in that: The shape information includes a target feature image, and the shape encoding module obtains the shape information by encoding the target three-dimensional image and the spatial information, specifically including: The target three-dimensional image is subjected to multi-layer convolution and pooling to obtain an intermediate feature image; The intermediate feature image and the spatial information are spliced ​​and fused to obtain a target feature image.

5. The medical information retrieval method according to claim 2, characterized in that: The classification information includes a residual image and a classification result, and the target three-dimensional image is input into the classification encoding module to obtain the classification information, which specifically includes: Inputting the target three-dimensional image into a residual network to obtain a residual image; A classification result is obtained according to the residual image.

6. The medical information retrieval method according to claim 1, characterized in that: The encoder training method comprises: Acquire a target three-dimensional image sample set, wherein the target three-dimensional image sample set includes a plurality of target three-dimensional image samples, each of the target three-dimensional image samples has a corresponding classification label; Inputting the target three-dimensional image samples into the encoder one by one in batches for multiple times to obtain corresponding low-dimensional vectors; According to the classification label, sampling is performed in the target three-dimensional image sample set to obtain triplets, each of the triplets includes three target three-dimensional image samples, and only two of the three target three-dimensional image samples have the same classification label; According to the triplet, parameters of the encoder are updated until a second predetermined end condition is reached, and the training is terminated to obtain a trained encoder.

7. The medical information retrieval method according to claim 6, characterized in that: The step of updating parameters of the encoder according to the triplet until a second predetermined end condition is reached, and the training is terminated to obtain a trained encoder, specifically comprising: Determine an intra-class distance and an inter-class distance of the target three-dimensional image samples in the triplet, wherein the intra-class distance is a distance between two target three-dimensional image samples with the same classification label, and the inter-class distance is a distance between two target three-dimensional image samples with different classification labels; The encoder is parameter updated according to the intra-class distance and the inter-class distance until a second predetermined end condition is reached, and the training is terminated to obtain a trained encoder.

8. A medical information retrieval device, characterized in that: The medical information retrieval device comprises: An image cutting module is used to input a medical image into a three-dimensional model to obtain a target three-dimensional image, wherein the target three-dimensional image is a three-dimensional image containing predetermined features; An information acquisition module, used for inputting the target three-dimensional image into an encoder to obtain shape information and classification information; A low-dimensional vector module, used for obtaining a low-dimensional vector according to the shape information and the classification information; The image output module is used to compare the low-dimensional vectors in the database to obtain a predetermined number of similar three-dimensional images.

9. A computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the medical information retrieval method according to any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement the medical information retrieval method as described in any one of claims 1 to 7.

Images