A similar surgical image retrieval method, system, device and medium

Abstract

A similar surgical picture retrieval method, system, device and medium relate to the field of image processing technology, and include preprocessing an input video stream and text content respectively; extracting original image features in the preprocessed video stream and original text features in the preprocessed text content respectively; using a feature fusion module to interactively fuse the original image features and the original text features; using a feature codebook module to map the fused features into a codebook vector; using a mapping feature decoding module to decompose the codebook vector into compressed image features and compressed text features, and finally decoding the compressed image features into similar images and decoding the compressed text features into text descriptions corresponding to the similar images; the surgical picture can be retrieved quickly and accurately.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method, system, device and medium for retrieving similar surgical images. Background Art

[0002] The application of similar image retrieval during surgery can not only provide a reference for intraoperative decision-making through real-time matching of historical surgical images, but also assist in the formulation of preoperative planning, effectively predicting and avoiding surgical risks. For young doctors, it provides a wealth of real surgical case learning resources and accelerates the improvement of surgical skills. In terms of quality control, by comparing and analyzing similar cases, deviations in surgical operations can be discovered in a timely manner, complications can be prevented, and surgical quality can be continuously improved. In addition, this technology has also promoted the accumulation and inheritance of surgical experience, promoted the standardization process of surgical operations, and provided strong technical support for improving the overall level of laparoscopic surgery.

[0003] The current technical framework can be summarized into three key steps: feature extraction, similarity calculation, and retrieval ranking [1,2,3,4,5,6]. Whether using traditional manual features or feature vectors extracted by deep learning, the essence is to convert the image into a computable numerical representation; then calculate the similarity between the query image and the database image through various measurement methods such as Euclidean distance and cosine similarity; finally, sort based on similarity and return the closest result;

[0004] However, the existing technology requires the construction of a large-scale database, which leads to high time complexity of data retrieval, and it is difficult to strike a balance between accuracy and efficiency, making it difficult to achieve real-time intraoperative retrieval. In addition, the existing technology can only realize the retrieval of macroscopic similarities of a single image, and cannot achieve image retrieval that truly understands the user's retrieval intention based on the image and the core of the user's concern. Summary of the invention

[0005] The purpose of the present invention is to provide a similar surgical screen retrieval method, system, device and medium, which can quickly and accurately retrieve surgical screens.

[0006] The present invention is achieved through the following technical solutions:

[0007] A similar surgical image retrieval method, comprising:

[0008] Preprocess the input video stream and text content respectively;

[0009] Extracting original image features from the preprocessed video stream and original text features from the preprocessed text content respectively;

[0010] The feature fusion module is used to interactively fuse the original image features and the original text features;

[0011] Using the feature codebook module, the fused features are mapped into codebook vectors;

[0012] The mapping feature decoding module is used to decompose the codebook vector into compressed image features and compressed text features. Finally, the compressed image features are decoded into similar images, and the compressed text features are decoded into text descriptions corresponding to the similar images.

[0013] Furthermore, the interactive fusion method of the feature fusion module is:

[0014] Normalize the original image features and original text features respectively;

[0015] The normalized features are stacked to obtain the initially connected image and text features;

[0016] The encoding layer of the feature encoder is used to perform weighted fusion on the stacked features.

[0017] Furthermore, the processing method of the feature code book module includes:

[0018] Construct the codebook structure:

[0019] Find the codebook vector closest to the fused feature in the codebook structure;

[0020] Replace the fused features with the codebook vector.

[0021] Furthermore, in the mapping feature decoding module, the method for decoding the compressed image features includes:

[0022] Adding position encoding to the mapped codebook vector;

[0023] The compressed image features are obtained by using feature weighting based on the correlation between the codebook vector itself and other vectors, and then through residual connection and normalization;

[0024] The compressed image features are reshaped using the image reconstruction head, and then upsampled to restore the image size to obtain a similar image.

[0025] Furthermore, in the mapping feature decoding module, the method for decoding the compressed text feature includes:

[0026] Adding position encoding to the mapped codebook vector;

[0027] The feature weighting based on the correlation between the codebook vector itself and other vectors is used, and then the compressed text features are obtained after residual connection and normalization;

[0028] The following formula is used to process the compressed text features to obtain the probability distribution corresponding to the vocabulary:

[0029] ;

[0030] in, is the vocabulary probability distribution, that is is a one-dimensional matrix with a length of , the matrix data is the probability corresponding to each word, and the data is a real number, is the output projection matrix, is the vocabulary size, It is a compressed text feature;

[0031] According to the probability distribution, the corresponding words are extracted from the vocabulary and the final text description is output.

[0032] A similar surgical image retrieval system, comprising:

[0033] A preprocessing module, used for preprocessing the input video stream and text content respectively;

[0034] An acquisition module, used to extract original image features in the preprocessed video stream and original text features in the preprocessed text content;

[0035] Feature fusion module, used to interactively fuse original image features and original text features;

[0036] The feature codebook module is used to map the fused features into a codebook vector;

[0037] The mapping feature decoding module is used to decompose the codebook vector into compressed image features and compressed text features, and finally decode the compressed image features into similar images and decode the compressed text features into text descriptions corresponding to the similar images.

[0038] A computer-readable storage medium stores a computer program, wherein the program implements the above-mentioned similar surgical image retrieval method when executed by a processor.

[0039] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable, wherein the processor implements the above-mentioned similar surgical screen retrieval method when executing the program.

[0040] The technical solution of the present invention has at least the following advantages and beneficial effects:

[0041] The present invention discloses a similar surgical screen retrieval method, system, device and medium, which realize accurate retrieval of similar screens in minimally invasive surgery by adopting a feature code book and corresponding decoding operations, and has higher clinical practicality than the existing technology.

[0042] In addition, by utilizing text information, similar images that meet the search purpose can be accurately retrieved.

[0043] In addition, the feature code book mapping can complete the image retrieval by decoding once, avoiding the complex calculation of database retrieval and improving the retrieval speed. The image and text decoding results of the mapping feature decoding module respectively verify the reliability of the retrieval results. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A schematic diagram of a method flow of the present invention;

[0045] Figure 2 A schematic diagram of a system structure flow chart of the present invention;

[0046] Figure 3 The figure is a schematic diagram of application results of a system of the present invention. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0048] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0050] In the description of the present invention, it should be noted that if the terms "center", "up", "down", "left", "right", "vertical", "horizontal", "inside", "outside" and the like appear, the orientation or position relationship indicated is based on the orientation or position relationship shown in the accompanying drawings, or is the orientation or position relationship in which the product of the application is usually placed when used. It is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0051] In the description of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0052] Example 1

[0053] As attached Figure 1 A similar surgical image retrieval method is shown, comprising:

[0054] The input video stream and text content are preprocessed respectively, where the video stream comes from the laparoscope, usually a 30fps RGB video;

[0055] The image preprocessing process includes: extracting frames of the surgical images that the user needs to retrieve, obtaining the retrieval image, adjusting it to a uniform resolution, such as 640x480 pixels, sampling center cropping and scaling to obtain 224x224 pixel input;

[0056] The text preprocessing process includes: performing unified preprocessing on the text description input by the user, such as cleaning characters, removing spaces, special symbols, unifying the text size, performing word segmentation, etc., to ensure the consistency of the input text standards;

[0057] Extracting original image features from the preprocessed video stream and original text features from the preprocessed text content respectively;

[0058] In particular, the original image features and the original text features can be extracted using a deep neural network method, such as using SwinTransformer as a backbone network to extract the original image features; the specific extraction process includes:

[0059] Input: preprocessed image, forming a 224x224x3 tensor;

[0060] Feature encoding: Encode the input image into 3D features to form 32x196x96 features;

[0061] Feature extraction at different stages: Through 4 stages, more advanced spatiotemporal features are gradually extracted;

[0062] Output: Generate multi-scale feature maps, including feature maps with 1 / 4, 1 / 8, 1 / 16, and 1 / 32 resolutions.

[0063] In addition, the Transformer encoder is used as the backbone network to extract the original text features; the specific extraction process includes:

[0064] Input: preprocessed text sequence, split into token sequences, with a uniform length of 77;

[0065] Feature encoding: The token sequence is encoded as an embedded vector, mapped to the vector space, and position encoding is added to form a 77×512 feature sequence;

[0066] Multi-layer feature extraction: Through the 12-layer Transformer encoder, more abstract semantic features are extracted layer by layer;

[0067] Output: vector representation of generated text;

[0068] The feature fusion module is used to interactively fuse the original image features and the original text features;

[0069] This module obtains the similarity between image features and text features by calculating the Euclidean distance and cosine similarity of different feature maps, trains and aligns the features of two different modalities, then concatenates the aligned image and text features, and further fully integrates the concatenated features using the encoding layer of feature weighting and residual mixed features.

[0070] Using the feature codebook module, the fused features are mapped into codebook vectors;

[0071] The feature codebook module is a collection of codebook vectors, which integrates the image and text knowledge learned by the entire model into a codebook. Finally, the mapped codebook vector can be used to directly decode and obtain the result. Its working principle is as follows: first, the dimension of the acquired fusion feature is adjusted to align with the code table dimension, and then the Euclidean distance between the fusion feature and each codebook vector is calculated to find the nearest neighbor codebook vector, and this vector is extracted to replace the original fusion feature, and the dimension size is restored for subsequent decoding. The whole process is to map the fusion feature obtained by encoding to the codebook vector to complete the feature replacement. It should be noted that the codebook vector is the corresponding vector of the similar image.

[0072] A mapping feature decoding module is used to decompose the codebook vector into compressed image features and compressed text features, and finally the compressed image features are decoded into similar images, and the compressed text features are decoded into text descriptions corresponding to the similar images;

[0073] The mapping feature decoding module aims to decompose the feature vector of the codebook mapping, that is, the codebook vector, into separate image features and text features, and then decode them using separate decoding methods for image and text respectively; its working principle is: first, the fused features are separated by using the feature self-weighting method to obtain the final compressed image features and compressed text features, and then the compressed image features are decoded separately through the feature weighted decoding layer, and the image is upsampled and restored; the compressed text features are decoded to restore the text description.

[0074] The output similar images will retain their original data, surgical videos, etc. in the database; thus ensuring the authenticity and traceability of the output images; and the text description will prove that the output similar images meet the retrieval target.

[0075] Example 2

[0076] The interactive fusion method of the feature fusion module is:

[0077] Normalize the original image features and original text features respectively;

[0078] The LayerNorm layer is used for normalization operation. For the input feature vector x, the calculation formula of LayerNorm is as follows:

[0079] ;

[0080] in: is the mean of the feature map in the channel dimension: ; is the variance of the feature map in the channel dimension: ; is a small constant (such as 1e-5) for numerical stability; γ and β are learnable scaling and translation parameters, and C is the number of channels;

[0081] Since LayerNorm normalizes each feature vector independently, the feature distribution is more stable

[0082] The normalized features are stacked to obtain the initially connected image and text features;

[0083] The encoding layer of the feature encoder is used to perform weighted fusion on the stacked features;

[0084] First input: image features: ,Right now It is a two-dimensional matrix with B rows and D columns, and the data in the matrix are real numbers, text features: ,Right now It is a two-dimensional matrix with B rows and D columns, and the data in the matrix are real numbers, where B is the batch size and D is the feature dimension;

[0085] Then stack: , obtain the initial connection of image and text features; is a three-dimensional matrix with B rows, 2 columns and D height, and the matrix contents are all real numbers;

[0086] Finally, by calculating the correlation between each feature element, a weighted fusion based on this similarity is achieved for the feature vectors from the image and text.

[0087] Example 3

[0088] The processing method of the feature code book module includes:

[0089] The fused features are defined as ,Right now It is a one-dimensional matrix, the length of the matrix is ​​the feature dimension D, and the matrix contains the fused data and is a real number;

[0090] Construct the codebook structure:

[0091] , ;

[0092] Where K is the codebook size, D is the dimension size of each codeword, represents the kth codebook vector, It is also a one-dimensional matrix, the length of the matrix is ​​the feature dimension D, and the matrix contains the fused data and is a real number;

[0093] Find the code in the codebook structure The codebook vector with the closest distance;

[0094] The calculation formula is:

[0095] ;

[0096] This formula is used to calculate the codebook vector and The Euclidean distance is then sorted to obtain the codebook vector index k of the nearest neighbor;

[0097] Replace the fused features with the codebook vector, that is, .

[0098] Example 4

[0099] In the mapping feature decoding module, a method for decoding similar images includes:

[0100] Adding position encoding to the mapped codebook vector;

[0101] The compressed image features are obtained by using feature weighting based on the correlation between the codebook vector itself and other vectors, and then through residual connection and normalization;

[0102] After reshaping the compressed image features using the image reconstruction head, upsampling is performed to restore the image size, thus obtaining a similar image.

[0103] Among them, the calculation formula for reshaping the compressed image feature vector is:

[0104]

[0105] Among them, H, W are the height and width of the original image, p is the patch size, and C is the number of channels.

[0106] The upsampling process is performed using transposed convolution, and the specific calculation formula is:

[0107] ;

[0108] ;

[0109] Where k is the convolution kernel size, s is the step size, and W is the learnable weight.

[0110] In addition, in the mapping feature decoding module, the method for decoding the text description includes:

[0111] Adding position encoding to the mapped codebook vector;

[0112] The feature weighting based on the correlation between the codebook vector itself and other vectors is used, and then the compressed text features are obtained after residual connection and normalization;

[0113] The following formula is used to process the compressed text features to obtain the probability distribution corresponding to the vocabulary:

[0114] ;

[0115] in, is the vocabulary probability distribution, that is is a one-dimensional matrix with a length of , the matrix data is the probability corresponding to each word, and the data is a real number, is the output projection matrix, is the vocabulary size, It is a compressed text feature;

[0116] According to the probability distribution, the corresponding words are extracted from the vocabulary and the final text description is output; the vocabulary is composed of each token sequence in the training process.

[0117] As attached Figure 2 A similar surgical image retrieval system shown includes:

[0118] A preprocessing module, used for preprocessing the input video stream and text content respectively;

[0119] An acquisition module, used to extract original image features in the preprocessed video stream and original text features in the preprocessed text content;

[0120] Feature fusion module, used to interactively fuse original image features and original text features;

[0121] A feature codebook module is used to map the fused features into a codebook vector;

[0122] A mapping feature decoding module is used to decompose the codebook vector into compressed image features and compressed text features, and finally decode the compressed image features into similar images and decode the compressed text features into text descriptions corresponding to the similar images;

[0123] It should be noted that according to the attached Figure 3 In the example shown, the system can retrieve similar surgical scenes based on the input video stream data and the manually input textual interpretation data of the video stream data, and give a simple textual description for the retrieval results;

[0124] In addition, before the actual application of this system, multiple videos will be used for training, and the training content will also be retained in the database of this system. Therefore, when the current surgical video is actually input, similar images in the previous training content can be matched and output; for example, in the initial stage of this system, the first surgical video input will still be output as the video, but the calculation process of the video will be retained in this system. If the fusion features of the nth video input in the subsequent calculation of this system are replaced by the codebook vector of the first surgical video, the output will be the content of the first surgical video;

[0125] As the number of training videos increases, the system can match historical surgical images in real time, provide a reference for intraoperative decision-making, assist in the formulation of preoperative planning, and effectively predict and avoid surgical risks.

[0126] A computer-readable storage medium stores a computer program, wherein the program implements the above-mentioned similar surgical image retrieval method when executed by a processor.

[0127] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable, wherein the processor implements the above-mentioned similar surgical screen retrieval method when executing the program.

[0128] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A similar surgical image retrieval method, characterized by: include: Preprocess the input video stream and text content respectively; Extracting original image features from the preprocessed video stream and original text features from the preprocessed text content respectively; The feature fusion module is used to interactively fuse the original image features and the original text features; Using the feature codebook module, the fused features are mapped into codebook vectors; The processing method of the feature code book module includes: The fused features are defined as ,Right now It is a one-dimensional matrix, the length of the matrix is ​​the feature dimension D, and the matrix contains the fused data and is a real number; Construct the codebook structure: ， ； Where K is the codebook size, D is the dimension size of each codeword, represents the kth codebook vector, It is also a one-dimensional matrix, the length of the matrix is ​​the feature dimension D, and the matrix contains the fused data and is a real number; Find the code in the codebook structure The codebook vector with the closest distance; The calculation formula is: ； This formula is used to calculate the codebook vector and The Euclidean distance is then sorted to obtain the codebook vector index k of the nearest neighbor; Replace the fused features with the codebook vector, that is, ; The mapping feature decoding module is used to decompose the codebook vector into compressed image features and compressed text features. Finally, the compressed image features are decoded into similar images, and the compressed text features are decoded into text descriptions corresponding to the similar images: In the mapping feature decoding module, the method for decoding similar images and corresponding text descriptions includes: Adding position encoding to the mapped codebook vector; Using feature weighting based on the correlation between the codebook vector itself and other vectors, and then after residual connection and normalization, the compressed image feature and compressed text feature are obtained respectively; After reshaping the compressed image features using the image reconstruction head, upsampling is performed to restore the image size, thus obtaining a similar image. Among them, the calculation formula for reshaping the compressed image feature vector is: Among them, H, W are the height and width of the original image, p is the patch size, and C is the number of channels; The upsampling process is performed using transposed convolution, and the specific calculation formula is: ； ； Where k is the convolution kernel size, s is the step size, and W is the learnable weight; The following formula is used to process the compressed text features to obtain the probability distribution corresponding to the vocabulary: ； in, is the vocabulary probability distribution, that is is a one-dimensional matrix with a length of , the matrix data is the probability corresponding to each word, and the data is a real number, is the output projection matrix, is the vocabulary size, It is a compressed text feature.

2. The similar surgical image retrieval method according to claim 1, characterized in that: The interactive fusion method of the feature fusion module is: Normalize the original image features and original text features respectively; The normalized features are stacked to obtain the initially connected image and text features; The encoding layer of the feature encoder is used to perform weighted fusion of the initially connected image and text features.

3. A similar surgery image retrieval system, implementing the similar surgery image retrieval method as claimed in any one of claims 1 to 2, characterized in that: include: A preprocessing module, used for preprocessing the input video stream and text content respectively; An acquisition module, used to extract original image features in the preprocessed video stream and original text features in the preprocessed text content; Feature fusion module, used to interactively fuse original image features and original text features; The feature codebook module is used to map the fused features into a codebook vector; The mapping feature decoding module is used to decompose the codebook vector into compressed image features and compressed text features, and finally decode the compressed image features into similar images and decode the compressed text features into text descriptions corresponding to the similar images.

4. A computer-readable storage medium storing a computer program, characterized in that: When the program is executed by a processor, the similar surgical image retrieval method as described in any one of claims 1-2 is implemented.

5. An electronic device comprising a memory, a processor and a computer program stored in the memory and executable, characterized in that: When the processor executes the program, the similar surgical screen retrieval method as described in any one of claims 1-2 is implemented.

Images