Image Processing Method and Apparatus

Abstract

The present application is applicable to the field of image processing technology and provides an image processing method and device, including: acquiring a target frame image, multiple memory frame images, and a guide frame image, wherein the multiple memory frame images include a mask of a lesion, and the difference in change between the target frame image and the guide frame image is within a preset range; performing local similarity calculation on the target frame image and each memory frame image according to the time dimension of the multiple memory frame images to obtain a time feature map; performing spatial similarity calculation on the target frame image and the guide frame image to obtain a spatial feature map; and determining a predicted mask of the lesion in the target frame image according to the time feature map and the spatial feature map. Thus, by decoupling the time dimension and the space dimension and performing parallel processing, the complexity and time cost of the calculation are reduced, computing power resources are saved, the detection speed and accuracy of the lesion area are improved, and the real-time and low-latency detection requirements are met.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and in particular relates to an image processing method and device. Background Art

[0002] With the continuous development of science and technology, computers can be used to segment the lesion area in ultrasound videos, such as providing information such as the mask and time of the lesion, which is crucial for clinical examination and treatment.

[0003] Ultrasound video generation is often accompanied by artifacts such as speckle noise, low contrast, and intensity inhomogeneity, which blur the boundaries between lesion and non-lesion areas. Therefore, a semi-supervised video segmentation network (STM) algorithm is proposed.

[0004] Currently, STM algorithms typically use a global attention matching mechanism to distinguish lesion areas from non-lesion areas, enabling segmentation of ultrasound videos. However, because lesion masks are often localized across multiple frames, this mechanism can introduce misleading information, resulting in a large computational load. Furthermore, the computational cost of the global attention matching mechanism itself is high, making it difficult to meet the requirements of real-time segmentation. Summary of the Invention

[0005] The present application provides an image processing method and device, which can solve the problem that related technologies have large computational complexity and cannot meet the needs of quickly segmenting lesion areas and non-lesion areas.

[0006] In a first aspect, the present application provides an image processing method, comprising:

[0007] Acquire a target frame image, multiple memory frame images, and a guide frame image, wherein the multiple memory frame images include a mask of the lesion, and the change difference between the target frame image and the guide frame image is within a preset range;

[0008] According to the time dimension of multiple memory frame images, the local similarity between the target frame image and each memory frame image is calculated to obtain a time feature map;

[0009] Calculate the spatial similarity between the target frame image and the guidance frame image to obtain a spatial feature map;

[0010] A predicted mask of the lesion in the target frame image is determined according to the temporal feature map and the spatial feature map.

[0011] Through the image processing method of the first aspect, a target frame image, multiple memory frame images and a guide frame image can be obtained. From the perspective of decoupling the time dimension and the space dimension, attention can be focused on the local areas of the target frame image and the multiple memory frame images, and the local similarity between the target frame image and each memory frame image is calculated to obtain a time feature map, which can determine the local similarity of the target frame image and the multiple memory frame images regarding the mask of the lesion. At the same time, attention can be focused on the global / overall area of the target frame image and the guide frame image, and the overall similarity between the target frame image and the guide frame image is calculated to obtain a spatial feature map, which can determine the global similarity between the target frame image and the guide frame image regarding the change difference (such as static background texture). Based on the time feature map and the spatial feature map, the predicted mask of the lesion in the target frame image is determined. Therefore, in this application, with the help of decoupling of the time dimension and the spatial dimension and parallel processing, the mask of the lesion in the image can be accurately predicted, which reduces the complexity and time cost of the calculation, saves computer computing resources, and effectively filters out the interference information in the image, thereby improving the detection speed and accuracy of the lesion area, and meeting the real-time and low-latency detection needs.

[0012] In one possible design, based on the time dimension of multiple memory frame images, local similarity calculation is performed between the target frame image and each memory frame image to obtain a temporal feature map, including:

[0013] According to the target frame image, a first key feature map and a first value feature map are obtained, wherein the first key feature map corresponds to the first value feature map;

[0014] According to each memory frame image, a second key feature map and a second value feature map are obtained, wherein the second key feature map corresponds to the second value feature map;

[0015] Dividing the first key feature map, the first value feature map, the plurality of second key feature maps, and the plurality of second value feature maps into a preset number of non-overlapping area blocks;

[0016] According to the time sequence of the multiple memory frame images, local similarity calculation is performed on each area block in the first key feature map and the area blocks corresponding to the same index in the multiple second key feature maps to obtain local similarity;

[0017] A temporal feature map is obtained according to the local similarity, each region block in the first value feature map, and each region block in the plurality of second value feature maps.

[0018] In one possible design, local similarity is calculated for each region block in the first key feature map and region blocks corresponding to the same index in multiple second key feature maps according to the time sequence of multiple memory frame images, to obtain the local similarity, including:

[0019] Sort each region block in the plurality of second key feature maps in chronological order according to the same index to obtain a first set;

[0020] A local similarity calculation and normalization process is performed on each region block in the first key feature map and the corresponding identical index in the first set to obtain a local similarity.

[0021] In one possible design, a temporal feature map is obtained according to the local similarity, each region block in the first value feature map, and each region block in the plurality of second value feature maps, including:

[0022] Weighting the local similarity and each region block in the plurality of second value feature maps to obtain a memory value feature map;

[0023] According to the same index of the region blocks, the memory value feature map is merged with each region block in the first value feature map to obtain a time feature map.

[0024] In one possible design, the method further includes:

[0025] According to the time characteristic graph, a third value characteristic graph is obtained;

[0026] Dividing the third value feature map into a preset number of non-overlapping area blocks;

[0027] The temporal feature map is updated according to the local similarity, each region block in the first value feature map, and each region block in the third value feature map.

[0028] In one possible design, the spatial similarity between the target frame image and the guidance frame image is calculated to obtain a spatial feature map, including:

[0029] Obtaining a fourth key feature map and a fourth value feature map according to the guidance frame image, wherein the fourth key feature map corresponds to the fourth value feature map;

[0030] Calculating spatial similarity between the first key feature map and the fourth key feature map to obtain spatial similarity;

[0031] Weighting the spatial similarity and the fourth value feature map to obtain the guidance value feature map;

[0032] The guidance value feature map is merged with the first value feature map to obtain a spatial feature map.

[0033] In one possible design, the method further includes:

[0034] According to the spatial feature map, a fifth value feature map is obtained;

[0035] Weighting the spatial similarity and the fifth value feature map to update the guidance value feature map;

[0036] The guidance value feature map is merged with the first value feature map to update the spatial feature map.

[0037] In a second aspect, an embodiment of the present application provides an image processing device, which is used to execute the image processing method in the above-mentioned first aspect or any possible design of the first aspect.

[0038] In a third aspect, embodiments of the present application provide an image processing device, comprising a memory and a processor. The memory is configured to store instructions; the processor executes the instructions stored in the memory, causing the device to perform the image processing method of the first aspect or any possible design of the first aspect.

[0039] In a fourth aspect, a computer-readable storage medium is provided, wherein instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes the image processing method in the first aspect or any possible design of the first aspect.

[0040] In a fifth aspect, a computer program product comprising instructions is provided, which, when executed on a device, causes the device to execute the image processing method in the first aspect or any possible design of the first aspect.

[0041] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions of the first aspect and any possible design of the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0043] Figure 1 A schematic diagram of an architecture block diagram of an image processing model provided in one embodiment of the present application;

[0044] Figure 2 A flowchart of an image processing method provided in one embodiment of the present application;

[0045] Figure 3 A flowchart of an image processing method provided in one embodiment of the present application;

[0046] Figure 4 A schematic diagram of a module of an image processing method provided in one embodiment of the present application;

[0047] Figure 5A schematic diagram of a module of an image processing method provided in one embodiment of the present application;

[0048] Figure 6 A schematic structural diagram of an image processing device provided in one embodiment of the present application;

[0049] Figure 7 A schematic structural diagram of an image processing device provided in one embodiment of the present application;

[0050] Figure 8 A schematic structural diagram of an image processing device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0051] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.

[0052] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections thereof.

[0053] It will also be understood that the term "and / or" used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0054] As used in this specification and the appended claims, the term "if" can be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0055] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0056] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present application. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0057] This application provides an image processing method, apparatus, device, computer-readable storage medium, and computer program product, and proposes the following inventive concepts:

[0058] For any frame image that needs to be detected, by using a large number of memory frame images and focusing on the similarity between local areas of the image in the time dimension, the features of the target in the frame image can be obtained, and the computational complexity is reduced; by using images with little or no change difference and focusing on the overall / global similarity of the image in the spatial dimension, the overall features of the frame image can be obtained.

[0059] Therefore, with the help of the decoupling of the time dimension and the spatial dimension and the above two lightweight parallel processing processes, based on obtaining the characteristics of the target in the frame image and the overall characteristics of the frame image, the area where the target is located in the frame image can be automatically obtained, which solves the technical problem that the related technology has large computational complexity and cannot meet the needs of fast segmentation, and achieves the technical effect of improving the target detection speed and detection accuracy, and reducing the computing cost.

[0060] The image processing method of the present application is applicable to a variety of imaging / image fields. For example, the image processing method of the present application is suitable for the imaging field of clinical medicine, providing more possibilities for application in this field.

[0061] For ease of explanation, in this application, the target is exemplified by a lesion. Accordingly, the target area is the mask of the lesion. This facilitates rapid and accurate differentiation between lesion areas and non-lesion areas.

[0062] See also Figure 1 , Figure 1 A schematic diagram of the architecture of an image processing model provided in one embodiment of the present application is shown.

[0063] like Figure 1As shown in the figure, the image processing model may include: an encoder (encode, referred to as Enc), a key-value output module (key-value output module), a block segmentation module (block segmentation module), a temporal attention module (Temporal Transformer), a spatial attention module (spatial transformer), a merging module (concate module), a feature fusion module (fea) and a decoder (decode, referred to as Dec).

[0064] The encoder is used to extract the feature map of any frame of image. The number and type of encoders are not limited in this application. For the sake of convenience, Figure 1 In the figure, the encoders are illustrated using Enc1, Enc2 and Enc3 as examples.

[0065] For example, the encoder of the present application may use ResNet50 as the backbone network. In addition, the encoder of the present application may also use other existing feature extraction networks, such as residual network (ResNet-152), neural network (AlexNet), deep convolutional neural network (VGGNet), convolutional neural network (Inception), etc.

[0066] It should be noted that each frame of image input to the encoder may include a four-dimensional channel, namely, a three-dimensional channel corresponding to the color and a one-dimensional channel corresponding to the mask after the color is removed. And / or, each frame of image input to the encoder may include only a one-dimensional channel corresponding to the mask after the color is removed.

[0067] The key-value pair output module includes two parallel convolutional layers (Conv), which are used to receive the feature map output by the encoder and generate two feature maps, namely, a key feature map and a value feature map. Figure 1 In FIG, the key-value pair output module is illustrated by taking Conv11 and Conv12, Conv21 and Conv22, and Conv31 and Conv32 as examples.

[0068] The region block segmentation module is used to segment the key feature map and the value feature map into a preset number of non-overlapping region blocks. The non-overlapping here refers to the non-overlapping between region blocks. The specific value of the preset number is not limited in this application. For the sake of convenience, Figure 1 In the example, the region block segmentation module is illustrated using block 1 and block 2 as examples.

[0069] The temporal attention module is a lightweight module that focuses on the attention of local areas of an image. It is used to calculate the local similarity between each feature map corresponding to a multi-frame image and the feature map corresponding to any frame image according to the time dimension of the multi-frame image (i.e., the time sequence of the captured images). The local similarity here refers to the similarity between the partial areas of the two feature maps. Among them, the present application can set the number of temporal attention modules to N, where N is a positive integer.

[0070] The spatial attention module is a lightweight module that focuses on the global / overall area of the image. It is used to calculate the spatial similarity between the feature maps corresponding to any two frames of images based on the spatial dimension of the image. The spatial similarity here refers to the similarity between the entire area (i.e., the entire area) of the two feature maps. Among them, the application can set the number of spatial attention modules to M, where M is a positive integer.

[0071] In addition, M and N may be the same or different, and this application does not limit this.

[0072] It should be noted that the temporal attention module and the spatial attention module are two parallel modules. The temporal attention module focuses on the time dimension, while the spatial attention module focuses on the space dimension.

[0073] The merging module is used to merge multiple feature maps according to their channels.

[0074] The feature fusion module is used to fuse features of multiple feature maps. For example, the feature fusion module can use a convolutional layer to achieve feature fusion.

[0075] The decoder is used to output the prediction result, which is recorded as pred, and can restore the image input to the encoder to achieve segmentation of the image. Among them, this application does not limit the number and type of decoders and other parameters.

[0076] In some embodiments, the decoder may include a set of residual convolution blocks, an interpolation module, and a resizing module. The residual convolution blocks are used to amplify feature maps, the interpolation module is used to perform processing on the feature maps, such as bipolar interpolation, and the resizing module is used to resize the feature maps to the same size as the image input to the encoder.

[0077] It should be noted that the feature image output to the decoder is a black and white image, and the output channel of the decoder is the one-dimensional channel corresponding to the mask after color removal.

[0078] Below, this application uses a device Figure 1 Taking the image processing model in as an example, the image processing method of this application is explained in detail in combination with the accompanying drawings and application scenarios.

[0079] The device may include a software module or a physical device of software and / or hardware.

[0080] See also Figure 2 , Figure 2 A flow chart of an image processing method provided in one embodiment of the present application is shown.

[0081] like Figure 2 As shown, the image processing method of the present application may include:

[0082] S201 : Acquire a target frame image, a plurality of memory frame images, and a guidance frame image.

[0083] The target frame image may be the current frame image in the ultrasound video, that is, the current frame image that the user wants to test, and the current frame image t may be marked using Query.

[0084] The multiple memory frame images are historical images collected before the target frame image in the ultrasound video. T memory frame images {t-1,…,tT} can be marked with Memory, where T is a positive integer.

[0085] The inclusion of the lesion mask in the multiple memory frame images facilitates providing a local basis for the characteristics of the lesion mask (i.e., the lesion area) for the target frame image. Furthermore, the lesion mask in the multiple memory frame images can be located in the spanned image and / or in a local area within the image. The spanned image herein refers to a discontinuous multi-frame image.

[0086] The guidance frame image can be a frame image in the ultrasound video acquired before the target frame image, such as the previous frame image. The guidance frame image t-1 can be labeled "Previous." The difference in change (or motion) between the target frame image and the guidance frame image can be within a preset range, that is, the two frames have no change or the difference in change is small, which facilitates providing a global / holistic reference / guidance for the characteristics of all regions (i.e., lesion areas and non-lesion areas) of the target frame image. It should be noted that the guidance frame image is included in multiple memory frame images.

[0087] Among them, this application does not limit the parameters such as format and content of the target frame image, multiple memory frame images and guidance frame image.

[0088] S202 , performing local similarity calculation between the target frame image and each memory frame image according to the time dimension of the multiple memory frame images to obtain a time feature map.

[0089] It can be understood that the time dimension of the multiple memory frame images refers to the time sequence of collecting the multiple memory frame images, which may correspond to the collection time of the multiple memory frame images from early to late.

[0090] Therefore, in the time dimension, the image processing model can be used to calculate the local similarity of each identical partial area between the target frame image and each memory frame image to obtain a time feature map.

[0091] Among them, the temporal feature map can be used to represent the local similarity between the target frame image and each memory frame image, that is, the similarity between each partial area of the target frame image and the corresponding partial area of each memory frame image.

[0092] S203 , performing spatial similarity calculation on the target frame image and the guidance frame image to obtain a spatial feature map.

[0093] The spatial feature map may be used to represent the spatial similarity between the target frame image and the guidance frame image, that is, the similarity between the entire region of the target frame image and the entire guidance frame image.

[0094] Therefore, in the spatial dimension, the image processing model can be used to calculate the spatial similarity of the entire area of the target frame image and the guidance frame image to obtain a spatial feature map.

[0095] It should be noted that there is no temporal order between the above S202 and S203, and S202 and S203 can be executed simultaneously or sequentially.

[0096] In some embodiments, the present application may execute S202 and S203 in parallel, thereby effectively saving computer computing resources, accelerating the speed of detecting the lesion mask, filtering out interference information, and improving the accuracy of detecting the lesion mask.

[0097] S204: Determine a predicted mask of the lesion in the target frame image according to the temporal feature map and the spatial feature map.

[0098] The image processing model can be used to distinguish the lesion area and / or non-lesion area of the target frame image according to the temporal feature map and the spatial feature map, thereby determining the predicted mask of the lesion in the target frame image.

[0099] In some embodiments, the image processing model may utilize Figure 1 The merging module, feature fusion module and decoder shown determine the predicted mask of the lesion in the target frame image.

[0100] The temporal feature map and the spatial feature map may be input into a merging module, which may merge the temporal feature map and the spatial feature map, and input the merged feature map into a feature fusion module.

[0101] For example, the temporal feature map is represented by a matrix 1 of H×W×C1, and the spatial feature map is represented by a matrix 2 of H×W×C2. The merged feature map can be represented by a matrix 3 of H×W×(C1+C2).

[0102] The feature fusion module performs feature fusion on the merged feature maps and inputs the fused feature maps into the decoder. The decoder decodes the fused feature maps and outputs a predicted mask of the lesion in the target frame image.

[0103] In some embodiments, a set of residual convolution blocks in the decoder may amplify the feature map step by step and input the amplified feature map into an interpolation module in the decoder.

[0104] For two adjacent residual convolution blocks, the input of the latter residual convolution block comes from the connection between the feature map output by the previous residual convolution block and the feature map of the target frame image output by the encoder.

[0105] The interpolation module in the decoder can perform bipolar interpolation processing on the amplified feature map, and input the interpolated feature map into the adjustment module in the decoder.

[0106] The adjustment module in the decoder adjusts the size of the interpolated feature map, that is, adjusts the size of the interpolated feature map to the same size as the target frame image input to the encoder.

[0107] Thus, the adjustment module in the decoder outputs a predicted mask of the lesion in the target frame image. In addition, the application can also use the predicted results output by the decoder and the actual results of the corresponding image to estimate the recognition accuracy of the image processing model and adjust the model parameters of the image processing model, thereby improving the accuracy of the mask for identifying the lesion.

[0108] In some embodiments, the predicted value corresponding to the predicted mask of the lesion in the target frame image is The ground truth (GT) Y corresponding to the true mask of the lesion in the target frame image can determine the loss and L between the two. all .

[0109] Among them, the loss and L all It can be expressed by the following formula:

[0110]

[0111] in, represents the binary cross entropy loss, Denotes Dice loss, loss and L all is the sum of binary cross entropy loss and Dice loss.

[0112] m represents each pixel in the prediction mask, Y + represents the lesion area in the predicted mask, and Y- represents the non-lesion area in the predicted mask. α is a hyperparameter greater than 0 that weighs the binary cross entropy loss and the Dice loss. This application sets it to 1.

[0113] It can be seen that the loss and L all The numerical value of can represent the recognition accuracy of the image processing model.

[0114] Among them, the loss and L all The smaller the value is, the smaller the predicted value corresponding to the predicted mask of the lesion in the target frame image is. The higher the degree of overlap between the true value Y and the true mask of the lesion in the target frame image, the higher the recognition accuracy of the image processing model.

[0115] Among them, the loss and L all The larger the value is, the larger the predicted value of the predicted mask of the lesion in the target frame image is. The lower the degree of overlap between the true value Y and the true mask of the lesion in the target frame image, the lower the recognition accuracy of the image processing model.

[0116] In addition, compared with the STM algorithm provided by related technologies, the image processing method provided by this application is based on the comparison of parameters such as the Jaccard index (Jaccard), statistics (F-measure), precision (Precision), and recall (Recall), which has better detection accuracy and faster detection speed. In addition, this application can also achieve real-time detection.

[0117] The image processing method provided by the present application is obtained by acquiring a target frame image, multiple memory frame images and a guide frame image. From the perspective of decoupling the time dimension and the space dimension, attention can be focused on the local area of the target frame image and the multiple memory frame images, and the local similarity between the target frame image and each memory frame image is calculated to obtain a time feature map, which can determine the local similarity of the target frame image and the multiple memory frame images with respect to the mask of the lesion. At the same time, attention can be focused on the global / overall area of the target frame image and the guide frame image, and the overall similarity between the target frame image and the guide frame image is calculated to obtain a spatial feature map, which can determine the global similarity between the target frame image and the guide frame image with respect to the change difference (such as static background texture). According to the time feature map and the spatial feature map, the predicted mask of the lesion in the target frame image is determined. Therefore, in this application, with the help of decoupling of the time dimension and the spatial dimension and parallel processing, the mask of the lesion in the image can be accurately predicted, which reduces the complexity and time cost of the calculation, saves computer computing resources, and effectively filters out the interference information in the image, thereby improving the detection speed and accuracy of the lesion area, and meeting the real-time and low-latency detection needs.

[0118] See also Figure 3 , Figure 3 A flow chart of an image processing method provided in one embodiment of the present application is shown.

[0119] like Figure 3 As shown, the image processing method of the present application may include:

[0120] S301 , acquiring a target frame image, a plurality of memory frame images, and a guide frame image, wherein the plurality of memory frame images include a mask of a lesion, and a change difference between the target frame image and the guide frame image is within a preset range.

[0121] Among them, S301 and Figure 2 The implementation of S201 in the embodiment is similar and will not be repeated here in this application.

[0122] S302. Obtain a first key feature map and a first value feature map according to the target frame image, where the first key feature map corresponds to the first value feature map.

[0123] S303 . Obtain a second key feature map and a second value feature map according to each memory frame image, where the second key feature map corresponds to the second value feature map.

[0124] S304: Divide the first key feature map, the first value feature map, the plurality of second key feature maps, and the plurality of second value feature maps into a preset number of non-overlapping area blocks.

[0125] S305 , performing local similarity calculation on each region block in the first key feature map and region blocks corresponding to the same index in the plurality of second key feature maps according to the time sequence of the plurality of memory frame images to obtain local similarity.

[0126] S306: Obtain a temporal feature map according to the local similarity, each region block in the first value feature map, and each region block in the plurality of second value feature maps.

[0127] S307. Obtain a third value feature map based on the time feature map; divide the third value feature map into a preset number of non-overlapping area blocks; and update the time feature map based on the local similarity, each area block in the first value feature map, and each area block in the third value feature map.

[0128] S308. Obtain a fourth key feature map and a fourth value feature map according to the guidance frame image, where the fourth key feature map corresponds to the fourth value feature map.

[0129] S309: Calculate spatial similarity between the first key feature map and the fourth key feature map to obtain spatial similarity.

[0130] S310 , weighting the spatial similarity and the fourth value feature map to obtain a guide value feature map.

[0131] S311. Merge the guidance value feature map and the first value feature map to obtain a spatial feature map.

[0132] S312. Obtain a fifth-value feature map based on the spatial feature map; weight the spatial similarity and the fifth-value feature map to update the guiding value feature map; merge the guiding value feature map with the first-value feature map to update the spatial feature map.

[0133] S313: Determine a predicted mask of the lesion in the target frame image according to the temporal feature map and the spatial feature map.

[0134] Among them, S313 and Figure 2 The implementation of S204 in the embodiment is similar and will not be repeated here in this application.

[0135] Among them, S302-S306 can be executed sequentially, and S308-S311 can be executed sequentially.

[0136] It should be noted that there is no temporal sequence between S302 - S306 and S308 - S311 , and S302 - S306 and S308 - S311 can be executed simultaneously or sequentially.

[0137] In S302-S306, the image processing model can be used Figure 1The encoder, key-value pair output module, region block segmentation module and time attention module shown in the figure obtain the time feature map.

[0138] In S302, the target frame image is input into Enc2. The feature layer in Enc2 extracts a feature map of the target frame image and inputs the feature map into Conv21 and Conv22. Conv21 and Conv22 generate a pair of key feature maps and value feature maps, namely, the corresponding first key feature map and first value feature map.

[0139] Among them, the first key feature map can be K Q Marking, the first value feature map can be used V Q Mark.

[0140] In S303, the memory frame image is input into Enc3. The feature layer in Enc3 extracts the feature map of the target frame image and inputs the feature map into Conv 31 and Conv 32. Conv 31 and Conv 32 generate a pair of key feature maps and value feature maps, namely, the corresponding second key feature map and second value feature map.

[0141] Among them, the second key feature map can be K M Marked, the second value feature map uses V M Mark.

[0142] In S304, the first key feature map K Q And the first value feature map V Q Input into block 1, multiple second key feature maps K M And the corresponding multiple second value feature maps V M Enter into block 2.

[0143] Block 1 takes the first key feature map K Q Along the first key feature map K Q The length and width are divided into S 2 Non-overlapping area blocks. Among them, the first key feature map is divided into S 2 The area blocks can be marked as

[0144] Block 1 takes the first value feature map V Q Along the first value feature map V Q The length and width are divided into S 2 Non-overlapping area blocks. Among them, the first value feature map is divided into S 2 The area blocks can be marked as

[0145] Block 2 combines multiple second key feature maps KM Divide the length and width of multiple second key feature maps into S 2 Non-overlapping area blocks. Among them, multiple second key feature maps are divided into S 2 The area blocks can be marked as

[0146] Block 2 takes multiple second value feature maps V M Divide the length and width of multiple second value feature maps into S 2 Non-overlapping area blocks. Among them, multiple second value feature maps are divided into S 2 The area blocks can be marked as

[0147] Where S is a positive integer, z∈[1,T] represents each memory frame image in T memory frame images, i,j∈[1,S] represents the index of the region block, that is, i is a positive integer greater than or equal to 1 and less than or equal to S, and j is a positive integer greater than or equal to 1 and less than or equal to S.

[0148] In addition, for any of the above feature maps, S 2 Any two area blocks in the area blocks do not overlap.

[0149] In S305, each region block in the plurality of second key feature maps is sorted in time order according to the same index to obtain a first set. It can be expressed as:

[0150]

[0151] in, represents the region block with index {i, j} in the first second key feature map, Represents the region block with index {i, j} in the T-th second key feature map.

[0152] Furthermore, for S in the first key feature map 2 A local similarity calculation is performed between the region block and the region block corresponding to the same index of each second key feature graph in the first set to obtain a local similarity.

[0153] For any region block of each second key feature map in the first set, S in the first key feature map 2 In the region blocks, the region block corresponding to the first key feature map with the same index as the region block is searched, and the two region blocks with the same index are subjected to dot product calculation and normalization processing to obtain the local similarity of the two region blocks with the same index.

[0154] Among them, local similarity It can be expressed by the following formula:

[0155]

[0156] Among them, Softmax represents normalization processing, exp represents the exponential function with the natural constant e as the base, Indicates that the dot product calculation is performed on the area blocks with the same index in the first key feature map and multiple second key feature maps. represents the region block with index {i, j} in the first key feature map, Represents the region block with index {i, j} in the second key feature map.

[0157] In addition, the present application is not limited to the above implementation. For example, the device can also perform local similarity calculation on each area block in the first key feature map and the area blocks corresponding to the same index in the multiple second key feature maps according to the time sequence of the multiple second key feature maps.

[0158] In S306, the local similarity is used as the weight of the multiple second-value feature maps. That is, the local similarity corresponding to each region block with the same index is used as the weight of the region block with the same index in each second-value feature map. Each region block in the multiple second-value feature maps is weighted according to the local similarity to calculate a memory value feature map.

[0159] Among them, the memory value characteristic map V T Includes the memory value feature map of each region block with the same index. The memory value feature map of the region block with index {i, j} Expressed as:

[0160]

[0161] in, represents the local similarity, Represents the region block with index {i, j} in the Zth second-value feature map.

[0162] Furthermore, according to the same index of the region block, the memory value feature map V T With the first value feature map V Q Each region block in is merged to obtain the temporal feature map y T .

[0163] Among them, the time feature map y T Expressed as:

[0164] y T =[V Q ,V T ].

[0165] S307 is an optional step.

[0166] In S307, the number of temporal attention modules is N, and this application does not limit the specific value of N.

[0167] When N is equal to 1, the temporal attention module can output a temporal feature map.

[0168] When N is greater than 1, the first temporal attention module can output a temporal feature map. For any temporal attention module from the second temporal attention module to the Nth temporal attention module, the target frame image and the temporal feature map output by the previous temporal attention module are input into the temporal attention module. The temporal attention module can update the temporal feature map based on the target frame image and the temporal feature map output by the previous temporal attention module.

[0169] Therefore, the N-1 superposition processing based on the temporal attention module makes the feature map output by the temporal feature map more expressive.

[0170] Next, combine Figure 4 , using 1 time attention module, Figure 3 The implementation of S302-S306 is described in detail.

[0171] See also Figure 4 , Figure 4 A module diagram of an image processing method provided in an embodiment of the present application is shown.

[0172] like Figure 4 As shown, in the time attention module, the first key feature map K Q and each second key feature map Perform dot multiplication and softmax operation to obtain the first key feature map K Q and the second key feature map The local similarity between .

[0173] The first key feature map K Q and the second key feature map The local similarity between them is used as the corresponding second value feature map The weight coefficient and the weight system and the corresponding second value feature map Perform dot multiplication to obtain the memory value feature map V T .

[0174] The memory value feature map V T With the first value feature map V Q Merge to get the time feature map y T .

[0175] In this application, the temporal feature map of the local area between images can be obtained through the temporal attention module, which is conducive to improving the detection accuracy and thus reducing the amount of calculation and complexity.

[0176] In S308-S311, the image processing model can be used Figure 1 The encoder, key-value pair output module, and spatial attention module shown in Figure 3 produce a spatial feature map.

[0177] In S308, the guidance frame image is input into Enc1. The feature layer in Enc1 extracts the feature map of the guidance frame image and inputs the feature map into Conv11 and Conv12. Conv11 and Conv12 generate a pair of key feature map and value feature map, namely the corresponding fourth key feature map and fourth value feature map.

[0178] Among them, the fourth key feature map can be K P Marking, the fourth value feature map can be used V P Mark.

[0179] In S309 , a global similarity calculation is performed on the first key feature map and the fourth key feature map to obtain a spatial similarity.

[0180] In some embodiments, dot product calculation and normalization processing are performed on the first key feature map and the fourth key feature map to obtain corresponding spatial similarity.

[0181] Among them, the spatial similarity f(K Q ,K P ) is expressed as:

[0182]

[0183] Among them, Softmax represents normalization processing, exp represents the exponential function with the natural constant e as the base, Indicates the dot product calculation of the first key feature map and the fourth key feature map.

[0184] In S310, the spatial similarity is used as the weight of the fourth value feature map. The fourth value feature map is weighted according to the spatial similarity to calculate a guiding value feature map.

[0185] Among them, the guidance value characteristic map V S Expressed as:

[0186] V S =f(K Q ,K P )V P ;

[0187] Among them, f(K Q ,KP ) represents the spatial similarity, V P Represents the fourth value feature map.

[0188] In S311, the guidance value feature map and the first value feature map are merged to obtain a spatial feature map.

[0189] Among them, the spatial feature map y S Expressed as:

[0190] y S =[V Q ,V S ]=[V Q ,f(K Q ,K P )V P ].

[0191] S312 is an optional step.

[0192] In S312, the number of spatial attention modules is M, and this application does not limit the specific value of M.

[0193] When M is equal to 1, the spatial attention module can output a temporal feature map.

[0194] When M is greater than 1, the first spatial attention module can output a spatial feature map. For any spatial attention module from the second spatial attention module to the Mth spatial attention module, the target frame image and the spatial feature map output by the previous spatial attention module are input into the spatial attention module. The spatial attention module can update the spatial feature map based on the target frame image and the spatial feature map output by the previous spatial attention module.

[0195] Therefore, the M-1 superposition processing based on the spatial attention module makes the feature map output by the spatial attention module more expressive.

[0196] Next, combine Figure 5 , using a spatial attention module, Figure 3 The implementation of S308-S311 is described in detail.

[0197] See also Figure 5 , Figure 5 A module diagram of an image processing method provided in an embodiment of the present application is shown.

[0198] like Figure 5 As shown, in the spatial attention module, the first key feature map K Q and the fourth key feature map K P Perform dot multiplication and softmax operation to obtain the first key feature map K Qand the fourth key feature map K P The spatial similarity between them.

[0199] The first key feature map K Q and the fourth key feature map K P The spatial similarity between them is used as the fourth value feature map V P The weight coefficient and the weight system and the fourth value characteristic map V P Perform dot multiplication to obtain the guidance value feature map V S .

[0200] The guidance value feature map V S With the first value feature map V Q Combined, we get the spatial feature map y S .

[0201] In this application, the spatial feature map of similar images can be obtained through the spatial attention module, which is beneficial to improving the detection accuracy.

[0202] Illustratively, an embodiment of the present application also provides an image processing device.

[0203] Figure 6 This is a structural diagram of an image processing device provided in one embodiment of the present application. The image processing device 10 is used to implement any of the above-mentioned method embodiments. Figure 1 The operation of the image processing model device shown is as Figure 6 As shown, the image processing device 10 may include: an acquisition module 11 , a first calculation module 12 , a second calculation module 13 and a determination module 14 .

[0204] An acquisition module 11 is configured to acquire a target frame image, a plurality of memory frame images, and a guide frame image, wherein the plurality of memory frame images include a mask of the lesion, and a change difference between the target frame image and the guide frame image is within a preset range;

[0205] A first calculation module 12 is configured to calculate a local similarity between the target frame image and each memory frame image according to a time dimension of the plurality of memory frame images to obtain a time feature map;

[0206] The second calculation module 13 is used to calculate the spatial similarity between the target frame image and the guidance frame image to obtain a spatial feature map;

[0207] The determination module 14 is configured to determine a predicted mask of the lesion in the target frame image according to the temporal feature map and the spatial feature map.

[0208] In some embodiments, the first computing module 12 is configured to:

[0209] According to the target frame image, a first key feature map and a first value feature map are obtained, wherein the first key feature map corresponds to the first value feature map;

[0210] According to each memory frame image, a second key feature map and a second value feature map are obtained, wherein the second key feature map corresponds to the second value feature map;

[0211] Dividing the first key feature map, the first value feature map, the plurality of second key feature maps, and the plurality of second value feature maps into a preset number of non-overlapping area blocks;

[0212] According to the time sequence of the multiple memory frame images, local similarity calculation is performed on each area block in the first key feature map and the area blocks corresponding to the same index in the multiple second key feature maps to obtain local similarity;

[0213] A temporal feature map is obtained according to the local similarity, each region block in the first value feature map, and each region block in the plurality of second value feature maps.

[0214] In some embodiments, the first computing module 12 is specifically configured to:

[0215] Sort each region block in the plurality of second key feature maps in chronological order according to the same index to obtain a first set;

[0216] A local similarity calculation and normalization process is performed on each region block in the first key feature map and the region blocks corresponding to the same index in the first set to obtain a local similarity.

[0217] In some embodiments, the first computing module 12 is specifically configured to:

[0218] Performing weighted calculation on the local similarity and each region block in the plurality of second value feature maps to obtain a memory value feature map;

[0219] According to the same index of the region blocks, the memory value feature map is merged with each region block in the first value feature map to obtain a time feature map.

[0220] In some embodiments, the second calculation module 13 is configured to:

[0221] Obtaining a fourth key feature map and a fourth value feature map according to the guidance frame image, wherein the fourth key feature map corresponds to the fourth value feature map;

[0222] Calculating spatial similarity between the first key feature map and the fourth key feature map to obtain spatial similarity;

[0223] Weighting the spatial similarity and the fourth value feature map to obtain the guidance value feature map;

[0224] The guidance value feature map is merged with the first value feature map to obtain a spatial feature map.

[0225] Figure 7 A schematic diagram of the structure of an image processing device provided in one embodiment of the present application is shown in FIG. Figure 7 As shown, in Figure 6 Based on the structure shown, the image processing device 10 may further include: a time updating module 15 and a space updating module 16 .

[0226] In some embodiments, the time updating module 15 is configured to:

[0227] According to the time characteristic graph, a third value characteristic graph is obtained;

[0228] Dividing the third value feature map into a preset number of non-overlapping area blocks;

[0229] The temporal feature map is updated according to the local similarity, each region block in the first value feature map, and each region block in the third value feature map.

[0230] In some embodiments, the space update module 16 is configured to:

[0231] According to the spatial feature map, a fifth value feature map is obtained;

[0232] Weighting the spatial similarity and the fifth value feature map to update the guidance value feature map;

[0233] The guidance value feature map is merged with the first value feature map to update the spatial feature map.

[0234] The image processing device of the present application can be used to perform Figure 1-Figure 5 The technical solutions of the devices in the method embodiments shown have similar implementation principles and technical effects, and the implementation operations of each module can be further referred to the relevant description of the method embodiments, which will not be repeated here. The modules here can also be replaced by components or circuits.

[0235] The present application can divide the image processing device into functional modules according to the above method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. In one implementation, the above device is part of an operating system. The above integrated module can be implemented in the form of hardware or in the form of a software functional module. It should be noted that the division of modules in the various embodiments of the present application is schematic and is only a logical functional division. There may be other division methods in actual implementation.

[0236] Figure 8 A schematic diagram of the structure of an image processing device provided in one embodiment of the present application is shown in FIG. Figure 8 As shown, the image processing device 20 includes:

[0237] The memory 21 is used to store program instructions. The memory 21 may be a flash memory. The memory 21 also stores the operating system of the device and applications in the device.

[0238] Processor 22 is used to call and execute program instructions in the memory to implement Figure 1-Figure 5 For details, please refer to the relevant description in the previous method embodiment.

[0239] An input / output interface 23 may also be included. The input / output interface 23 may include an independent output interface and input interface, or may be an integrated interface integrating input and output. The output interface is used to output data, and the input interface is used to obtain input data. The output data is a general term for the output in the above method embodiment, and the input data is a general term for the input in the above method embodiment.

[0240] The image processing device 20 can be used to execute the various steps and / or processes corresponding to the corresponding devices in the above method embodiments.

[0241] The present application also provides a readable storage medium, in which an execution instruction is stored. When at least one processor of a terminal device executes the execution instruction, the terminal device executes the image processing method in the above method embodiment.

[0242] The present application also provides a program product, the program product including execution instructions stored in a readable storage medium. At least one processor of a device can read the execution instructions from the readable storage medium, and at least one processor executes the execution instructions so that the device implements the image processing method in the above method embodiment.

[0243] The present application also provides a chip, which is connected to a memory, or has a memory integrated thereon. When the software program stored in the memory is executed, the image processing method in the above method embodiment is implemented.

[0244] Those skilled in the art will appreciate that in the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments may be implemented in the form of a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to the embodiments of the present application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated therein. Available media may be magnetic media (eg, floppy disks, hard disks, magnetic tapes), optical media (eg, DVDs), or semiconductor media (eg, solid state disks (SSDs)).

Claims

1. An image processing method, characterized in that: include: Acquire a target frame image, multiple memory frame images, and a guide frame image, wherein the multiple memory frame images include a mask of the lesion, and a change difference between the target frame image and the guide frame image is within a preset range; According to the time dimension of the plurality of memory frame images, local similarity calculation is performed on the target frame image and each memory frame image to obtain a time feature map; Performing spatial similarity calculation on all regions of the target frame image and the guide frame image to obtain a spatial feature map; Determining a predicted mask of a lesion in the target frame image according to the temporal feature map and the spatial feature map; The method of calculating the local similarity between the target frame image and each memory frame image according to the time dimension of the plurality of memory frame images to obtain a time feature map includes: Obtaining a first key feature map and a first value feature map according to the target frame image, wherein the first key feature map corresponds to the first value feature map; Obtaining a second key feature map and a second value feature map according to each memory frame image, wherein the second key feature map corresponds to the second value feature map; Dividing the first key feature map, the first value feature map, the plurality of second key feature maps, and the plurality of second value feature maps into a preset number of non-overlapping area blocks; According to the time sequence of the plurality of memory frame images, local similarity calculation is performed on each area block in the first key feature map and the area blocks corresponding to the same index in the plurality of second key feature maps to obtain local similarity; Obtaining the temporal feature map according to the local similarity, each region block in the first value feature map, and each region block in the plurality of second value feature maps; The method further comprises: Obtaining a third value characteristic graph according to the time characteristic graph; Dividing the third value feature map into the preset number of non-overlapping area blocks; The temporal feature map is updated according to the local similarity, each region block in the first value feature map, and each region block in the third value feature map.

2. The method according to claim 1, wherein The method of calculating the local similarity of each region block in the first key feature map and the region blocks corresponding to the same index in the plurality of second key feature maps according to the time sequence of the plurality of memory frame images to obtain the local similarity includes: Sorting each region block in the plurality of second key feature maps in chronological order according to the same index to obtain a first set; Local similarity calculation and normalization processing are performed on each area block in the first key feature map and the area blocks corresponding to the same index in the first set to obtain the local similarity.

3. The method according to claim 1, wherein The obtaining of the temporal feature map according to the local similarity, each region block in the first value feature map, and each region block in the plurality of second value feature maps includes: Performing weighted calculation on the local similarity and each region block in the plurality of second value feature maps to obtain a memory value feature map; According to the same index of the region blocks, the memory value feature map is merged with each region block in the first value feature map to obtain the time feature map.

4. The method according to claim 1, wherein The performing spatial similarity calculation on the target frame image and the guide frame image to obtain a spatial feature map includes: Obtaining a fourth key feature map and a fourth value feature map according to the guidance frame image, wherein the fourth key feature map corresponds to the fourth value feature map; Performing spatial similarity calculation on the first key feature graph and the fourth key feature graph to obtain spatial similarity; Weighting the spatial similarity and the fourth value feature map to obtain a guidance value feature map; The guidance value feature map and the first value feature map are merged to obtain the spatial feature map.

5. The method according to claim 4, wherein The method further comprises: Obtaining a fifth value feature map according to the spatial feature map; Weighting the spatial similarity and the fifth value feature map to update the guidance value feature map; The guidance value feature map is merged with the first value feature map to update the spatial feature map.

6. An image processing device comprising a module for executing the image processing method according to any one of claims 1 to 5.

7. An image processing device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the image processing method according to any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the image processing method according to any one of claims 1 to 5 is implemented.

Images