Image processing method and device, electronic device and storage medium

Abstract

The present disclosure relates to an image processing method and device, an electronic device and a storage medium, the method comprising: obtaining local images of one or more scales based on an image to be processed; performing feature extraction processing on the one or more local images respectively to obtain feature maps corresponding to the local images of one or more scales; segmenting the image to be processed based on the feature maps corresponding to the local images of one or more scales to obtain a segmentation result. According to the image processing method of an embodiment of the present disclosure, by performing screenshot processing of the image to be processed at one or more scales and performing feature extraction processing on the local image, feature maps of one or more scales can be obtained, which is conducive to obtaining local fine features and global features, and can simultaneously obtain fine features of smaller targets and more complex global distribution features, thereby improving the accuracy of segmentation processing.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to an image processing method and device, an electronic device, and a storage medium. Background Art

[0002] In the process of image processing, it is difficult to obtain good segmentation effects when the targets are small and complexly distributed. For example, in the field of medical image processing, in computed tomography angiography (CTA) images, it is difficult to accurately segment the coronary arteries because the coronary arteries are small, complexly distributed, and easily affected by noise. Summary of the invention

[0003] The present disclosure provides an image processing method and device, an electronic device and a storage medium.

[0004] According to one aspect of the present disclosure, there is provided an image processing method, comprising: acquiring local images of one or more scales based on an image to be processed; performing feature extraction processing on the one or more local images respectively to obtain feature maps corresponding to the local images of the one or more scales; and segmenting the image to be processed based on the feature maps corresponding to the local images of the one or more scales to obtain a segmentation result.

[0005] According to the image processing method of the embodiment of the present disclosure, by performing screenshot processing on the image to be processed at one or more scales and performing feature extraction processing on the local image, a feature map at one or more scales can be obtained, which is conducive to obtaining local fine features and global features. It can simultaneously obtain the fine features of smaller targets and more complex global distribution features, thereby improving the accuracy of the segmentation processing.

[0006] In a possible implementation, acquiring local images of one or more scales based on the image to be processed includes: performing screenshot processing on the image to be processed at multiple scales to obtain multiple local images, wherein the multiple local images include a first local image of a reference size and a second local image larger than the reference size, and the image centers of the multiple local images are the same.

[0007] In a possible implementation, the performing feature extraction processing on the one or more local images respectively to obtain feature maps corresponding to the local images of the one or more scales includes: performing feature extraction processing on the multiple local images respectively to obtain a first feature map corresponding to the first local image and a second feature map corresponding to the second local image.

[0008] In a possible implementation, the segmenting of the image to be processed based on the feature map corresponding to the local image at one or more scales to obtain a segmentation result includes: superimposing the first feature map and the second feature map to obtain a third feature map; and activating the third feature map to obtain a segmentation result of the target area in the image to be processed.

[0009] In a possible implementation, feature extraction is performed on the multiple local images respectively to obtain a first feature map corresponding to the first local image and a second feature map corresponding to the second local image, including: downsampling the second local image to obtain a third local image of a reference size; feature extraction is performed on the first local image and the third local image respectively to obtain a first feature map corresponding to the first local image and a second feature map corresponding to the second local image.

[0010] In this way, local images of different sizes can be down-sampled to meet the input requirements of the feature extraction network, thereby obtaining feature maps of multiple scales, which is conducive to obtaining feature information of multiple scales.

[0011] In a possible implementation, the first feature map and the second feature map are superimposed to obtain a third feature map, including: upsampling the second feature map to obtain a fourth feature map, the ratio of the sizes of the fourth feature map and the first feature map is the same as the ratio of the sizes of the second local image and the first local image; cropping the fourth feature map to obtain a fifth feature map, the fifth feature map is consistent in size with the first feature map; and weighted summing the first feature map and the fifth feature map to obtain the third feature map.

[0012] In this way, the sizes of multiple feature maps can be unified through upsampling and cropping, and weighted averaging can be performed to fuse multiple feature maps, so that the obtained third feature map contains more feature information, which is conducive to determining the details and distribution of the target, reducing noise interference, and improving segmentation accuracy.

[0013] In one possible implementation, the image processing method is implemented by a neural network, and the neural network includes multiple feature extraction networks, overlay networks and activation layers, wherein the method also includes: training the multiple feature extraction networks, overlay networks and activation layers through sample images to obtain multiple feature extraction networks, overlay networks and activation layers after training.

[0014] In a possible implementation, the plurality of feature extraction networks, overlay networks, and activation layers are trained through sample images to obtain the plurality of feature extraction networks, overlay networks, and activation layers after training, including: performing screenshot processing on the sample image at multiple scales to obtain a fourth sample local image of a reference size, and a fifth sample local image larger than the reference size; performing downsampling processing on the fifth sample local image to obtain a sixth sample local image of the reference size; respectively inputting the fourth sample local image and the sixth sample local image into corresponding feature extraction networks for feature extraction processing to obtain a third sample feature map corresponding to the fourth sample local image, and a fifth sample local image corresponding to the sixth sample local image; a fourth sample feature map corresponding to the local image; performing upsampling and cropping on the fourth sample feature map to obtain a fifth sample feature map; inputting the third sample feature map and the fifth sample feature map into the overlay network to obtain a sixth sample feature map; inputting the sixth sample feature map into the activation layer for activation processing to obtain a second sample target area of ​​the sample image; determining a second network loss of the neural network based on the annotation information of the second sample target area and the sample image; and training the multiple feature extraction networks, overlay networks and activation layers based on the second network loss to obtain multiple feature extraction networks, overlay networks and activation layers after training.

[0015] In this way, the neural network can be trained through the pre-trained feature extraction network and sample images to improve the training accuracy of the neural network. Furthermore, by training the superposition network to obtain the superimposed weights, appropriate weight parameters can be selected during the training process to improve the effect of feature fusion, optimize the detailed features and global features, and help improve the accuracy of the neural network.

[0016] In a possible implementation, the third sample feature map and the fifth sample feature map are input into the overlay network to obtain a sixth sample feature map, including: performing weighted sum processing on the third sample feature map and the fifth sample feature map through the overlay network to obtain the sixth sample feature map.

[0017] In a possible implementation, the multiple feature extraction networks, overlay networks, and activation layers are trained through sample images to obtain the trained multiple feature extraction networks, overlay networks, and activation layers, and also includes: activating the fourth sample feature map to obtain a third sample target area; determining the third network loss of the feature extraction network corresponding to the fourth sample feature map based on the annotation information of the third sample target area and the sample image; and training the feature extraction network corresponding to the fourth sample feature map based on the third network loss.

[0018] In this way, the prediction information in the larger local image can be used to further train the feature extraction network, which can further improve the accuracy of the neural network.

[0019] According to another aspect of the present disclosure, there is provided an image processing device, comprising: a local image module, used to obtain local images of one or more scales based on an image to be processed; a feature extraction module, used to perform feature extraction processing on the one or more local images respectively to obtain feature maps corresponding to the local images of the one or more scales; and a segmentation module, used to segment the image to be processed based on the feature maps corresponding to the local images of the one or more scales to obtain a segmentation result.

[0020] In a possible implementation, the local image module is further configured to: perform screenshot processing on the image to be processed at multiple scales to obtain multiple local images, wherein the multiple local images include a first local image of a reference size and a second local image larger than the reference size, and the image centers of the multiple local images are the same.

[0021] In a possible implementation, the feature extraction module is further configured to: perform feature extraction processing on the multiple partial images respectively to obtain a first feature map corresponding to the first partial image and a second feature map corresponding to the second partial image.

[0022] In a possible implementation, the segmentation module is further configured to: superimpose the first feature map and the second feature map to obtain a third feature map; and activate the third feature map to obtain a segmentation result of the target area in the image to be processed.

[0023] In a possible implementation, the feature extraction module is further configured to: downsample the second local image to obtain a third local image of a reference size; perform feature extraction on the first local image and the third local image respectively to obtain a first feature map corresponding to the first local image and a second feature map corresponding to the second local image.

[0024] In a possible implementation, the segmentation module is further configured to: upsample the second feature map to obtain a fourth feature map, and the ratio of the sizes of the fourth feature map to the first feature map is the same as the ratio of the sizes of the second local image to the first local image; crop the fourth feature map to obtain a fifth feature map, and the fifth feature map is consistent in size with the first feature map; perform weighted summation on the first feature map and the fifth feature map to obtain the third feature map.

[0025] In one possible implementation, the image processing device is executed through a neural network, and the neural network includes multiple feature extraction networks, overlay networks and activation layers. The device also includes: a training module, which is used to train the multiple feature extraction networks, overlay networks and activation layers through sample images to obtain multiple feature extraction networks, overlay networks and activation layers after training.

[0026] In a possible implementation, the training module is further configured to: perform screenshot processing on the sample image at multiple scales to obtain a fourth sample local image of a reference size and a fifth sample local image larger than the reference size; perform downsampling processing on the fifth sample local image to obtain a sixth sample local image of the reference size; input the fourth sample local image and the sixth sample local image into corresponding feature extraction networks for feature extraction processing, respectively, to obtain a third sample feature map corresponding to the fourth sample local image, and a fourth sample feature map corresponding to the sixth sample local image; perform upsampling processing and cropping processing on the fourth sample feature map to obtain a fifth sample feature map; input the third sample feature map and the fifth sample feature map into the overlay network to obtain a sixth sample feature map; input the sixth sample feature map into the activation layer for activation processing to obtain a second sample target area of ​​the sample image; determine the second network loss of the neural network according to the annotation information of the second sample target area and the sample image; train the multiple feature extraction networks, overlay networks and activation layers according to the second network loss to obtain multiple feature extraction networks, overlay networks and activation layers after training.

[0027] In a possible implementation, the training module is further configured to: perform weighted sum processing on the third sample feature map and the fifth sample feature map through the overlay network to obtain the sixth sample feature map.

[0028] In a possible implementation, the training module is further configured to: activate the fourth sample feature map to obtain a third sample target area; determine the third network loss of the feature extraction network corresponding to the fourth sample feature map based on the annotation information of the third sample target area and the sample image; and train the feature extraction network corresponding to the fourth sample feature map based on the third network loss.

[0029] According to another aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to call the instructions stored in the memory to execute the above method.

[0030] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored, and the computer program instructions implement the above method when executed by a processor.

[0031] It should be understood that the above general description and the following detailed description are exemplary and explanatory only and do not limit the present disclosure. Other features and aspects of the present disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used to illustrate the technical solutions of the present disclosure together with the specification.

[0033] Figure 1 A flowchart showing an image processing method according to an embodiment of the present disclosure;

[0034] Figure 2A , Figure 2B , Figure 2C and Figure 2D A schematic diagram showing a screenshot process of an image to be processed according to an embodiment of the present disclosure;

[0035] Figure 3 A schematic diagram showing an application of an image processing method according to an embodiment of the present disclosure;

[0036] Figure 4 A block diagram showing an image processing apparatus according to an embodiment of the present disclosure;

[0037] Figure 5 A block diagram showing an electronic device according to an embodiment of the present disclosure;

[0038] Figure 6 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0039] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0040] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0041] The term "and / or" herein is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can represent including any one or more elements selected from the set consisting of A, B, and C.

[0042] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0043] Figure 1 A flowchart of an image processing method according to an embodiment of the present disclosure is shown. Figure 1 As shown, the method includes:

[0044] In step S11, a local image of one or more scales is acquired based on the image to be processed;

[0045] In step S12, feature extraction processing is performed on the one or more local images respectively to obtain feature maps corresponding to the local images of the one or more scales;

[0046] In step S13, the image to be processed is segmented based on the feature map corresponding to the local image of the one or more scales to obtain a segmentation result.

[0047] According to the image processing method of the embodiment of the present disclosure, by performing screenshot processing on the image to be processed at one or more scales and performing feature extraction processing on the local image, a feature map at one or more scales can be obtained, which is conducive to obtaining local fine features and global features. It can simultaneously obtain the fine features of smaller targets and more complex global distribution features, thereby improving the accuracy of the segmentation processing.

[0048] In a possible implementation, the image processing method may be executed by an electronic device such as a terminal device or a server, and the terminal device may be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The method may be implemented by a processor calling a computer-readable instruction stored in a memory. Alternatively, the method may be executed by a server.

[0049] In a possible implementation, the image to be processed may include a medical image, such as a computed tomography angiography (CTA) image, and the target area includes the area where the coronary vessels and other targets are located. The coronary vessels are relatively small, have a complex distribution, and are connected to other vessels (such as pulmonary vessels), and are susceptible to noise interference from other vessels. It should be understood that the image to be processed may also be other images, such as portrait images, street view images, etc. The targets in the image to be processed may include human facial features, pedestrians on the street, vehicles, etc. The present disclosure does not limit the types of images to be processed and the types of targets.

[0050] In a possible implementation, in step S11, the image to be processed may be captured at one or more scales to obtain a local image at one or more scales. The image to be processed may be captured at any scale to obtain a local image at that scale, for example, a smaller scale may be captured to obtain a local image representing detail features, or a larger scale may be captured to obtain a local image representing global features.

[0051] In a possible implementation, screenshots of the image to be processed may be performed at multiple scales to obtain local images of multiple scales. Step S11 may include: taking screenshots of the image to be processed at multiple scales to obtain multiple local images, wherein the multiple local images include a first local image of a reference size and a second local image larger than the reference size, and the multiple local images have the same image centers.

[0052] In a possible implementation, the image to be processed can be screenshotted at multiple scales to obtain multiple local images of different sizes, wherein the local images include a first local image of a reference size and a second local image larger than the reference size. When taking screenshots, the image center can be kept consistent. For example, the screenshot can be taken with the image center of the image to be processed as the center, and the image center of each local image obtained is consistent with the image center of the image to be processed. Alternatively, any pixel point of the image to be processed can be taken as the image center of the local image, and the image center of each local image obtained is the pixel point. Among the multiple local images obtained, the second local image includes more content than the first local image. The present disclosure does not limit the selection of the reference size and the shape of the local image.

[0053] Figure 2A , Figure 2B , Figure 2C and Figure 2D FIG. 2 is a schematic diagram showing a screenshot process of an image to be processed according to an embodiment of the present disclosure. Figure 2AAs shown, the image to be processed can be processed by screenshot processing of multiple scales, that is, the screenshot processing is performed with the reference size as the screenshot size to obtain the first local image x1 of the reference size (such as Figure 2B As shown), and a screenshot processing is performed with a screenshot size larger than the reference size to obtain a second partial image x2 (as shown) larger than the reference size. Figure 2C as shown) and the second local image x3 (as shown Figure 2D ). The first local image x1 and the second local images x2 and x3 have the same image center, the second local image x2 contains more content than the first local image x1, and the second local image x3 contains more content than the second local image x2. The first local image x1 of the reference size contains finer local detail features (e.g., detail features of the coronary vessels themselves), and the second local images x2 and x3 contain more global distribution features (e.g., the distribution of coronary vessels and the connection with other vessels), for example, the connection between the target in the first local image x1 and other areas in the second local image x2 or x3 (e.g., the connection between the coronary vessels in the first local image x1 and the vessels in other areas in the second local image x2 or x3).

[0054] In a possible implementation, in step S12, if a local image of one scale is obtained by the screenshot, feature extraction processing may be performed on the local image to obtain a feature map. If local images of multiple scales are obtained, feature extraction processing may be performed on the multiple local images respectively to obtain a feature map of each local image. Step S12 may include: performing feature extraction processing on the multiple local images respectively to obtain a first feature map corresponding to the first local image and a second feature map corresponding to the second local image.

[0055] In the example, feature extraction processing can be performed on each local image through a feature extraction network. The feature extraction network can be a deep learning neural network such as a convolutional neural network. The present disclosure does not limit the type of the feature extraction network.

[0056] In the example, the number of feature extraction networks and the number of local images may be the same, that is, each feature extraction network extracts a local image. For example, the first local image x1 may be input to feature extraction network 1 for feature extraction processing, the second local image x2 may be input to feature extraction network 2 for feature extraction processing, and the second local image x3 may be input to feature extraction network 3 for feature extraction processing.

[0057] In one possible implementation, multiple feature extraction networks can be the same feature extraction network, the size of the image input to the feature extraction network can be kept consistent, the reference size of the first local image can be used as the input size of the input feature extraction network, and the second local image larger than the reference size can be processed to reduce the size of the second local image so that it meets the input requirements of the feature extraction network. In the example, other sizes can also be used as the input size of the input feature extraction network. For example, a size can be preset as the input size, and local images that are inconsistent with the input size can be processed to meet the input requirements of the feature extraction network. The present disclosure does not limit the selection of the input size.

[0058] In a possible implementation, when the reference size is used as the input size, feature extraction processing is performed on the multiple local images respectively to obtain a first feature map corresponding to the first local image and a second feature map corresponding to the second local image, which may include: downsampling the second local image to obtain a third local image of the reference size; and feature extraction processing is performed on the first local image and the third local image respectively to obtain a first feature map corresponding to the first local image and a second feature map corresponding to the second local image.

[0059] In one possible implementation, the size of the second local image is larger than the reference size, that is, larger than the input size, and the second local image can be downsampled to obtain a third local image of the reference size, that is, the third local image meets the input size requirement of the feature extraction network.

[0060] In a possible implementation, feature extraction processing may be performed on the first local image and the third local image, respectively. In the example, they may be respectively input into the corresponding feature extraction networks for feature extraction processing. For example, the first local image x1 may be input into feature extraction network 1 for feature extraction processing to obtain a first feature map, the third local image corresponding to the second local image x2 may be input into feature extraction network 2 for feature extraction processing to obtain a second feature map corresponding to x2, and the third local image corresponding to the second local image x3 may be input into feature extraction network 3 for feature extraction processing to obtain a second feature map corresponding to x3.

[0061] In one possible implementation, the first feature map may include fine local detail features (e.g., detail features of the coronary arteries themselves), such as detail features (e.g., shape, contour, etc.) of the target (e.g., coronary arteries) in the first local image x1, and the second feature map may have a larger receptive field and include more global distribution features, such as the distribution of coronary arteries in the second local image x2 or x3 and the connection between coronary arteries and other blood vessels.

[0062] In this way, local images of different sizes can be down-sampled to meet the input requirements of the feature extraction network, thereby obtaining feature maps of multiple scales, which is conducive to obtaining feature information of multiple scales.

[0063] In one possible implementation, multiple feature extraction networks are the same feature extraction network, and when the input first partial image and the third partial image have the same size, the feature maps output by each feature extraction network, that is, the sizes of the first feature map and the second feature map are the same, but the second feature map is the processing result of the second partial image. Therefore, the size of the area corresponding to the second feature map in the original image to be processed is larger than the size of the area corresponding to the first feature map in the original image to be processed, and the feature information contained in the second feature map is different from that of the first feature map.

[0064] In a possible implementation, the image to be processed may be segmented according to a feature map including different feature information to obtain a segmentation result. Step S13 may include: superimposing the first feature map and the second feature map to obtain a third feature map; activating the third feature map to obtain a segmentation result of the target area in the image to be processed.

[0065] In a possible implementation, superimposing the first feature map and the second feature map to obtain a third feature map may include: upsampling the second feature map to obtain a fourth feature map, wherein the ratio of the sizes of the fourth feature map and the first feature map is the same as the ratio of the sizes of the second local image and the first local image; cropping the fourth feature map to obtain a fifth feature map, wherein the fifth feature map has the same size as the first feature map; and performing weighted summation on the first feature map and the fifth feature map to obtain the third feature map.

[0066] In a possible implementation, the second feature map may be upsampled to obtain a fourth feature map. For example, the upsampling may be performed by interpolation or the like. The present disclosure does not limit the upsampling method.

[0067] In a possible implementation, the first partial image and the second partial image are both part of the image to be processed, the size of the first partial image is smaller than that of the second partial image, and the first partial image and the second partial image have the same center, that is, the first partial image is part of the second partial image. The first feature map is the feature map of the first partial image, and the fourth feature map is the feature map of the second partial image, so that the ratio of the size of the fourth feature map to the first feature map is the same as the ratio of the size of the second partial image to the first partial image.

[0068] In the example, the size of the second local image x2 is 8 times that of the first local image x1 (three-dimensional image), and the first local image is a part of the second local image (the central areas of the first local image x1 and the second local image x2 are consistent, and the length and width of the first local image x1 are both half of the second local image x2), then the size ratio of the fourth feature map corresponding to the second local image x2 and the first feature map is also 8, the length and width of the first feature map are both half of the fourth feature map corresponding to the second local image x2, and the central area of ​​the fourth feature map corresponding to the second local image x2 corresponds to the same area in the image to be processed as the first local image x1. In the example, the size of the second local image x3 is 27 times that of the first local image x1, and the first local image is a part of the second local image (the central areas of the first local image x1 and the second local image x3 are consistent, and the length and width of the first local image x1 are each one-third of the second local image x3). Then, the size ratio of the fourth feature map corresponding to the second local image x3 and the first feature map is also 27, the length and width of the first feature map are each one-third of the fourth feature map corresponding to the second local image x3, and the central area of ​​the fourth feature map corresponding to the second local image x3 corresponds to the same area in the image to be processed as the first local image x1.

[0069] In the example, the size of the first feature map may be consistent with the size of the first partial image x1, the fourth feature map corresponding to the second partial image x2 may be consistent with the size of the second partial image x2, and the fourth feature map corresponding to the second partial image x3 may be consistent with the size of the second partial image x3. The present disclosure does not limit the sizes of the first feature map and the fourth feature map.

[0070] In a possible implementation, the fourth feature map may be cropped to obtain a fifth feature map of the same size as the first feature map. In the example, the central area of ​​the fourth feature map may be retained (the area corresponding to the central area in the image to be processed is the same as the area corresponding to the first feature map in the image to be processed), and other areas may be cropped to obtain the fifth feature map. In the example, the fifth feature map corresponding to the second local image x2 is the central area of ​​the fourth feature map corresponding to the second local image x2 (the corresponding area in the image to be processed is the same as the first local image x1), and the fifth feature map corresponding to the second local image x3 is the central area of ​​the fourth feature map corresponding to the second local image x3 (the corresponding area in the image to be processed is the same as the first local image x1). The fifth feature map includes some global features of the central area of ​​the second local image, for example, the global features of the central area of ​​the second local image x2 or x3. Specifically, the areas other than the central area in the fourth feature map are clipped, and only the central area (i.e., the fifth feature map) is retained. Therefore, the fifth feature map includes the features of the central area of ​​the fourth feature map. For example, the receptive field of the fifth feature map is larger than that of the first feature map, and may include, for example, the distribution information of the coronary vessels in the area corresponding to x1 of the image to be processed, which is conducive to determining the distribution of the coronary vessels and reducing the noise interference of other vessels (e.g., pulmonary vessels). That is, the fifth feature map and the first feature map are both feature maps of the area corresponding to x1, but because the parameters (weights, receptive fields, etc.) of each feature extraction network are different, the features of the fifth feature map are different from those of the first feature map. The feature information of the area corresponding to x1 can be increased in the above manner, providing a richer basis for the segmentation process and improving the accuracy of the segmentation process.

[0071] In a possible implementation, the first feature map and the fifth feature map may be weighted and summed to obtain a third feature map. For example, the first feature map and the fifth feature map may be weighted and summed pixel by pixel. In the example, the weight of the first feature map is α, the weight of the fifth feature map corresponding to the second local image x2 is β, and the weight of the fifth feature map corresponding to the second local image x3 is γ. According to the above weights, the first feature map, the fifth feature map corresponding to the second local image x2, and the fifth feature map corresponding to the second local image x3 may be weighted and summed pixel by pixel to obtain a third feature map. The third feature map contains both the detail features of the first feature map and the global features of the fifth feature map.

[0072] In the example, the weighted summation process may be performed by an overlay network, and the overlay network may be a deep learning neural network such as a convolutional neural network. The present disclosure does not limit the type of the overlay network.

[0073] In this way, the sizes of multiple feature maps can be unified through upsampling and cropping, and weighted averaging can be performed to fuse multiple feature maps, so that the obtained third feature map contains more feature information, which is conducive to determining the details and distribution of the target, reducing noise interference, and improving segmentation accuracy.

[0074] In a possible implementation, the third feature map may be activated. For example, the activation may be performed according to the feature information included in the third feature map to obtain a segmentation result of the target area where the target (e.g., coronary vessels) is located. For example, the third feature map may be activated by a softmax activation function to obtain a probability map. Other activation functions may also be used for activation, such as a RELU activation function, etc. The present disclosure does not limit the activation process. In the probability map, the probability of each pixel represents the probability that the pixel is located in the target area (for example, when the probability is greater than or equal to a probability threshold (e.g., 50%), the pixel may be considered to be located in the target area). The location of the target area where the target is located may be determined based on the probability of each pixel in the probability map. In the example, the segmentation may be performed by an activation layer. In the example, the target area may be the target area in the area corresponding to x1 in the image to be processed. For example, the coronary vessels in the area corresponding to x1 may be segmented.

[0075] In a possible implementation, the image processing method is implemented by a neural network, and the neural network includes multiple feature extraction networks, superposition networks, and activation layers. For example, feature extraction processing can be performed on multiple local images respectively through the feature extraction network, superposition processing can be performed on the first feature map and the fifth feature map through the superposition network, and segmentation processing can be performed on the third feature map through the activation layer. Before using the neural network to perform the above-mentioned image processing method, the neural network can be trained.

[0076] In a possible implementation, the method further includes: training the multiple feature extraction networks, overlay networks, and activation layers through sample images to obtain the trained multiple feature extraction networks, overlay networks, and activation layers.

[0077] In a possible implementation, the sample image may include a medical image, such as a CTA image, and the sample image may include annotation information of a target (eg, a coronary vessel).

[0078] In a possible implementation, the feature extraction network may be pre-trained. The sample image may be screenshotted at multiple scales to obtain a first sample local image of a reference size, and a second sample local image larger than the reference size. The total number of the first sample local images and the second sample local images is the same as the number of feature extraction networks. In the example, the image centers of the first sample local image and the second sample local image are the same, and the area captured by the first sample local image is the central area of ​​the second sample local image. For example, the sample image may be screenshotted to obtain a first sample local image y1 of a reference size, a second sample local image y2 larger than the reference size, and a second sample local image y3 larger than the size of the second sample local image y2.

[0079] In a possible implementation, the second sample local image may be downsampled to obtain a third sample local image of a reference size. Multiple feature extraction networks may be the same feature extraction network, and the size of the image input to the feature extraction network may be kept consistent. For example, the reference size of the first sample local image may be used as the input size of the input feature extraction network, and the second sample local image larger than the reference size may be downsampled to reduce the size of the second sample local image to obtain a third sample local image so that it meets the input requirements of the feature extraction network, that is, the size of the third sample local image is equal to the reference size. In the example, the second sample local images y2 and y3 may be downsampled respectively to obtain a third sample local image of the reference size.

[0080] In a possible implementation, the first sample local image and the third sample local image may be respectively input into corresponding feature extraction networks for feature extraction processing to obtain a first sample feature map corresponding to the first sample local image and a second sample feature map corresponding to the second sample local image. In the example, the first sample local image y1 may be input into feature extraction network 1 for feature extraction processing to obtain a first sample feature map, the third sample local image corresponding to the second sample local image y2 may be input into feature extraction network 2 for feature extraction processing to obtain a second sample feature map corresponding to y2, and the third sample local image corresponding to the second sample local image y3 may be input into feature extraction network 3 for feature extraction processing to obtain a second sample feature map corresponding to y3.

[0081] In a possible implementation, the first sample feature map and the second sample feature map may be activated, for example, by a softmax function, to obtain first sample target regions corresponding to the plurality of feature extraction networks. In the example, the first sample feature map, the second sample feature map corresponding to y2, and the second sample feature map corresponding to y3 may be activated, to obtain the first sample target region in the first sample local image y1, the first sample target region in the second sample local image y2, and the first sample target region in the second sample local image y3. That is, through the feature extraction and activation processing of the feature extraction network, the sample target regions are determined in the plurality of sample local images, and there may be errors in the sample target regions.

[0082] In a possible implementation, the first network losses of multiple feature extraction networks may be determined respectively according to the annotation information of the sample image and the first sample target area. In the example, the first network loss of feature extraction network 1 may be determined according to the annotation information of the area corresponding to y1 in the sample image and the first sample target area in the first sample local image y1. The first network loss of feature extraction network 2 may be determined according to the annotation information of the area corresponding to y2 in the sample image and the first sample target area in the second sample local image y2. The first network loss of feature extraction network 3 may be determined according to the annotation information of the area corresponding to y3 in the sample image and the first sample target area in the second sample local image y3. In the example, the first network loss may include cross entropy loss and set similarity loss (diceloss), and the cross entropy loss and set similarity loss of each feature extraction network may be determined respectively according to the first sample target area and the corresponding annotation information, and the cross entropy loss and the set similarity loss are weighted and summed to obtain the first network loss of each feature extraction network. The present disclosure does not limit the method for determining the first network loss.

[0083] In a possible implementation, the multiple feature extraction networks can be trained according to the first network loss to obtain multiple pre-trained feature extraction networks. In the example, each feature extraction network can be trained according to the first network loss of each feature extraction network, for example, feature extraction network 1 can be trained according to the first network loss of feature extraction network 1, feature extraction network 2 can be trained according to the first network loss of feature extraction network 2, and feature extraction network 3 can be trained according to the first network loss of feature extraction network 3. In the example, the first network loss can be back-propagated by the gradient descent method to adjust the network parameters of the feature extraction network. The training method can be iteratively executed multiple times until the first training condition is met. The first training condition may include the number of training times, that is, when the number of training times is greater than or equal to the preset number of times, the first training condition is met. The first training condition may include the size or convergence of the first network loss. For example, when the first network loss is less than or equal to the preset threshold, or converges to a preset interval, the first training condition is met. The present disclosure does not limit the first training condition. When the first training condition is met, a pre-trained feature extraction network can be obtained.

[0084] In a possible implementation, the neural network may be trained based on a pre-trained feature extraction network.

[0085] In a possible implementation, the plurality of feature extraction networks, overlay networks, and activation layers are trained through sample images to obtain the plurality of feature extraction networks, overlay networks, and activation layers after training, including: performing screenshot processing on the sample image at multiple scales to obtain a fourth sample local image of a reference size, and a fifth sample local image larger than the reference size; performing downsampling processing on the fifth sample local image to obtain a sixth sample local image of the reference size; respectively inputting the fourth sample local image and the sixth sample local image into corresponding feature extraction networks for feature extraction processing to obtain a third sample feature map corresponding to the fourth sample local image, and a fifth sample local image corresponding to the sixth sample local image; a fourth sample feature map corresponding to the local image; performing upsampling and cropping on the fourth sample feature map to obtain a fifth sample feature map; inputting the third sample feature map and the fifth sample feature map into the overlay network to obtain a sixth sample feature map; inputting the sixth sample feature map into the activation layer for activation processing to obtain a second sample target area of ​​the sample image; determining a second network loss of the neural network based on the annotation information of the second sample target area and the sample image; and training the multiple feature extraction networks, overlay networks and activation layers based on the second network loss to obtain multiple feature extraction networks, overlay networks and activation layers after training.

[0086] In a possible implementation, the sample image may include a medical image, such as a CTA image, and the sample image may include annotation information of a target (eg, a coronary vessel).

[0087] In a possible implementation, the sample image may be screenshotted at multiple scales to obtain a fourth sample local image of a reference size, and a fifth sample local image larger than the reference size. The total number of the fourth sample local image and the fifth sample local image is the same as the number of feature extraction networks. In the example, the image centers of the fourth sample local image and the fifth sample local image are the same, and the area captured by the fourth sample local image is the central area of ​​the fifth sample local image. For example, the sample image may be screenshotted to obtain a fourth sample local image z1 of a reference size, a fifth sample local image z2 larger than the reference size, and a fifth sample local image z3 larger than the size of the fifth sample local image z2.

[0088] In a possible implementation, the fifth sample local image may be downsampled to obtain a sixth sample local image of a reference size. The multiple feature extraction networks may be feature extraction networks of the same structure, and the size of the image input to the feature extraction network may be kept consistent. For example, the reference size of the fourth sample local image may be used as the input size of the input feature extraction network, and the fifth sample local image larger than the reference size may be downsampled to reduce the size of the fifth sample local image to obtain a sixth sample local image so that it meets the input requirements of the feature extraction network, that is, the size of the sixth sample local image is equal to the reference size. In the example, the fifth sample local images z2 and z3 may be downsampled respectively to obtain a sixth sample local image of the reference size.

[0089] In a possible implementation, the fourth sample local image and the sixth sample local image may be respectively input into corresponding feature extraction networks for feature extraction processing to obtain a third sample feature map corresponding to the fourth sample local image and a fourth sample feature map corresponding to the sixth sample local image. In the example, the fourth sample local image z1 may be input into feature extraction network 1 for feature extraction processing to obtain a third sample feature map, the sixth sample local image corresponding to the fifth sample local image z2 may be input into feature extraction network 2 for feature extraction processing to obtain a fourth sample feature map corresponding to z2, and the sixth sample local image corresponding to the fifth sample local image z3 may be input into feature extraction network 3 for feature extraction processing to obtain a fourth sample feature map corresponding to z3.

[0090] In a possible implementation, the fourth sample feature map may be upsampled and cropped to obtain a fifth sample feature map. In the example, the multiple feature extraction networks are the same feature extraction network, and the sizes of the input fourth sample local image and the sixth sample local image are the same, so the obtained third sample feature map and the fourth sample feature map have the same size. The fourth sample feature map may be upsampled so that the ratio of the sizes of the upsampled fourth sample feature map and the third sample feature map is the same as the ratio of the fifth sample local image to the fourth sample local image. For example, the third sample feature map and the fourth sample feature map may be consistent in size with the fourth sample local image, and the upsampled fourth sample feature map may be consistent in size with the fifth sample local image.

[0091] In the example, the central area of ​​the fifth sample feature map can be retained and other areas can be cropped to obtain the fifth sample feature map. The area corresponding to the fifth sample feature map in the sample image is consistent with the area corresponding to the third sample feature map in the sample image. For example, both are areas corresponding to the fourth sample local image z1 in the sample image.

[0092] In a possible implementation, the third sample feature map and the fifth sample feature map may be input into the overlay network to obtain a sixth sample feature map. The step of inputting the third sample feature map and the fifth sample feature map into the overlay network to obtain the sixth sample feature map includes: performing weighted sum processing on the third sample feature map and the fifth sample feature map through the overlay network to obtain the sixth sample feature map.

[0093] In the example, the overlay network can perform pixel-by-pixel weighted summation of the three sample feature maps and the fifth sample feature map. In the example, the weight of the third sample feature map is α, the weight of the fifth sample feature map corresponding to the fifth sample local image z2 is β, and the weight of the fifth sample feature map corresponding to the fifth sample local image z3 is γ. α, β and γ can be network parameters of the overlay network, and the weight values ​​can be determined by training the overlay network. After being processed by the overlay network, the sixth sample feature map can be obtained, and the sixth sample feature map can contain more feature information.

[0094] In a possible implementation, the sixth sample feature map may be input into the activation layer to obtain a second sample target region of the sample image. In the example, the activation layer may activate the sixth sample feature map through a softmax function to obtain a target region where a target (e.g., a coronary vessel) is located. The target region may be the target region in the region corresponding to the fourth sample local image z1 in the sample image. For example, the region where the coronary vessel is located in the region corresponding to z1 in the sample image may be segmented, that is, the second sample target region. The second sample target region may have an error.

[0095] In a possible implementation, the second network loss of the neural network may be determined based on the annotation information of the second sample target area and the sample image. The second network loss may be determined based on the annotation information of the sample image (for example, the annotation information of z1 in the area corresponding to the sample image) and the second sample target area obtained by the neural network. In the example, the second network loss may include cross entropy loss and set similarity loss (dice loss). The cross entropy loss and set similarity loss of each neural network may be determined based on the second sample target area and the annotation information, and the cross entropy loss and the set similarity loss are weighted and summed to obtain the second network loss of the neural network. The present disclosure does not limit the method for determining the second network loss.

[0096] In a possible implementation, the multiple feature extraction networks, superposition networks and activation layers can be trained according to the second network loss to obtain multiple feature extraction networks, superposition networks and activation layers after training. In the example, the second network loss can be back-propagated by the gradient descent method to adjust the network parameters of the neural network. The training method can be iteratively executed multiple times until the second training condition is met. The second training condition may include the number of training times, that is, when the number of training times is greater than or equal to the preset number of times, the second training condition is met. The second training condition may include the size or convergence of the second network loss. For example, when the second network loss is less than or equal to a preset threshold, or converges to a preset interval, the second training condition is met. The present disclosure does not limit the second training condition. After the second training condition is met, multiple feature extraction networks, superposition networks and activation layers after training can be obtained, and can be used to segment targets in images.

[0097] In this way, the neural network can be trained through the pre-trained feature extraction network and sample images to improve the training accuracy of the neural network. Furthermore, by training the superposition network to obtain the superimposed weights, appropriate weight parameters can be selected during the training process to improve the effect of feature fusion, optimize the detailed features and global features, and help improve the accuracy of the neural network.

[0098] In a possible implementation, the parameters of the feature extraction network can also be subsequently adjusted to further improve the accuracy of the neural network. The method of training the multiple feature extraction networks, overlay networks, and activation layers through sample images to obtain multiple feature extraction networks, overlay networks, and activation layers after training also includes: activating the fourth sample feature map to obtain a third sample target area; determining the third network loss of the feature extraction network corresponding to the fourth sample feature map based on the annotation information of the third sample target area and the sample image; and training the feature extraction network corresponding to the fourth sample feature map based on the third network loss.

[0099] In the example, before upsampling and cropping the fourth sample feature map, the fourth sample feature map may be activated (for example, the activation may be performed by a softmax function) to obtain the third sample target region. For example, the third sample target region is predicted by the fourth sample feature map obtained by the feature extraction network 2 and the feature extraction network 3. In the example, the feature extraction network 2 may predict the third sample target region in the fifth sample partial image z2, and the feature extraction network 3 may predict the third sample target region in the fifth sample partial image z3.

[0100] In the example, the third network loss of the feature extraction network 2 can be obtained by utilizing the labeling information of the corresponding area of ​​z2 in the sample image and the third sample target area in the fifth sample local image z2, and the third network loss of the feature extraction network 3 can be obtained by utilizing the labeling information of the corresponding area of ​​z3 in the sample image and the third sample target area in the fifth sample local image z3. In the example, the third network loss may include cross entropy loss and set similarity loss, and the present disclosure does not limit the third network loss.

[0101] In the example, the third network loss of each feature extraction network can be used to train each feature extraction network to utilize the sample target area predicted in a larger local image (e.g., z2 and z3) and the annotation information to continue adjusting the parameters of the feature extraction network and improve the accuracy of the neural network.

[0102] In this way, the prediction information in the larger local image can be used to further train the feature extraction network, which can further improve the accuracy of the neural network.

[0103] After the above training is completed, the trained neural network can be used in image segmentation processing to segment the target area in the image (for example, CTA image).

[0104] According to the image processing method of the embodiment of the present disclosure, by performing screenshot processing of multiple scales on the image to be processed, and by downsampling, local images of different sizes can meet the input requirements of the feature extraction network, thereby obtaining feature maps of multiple scales. The sizes of multiple feature maps can be unified through upsampling and cropping, and weighted averaging can be performed to fuse multiple feature maps, so that the third feature map obtained contains more feature information, which is conducive to determining the details of the target, and is conducive to determining the distribution of the target and reducing noise interference. Furthermore, the neural network can be trained through the pre-trained feature extraction network and sample images to improve the training accuracy of the neural network. And by training the superimposed weights obtained by superimposed networks, suitable weight parameters can be selected during the training process to improve the effect of feature fusion, optimize detail features and global features, and use the prediction information in larger local areas to further train the feature extraction network, which helps to improve the accuracy of the neural network and improve the accuracy of the segmentation process.

[0105] Figure 3 A schematic diagram showing an application of the image processing method according to an embodiment of the present disclosure is shown as follows: Figure 3 As shown, the neural network for image segmentation may include feature extraction network 1, feature extraction network 2, feature extraction network 3, a superposition network and an activation layer.

[0106] In a possible implementation, the neural network may be trained by sample images, and the sample images may include medical images, such as CTA images, and the sample images may include annotation information of targets (e.g., coronary vessels). The sample images may be preprocessed, for example, the sample images may be resampled, normalized, and the preprocessed sample images may be screenshotted to obtain a local image x1 of a reference size, a local image x2 of a size larger than the reference size, and a local image x3 of a size larger than x2. The screenshot method is described in detail in Figure 3 , three local images with the same image center but different sizes, i.e., x1, x2, and x3, can be captured from the sample image. Further, the local images x2 and x3 can be downsampled to reduce their sizes to the reference size, and the local images x1, x2 of the reference size, and x3 of the reference size are respectively input into feature extraction network 1, feature extraction network 2, and feature extraction network 3 for training (for example, training is performed based on the output results of each feature extraction network and the annotation information of the sample image), to obtain pre-trained feature extraction network 1, feature extraction network 2, and feature extraction network 3.

[0107] In a possible implementation, the neural network can be further trained through the pre-trained feature extraction network 1, feature extraction network 2 and feature extraction network 3, and sample images (which may be the same as the above sample images or other sample images). In the example, the sample images can be pre-processed and screenshot processed to obtain a local image x1 of a reference size, a local image x2 larger than the reference size, and a local image x3 larger than x2. The local images x2 and x3 can be downsampled to reduce their size to the reference size, and the local images x1, x2 of the reference size, and x3 of the reference size are respectively input into the feature extraction network 1, feature extraction network 2, and feature extraction network 3, and the feature extraction network 1 can obtain a feature map corresponding to the local image x1, the feature extraction network 2 can obtain a feature map corresponding to the local image x2, and the feature extraction network 3 can obtain a feature map corresponding to the local image x3.

[0108] In a possible implementation, the feature map corresponding to the local image x2 and the feature map corresponding to the local image x3 may be upsampled, and the central area corresponding to the local image x1 may be cropped. Furthermore, the feature map corresponding to the local image x1 may be superimposed with the two cropped feature maps. For example, the three feature maps may be weighted and summed with weights α, β, and γ, respectively, through a superimposed network, and the superimposed result may be input into an activation layer for activation processing to obtain a second sample target area, that is, the target area predicted by the local image x1 in the corresponding area of ​​the sample image. The network loss of the neural network may be determined based on the annotation information of the target area and the sample image, and the network loss may be used to train the neural network.

[0109] In a possible implementation, the feature maps output by the feature extraction network 2 and the feature extraction network 3 can be activated respectively to obtain the third sample target area, that is, the target area predicted by x2 in the corresponding area of ​​the sample image and the target area predicted by x3 in the corresponding area of ​​the sample image. Furthermore, the annotation information of the third sample target area and the sample image can be used to determine the network losses of the feature extraction network 2 and the feature extraction network 3 respectively, and train the feature extraction network 2 and the feature extraction network 3.

[0110] In a possible implementation, after the training process is completed, the neural network can be used to segment the image to be processed. For example, the image to be processed can be preprocessed and the image to be processed can be screenshotted to obtain a local image x1 of a reference size, a local image x2 larger than the reference size, and a local image x3 larger than x2. The local images x2 and x3 can be downsampled to reduce their size to the reference size, and the local image x1, the reference size x2, and the reference size x3 can be input into the feature extraction network 1, the feature extraction network 2, and the feature extraction network 3, respectively. Furthermore, after the feature maps output by the feature extraction network 2 and the feature extraction network 3 are upsampled and cropped, they are superimposed with the feature maps output by the feature extraction network 1, that is, input into the superposition network. The superposition result output by the superposition network can be input into the activation layer for activation processing, and the target area where the target is located in the corresponding area of ​​x1 in the image to be processed can be obtained. For example, the image to be processed is a computed tomography angiography (CTA) image, and the above neural network can segment the area where the coronary blood vessels are located in the corresponding area of ​​x1 in the image to be processed.

[0111] In a possible implementation, the image processing method can be used in the segmentation of coronary vessels, and feature information at multiple scales can be used to improve segmentation accuracy, obtain the region where the coronary vessels are located, and provide a basis for subsequent diagnosis (for example, diagnosis of plaques, vascular blockage, stenosis, and other diseases in the blood vessels). The image processing method can also be used in image segmentation processing in other fields, for example, in the fields of portrait segmentation, object segmentation, etc., and the present disclosure does not limit the application field of the image processing method.

[0112] Figure 4 A block diagram of an image processing device according to an embodiment of the present disclosure is shown. Figure 4 As shown, the device includes: a local image module 11, used to obtain local images of one or more scales based on the image to be processed; a feature extraction module 12, used to perform feature extraction processing on the one or more local images respectively to obtain feature maps corresponding to the local images of the one or more scales; a segmentation module 13, used to segment the image to be processed based on the feature maps corresponding to the local images of the one or more scales to obtain a segmentation result.

[0113] In a possible implementation, the local image module is further configured to: perform screenshot processing on the image to be processed at multiple scales to obtain multiple local images, wherein the multiple local images include a first local image of a reference size and a second local image larger than the reference size, and the image centers of the multiple local images are the same.

[0114] In a possible implementation, the feature extraction module is further configured to: perform feature extraction processing on the multiple partial images respectively to obtain a first feature map corresponding to the first partial image and a second feature map corresponding to the second partial image.

[0115] In a possible implementation, the segmentation module is further configured to: superimpose the first feature map and the second feature map to obtain a third feature map; and activate the third feature map to obtain a segmentation result of the target area in the image to be processed.

[0116] In a possible implementation, the feature extraction module is further configured to: downsample the second local image to obtain a third local image of a reference size; perform feature extraction on the first local image and the third local image respectively to obtain a first feature map corresponding to the first local image and a second feature map corresponding to the second local image.

[0117] In a possible implementation, the segmentation module is further configured to: upsample the second feature map to obtain a fourth feature map, and the ratio of the sizes of the fourth feature map to the first feature map is the same as the ratio of the sizes of the second local image to the first local image; crop the fourth feature map to obtain a fifth feature map, and the fifth feature map is consistent in size with the first feature map; perform weighted summation on the first feature map and the fifth feature map to obtain the third feature map.

[0118] In one possible implementation, the image processing device is executed through a neural network, and the neural network includes multiple feature extraction networks, overlay networks and activation layers. The device also includes: a training module, which is used to train the multiple feature extraction networks, overlay networks and activation layers through sample images to obtain multiple feature extraction networks, overlay networks and activation layers after training.

[0119] In a possible implementation, the training module is further configured to: perform screenshot processing on the sample image at multiple scales to obtain a fourth sample local image of a reference size and a fifth sample local image larger than the reference size; perform downsampling processing on the fifth sample local image to obtain a sixth sample local image of the reference size; input the fourth sample local image and the sixth sample local image into corresponding feature extraction networks for feature extraction processing, respectively, to obtain a third sample feature map corresponding to the fourth sample local image, and a fourth sample feature map corresponding to the sixth sample local image; perform upsampling processing and cropping processing on the fourth sample feature map to obtain a fifth sample feature map; input the third sample feature map and the fifth sample feature map into the overlay network to obtain a sixth sample feature map; input the sixth sample feature map into the activation layer for activation processing to obtain a second sample target area of ​​the sample image; determine the second network loss of the neural network according to the annotation information of the second sample target area and the sample image; train the multiple feature extraction networks, overlay networks and activation layers according to the second network loss to obtain multiple feature extraction networks, overlay networks and activation layers after training.

[0120] In a possible implementation, the training module is further configured to: perform weighted sum processing on the third sample feature map and the fifth sample feature map through the overlay network to obtain the sixth sample feature map.

[0121] In a possible implementation, the training module is further configured to: activate the fourth sample feature map to obtain a third sample target area; determine the third network loss of the feature extraction network corresponding to the fourth sample feature map based on the annotation information of the third sample target area and the sample image; and train the feature extraction network corresponding to the fourth sample feature map based on the third network loss.

[0122] It can be understood that the above-mentioned various method embodiments mentioned in the present disclosure can be combined with each other to form a combined embodiment without violating the principle logic. Due to space limitations, the present disclosure will not repeat them. It can be understood by those skilled in the art that in the above-mentioned method of the specific implementation method, the specific execution order of each step should be determined according to its function and possible internal logic.

[0123] In addition, the present disclosure also provides an image processing device, an electronic device, a computer-readable storage medium, and a program, all of which can be used to implement any image processing method provided by the present disclosure. The corresponding technical solutions and descriptions are referred to the corresponding records in the method part and will not be repeated here.

[0124] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0125] The embodiment of the present disclosure also provides a computer-readable storage medium, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the above method is implemented. The computer-readable storage medium may be a non-volatile computer-readable storage medium.

[0126] An embodiment of the present disclosure further proposes an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to call the instructions stored in the memory to execute the above method.

[0127] The embodiments of the present disclosure also provide a computer program product, including a computer-readable code. When the computer-readable code is executed on a device, a processor in the device executes instructions for implementing the image processing method provided in any of the above embodiments.

[0128] The embodiment of the present disclosure also provides another computer program product for storing computer-readable instructions, which, when executed, enables a computer to perform the operations of the image processing method provided by any of the above embodiments.

[0129] The electronic device may be provided as a terminal, a server, or a device in other forms.

[0130] Figure 5 A block diagram of an electronic device 800 according to an embodiment of the present disclosure is shown. For example, the electronic device 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, or the like.

[0131] Reference Figure 5 , the electronic device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0132] The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0133] The memory 804 is configured to store various types of data to support operations on the electronic device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0134] The power supply component 806 provides power to the various components of the electronic device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 800.

[0135] The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the edge of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the electronic device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0136] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the electronic device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0137] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0138] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the electronic device 800. For example, the sensor assembly 814 can detect the open / closed state of the electronic device 800, the relative positioning of the components, such as the display and keypad of the electronic device 800, and the sensor assembly 814 can also detect the position change of the electronic device 800 or a component of the electronic device 800, the presence or absence of contact between the user and the electronic device 800, the orientation or acceleration / deceleration of the electronic device 800, and the temperature change of the electronic device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0139] The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0140] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0141] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 804 including computer program instructions, which can be executed by a processor 820 of an electronic device 800 to perform the above method.

[0142] Figure 6 1 is a block diagram of an electronic device 1900 according to an embodiment of the present disclosure. For example, the electronic device 1900 may be provided as a server. Figure 6 , the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0143] The electronic device 1900 may also include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output (I / O) interface 1958. The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server 2000. TM , Mac OS X TM , Unix TM ,Linux TM , FreeBSD TM or similar.

[0144] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the electronic device 1900 to perform the above method.

[0145] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0146] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0147] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0148] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0149] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0150] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0151] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0152] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0153] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK) and the like.

[0154] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. An image processing method, characterized in that: include: Acquire local images of multiple scales based on the image to be processed; Performing feature extraction processing on the multiple local images respectively to obtain feature maps corresponding to the local images of the multiple scales; Segmenting the image to be processed based on the feature maps corresponding to the local images of the multiple scales to obtain a segmentation result; The step of performing feature extraction processing on the multiple local images respectively to obtain feature maps corresponding to the local images of the multiple scales comprises: Performing feature extraction processing on the multiple partial images respectively to obtain a first feature map corresponding to the first partial image and a second feature map corresponding to the second partial image; The step of segmenting the image to be processed based on the feature maps corresponding to the local images of the multiple scales to obtain a segmentation result includes: Performing upsampling processing on the second feature map to obtain a fourth feature map, wherein a ratio of the sizes of the fourth feature map to the first feature map is the same as a ratio of the sizes of the second partial image to the first partial image; performing cropping processing on the fourth feature map to obtain a fifth feature map, wherein the size of the fifth feature map is consistent with that of the first feature map; Performing weighted sum processing on the first feature map and the fifth feature map to obtain a third feature map; The third feature map is activated to obtain a segmentation result of the target area in the image to be processed.

2. The image processing method according to claim 1, characterized in that: The step of acquiring local images of multiple scales based on the image to be processed comprises: The image to be processed is screenshoted at multiple scales to obtain multiple local images, wherein the multiple local images include a first local image of a reference size and a second local image larger than the reference size, and the image centers of the multiple local images are the same.

3. The method according to claim 1, characterized in that Performing feature extraction processing on the multiple partial images respectively to obtain a first feature map corresponding to the first partial image and a second feature map corresponding to the second partial image, including: Performing downsampling processing on the second partial image to obtain a third partial image of a reference size; Feature extraction processing is performed on the first partial image and the third partial image respectively to obtain a first feature map corresponding to the first partial image and a second feature map corresponding to the second partial image.

4. The method according to claim 1, characterized in that: The image processing method is implemented by a neural network, which includes a plurality of feature extraction networks, a superposition network and an activation layer. Wherein, the method further comprises: The multiple feature extraction networks, overlay networks and activation layers are trained using sample images to obtain the trained multiple feature extraction networks, overlay networks and activation layers.

5. The method according to claim 4, characterized in that The plurality of feature extraction networks, overlay networks and activation layers are trained by using sample images to obtain the trained plurality of feature extraction networks, overlay networks and activation layers, including: Performing screenshot processing on the sample image at multiple scales to obtain a fourth sample partial image of a reference size and a fifth sample partial image larger than the reference size; Performing downsampling processing on the fifth sample partial image to obtain a sixth sample partial image of a reference size; Inputting the fourth sample partial image and the sixth sample partial image into corresponding feature extraction networks for feature extraction processing respectively, to obtain a third sample feature map corresponding to the fourth sample partial image and a fourth sample feature map corresponding to the sixth sample partial image; Performing upsampling and cropping processing on the fourth sample feature map to obtain a fifth sample feature map; Inputting the third sample feature map and the fifth sample feature map into the overlay network to obtain a sixth sample feature map; Inputting the sixth sample feature map into the activation layer for activation processing to obtain a second sample target area of ​​the sample image; Determining a second network loss of the neural network according to the second sample target area and the annotation information of the sample image; According to the second network loss, the multiple feature extraction networks, superposition networks and activation layers are trained to obtain the trained multiple feature extraction networks, superposition networks and activation layers.

6. The method according to claim 5, characterized in that Inputting the third sample feature map and the fifth sample feature map into the overlay network to obtain a sixth sample feature map includes: The third sample feature map and the fifth sample feature map are weightedly summed up by the overlay network to obtain the sixth sample feature map.

7. The method according to claim 5, characterized in that The method further comprises: training the plurality of feature extraction networks, overlay networks and activation layers through sample images to obtain the trained plurality of feature extraction networks, overlay networks and activation layers; Performing activation processing on the fourth sample feature map to obtain a third sample target area; Determining a third network loss of a feature extraction network corresponding to the fourth sample feature map according to the annotation information of the third sample target area and the sample image; A feature extraction network corresponding to the fourth sample feature map is trained according to the third network loss.

8. An image processing device, characterized in that: include: A local image module, used for obtaining local images of multiple scales based on the image to be processed; A feature extraction module, used to perform feature extraction processing on the multiple local images respectively to obtain feature maps corresponding to the local images of the multiple scales; A segmentation module, used for segmenting the image to be processed based on the feature maps corresponding to the local images of the multiple scales to obtain a segmentation result; The feature extraction module is further configured to: perform feature extraction processing on the multiple partial images respectively to obtain a first feature map corresponding to the first partial image and a second feature map corresponding to the second partial image; The segmentation module is further configured to: perform upsampling processing on the second feature map to obtain a fourth feature map, wherein the ratio of the size of the fourth feature map to the first feature map is the same as the ratio of the size of the second partial image to the first partial image; perform cropping processing on the fourth feature map to obtain a fifth feature map, wherein the size of the fifth feature map is consistent with that of the first feature map; Performing weighted sum processing on the first feature map and the fifth feature map to obtain a third feature map; The third feature map is activated to obtain a segmentation result of the target area in the image to be processed.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method described in any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images